Fabric data engineering agent (Project Osmos): what to check before you hand it a task

Microsoft has put an autonomous data engineering agent into Microsoft Fabric. It is called Project Osmos, it is in preview, and the idea is simple: you describe a complete data engineering outcome, and the agent plans, builds, runs and validates it against a lakehouse.

This is not another chat box that suggests a notebook. The task runs inside Fabric, on Spark, and keeps going after you close your laptop. That makes it more useful, and it also means there are a few things to understand before you let it near real data.

What it is

You start a task from a command-line client. Microsoft Learn lists GitHub Copilot CLI, Codex and Claude Code as supported. You give it a project outcome and point it at a lakehouse. From there the agent can inspect data, create and test Spark notebooks, transform data, write Delta tables and validate the result.

A few details from the documentation are worth knowing:

• Code and data operations run on Fabric Spark and OneLake, not on your machine.
• The task has persistent state, so it can be resumed instead of restarted.
• The selected lakehouse is the default lakehouse for the Spark session. It is not automatically the only source or destination.
• Anyone with access to that lakehouse can see the task's progress and give input.
• It is built for data engineering outcomes. Microsoft states that it is not meant for a quick query or a Power BI report change.

The part I like: review before it runs

Before execution, the agent classifies the task and proposes settings for you to review. They cover the permission boundary, the write safety pattern (staging, clone-and-promote or direct iteration), how a rerun behaves (fail, append, deduplicate or replace), whether the schema may change, where artifacts are saved, and how much reasoning effort to spend.

That list is basically the questions a good engineer asks before touching a production table. It is good to see them forced into the workflow.

One line in the documentation deserves to be read twice. The permission boundary guides the task, but Fabric and OneLake permissions remain the enforcement boundary. In other words, a sentence in a prompt is not a security control. Protect sensitive data with real permissions.

What you need

The prerequisites are more than a licence checkbox:

1. A workspace on an eligible Fabric capacity. Trial SKUs are not supported.
2. A lakehouse, and Contributor or higher on the workspace.
3. The tenant setting for Copilot and other features powered by Azure OpenAI enabled. Microsoft also recommends the setting for OpenAI as a Microsoft Subprocessor, but does not require it.
4. Azure CLI, plus one of the supported local agents.
5. The Skills for Fabric plug-in installed, which includes the Project Osmos skill.

There is one documented limitation at the time of writing: it is not available for workspaces that have outbound access protection enabled. If you work with locked-down workspaces, check that first.

Two bills, not one

This is the catch I would raise with any capacity owner. A task uses language model tokens and Spark compute, and they are reported separately.

• Language model usage shows up under the Project Osmos operation in the Capacity Metrics app, as a background operation on the Copilot and AI billing meter.
• Spark usage, including test and validation runs, is reported under the normal Spark billing model. Microsoft warns not to read the Project Osmos operation as the total cost of a task.

On a shared capacity, this work competes with everything else. Microsoft's own guidance is to start with a bounded outcome on representative data, measure both components, stagger submissions and limit concurrent tasks. A separate workspace on the same capacity does not give you compute isolation.

What to do next

If you want to try it, I would keep the first run boring:

1. Use a development workspace on a capacity that is not carrying month-end reporting.
2. Start with the retail medallion example from the quickstart. It needs only an empty lakehouse and creates its own sample data.
3. Read every proposed setting. Pay most attention to rerun behaviour and what is writable.
4. Open the Capacity Metrics app afterwards and find both the Project Osmos operation and the Spark usage.
5. Only then try an outcome on your own data, with success criteria written in the prompt: row counts, key uniqueness, reconciliation to source.

The documentation recommends exactly that last step. A project outcome should state the goal, sources, transformations, outputs and how to prove success. If you cannot write the validation down, you are not ready to delegate the task.

Takeaway

After two decades in data, I have learned that the hard part of an engineering task is rarely the code. It is agreeing on what correct means. Project Osmos does not remove that work. It moves it to the front, into the outcome and the validation rules, and it executes the rest.

It is preview, not for production use and with no SLA, so treat it as something to evaluate rather than rely on. But it is free of hype in the documentation, which is a good sign.

Which task in your backlog could you describe precisely enough to hand to an agent?

Sources

• What is Fabric data engineering agent (Project Osmos)? (Microsoft Learn)
• Get started with Fabric data engineering agent (Project Osmos) (Microsoft Learn)
• Project Osmos capacity consumption (Microsoft Learn)
• Best practices for the data engineering agent (Microsoft Learn)