Data Lineage in Microsoft Purview, Explained From Zero

The first article in this series made the case that data lineage is a family tree for facts, and that a complete map of it is one of the most useful documents a security team can own. This article looks at one real, widely deployed system built specifically to draw that family tree: Microsoft Purview. Everything here builds on the first article’s ideas, but nothing from it is assumed — if this is the first thing you read in the series, it still stands on its own.

1 · What Microsoft Purview actually is

Strip away the product name, and Microsoft Purview is simply Microsoft’s cloud service for three related jobs a large company otherwise struggles to do by hand: finding out what data it has, understanding what that data means and where it came from, and protecting the data that turns out to be sensitive. It looks across a company’s entire “data estate” — a plain way of saying “everywhere the company keeps data”: databases running in the cloud, files sitting in a data lake, reports built in Power BI, and systems still running on the company’s own on-premises servers.

Purview does not store a second copy of your actual data. It stores metadata — facts about the data, rather than the data itself: a table’s name, its column names, when it was last scanned, who owns it, and, central to this article, everywhere it came from and everywhere it went.

2 · The Data Map: the foundation everything else sits on

Everything Purview does rests on one underlying structure called the Data Map. Think of it as an enormous, constantly updated card catalog, the way a library keeps one card per book so a librarian can find any title without walking every shelf. The Data Map holds one entry for every table, file, report, and pipeline Purview knows about, across the whole company.

That catalog is kept current by a scan: an automated process, run on a schedule, that walks through a connected data source the way a librarian walks the shelves, noting what it finds. A scan also looks for patterns inside the data itself — more on that later, under classification — and records what it finds directly into the Data Map.

Everything else people think of as “Purview” — the searchable catalog you browse, the lineage diagrams you look at, the reports on your whole data estate — are all views built on top of this one underlying map. Get the Data Map wrong or incomplete, and every view built on top of it inherits the same gap.

3 · Two kinds of things: Datasets and Processes

Inside the Data Map, Purview only ever deals with two basic kinds of entries, and understanding this pair unlocks almost everything else in this article.

• A Dataset is anything that holds data — a database table, a file sitting in storage, a Power BI report.
• A Process is anything that does something to data — a pipeline, a query, a transformation step.

Lineage, underneath everything, is just arrows drawn between these two kinds of boxes: a Dataset feeding into a Process, and that same Process producing a new Dataset out the other side. Purview even has a shortcut for when the exact process in between is not known: a direct arrow straight from one Dataset to another, meaning “the first clearly came before the second, even though we cannot show exactly how.”

4 · How the arrows get drawn automatically

For a specific, defined list of tools, Purview draws these Dataset-Process-Dataset arrows completely on its own, with nobody touching a keyboard. Microsoft calls these lineage connections, and the most common ones are Azure Data Factory and Azure Synapse pipelines (both tools that move and reshape data on a schedule), Power BI (reports and the dataflows that feed them), and, more recently, open-source tools like Airflow, through a shared open standard called OpenLineage.

The mechanism is a simple push model. The moment a supported pipeline or report actually runs, it automatically reports its own inputs, itself, and its outputs straight into the Data Map — the same way a delivery driver might radio ahead automatically every time a truck leaves a warehouse, without anyone having to phone and ask. If the source or destination was not already in the Data Map, Purview quietly adds it, so the map keeps growing on its own as real work happens.

5 · Where the arrows don’t get drawn automatically

Read that last sentence again: automatic lineage only exists for tools Purview has been built to listen to. The moment work happens somewhere outside that list — a custom Spark job someone wrote by hand, a third-party ETL tool without a lineage connection, or a script running on an old on-premises server — nothing gets reported, and the Data Map simply does not know that link ever existed.

This is the exact same gap the first article in this series described in the abstract, now with concrete, real names attached to both sides of it. It is not a flaw unique to Purview — it is a structural consequence of any push-model system: coverage can only ever be as wide as the list of things that have been taught to push.

6 · Closing the gap: manual and custom lineage

Purview gives you exactly two ways to fill in the arrows automatic lineage misses, and which one makes sense depends entirely on scale.

• Manual lineage, drawn directly inside the Purview portal by clicking to connect one existing entry to another. This is genuinely practical for a small number of missing links — realistically, a couple of dozen at a time — and needs no code at all.
• Custom lineage, reported by calling Purview’s own programming interface directly from inside your own pipeline or script, the same way a supported tool would automatically report itself. This scales to any number of links, and is the only realistic option once you are talking about hundreds or thousands of them.

Underneath, both paths simply create the same Dataset-and-Process entries and draw the same kind of arrows described in Figure 2 — there is no separate, second-class kind of lineage. A manually or programmatically reported arrow shows up in the exact same lineage diagram, drawn the exact same way, as one Purview captured entirely on its own.

7 · Classification and sensitivity labels

Purview has two related but genuinely different ways of marking what is sensitive, and mixing them up is one of the most common points of confusion.

A classification is a pattern match found during a scan — a column that looks like a social security number, a bank account number, a person’s name. Purview ships with over two hundred of these built in, and you can define your own. The important limitation: a classification only lives inside the one Data Map that found it. If the same data moves somewhere covered by a completely separate Purview Data Map, that classification does not travel with it — it has to be found again, from scratch.

A sensitivity label is different in exactly the way that matters here. Labels like Public, Confidential, or Highly Confidential come from a related Microsoft capability and are designed to travel with the data itself, embedded as metadata in the file or record, wherever it goes next — into Azure storage, into SharePoint, into Power BI, into a SQL column. A label applied once keeps showing up automatically everywhere that same data is later scanned.

8 · A worked example: tracing a labelled table

Bring the last three sections together with one example. Say a customer table in a data lake carries a Highly Confidential sensitivity label, and an Azure Data Factory pipeline, one of the automatically connected tools, copies part of it into a Power BI dataset every night.

Because the pipeline is a supported lineage connection, Purview automatically draws the Dataset-Process-Dataset arrow from the source table, through the pipeline, into the Power BI dataset the moment it runs — no one has to ask it to. Because the label travels with the data, that same Highly Confidential marking should reappear on the Power BI dataset the next time it is scanned. If an incident ever calls that source table into question, the lineage graph instantly shows the Power BI dataset as downstream and therefore suspect, and the label tells the response team exactly how seriously to treat it — two separate Purview features, lineage and labelling, doing their jobs together, automatically, without anyone having built anything custom for this specific case.

Now change one detail: suppose that nightly copy instead runs through a hand-written script on an old on-premises server, not through Data Factory. The label would still travel, because it is embedded in the data itself — but the lineage arrow would simply not exist, because no supported tool ever reported it. The Power BI dataset would carry the right label, sitting completely disconnected from the map that should have explained where it came from. That is the coverage gap from Section 5, made concrete.

9 · A simple test you can run in your own tenant