Skip to content
Source on GitHubVisitors

Databricks pipeline

Trace how data flows through the tables and views of a Databricks declarative pipeline.

Diagram · origin tag databricks/pipeline · Data platforms & pipeline orchestration

Availability

Availability per host
HostStateHow to get it
StandalonePrototypeNot yet released
DesktopPrototypeNot yet released
IntelliJ PlatformNot plannedNot planned
Visual Studio CodeNot plannedNot planned
EclipseNot plannedNot planned
ThinkRailNot plannedNot planned
NotionNot plannedNot planned
ClaudeNot plannedNot planned

What it is for

Databricks declarative pipeline (the datasets, streaming tables and materialized views it declares, and the flows between them)

Why a specialized view helps

Declared datasets only reveal their lineage when drawn as a graph; a specialized view follows each flow from source to materialized view.

File formats

Not described yet.

Screenshots

Three source libraries flowing into the Bronze to gold pipeline and on to its target catalog, with the compute and notification cards below. Nothing selected.

Standalone, at 3e2df46

What is visible: Three source libraries flowing into the Bronze to gold pipeline and on to its target catalog, with the compute and notification cards below. Nothing selected.

Why it matters: Source libraries flow into the pipeline and on to its target catalog, with compute and notifications alongside, so the data's path through the declarative pipeline is traceable from start to end.

Licence: Apache-2.0

Background

Focus areas

Sources