Repository navigation
backport(2.0): Fixes #34291: Route one-sided Databricks lineage rows through the job (#34292) - #34732
Conversation
Databricks records a job that reads and writes in separate statements as a read row (no target) and a write row (no source) in system.access.table_lineage. The pipeline connector required both sides, so that lineage was dropped while Catalog Explorer still showed it through the job node. Read-only rows now become table -> job edges and write-only rows job -> table edges; tables already on a table -> table edge of the same job get no redundant hop. (cherry picked from commit de37bbd)
A self-referencing row or a table edge whose table is not in OpenMetadata produces no edge, but still marked its tables as linked, which hid the one-sided job edge as well. Process two-sided rows first and mark tables linked only once their table edge is emitted. Also switch the new warning to lazy %-style logging (G004). (cherry picked from commit b7eef9f)
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
Code Review ✅ Approved🟡 Medium risk · Databricks lineage ingestion now routes one-sided rows through jobs Backport of Databricks lineage fix that routes one-sided rows through the job connector instead of dropping them. The lineage query now preserves reads and writes from separate statements, emitting OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
|
✅ Playwright Results — workflow succeededValidated commit ✅ 110 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 48m 48s ⏱️ Max setup 3m 11s · max shard execution 11m 57s · max shard-job elapsed before upload 18m 1s · reporting 4s 🌐 210.19 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |



Backport of #34292 to
2.0.Fixes #34291
Why
A Databricks job that reads and writes in separate statements leaves only one-sided rows in
system.access.table_lineage(a read with no target, a write with no source). The Databricks Pipeline connector required both sides, so this lineage was dropped even though Catalog Explorer shows it astable -> job -> table.What
-x) of both commits from Fixes #34291: Route one-sided Databricks lineage rows through the job #34292. The lineage query keeps one-sided rows. Reads emittable -> pipelineand writes emitpipeline -> table. Tables already on an emittedtable -> tableedge of the same job get no extra hop through the job.metadata.pystill usesOptional[...] # noqa: UP045annotations, so the new_lookup_system_tablehelper and_lookup_tablefollow that style. Nothing else changed.Verification
pytest tests/unit/topology/pipeline/test_databricks_self_referencing_lineage.py: 11 passedruff format --check/ruff checkon the changed files: clean🤖 Generated with Claude Code