Skip to content

backport(2.0): Fixes #34291: Route one-sided Databricks lineage rows through the job (#34292) - #34732

Merged
ulixius9 merged 2 commits into
2.0from
ulixius9/collate-lineage-missing-2.0
Oct 6, 2026
Merged

ulixius9 merged 2 commits into
2.0from
ulixius9/collate-lineage-missing-2.0

Conversation

@ulixius9

@ulixius9 ulixius9 commented Oct 6, 2026

Copy link
Copy Markdown
Member

Backport of #34292 to 2.0.

Fixes #34291

Why

A Databricks job that reads and writes in separate statements leaves only one-sided rows in system.access.table_lineage (a read with no target, a write with no source). The Databricks Pipeline connector required both sides, so this lineage was dropped even though Catalog Explorer shows it as table -> job -> table.

What

  • Cherry-pick (-x) of both commits from Fixes #34291: Route one-sided Databricks lineage rows through the job #34292. The lineage query keeps one-sided rows. Reads emit table -> pipeline and writes emit pipeline -> table. Tables already on an emitted table -> table edge of the same job get no extra hop through the job.
  • Conflict resolution: 2.0's metadata.py still uses Optional[...] # noqa: UP045 annotations, so the new _lookup_system_table helper and _lookup_table follow that style. Nothing else changed.

Verification

  • pytest tests/unit/topology/pipeline/test_databricks_self_referencing_lineage.py: 11 passed
  • ruff format --check / ruff check on the changed files: clean

🤖 Generated with Claude Code

Databricks records a job that reads and writes in separate statements as a
read row (no target) and a write row (no source) in system.access.table_lineage.
The pipeline connector required both sides, so that lineage was dropped while
Catalog Explorer still showed it through the job node. Read-only rows now become
table -> job edges and write-only rows job -> table edges; tables already on a
table -> table edge of the same job get no redundant hop.

(cherry picked from commit de37bbd)
A self-referencing row or a table edge whose table is not in OpenMetadata
produces no edge, but still marked its tables as linked, which hid the
one-sided job edge as well. Process two-sided rows first and mark tables
linked only once their table edge is emitted. Also switch the new warning
to lazy %-style logging (G004).

(cherry picked from commit b7eef9f)
@ulixius9
ulixius9 requested a review from a team as a code owner October 6, 2026 08:37
@ulixius9
ulixius9 requested review from Khairajani and removed request for a team October 6, 2026 08:37
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Oct 6, 2026
@gitar-bot

gitar-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

🟡 Medium risk · Databricks lineage ingestion now routes one-sided rows through jobs

Backport of Databricks lineage fix that routes one-sided rows through the job connector instead of dropping them. The lineage query now preserves reads and writes from separate statements, emitting table -> pipeline and pipeline -> table edges as shown in Catalog Explorer. No issues found.

Review coverage

📋 Rules No rules evaluated

🧪 Functional validation Not enabled · Set up

🤖 Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@sonarqubecloud

sonarqubecloud Bot commented Oct 6, 2026

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 3d2c6da9b6e19f29ac34d2819621d787232c0830 in Playwright run 37437355579, attempt 1.

✅ 110 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 48m 48s

⏱️ Max setup 3m 11s · max shard execution 11m 57s · max shard-job elapsed before upload 18m 1s · reporting 4s

🌐 210.19 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

  • Browser traffic was 210.19 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 1.79 per UI scenario (216 boots / 121 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 46 0 0 0 0 0
✅ Shard ingestion-01 26 0 0 0 0 0
✅ Shard ingestion-02 38 0 0 0 0 0

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@ulixius9
ulixius9 enabled auto-merge (squash) October 6, 2026 10:17
@ulixius9
ulixius9 merged commit ba3d445 into 2.0 Oct 6, 2026
98 of 99 checks passed
@ulixius9
ulixius9 deleted the ulixius9/collate-lineage-missing-2.0 branch October 6, 2026 10:17

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants