Skip to content

fix(lineage): keep an edge writable after its pipeline is deleted [2.0] - #35315

Merged
harshsoni2024 merged 1 commit into
2.0from
backport-34816-2.0
Oct 13, 2026
Merged

harshsoni2024 merged 1 commit into
2.0from
backport-34816-2.0

Conversation

@harshsoni2024

@harshsoni2024 harshsoni2024 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

backport of 5620522 on 2.0


Summary by Gitar

  • Fixed lineage edge pipeline validation:
    • Kept soft-deleted pipelines already referenced by an edge so unchanged writes and patches succeed
    • Dropped pipelines hard-deleted from an edge instead of failing the write
  • Added integration tests:
    • Added LineageDeletedPipelineIT covering PUT and PATCH on edges with deleted pipelines

This will update automatically on new commits.

…4816)

Ingestion copies an edge's stored pipeline onto every lineage write that
carries none (SQL-parsed and query-log lineage), so those writes do not
strip the pipeline a pipeline connector attached. addLineage and
patchLineageEdge re-validated that pipeline with Include.NON_DELETED on
every write. Once the pipeline was soft-deleted, for example because its
dbt Cloud job was removed, every later write to the edge failed with
"pipeline instance for <id> not found", nothing was saved, and the agent
logged the same 404 on every run.

Only a pipeline newly linked to an edge now has to exist and not be
deleted. A pipeline the edge already references is kept when it was
soft-deleted, and dropped when it was hard-deleted, since hard delete
removes only PipelineLineage edges and other edges can still reference it.
@harshsoni2024
harshsoni2024 requested a review from a team as a code owner October 13, 2026 07:51
@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Oct 13, 2026
@gitar-bot

gitar-bot Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Code Review ✅ Approved

🟡 Medium risk · Changing lineage writes to retain soft-deleted pipeline references and drop hard-deleted ones could leave stale or missing pipeline attribution on edges.

Keeps lineage edges writable when their referenced pipeline has been soft-deleted, so unchanged PUT and PATCH writes succeed, and drops pipelines that were hard-deleted from an edge instead of failing the write. Adds LineageDeletedPipelineIT covering both PUT and PATCH on edges with deleted pipelines. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@harshsoni2024
harshsoni2024 merged commit 3486373 into 2.0 Oct 13, 2026
64 of 66 checks passed
@harshsoni2024
harshsoni2024 deleted the backport-34816-2.0 branch October 13, 2026 08:42

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants