Skip to content

fix(grafana): ingest panels inside collapsed rows - #31611

Merged
IceS2 merged 12 commits into
open-metadata:mainfrom
zerafachris:fix/grafana-collapsed-row-panels-31474
Sep 4, 2026
Merged

IceS2 merged 12 commits into
open-metadata:mainfrom
zerafachris:fix/grafana-collapsed-row-panels-31474

Conversation

@zerafachris

@zerafachris zerafachris commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

Fixes #31474

When a Grafana dashboard row is collapsed, the Grafana REST API nests its child panels inside the row panel's own panels field instead of the top-level dashboard.panels list. The connector iterated only the top-level list, so every panel under a collapsed row was silently skipped — those panels never appeared as charts and were excluded from lineage.

Root cause

GrafanaPanel had no collapsed or panels fields, so the nested panels were discarded during JSON deserialisation. Even if they had been preserved, yield_dashboard_chart and yield_dashboard_lineage_details iterated dashboard.panels directly without unwrapping row children.

What changed

File Change
grafana/models.py Added collapsed: Optional[bool] and panels: Optional[List["GrafanaPanel"]] to GrafanaPanel; added GrafanaPanel.model_rebuild() to resolve the self-referential forward reference
grafana/metadata.py Added GrafanaSource._flatten_panels() — a static helper that replaces collapsed row panels with their children, leaving expanded rows (children already at top level) unchanged; updated yield_dashboard_chart and yield_dashboard_lineage_details to pass the panel list through this helper before iterating
fixtures/grafana_fixtures.py Added DASHBOARD_WITH_COLLAPSED_ROW fixture covering: a top-level panel, a collapsed row with two nested panels, an expanded row with no nested panels, and a top-level panel that follows the expanded row
test_grafana_simple.py Added three unit tests: collapsed-row panels are surfaced, expanded-row panels are unchanged, empty/None input does not raise

Behaviour

Before: panels inside a collapsed Grafana row → silently missing from OpenMetadata charts and lineage.
After: _flatten_panels surfaces those panels; the rest of the pipeline handles them identically to top-level panels.

Expanded rows are unaffected — when a row is expanded, Grafana already places child panels at the top level and the row's own panels list is empty, so _flatten_panels is a no-op for that case.

Testing

Unit tests added in test_grafana_simple.py cover all three scenarios. Full integration test suite requires the OpenMetadata build environment; CI will validate.


Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

Greptile Summary

The PR updates Grafana ingestion to deserialize panels nested inside collapsed rows and process them as ordinary dashboard charts and lineage inputs.

  • Adds recursive panel model fields for collapsed-row payloads.
  • Flattens collapsed-row children before chart and lineage extraction.
  • Adds regression coverage for collapsed and expanded rows, empty input, chart output, and lineage processing.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
ingestion/src/metadata/ingestion/source/dashboard/grafana/metadata.py Adds collapsed-row panel flattening to both chart and lineage extraction paths without an eligible blocking issue.
ingestion/src/metadata/ingestion/source/dashboard/grafana/models.py Extends Grafana panel deserialization with collapsed state and self-referential child panels.
ingestion/tests/unit/topology/dashboard/test_grafana.py Expands connector-level regression coverage for collapsed-row chart and lineage processing while converting the suite to pytest style.
ingestion/tests/unit/topology/dashboard/test_grafana_simple.py Adds focused flattening tests for collapsed rows, expanded rows, and empty input.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    API[Grafana dashboard response] --> Parse[Deserialize dashboard panels]
    Parse --> Flatten[Replace collapsed rows with child panels]
    Flatten --> Charts[Create chart requests]
    Flatten --> Lineage[Extract dashboard lineage]
    Charts --> Catalog[OpenMetadata catalog]
    Lineage --> Catalog
Loading

Reviews (5): Last reviewed commit: "fix(grafana): make test_grafana_simple s..." | Re-trigger Greptile

Context used:

When a Grafana dashboard row is collapsed the API nests child panels
inside the row panel's `panels` field instead of the top-level
`dashboard.panels` list.  The ingestion connector iterated only the
top-level list, so all panels under a collapsed row were silently
skipped — they never appeared as charts and were excluded from lineage.

Changes:
- `GrafanaPanel`: add optional `collapsed` and `panels` fields to
  represent the nested structure returned by the Grafana API; call
  `model_rebuild()` to resolve the self-referential forward reference.
- `GrafanaSource._flatten_panels()`: new static helper that walks the
  panel list and replaces collapsed row panels with their children.
  Expanded rows (where child panels are already at the top level) are
  left unchanged.
- `yield_dashboard_chart` / `yield_dashboard_lineage_details`: call
  `_flatten_panels` before iterating so both code paths see every panel.
- Unit tests: added `DASHBOARD_WITH_COLLAPSED_ROW` fixture and three
  test methods covering collapsed rows, expanded rows, and empty input.

Prepared with AI assistance (Claude Code, Anthropic), reviewed for
correctness before submission.
@zerafachris
zerafachris requested a review from a team as a code owner August 17, 2026 08:10
@github-actions

github-actions Bot commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@IceS2

IceS2 commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Hey @zerafachris, thanks for the contrib!

The production change looks good, but the regression tests currently exercise _flatten_panels() directly rather than the behavior affected by the bug.

This leaves an important gap: if the _flatten_panels() call were removed from either yield_dashboard_chart() or yield_dashboard_lineage_details(), all the new tests would still pass while collapsed-row charts or their lineage would disappear again.

Could you add behavioral coverage for both consumers?

  • For chart ingestion, call yield_dashboard_chart() with the collapsed-row fixture

  • For lineage, call yield_dashboard_lineage_details() and assert that the nested panels contribute the expected
    lineage.

  • Keep the expanded-row case to prove its chart set remains unchanged and is not duplicated.

Also, since you are touching those tests, can you please migrate them from unittest to pytest?

Thanks

@zerafachris

Copy link
Copy Markdown
Contributor Author

Thanks @IceS2! I've pushed a follow-up commit (0c7db36) that addresses all three points:

  • Chart ingestion behavioral test (): calls yield_dashboard_chart() with a fixture that has both an expanded row and a collapsed row containing two nested panels. Asserts the nested panels (id=21, 22) are yielded and neither the collapsed-row wrapper (id=20) nor the expanded row (id=4) appears as a chart.
  • Expanded-row regression (test_yield_dashboard_chart_expanded_row_is_not_yielded): kept as a standalone test to guard against the original bug re-appearing.
  • Lineage behavioral test (test_yield_dashboard_lineage_collapsed_row_panels_contribute): calls yield_dashboard_lineage_details() with the same fixture and patches _process_panel_lineage to record which panels were dispatched to it. Asserts panels 21 and 22 are processed and the row wrapper (id=20) is not.
  • Migrated from unittest.TestCase to pytest: removed TestCase inheritance, converted __init__ to an autouse fixture, replaced all self.assertEqual/self.assertIn/etc. with plain assert statements.

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

4 similar comments
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@zerafachris

Copy link
Copy Markdown
Contributor Author

Applied make py_format (Black) to resolve the py-checkstyle CI failure — no logic changes, formatting only.

@IceS2
IceS2 self-requested a review August 28, 2026 09:49
akashverma0786 and others added 2 commits August 31, 2026 19:43
Builds on PR open-metadata#31611 (open-metadata#31474):
- reformat with ruff (supersedes the black commits) so py-checkstyle passes; fix F401/B905
- models.py: collapsed/panels as `| None` (drop Optional); keep GrafanaPanel.model_rebuild
- metadata.py: parameterized (%) logging — no f-strings in logger calls
- test_grafana.py: restore test_complete_json_parsing targeted assertions
  (nested targets, format 0-vs-None, per-target datasource, meta fields)
- test_grafana_simple.py: migrate unittest -> pytest
…ion)

The 'from tests.unit.topology.dashboard.fixtures...' import failed under the
repo's pytest invocation (ModuleNotFoundError: No module named 'tests.unit') —
it was the only 'from tests.*' import in the suite. Inline the collapsed-row
panels in the flatten test (matching test_grafana.py) and drop the now-unused
DASHBOARD_WITH_COLLAPSED_ROW fixture.
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Aug 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@IceS2 IceS2 added the safe to test Add this label to run secure Github workflows on PRs label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@gitar-bot

gitar-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 1 resolved / 1 findings

Grafana dashboard ingestion now surfaces panels nested inside collapsed rows by adding recursive panel fields to the model and flattening collapsed-row children before chart and lineage extraction. Comprehensive fixtures and unit tests cover collapsed rows, expanded rows, and edge cases. No issues found.

✅ 1 resolved
✅ Quality: test_complete_json_parsing weakened from full-object equality

📄 ingestion/tests/unit/topology/dashboard/test_grafana.py:690-694
The rewritten test replaces the previous full GrafanaDashboardResponse == expected_output equality check with four shallow assertions (uid, title, panel count, folderTitle). This drops verification of nested target parsing (rawSql, format=0 vs None coercion, per-target datasource) and meta fields that the old test guarded, so regressions in deserialization of those nested structures would no longer be caught. Consider asserting on the parsed panel targets (e.g. parsed_response.dashboard.panels[0].targets[1].rawSql and format) to retain that coverage.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@IceS2 IceS2 added the safe to test Add this label to run secure Github workflows on PRs label Sep 3, 2026
@sonarqubecloud

sonarqubecloud Bot commented Sep 3, 2026

Copy link
Copy Markdown

@IceS2
IceS2 enabled auto-merge September 3, 2026 16:23
@IceS2
IceS2 added this pull request to the merge queue Sep 3, 2026
Merged via the queue into open-metadata:main with commit a18e124 Sep 4, 2026
160 of 171 checks passed

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grafana connector: panels inside collapsed rows are not ingested

3 participants