Skip to content

Fixes #32548: Support BigQuery STRUCT profiling - #32549

Merged
pmbrull merged 1 commit into
mainfrom
pmbrull/fix-bigquery-profiler
Sep 4, 2026
Merged

pmbrull merged 1 commit into
mainfrom
pmbrull/fix-bigquery-profiler

Conversation

@pmbrull

@pmbrull pmbrull commented Sep 3, 2026 •

Copy link
Copy Markdown
Member

Describe your changes:

Fixes #32548

I updated BigQuery STRUCT-column expansion to pass SQLAlchemy 2's required parent-registration context and added a regression test because the old SQLAlchemy 1.x call crashes every profiled table containing a STRUCT.

Type of change:

  • Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

  • BigQuery profiling attaches a flattened STRUCT leaf column to the profiled SQLAlchemy table.
  • Nested STRUCT paths continue to compile with the real BigQuery dialect.

Unit tests

  • I added unit tests for the new/changed logic.
  • Files updated: ingestion/tests/unit/observability/profiler/sqlalchemy/bigquery/test_bigquery_profiler_sql.py
  • BigQuery profiler suite: 12 passed with sqlalchemy-bigquery==1.16.0.
  • Focused tests: 8 passed with sqlalchemy-bigquery==1.17.2.

Backend integration tests

  • Not applicable (no backend API changes).

Ingestion integration tests

  • Ran a local YAML profiler workflow against the Spider2 BigQuery service on OpenMetadata at localhost:8585.
  • All-table STRUCT smoke run (metrics: [rowCount]): 92/92 tables profiled, 184 profiler records, 92 REST writes, zero warnings/errors, exit code 0.
  • The OpenMetadata API returned a fresh profile with a row count for all 92 tables.
  • A default-metric spot check processed 4 GA4 tables and 519 profiler records with zero profiler errors before switching to the bounded all-table smoke run.

Playwright (UI) tests

  • Not applicable (no UI changes).

Manual testing performed

  • make py_format_check
  • git diff --check origin/main...HEAD
  • Live BigQuery profiling with the supplied service-account key and a separate billing project.

UI screen recording / screenshots:

Not applicable.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
  • For UI changes: I attached a screen recording and/or screenshots above.
  • I have added tests (unit / integration / Playwright as applicable) and listed them above.
  • I have added a test that covers the exact scenario we are fixing.

Pass SQLAlchemy 2's required parent-registration context when
flattening nested BigQuery fields.

Refs #32548
@pmbrull
pmbrull requested a review from a team as a code owner September 3, 2026 09:16
Copilot AI lite review requested due to automatic review settings September 3, 2026 09:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@gitar-bot

gitar-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

Fixes BigQuery STRUCT column profiling by passing SQLAlchemy 2's required parent-registration context to the column expansion logic. Includes regression test covering STRUCT profiling and nested path compilation. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@github-actions github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit 7a6cd76d0064ebbeb29150a63c5e5f532ddde965 in Playwright run 33737950708, attempt 1.

✅ 109 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 1h 24m 31s

⏱️ Max setup 4m 54s · max shard execution 12m 7s · max shard-job elapsed before upload 17m 31s · reporting 4s

🌐 219.42 requests/attempt · 1.78 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

  • Browser traffic was 219.42 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 1.78 per UI scenario (217 boots / 122 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 46 0 0 0 0 0
✅ Shard ingestion-01 29 0 0 0 0 0
🟡 Shard ingestion-02 34 0 1 0 0 0
🟡 1 flaky test(s) (passed on retry)
  • Features/DataQuality/DataQuality.spec.ts › Pagination functionality in test cases list (shard ingestion-02, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@sonarqubecloud

sonarqubecloud Bot commented Sep 3, 2026

Copy link
Copy Markdown

@pmbrull
pmbrull added this pull request to the merge queue Sep 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

🚦 Removed from the merge queue — failed_checks (2026-09-04T03:39:38Z)

Blocked the queue: playwright-summary

@pmbrull
pmbrull added this pull request to the merge queue Sep 4, 2026
Merged via the queue into main with commit 6b0d26b Sep 4, 2026
108 checks passed
@pmbrull
pmbrull deleted the pmbrull/fix-bigquery-profiler branch September 4, 2026 22:18
pmbrull added a commit that referenced this pull request Sep 7, 2026
Pass SQLAlchemy 2's required parent-registration context when
flattening nested BigQuery fields.

Refs #32548

(cherry picked from commit 6b0d26b)

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQuery profiler fails on STRUCT columns with SQLAlchemy 2

4 participants