Repository navigation
fix(bigquery): widen auto-detected partition window to avoid silent empty samples - #33221
Conversation
…mpty samples (open-metadata#33084) Time-unit and ingestion-time BigQuery partitions previously used a 1-unit window (1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window — GA4 exports (~2-day lag), weekly aggregates, slow-loading sources — was silently sampled as an empty result. The profiler still reported success, so tables that were never actually examined were indistinguishable from tables that were examined and found clean. Changes: • `partition.py` — increase the auto-detected default from 1 to 3 days for DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for HOUR-granularity partitions. Per-table profiler config overrides are unaffected: any explicitly set `partitionInterval` continues to take precedence via `get_partition_details`. • `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table COUNT(*) returns 0 rows, naming the table, current window, and the docs link to widen it. This makes any remaining narrow-window case visible in run logs instead of silent. • `tests/unit/test_partition.py` — update the two existing INGESTION_TIME assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals` covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitions. Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
…H/YEAR MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were given the same 3-day window as DAY partitions, which silently produced empty profiler samples for any monthly/yearly table whose most-recent partition is older than 3 days — exactly the failure this PR already fixed for hourly/daily tables. Introduces `_bigquery_window_for_granularity()` mapping each BigQuery partition granularity to an appropriately-sized window: - HOUR → 24 h - DAY → 3 days - MONTH → 35 days (covers a full month + buffer) - YEAR → 370 days (covers a full year + buffer) Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
| if row_count == 0: | ||
| logger.warning( | ||
| "Partition filter on table '%s' returned 0 rows. " | ||
| "The partition window (interval=%s %s) may not cover the table's most-recent data. " | ||
| "Override the profiler partition config for this table to widen the window. " | ||
| "See https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/" | ||
| "profiler/workflow#4.-updating-profiler-setting-at-the-table-level", | ||
| getattr(self.raw_dataset, "__tablename__", "unknown"), | ||
| getattr(self.partition_details, "partitionInterval", "?"), | ||
| getattr(self.partition_details, "partitionIntervalUnit", "?"), |
There was a problem hiding this comment.
Misleading Integer Partition Warning
When an INTEGER_RANGE partition returns no rows, this warning reports interval=None None and advises operators to widen the time window. Integer partitions instead use partitionIntegerRangeStart and partitionIntegerRangeEnd, so this guidance points operators toward a configuration change that cannot fix the problem. Please tailor the warning to the partition type or limit this message to time-based partitions.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…pling path The zero-row partition warning was only reached from _get_asset_row_count, which runs on the DYNAMIC sample-size path. A default profiler or auto-classification run sets no profileSampleConfig, so that method is never called and the warning never fired for the exact silent-empty-sample case in issue open-metadata#33084. Move the warning to fetch_sample_data, where every sampling path reads the sample rows, so it fires regardless of sample config and reuses the already fetched rows instead of running an extra COUNT(*). Tailor the message to the partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the misleading "widen the window" time-window guidance. Update test_profiler_partitions.py for the widened auto-detected defaults (DAY interval 3, HOUR interval 24).
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Add unit tests for the empty-partition warning added for issue open-metadata#33084. The warning was previously untested (flagged by gitar-bot): a refactor could drop it with no test failing. Tests pin the time-based branch (TIME_UNIT and INGESTION_TIME surface the window interval and unit), the non-time-based branch (INTEGER_RANGE and other types get the generic message, not misleading time-window guidance), and edge cases (non-enum interval unit, missing __tablename__).
…artition basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on self.partition_details.partitionInterval / .partitionIntervalUnit because partition_details is Optional and the method did not narrow it. Bind it to a local and early-return on None so the attribute access is type-safe; behaviour is unchanged (the caller only invokes this when partition_details is set).
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Code Review ✅ Approved 2 closed / 2 findings🟡 Medium risk · BigQuery profiling widens automatic partition windows and adds empty-sample warnings Widens BigQuery auto-detected partition windows from 1 day to 24 hours (HOUR) or 3 days (DAY/MONTH/YEAR) to avoid silent empty samples, and adds logging when partitioned tables return zero rows with override guidance. Explicitly configured partition settings are preserved. Unit tests cover the new TIME_UNIT and INGESTION_TIME defaults. ✅ 2 closed✅ Edge Case: 3-day window still silently empty for MONTH/YEAR partitions
✅ Quality: New empty-partition warning in fetch_sample_data has no tests
OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
|
316e84c
|
🤖 Backport to |
|
🤖 Backport to |
|
🤖 Backport to |
|
🤖 Backport to |
…mpty samples (#33221) (#34087) * fix(bigquery): widen auto-detected partition window to avoid silent empty samples (#33084) Time-unit and ingestion-time BigQuery partitions previously used a 1-unit window (1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window — GA4 exports (~2-day lag), weekly aggregates, slow-loading sources — was silently sampled as an empty result. The profiler still reported success, so tables that were never actually examined were indistinguishable from tables that were examined and found clean. Changes: • `partition.py` — increase the auto-detected default from 1 to 3 days for DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for HOUR-granularity partitions. Per-table profiler config overrides are unaffected: any explicitly set `partitionInterval` continues to take precedence via `get_partition_details`. • `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table COUNT(*) returns 0 rows, naming the table, current window, and the docs link to widen it. This makes any remaining narrow-window case visible in run logs instead of silent. • `tests/unit/test_partition.py` — update the two existing INGESTION_TIME assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals` covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitions. Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission. * fix(bigquery): scale default partition window to granularity for MONTH/YEAR MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were given the same 3-day window as DAY partitions, which silently produced empty profiler samples for any monthly/yearly table whose most-recent partition is older than 3 days — exactly the failure this PR already fixed for hourly/daily tables. Introduces `_bigquery_window_for_granularity()` mapping each BigQuery partition granularity to an appropriately-sized window: - HOUR → 24 h - DAY → 3 days - MONTH → 35 days (covers a full month + buffer) - YEAR → 370 days (covers a full year + buffer) Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY cases. * fix(profiler): fire empty-partition sample warning on the default sampling path The zero-row partition warning was only reached from _get_asset_row_count, which runs on the DYNAMIC sample-size path. A default profiler or auto-classification run sets no profileSampleConfig, so that method is never called and the warning never fired for the exact silent-empty-sample case in issue #33084. Move the warning to fetch_sample_data, where every sampling path reads the sample rows, so it fires regardless of sample config and reuses the already fetched rows instead of running an extra COUNT(*). Tailor the message to the partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the misleading "widen the window" time-window guidance. Update test_profiler_partitions.py for the widened auto-detected defaults (DAY interval 3, HOUR interval 24). * test(sampler): cover _warn_empty_partition wording branches Add unit tests for the empty-partition warning added for issue #33084. The warning was previously untested (flagged by gitar-bot): a refactor could drop it with no test failing. Tests pin the time-based branch (TIME_UNIT and INGESTION_TIME surface the window interval and unit), the non-time-based branch (INTEGER_RANGE and other types get the generic message, not misleading time-window guidance), and edge cases (non-enum interval unit, missing __tablename__). * fix(sampler): narrow partition_details before access in _warn_empty_partition basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on self.partition_details.partitionInterval / .partitionIntervalUnit because partition_details is Optional and the method did not narrow it. Bind it to a local and early-return on None so the attribute access is type-safe; behaviour is unchanged (the caller only invokes this when partition_details is set). --------- (cherry picked from commit 316e84c) Co-authored-by: zerafachris <christopher.zerafa@blocklabs.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Akash Verma <138790903+akashverma0786@users.noreply.github.com> Co-authored-by: Akash Verma <akashverma0768@gmail.com> Co-authored-by: Akash Verma <akku9711840768@gmail.com>
…mpty samples (#33221) (#34088) * fix(bigquery): widen auto-detected partition window to avoid silent empty samples (#33084) Time-unit and ingestion-time BigQuery partitions previously used a 1-unit window (1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window — GA4 exports (~2-day lag), weekly aggregates, slow-loading sources — was silently sampled as an empty result. The profiler still reported success, so tables that were never actually examined were indistinguishable from tables that were examined and found clean. Changes: • `partition.py` — increase the auto-detected default from 1 to 3 days for DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for HOUR-granularity partitions. Per-table profiler config overrides are unaffected: any explicitly set `partitionInterval` continues to take precedence via `get_partition_details`. • `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table COUNT(*) returns 0 rows, naming the table, current window, and the docs link to widen it. This makes any remaining narrow-window case visible in run logs instead of silent. • `tests/unit/test_partition.py` — update the two existing INGESTION_TIME assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals` covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitions. Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission. * fix(bigquery): scale default partition window to granularity for MONTH/YEAR MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were given the same 3-day window as DAY partitions, which silently produced empty profiler samples for any monthly/yearly table whose most-recent partition is older than 3 days — exactly the failure this PR already fixed for hourly/daily tables. Introduces `_bigquery_window_for_granularity()` mapping each BigQuery partition granularity to an appropriately-sized window: - HOUR → 24 h - DAY → 3 days - MONTH → 35 days (covers a full month + buffer) - YEAR → 370 days (covers a full year + buffer) Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY cases. * fix(profiler): fire empty-partition sample warning on the default sampling path The zero-row partition warning was only reached from _get_asset_row_count, which runs on the DYNAMIC sample-size path. A default profiler or auto-classification run sets no profileSampleConfig, so that method is never called and the warning never fired for the exact silent-empty-sample case in issue #33084. Move the warning to fetch_sample_data, where every sampling path reads the sample rows, so it fires regardless of sample config and reuses the already fetched rows instead of running an extra COUNT(*). Tailor the message to the partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the misleading "widen the window" time-window guidance. Update test_profiler_partitions.py for the widened auto-detected defaults (DAY interval 3, HOUR interval 24). * test(sampler): cover _warn_empty_partition wording branches Add unit tests for the empty-partition warning added for issue #33084. The warning was previously untested (flagged by gitar-bot): a refactor could drop it with no test failing. Tests pin the time-based branch (TIME_UNIT and INGESTION_TIME surface the window interval and unit), the non-time-based branch (INTEGER_RANGE and other types get the generic message, not misleading time-window guidance), and edge cases (non-enum interval unit, missing __tablename__). * fix(sampler): narrow partition_details before access in _warn_empty_partition basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on self.partition_details.partitionInterval / .partitionIntervalUnit because partition_details is Optional and the method did not narrow it. Bind it to a local and early-return on None so the attribute access is type-safe; behaviour is unchanged (the caller only invokes this when partition_details is set). --------- (cherry picked from commit 316e84c) Co-authored-by: zerafachris <christopher.zerafa@blocklabs.io> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Akash Verma <138790903+akashverma0786@users.noreply.github.com> Co-authored-by: Akash Verma <akashverma0768@gmail.com> Co-authored-by: Akash Verma <akku9711840768@gmail.com>
…ay window (open-metadata#34185) PR open-metadata#33221 widened the auto-detected partition window for DAY-granularity BigQuery partitions from 1 day to 3 days. The partition predicate is `partition_col >= CAST(CURRENT_DATE - interval N DAY AS DATE)`, and the E2E fixture table holds one row per day, so the profiler now samples 3 rows where it previously sampled 1. That PR updated the affected unit tests but not this E2E assertion, which is schedule-only and therefore did not run on the PR. Fixes open-metadata#34184 Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>



Why
Fixes #33084
BigQuery TIME_UNIT and INGESTION_TIME partitions were auto-detected with a
partitionInterval=1(1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window is silently sampled as an empty result — the profiler reports success, so tables that were never actually examined are indistinguishable from tables that were examined and found clean.Common sources of this silent failure:
What changed
ingestion/src/metadata/sampler/partition.pyTIME_UNITauto-detection:partitionInterval=1→24(HOUR) or3(DAY/MONTH/YEAR)INGESTION_TIMEauto-detection: same changepartitionIntervalintableProfilerConfig.partitioningcontinues to take precedence via the existingget_partition_detailsearly-return pathingestion/src/metadata/sampler/sqlalchemy/sampler.py_get_asset_row_count(): emitlogger.warningwhen a partitioned-tableCOUNT(*)returns 0 rows, naming the table, current window, and the docs link to override it. This makes any remaining narrow-window case visible in run logs rather than silent.ingestion/tests/unit/test_partition.py1→24(HOUR) and3(DAY)test_bigquery_time_unit_partition_default_intervalscovering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitionsTest evidence
The OpenMetadata ingestion package requires generated schema files not present in a shallow clone, which prevented running pytest in isolation. Logic was verified via a standalone script that mocks the pydantic models and calls the same branching logic directly — all 5 cases pass:
Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.
The PR appears safe to merge, with a non-blocking correction recommended for the warning shown on empty integer-range partitions.
Findings
Summary
This PR widens automatically inferred BigQuery profiling windows and adds visibility when a partition-filtered count remains empty.
Reviews (1) · Last reviewed commit: "Merge branch 'main' into fix/bigquery-pa..."