Skip to content

fix(bigquery): widen auto-detected partition window to avoid silent empty samples - #33221

Merged
akashverma0786 merged 9 commits into
open-metadata:mainfrom
zerafachris:fix/bigquery-partition-zero-row-warning
Sep 27, 2026
Merged

akashverma0786 merged 9 commits into
open-metadata:mainfrom
zerafachris:fix/bigquery-partition-zero-row-warning

Conversation

@zerafachris

@zerafachris zerafachris commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Why

Fixes #33084

BigQuery TIME_UNIT and INGESTION_TIME partitions were auto-detected with a partitionInterval=1 (1 day for DAY granularity, 1 hour for HOUR granularity). Any table whose most-recent partition is older than that window is silently sampled as an empty result — the profiler reports success, so tables that were never actually examined are indistinguishable from tables that were examined and found clean.

Common sources of this silent failure:

  • GA4 export tables (~2-day ingestion lag)
  • Weekly or monthly aggregate tables
  • Any source with a slow or batched load cadence

What changed

ingestion/src/metadata/sampler/partition.py

  • TIME_UNIT auto-detection: partitionInterval=1 → 24 (HOUR) or 3 (DAY/MONTH/YEAR)
  • INGESTION_TIME auto-detection: same change
  • Per-table profiler config overrides are fully preserved — any explicitly set partitionInterval in tableProfilerConfig.partitioning continues to take precedence via the existing get_partition_details early-return path

ingestion/src/metadata/sampler/sqlalchemy/sampler.py

  • _get_asset_row_count(): emit logger.warning when a partitioned-table COUNT(*) returns 0 rows, naming the table, current window, and the docs link to override it. This makes any remaining narrow-window case visible in run logs rather than silent.

ingestion/tests/unit/test_partition.py

  • Updated existing INGESTION_TIME assertions from 1 → 24 (HOUR) and 3 (DAY)
  • Added test_bigquery_time_unit_partition_default_intervals covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT partitions

Test evidence

The OpenMetadata ingestion package requires generated schema files not present in a shallow clone, which prevented running pytest in isolation. Logic was verified via a standalone script that mocks the pydantic models and calls the same branching logic directly — all 5 cases pass:

PASS TIME_UNIT DAY: interval=3 unit=DAY
PASS TIME_UNIT HOUR: interval=24 unit=HOUR
PASS INGESTION_TIME DAY: interval=3 col=_PARTITIONDATE
PASS INGESTION_TIME HOUR: interval=24 col=_PARTITIONTIME
PASS INTEGER_RANGE: interval=None rangeStart=1

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

RetriggerConfidence Score: 4/5

The PR appears safe to merge, with a non-blocking correction recommended for the warning shown on empty integer-range partitions.

Findings

  1. P2 Misleading Integer Partition Warning ▶
Summary

This PR widens automatically inferred BigQuery profiling windows and adds visibility when a partition-filtered count remains empty.

  • Maps hourly partitions to a 24-hour window, daily partitions to three days, and monthly/yearly partitions to wider day-based windows.
  • Preserves explicitly configured table-level partition settings.
  • Logs zero-row partition counts with override guidance, although that guidance currently also applies incorrectly to integer-range partitions.
  • Extends unit coverage for BigQuery time-unit and ingestion-time defaults.

Reviews (1) · Last reviewed commit: "Merge branch 'main' into fix/bigquery-pa..."

…mpty samples (open-metadata#33084)

Time-unit and ingestion-time BigQuery partitions previously used a 1-unit
window (1 day for DAY granularity, 1 hour for HOUR granularity).  Any table
whose most-recent partition is older than that window — GA4 exports (~2-day
lag), weekly aggregates, slow-loading sources — was silently sampled as an
empty result.  The profiler still reported success, so tables that were never
actually examined were indistinguishable from tables that were examined and
found clean.

Changes:
• `partition.py` — increase the auto-detected default from 1 to 3 days for
  DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for
  HOUR-granularity partitions.  Per-table profiler config overrides are
  unaffected: any explicitly set `partitionInterval` continues to take
  precedence via `get_partition_details`.
• `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table
  COUNT(*) returns 0 rows, naming the table, current window, and the docs link
  to widen it.  This makes any remaining narrow-window case visible in run logs
  instead of silent.
• `tests/unit/test_partition.py` — update the two existing INGESTION_TIME
  assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals`
  covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT
  partitions.

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.
@zerafachris
zerafachris requested review from a team and akashverma0786 as code owners September 11, 2026 14:23
@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

…H/YEAR

MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were
given the same 3-day window as DAY partitions, which silently produced empty
profiler samples for any monthly/yearly table whose most-recent partition is
older than 3 days — exactly the failure this PR already fixed for hourly/daily
tables.

Introduces `_bigquery_window_for_granularity()` mapping each BigQuery
partition granularity to an appropriately-sized window:
- HOUR  → 24 h
- DAY   →  3 days
- MONTH → 35 days (covers a full month + buffer)
- YEAR  → 370 days (covers a full year + buffer)

Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY
cases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@akashverma0786 akashverma0786 added the safe to test Add this label to run secure Github workflows on PRs label Sep 21, 2026
Comment on lines +207 to +216
if row_count == 0:
logger.warning(
"Partition filter on table '%s' returned 0 rows. "
"The partition window (interval=%s %s) may not cover the table's most-recent data. "
"Override the profiler partition config for this table to widen the window. "
"See https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/"
"profiler/workflow#4.-updating-profiler-setting-at-the-table-level",
getattr(self.raw_dataset, "__tablename__", "unknown"),
getattr(self.partition_details, "partitionInterval", "?"),
getattr(self.partition_details, "partitionIntervalUnit", "?"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Misleading Integer Partition Warning

When an INTEGER_RANGE partition returns no rows, this warning reports interval=None None and advises operators to widen the time window. Integer partitions instead use partitionIntegerRangeStart and partitionIntegerRangeEnd, so this guidance points operators toward a configuration change that cannot fix the problem. Please tailor the warning to the partition type or limit this message to time-based partitions.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

…pling path

The zero-row partition warning was only reached from _get_asset_row_count,
which runs on the DYNAMIC sample-size path. A default profiler or
auto-classification run sets no profileSampleConfig, so that method is never
called and the warning never fired for the exact silent-empty-sample case in
issue open-metadata#33084.

Move the warning to fetch_sample_data, where every sampling path reads the
sample rows, so it fires regardless of sample config and reuses the already
fetched rows instead of running an extra COUNT(*). Tailor the message to the
partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the
misleading "widen the window" time-window guidance.

Update test_profiler_partitions.py for the widened auto-detected defaults
(DAY interval 3, HOUR interval 24).
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@akashverma0786 akashverma0786 added safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch labels Sep 26, 2026
Add unit tests for the empty-partition warning added for issue open-metadata#33084. The
warning was previously untested (flagged by gitar-bot): a refactor could drop
it with no test failing. Tests pin the time-based branch (TIME_UNIT and
INGESTION_TIME surface the window interval and unit), the non-time-based branch
(INTEGER_RANGE and other types get the generic message, not misleading
time-window guidance), and edge cases (non-enum interval unit, missing
__tablename__).
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 26, 2026
…artition

basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on
self.partition_details.partitionInterval / .partitionIntervalUnit because
partition_details is Optional and the method did not narrow it. Bind it to a
local and early-return on None so the attribute access is type-safe; behaviour
is unchanged (the caller only invokes this when partition_details is set).
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@gitar-bot

gitar-bot Bot commented Sep 27, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 2 closed / 2 findings

🟡 Medium risk · BigQuery profiling widens automatic partition windows and adds empty-sample warnings

Widens BigQuery auto-detected partition windows from 1 day to 24 hours (HOUR) or 3 days (DAY/MONTH/YEAR) to avoid silent empty samples, and adds logging when partitioned tables return zero rows with override guidance. Explicitly configured partition settings are preserved. Unit tests cover the new TIME_UNIT and INGESTION_TIME defaults.

✅ 2 closed
✅ Edge Case: 3-day window still silently empty for MONTH/YEAR partitions

📄 ingestion/src/metadata/sampler/partition.py:124 📄 ingestion/src/metadata/sampler/partition.py:136
For TIME_UNIT/INGESTION_TIME partitions with MONTH or YEAR granularity, is_hourly is False so the config gets partitionIntervalUnit=DAY, partitionInterval=3, which build_partition_predicate turns into partition_column >= CURRENT_DATE - INTERVAL 3 DAY. A monthly/yearly table whose freshest data is older than 3 days is still sampled as empty — the exact silent-empty failure this PR targets. The comment claims 3 days covers "daily/monthly/yearly", but 3 days is far narrower than a month or year; consider scaling the interval to the granularity (e.g. ~35 days for MONTH, ~370 for YEAR). The new 0-row warning does at least surface the case now.

✅ Quality: New empty-partition warning in fetch_sample_data has no tests

📄 ingestion/src/metadata/sampler/sqlalchemy/sampler.py:339-353
This commit adds _warn_empty_partition and calls it from fetch_sample_data when partition_details is set and the sample is empty. That is the fix for the silent-empty-sample problem in #33084, but no test covers it. A search of ingestion/tests finds no reference to _warn_empty_partition or the 'returned 0 rows' message, so nothing checks either wording branch (TIME_UNIT/INGESTION_TIME vs. other types) or that the warning stays silent when rows come back. A later refactor could drop the warning without any test failing. Add a pytest test (plain assert, fixtures, a Test* class that does not inherit TestCase) that mocks the session to return no rows and uses caplog to check the warning for a TIME_UNIT config and for an INTEGER_RANGE config.

Review coverage

📋 Rules No rules evaluated

🧪 Functional validation Not enabled · Set up

🤖 Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@sonarqubecloud

Copy link
Copy Markdown

@akashverma0786
akashverma0786 added this pull request to the merge queue Sep 27, 2026
Merged via the queue into open-metadata:main with commit 316e84c Sep 27, 2026
178 of 192 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Backport to 2.0 is under way. It is built from openmetadata-collate, which holds the credentials needed to open a PR here, and it will comment again with the result — nothing is needed from you unless it reports otherwise.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 Backport to 1.13 is under way. It is built from openmetadata-collate, which holds the credentials needed to open a PR here, and it will comment again with the result — nothing is needed from you unless it reports otherwise.

@gh-bot-collate

Copy link
Copy Markdown

🤖 Backport to 1.13 opened as #34087. Claude resolved a cherry-pick conflict to build it. It merges itself once the checks pass, so read the diff now if this change is a sensitive one.

@gh-bot-collate

Copy link
Copy Markdown

🤖 Backport to 2.0 opened as #34088. Claude resolved a cherry-pick conflict to build it. It merges itself once the checks pass, so read the diff now if this change is a sensitive one.

gh-bot-collate Bot added a commit that referenced this pull request Sep 28, 2026
…mpty samples (#33221) (#34087)

* fix(bigquery): widen auto-detected partition window to avoid silent empty samples (#33084)

Time-unit and ingestion-time BigQuery partitions previously used a 1-unit
window (1 day for DAY granularity, 1 hour for HOUR granularity).  Any table
whose most-recent partition is older than that window — GA4 exports (~2-day
lag), weekly aggregates, slow-loading sources — was silently sampled as an
empty result.  The profiler still reported success, so tables that were never
actually examined were indistinguishable from tables that were examined and
found clean.

Changes:
• `partition.py` — increase the auto-detected default from 1 to 3 days for
  DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for
  HOUR-granularity partitions.  Per-table profiler config overrides are
  unaffected: any explicitly set `partitionInterval` continues to take
  precedence via `get_partition_details`.
• `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table
  COUNT(*) returns 0 rows, naming the table, current window, and the docs link
  to widen it.  This makes any remaining narrow-window case visible in run logs
  instead of silent.
• `tests/unit/test_partition.py` — update the two existing INGESTION_TIME
  assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals`
  covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT
  partitions.

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

* fix(bigquery): scale default partition window to granularity for MONTH/YEAR

MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were
given the same 3-day window as DAY partitions, which silently produced empty
profiler samples for any monthly/yearly table whose most-recent partition is
older than 3 days — exactly the failure this PR already fixed for hourly/daily
tables.

Introduces `_bigquery_window_for_granularity()` mapping each BigQuery
partition granularity to an appropriately-sized window:
- HOUR  → 24 h
- DAY   →  3 days
- MONTH → 35 days (covers a full month + buffer)
- YEAR  → 370 days (covers a full year + buffer)

Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY
cases.



* fix(profiler): fire empty-partition sample warning on the default sampling path

The zero-row partition warning was only reached from _get_asset_row_count,
which runs on the DYNAMIC sample-size path. A default profiler or
auto-classification run sets no profileSampleConfig, so that method is never
called and the warning never fired for the exact silent-empty-sample case in
issue #33084.

Move the warning to fetch_sample_data, where every sampling path reads the
sample rows, so it fires regardless of sample config and reuses the already
fetched rows instead of running an extra COUNT(*). Tailor the message to the
partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the
misleading "widen the window" time-window guidance.

Update test_profiler_partitions.py for the widened auto-detected defaults
(DAY interval 3, HOUR interval 24).

* test(sampler): cover _warn_empty_partition wording branches

Add unit tests for the empty-partition warning added for issue #33084. The
warning was previously untested (flagged by gitar-bot): a refactor could drop
it with no test failing. Tests pin the time-based branch (TIME_UNIT and
INGESTION_TIME surface the window interval and unit), the non-time-based branch
(INTEGER_RANGE and other types get the generic message, not misleading
time-window guidance), and edge cases (non-enum interval unit, missing
__tablename__).

* fix(sampler): narrow partition_details before access in _warn_empty_partition

basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on
self.partition_details.partitionInterval / .partitionIntervalUnit because
partition_details is Optional and the method did not narrow it. Bind it to a
local and early-return on None so the attribute access is type-safe; behaviour
is unchanged (the caller only invokes this when partition_details is set).

---------





(cherry picked from commit 316e84c)

Co-authored-by: zerafachris <christopher.zerafa@blocklabs.io>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Akash Verma <138790903+akashverma0786@users.noreply.github.com>
Co-authored-by: Akash Verma <akashverma0768@gmail.com>
Co-authored-by: Akash Verma <akku9711840768@gmail.com>
gh-bot-collate Bot added a commit that referenced this pull request Sep 28, 2026
…mpty samples (#33221) (#34088)

* fix(bigquery): widen auto-detected partition window to avoid silent empty samples (#33084)

Time-unit and ingestion-time BigQuery partitions previously used a 1-unit
window (1 day for DAY granularity, 1 hour for HOUR granularity).  Any table
whose most-recent partition is older than that window — GA4 exports (~2-day
lag), weekly aggregates, slow-loading sources — was silently sampled as an
empty result.  The profiler still reported success, so tables that were never
actually examined were indistinguishable from tables that were examined and
found clean.

Changes:
• `partition.py` — increase the auto-detected default from 1 to 3 days for
  DAY-granularity TIME_UNIT / INGESTION_TIME partitions, and from 1 to 24 for
  HOUR-granularity partitions.  Per-table profiler config overrides are
  unaffected: any explicitly set `partitionInterval` continues to take
  precedence via `get_partition_details`.
• `sqlalchemy/sampler.py` — emit a `logger.warning` when a partitioned-table
  COUNT(*) returns 0 rows, naming the table, current window, and the docs link
  to widen it.  This makes any remaining narrow-window case visible in run logs
  instead of silent.
• `tests/unit/test_partition.py` — update the two existing INGESTION_TIME
  assertions for the new defaults; add `test_bigquery_time_unit_partition_default_intervals`
  covering both DAY (3-day window) and HOUR (24-hour window) for TIME_UNIT
  partitions.

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

* fix(bigquery): scale default partition window to granularity for MONTH/YEAR

MONTH- and YEAR-granularity TIME_UNIT and INGESTION_TIME partitions were
given the same 3-day window as DAY partitions, which silently produced empty
profiler samples for any monthly/yearly table whose most-recent partition is
older than 3 days — exactly the failure this PR already fixed for hourly/daily
tables.

Introduces `_bigquery_window_for_granularity()` mapping each BigQuery
partition granularity to an appropriately-sized window:
- HOUR  → 24 h
- DAY   →  3 days
- MONTH → 35 days (covers a full month + buffer)
- YEAR  → 370 days (covers a full year + buffer)

Two new test cases (MONTH and YEAR) added alongside the existing HOUR/DAY
cases.



* fix(profiler): fire empty-partition sample warning on the default sampling path

The zero-row partition warning was only reached from _get_asset_row_count,
which runs on the DYNAMIC sample-size path. A default profiler or
auto-classification run sets no profileSampleConfig, so that method is never
called and the warning never fired for the exact silent-empty-sample case in
issue #33084.

Move the warning to fetch_sample_data, where every sampling path reads the
sample rows, so it fires regardless of sample config and reuses the already
fetched rows instead of running an extra COUNT(*). Tailor the message to the
partition type so INTEGER_RANGE and COLUMN_VALUE partitions no longer get the
misleading "widen the window" time-window guidance.

Update test_profiler_partitions.py for the widened auto-detected defaults
(DAY interval 3, HOUR interval 24).

* test(sampler): cover _warn_empty_partition wording branches

Add unit tests for the empty-partition warning added for issue #33084. The
warning was previously untested (flagged by gitar-bot): a refactor could drop
it with no test failing. Tests pin the time-based branch (TIME_UNIT and
INGESTION_TIME surface the window interval and unit), the non-time-based branch
(INTEGER_RANGE and other types get the generic message, not misleading
time-window guidance), and edge cases (non-enum interval unit, missing
__tablename__).

* fix(sampler): narrow partition_details before access in _warn_empty_partition

basedpyright (static-checks, 3.10) flagged reportOptionalMemberAccess on
self.partition_details.partitionInterval / .partitionIntervalUnit because
partition_details is Optional and the method did not narrow it. Bind it to a
local and early-return on None so the attribute access is type-safe; behaviour
is unchanged (the caller only invokes this when partition_details is set).

---------





(cherry picked from commit 316e84c)

Co-authored-by: zerafachris <christopher.zerafa@blocklabs.io>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Akash Verma <138790903+akashverma0786@users.noreply.github.com>
Co-authored-by: Akash Verma <akashverma0768@gmail.com>
Co-authored-by: Akash Verma <akku9711840768@gmail.com>
Chronillogical-Potato pushed a commit to Chronillogical-Potato/OpenMetadata that referenced this pull request Sep 30, 2026
…ay window (open-metadata#34185)

PR open-metadata#33221 widened the auto-detected partition window for DAY-granularity
BigQuery partitions from 1 day to 3 days. The partition predicate is
`partition_col >= CAST(CURRENT_DATE - interval N DAY AS DATE)`, and the E2E
fixture table holds one row per day, so the profiler now samples 3 rows where
it previously sampled 1.

That PR updated the affected unit tests but not this E2E assertion, which is
schedule-only and therefore did not run on the PR.

Fixes open-metadata#34184

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQuery auto-detected partition window is 1 day, so tables not updated daily sample zero rows and classify as clean

2 participants