Skip to content

[ESQL] Normalize local StorageObject path for stats reconcile - #152837

Merged
costin merged 2 commits into
elastic:mainfrom
costin:esql/fix-external-source-cache-reconcile-windows
Jul 3, 2026
Merged

costin merged 2 commits into
elastic:mainfrom
costin:esql/fix-external-source-cache-reconcile-windows

Conversation

@costin

@costin costin commented Jul 3, 2026

Copy link
Copy Markdown
Member

LocalStorageObject built its StoragePath via "file://" + toAbsolutePath(), keeping backslashes on Windows (file://C:\dir\file), while every other producer uses StoragePath.fileUri (file:///C:/dir/file). So the stats-capture key (object.path()) never matched the planning-side SchemaCacheKey canonical path on Windows, and reconcileSourceStats silently dropped every contribution — warm queries re-scanned instead of short-circuiting. Use fileUri in the constructor (no-op on POSIX), add a cross-platform regression test. Unmutes the pushdown cold-then-warm suite.

@costin
costin requested a review from bpintea July 3, 2026 07:37
@costin
costin enabled auto-merge (squash) July 3, 2026 07:37
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 3, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @costin, I've created a changelog YAML for you.

@github-actions

github-actions Bot commented Jul 3, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Path file = createTempFile();
LocalStorageObject obj = new LocalStorageObject(file);
assertEquals(StoragePath.fileUri(file), obj.path().toString());
assertEquals(StoragePath.of(StoragePath.fileUri(file)), obj.path());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On POSIX, the old "file://" + toAbsolutePath() and the new fileUri(...) produce byte-identical strings (no backslashes to normalize), so this assertion holds whether or not the fix is applied. Since it compares against fileUri() itself rather than a literal expected value, could this pass on non-Windows CI even without the fix?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now asserts no-backslash + file:/// shape; added test-windows lane.

Comment thread muted-tests.yml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Were these confirmed to have failed on Windows specifically? The fix is a Windows-only backslash normalization (identical output on POSIX before/after), but this PR isn't labeled test-windows, so the Windows CI lane won't exercise it here. If the original mutes were from Linux CI failures, would unmuting still be safe?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows-lane failures; added test-windows label to exercise here.

// SchemaCacheKey canonicalPath, so ExternalSourceCacheService.reconcileSourceStats silently
// dropped every captured contribution and warm queries re-scanned. On POSIX both forms
// already coincide, so this is a no-op there.
this.storagePath = StoragePath.of(StoragePath.fileUri(filePath));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates LocalStorageProvider.toStoragePath's StoragePath.of(StoragePath.fileUri(...)) one-liner. Since divergence between these two producers is exactly what caused this bug, would a shared factory (e.g. StoragePath.ofLocalPath(Path)) be worth extracting so they can't drift apart again?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extracted shared StoragePath.ofLocalPath; both producers now use it.

@bpintea

bpintea commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

BTW, same as before, you might want to include Closes #... to avoid doing it maually.

@costin costin added the test-windows Trigger CI checks on Windows label Jul 3, 2026
LocalStorageObject built its StoragePath via "file://" + toAbsolutePath(),
which keeps backslashes on Windows (file://C:\dir\file), while every other
producer uses StoragePath.fileUri (file:///C:/dir/file). The stats-capture
key (object.path()) therefore never matched the planning-side
SchemaCacheKey canonicalPath on Windows, so reconcileSourceStats silently
dropped every captured contribution and warm queries re-scanned instead of
short-circuiting. Use fileUri in the constructor (no-op on POSIX) and add a
cross-platform regression test. Unmutes the pushdown cold-then-warm suite.
@costin
costin force-pushed the esql/fix-external-source-cache-reconcile-windows branch from dd4fae3 to 16c6e61 Compare July 3, 2026 11:23
@costin
costin disabled auto-merge July 3, 2026 17:12
@costin
costin merged commit e39b505 into elastic:main Jul 3, 2026
46 of 48 checks passed
@costin
costin deleted the esql/fix-external-source-cache-reconcile-windows branch July 3, 2026 17:23
szybia added a commit to szybia/elasticsearch that referenced this pull request Jul 3, 2026
…uisition

* upstream/main: (65 commits)
  Use forceEvictEntries for all forceEvict methods (elastic#152846)
  [ESQL] Normalize local StorageObject path for stats reconcile (elastic#152837)
  Report vectordb_document usage in _xpack/usage (elastic#152780)
  Mute org.elasticsearch.common.lucene.LuceneTests testSearchWithoutBulkScorer elastic#152898
  Make DiskBBQ CentroidSupplier generic in the indexing method (elastic#152863)
  AGGS: Prevent OOM in test (elastic#152690)
  [CI] Fix bc upgrade disk size (elastic#152895)
  Mute org.elasticsearch.xpack.transform.integration.TransformIT testStopWaitForCheckpoint elastic#149663
  Always report script queries as maximum cost (elastic#152856)
  test(tsdb): cover `ES95` ordinal doc values (elastic#152802)
  Skip semantic_field source yaml tests in release (elastic#152852)
  Mute org.elasticsearch.xpack.esql.action.NdJsonScalarObjectConflictIT testSkipRowPolicyNullFillsAndWarnsClient elastic#152891
  [ESQL] Async STOP / cancel / DELETE contract for EXTERNAL queries (elastic#152585)
  Unmute rolling upgrade tests (elastic#152715)
  [ESQL] Fix ExternalSourceExec BWC serialization test types (elastic#152836)
  Add AllocateReshardSplitTargetPrimaryCommand (elastic#152806)
  [ESQL] Reconcile temporal stat units in cross-file merge (elastic#152859)
  ES|QL: error on `match` options w/ runtime field (elastic#152597)
  Improve allocation explain API for canRemain decisions (elastic#152121)
  [Inference API] Remove fallbackRegion from RegionPolicy (elastic#152798)
  ...
burqen pushed a commit to burqen/elasticsearch that referenced this pull request Jul 7, 2026
…c#152837)

`LocalStorageObject` built its `StoragePath` via `"file://" + toAbsolutePath()`, keeping backslashes on Windows (`file://C:\dir\file`), while every other producer uses `StoragePath.fileUri` (`file:///C:/dir/file`). So the stats-capture key (`object.path()`) never matched the planning-side `SchemaCacheKey` canonical path on Windows, and `reconcileSourceStats` silently dropped every contribution — warm queries re-scanned instead of short-circuiting. Use `fileUri` in the constructor (no-op on POSIX), add a cross-platform regression test. Unmutes the pushdown cold-then-warm suite.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >bug ES|QL|DS ES|QL datasources Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) test-windows Trigger CI checks on Windows v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants