Skip to content

[ES|QL] Keep LIMIT BY above MV_EXPAND when needed - #149911

Merged
MattAlp merged 3 commits into
mainfrom
matt/esql-limitby-mvexpand-stack-B
Jun 26, 2026
Merged

MattAlp merged 3 commits into
mainfrom
matt/esql-limitby-mvexpand-stack-B

Conversation

@MattAlp

@MattAlp MattAlp commented May 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR fixes a logical optimization bug where LIMIT BY could be pushed below MV_EXPAND even when grouping attributes are produced by the expand target.

Why

For plans like:

ROW x = 1
| MV_EXPAND x
| LIMIT 1 BY x

pushing LIMIT BY below MV_EXPAND can produce unbound grouping references and fail with missing-reference errors.

This change keeps LIMIT BY above MV_EXPAND when grouping attributes are defined by the child, preserving attribute liveness through optimization.

Closes #148513

@MattAlp MattAlp changed the title Guard LIMIT BY pushdown across MV_EXPAND [ES|QL] Keep LIMIT BY above MV_EXPAND when needed May 26, 2026
@MattAlp MattAlp added >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL priority:high A label for assessing bug priority to be used by ES engineers and removed priority:high A label for assessing bug priority to be used by ES engineers labels May 26, 2026
@MattAlp
MattAlp force-pushed the matt/esql-limitby-mvexpand-stack-B branch 3 times, most recently from cfd4fed to 0ec59c1 Compare May 27, 2026 14:32
@MattAlp
MattAlp force-pushed the matt/esql-limitby-mvexpand-stack-B branch 2 times, most recently from 55d0b90 to ff4d5f5 Compare May 28, 2026 13:28
@MattAlp
MattAlp changed the base branch from matt/esql-analyzer-stack-A to main May 28, 2026 13:30
@MattAlp
MattAlp marked this pull request as ready for review May 28, 2026 14:13
@MattAlp
MattAlp requested a review from fang-xing-esql May 28, 2026 14:13
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@MattAlp MattAlp added the v9.5.0 label May 28, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @MattAlp, I've created a changelog YAML for you.

@github-actions

github-actions Bot commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

@github-actions

Copy link
Copy Markdown
Contributor

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

@ivancea ivancea left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ivancea

ivancea commented May 28, 2026

Copy link
Copy Markdown
Contributor

Oh, and also remove the "error-skip" from the generative tests here:

/**
* See https://github.com/elastic/elasticsearch/issues/148513
* <p>
* The same root cause manifests as either {@code LimitBy[...]} (no upstream SORT) or {@code TopNBy[...]}
* (when an upstream SORT gets combined with the LIMIT BY into a TopNBy). DEDUP uses the same LimitBy
* plan internally, so it has the same missing-reference failure after MV_EXPAND.
*/
static boolean isLimitByMvExpandBug(String errorMessage, String query) {
if (errorMessage == null || query == null) {
return false;
}
if (OPTIMIZED_INCORRECTLY_LIMITBY_PATTERN.matcher(errorMessage).matches() == false) {
return false;
}
return MV_EXPAND_COMMAND_PATTERN.matcher(query).find()
&& (LIMIT_BY_COMMAND_PATTERN.matcher(query).find() || DEDUP_COMMAND_PATTERN.matcher(query).find());
}

Or it will remain skipped after this is merged

@MattAlp

MattAlp commented May 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review

@MattAlp
MattAlp force-pushed the matt/esql-limitby-mvexpand-stack-B branch from 7ec7331 to 861153f Compare May 28, 2026 15:54
@MattAlp
MattAlp force-pushed the matt/esql-limitby-mvexpand-stack-B branch 2 times, most recently from 6e086d4 to 37cf98b Compare June 22, 2026 14:47
MattAlp and others added 3 commits June 26, 2026 08:16
Keep LIMIT BY above MV_EXPAND when the grouping keys are produced by
the expand target. This avoids producing plans with unbound grouping
references after pushdown and locks in the behavior with a regression
test.
Add a dedicated ESQL capability and require it for the new LIMIT BY
plus MV_EXPAND CSV regression cases so BWC lanes running older nodes
skip scenarios that are known-unfixed there. Also drop stale generative
tests that referenced a helper removed by this PR.

Co-authored-by: Cursor <cursoragent@cursor.com>
@MattAlp
MattAlp force-pushed the matt/esql-limitby-mvexpand-stack-B branch from 37cf98b to 40bfe5d Compare June 26, 2026 12:16
@MattAlp
MattAlp merged commit 34c6b3b into main Jun 26, 2026
41 checks passed
@MattAlp
MattAlp deleted the matt/esql-limitby-mvexpand-stack-B branch June 26, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >bug Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ESQL: LIMIT BY with MV_EXPAND fails with plan optimized incorrectly

3 participants