Skip to content

Improved bulk loading for binary doc values - #138631

Merged
martijnvg merged 9 commits into
elastic:mainfrom
parkertimmins:parker/improved-bulk-loading-compressed-binary-dv-dense
Nov 27, 2025
Merged

martijnvg merged 9 commits into
elastic:mainfrom
parkertimmins:parker/improved-bulk-loading-compressed-binary-dv-dense

Conversation

@parkertimmins

@parkertimmins parkertimmins commented Nov 25, 2025 •

Copy link
Copy Markdown
Contributor

By decompressing all matching blocks into one byte array and appending that directly into the block builder.

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Hi @parkertimmins, I've created a changelog YAML for you.

@martijnvg martijnvg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

int blockDocOffset = 0;
int blockByteOffset = 0;
long[] offsets = new long[count + 1];
List<BytesRef> decompressedBlocks = new ArrayList<>();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a TODO, to account memory usage of decompressedBlocks? This could grow and have a significant jvm heap memory usage.

Eventually this will be accounted, but that happens when we hand over byte[] to the builder.

@martijnvg

Copy link
Copy Markdown
Member

I suspect most of the bwc failures are because of:

Caused by: java.lang.IllegalStateException: Node Sb8dLoCzTniI51zwITrA7A is missing required features [semantic_text.highlighter.vector_similarity_support]

@martijnvg martijnvg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM2

@martijnvg martijnvg mentioned this pull request Nov 27, 2025
3 of 6 tasks
@martijnvg
martijnvg enabled auto-merge (squash) November 27, 2025 09:18
@martijnvg
martijnvg merged commit 65426e8 into elastic:main Nov 27, 2025
30 checks passed
martijnvg added a commit that referenced this pull request Dec 15, 2025
This change iterates on top of #138631 and attempts to simplify the bulk read logic and improves testing.
It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder.

Relates to #138983
parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this pull request Dec 17, 2025
This change iterates on top of elastic#138631 and attempts to simplify the bulk read logic and improves testing.
It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder.

Relates to elastic#138983
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants