Skip to content

Tidy up es819 tsdb doc values binary bulk decoding - #139176

Merged
martijnvg merged 10 commits into
elastic:mainfrom
martijnvg:iterate_BinaryDecoderDecodeBulk
Dec 15, 2025
Merged

martijnvg merged 10 commits into
elastic:mainfrom
martijnvg:iterate_BinaryDecoderDecodeBulk

Conversation

@martijnvg

@martijnvg martijnvg commented Dec 7, 2025 •

Copy link
Copy Markdown
Member

This change iterates on top of #138631 and attempts to simplify the bulk read logic and improves testing.
It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder.

Relates to #138983

@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

throws IOException {
// Lookup the first blockId using binary search and subsequent blocks can be scanned because query and values are dense
// Also lookup the last blockId using binary search so that the buffer for values to be collected can be sized accordingly.
final long firstBlockId = findBlock(firstDocId, numBlocks, lastBlockId + 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to start from lastBlockId since we could be reading more values from the same block.
From our offline discussion, I think it should be lastBlockId == -1 ? 0 : lastBlockId.

lastBlockId + 1 worked in the existing code's binary search since it was only called when updating to the next block, due to this gate:

System.arraycopy(uncompressedBlock, startOffset, bytes, blockByteOffset, lenValuesInBlock);
lastBlockId = endBlockId;
startDocNumForBlock = docOffsets.get(endBlockId);
limitDocNumForBlock = docOffsets.get(endBlockId + 1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it will ever be needed to intersperse calls to decode and decodeBulk? If not, since startDocNumForBlock and limitDocNumForBlock are no longer used in decodeBulk we could not set them here. This would be nice, since having multiple state variables is confusing. On the other hand, decode could not be called after decodeBulk since the state would be inconsistent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is currently no guarantee that compute engine would not interleave load a single value and bulk load values in one operation. (iirc sometimes depending on query, row and column oriented reading can be combined, even on a doc values field)

Perhaps we can let decode reuse decodeBulk? I will add a TODO, we can look into this in a follow up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit worried about the overhead of calling decodeBulk for each call to decode, since decodeBulk allocates a large buffer on each call. Also, we'd want to make use of limitDocNumForBlock within decodeBulk to avoid reading from docOffsets for each value within a block. decode originally read from docOffsets directly, but caching in limitDocNumForBlock provided a good speedup. It would be easy to add this to decodeBulk, but I think that is unnecessary as we expect bulk reads to be large.

@parkertimmins

Copy link
Copy Markdown
Contributor

This really improves the clarity of the code. I like it! I held off from approving, due to the lastBlockId + 1 line, but other than that, this looks good.

@martijnvg
martijnvg enabled auto-merge (squash) December 15, 2025 08:45
@martijnvg
martijnvg merged commit 3670747 into elastic:main Dec 15, 2025
35 checks passed
parkertimmins pushed a commit to parkertimmins/elasticsearch that referenced this pull request Dec 17, 2025
This change iterates on top of elastic#138631 and attempts to simplify the bulk read logic and improves testing.
It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder.

Relates to elastic#138983
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants