Repository navigation
Tidy up es819 tsdb doc values binary bulk decoding - #139176
Conversation
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
| throws IOException { | ||
| // Lookup the first blockId using binary search and subsequent blocks can be scanned because query and values are dense | ||
| // Also lookup the last blockId using binary search so that the buffer for values to be collected can be sized accordingly. | ||
| final long firstBlockId = findBlock(firstDocId, numBlocks, lastBlockId + 1); |
There was a problem hiding this comment.
I think we want to start from lastBlockId since we could be reading more values from the same block.
From our offline discussion, I think it should be lastBlockId == -1 ? 0 : lastBlockId.
lastBlockId + 1 worked in the existing code's binary search since it was only called when updating to the next block, due to this gate:
| System.arraycopy(uncompressedBlock, startOffset, bytes, blockByteOffset, lenValuesInBlock); | ||
| lastBlockId = endBlockId; | ||
| startDocNumForBlock = docOffsets.get(endBlockId); | ||
| limitDocNumForBlock = docOffsets.get(endBlockId + 1); |
There was a problem hiding this comment.
Do you think it will ever be needed to intersperse calls to decode and decodeBulk? If not, since startDocNumForBlock and limitDocNumForBlock are no longer used in decodeBulk we could not set them here. This would be nice, since having multiple state variables is confusing. On the other hand, decode could not be called after decodeBulk since the state would be inconsistent.
There was a problem hiding this comment.
I think there is currently no guarantee that compute engine would not interleave load a single value and bulk load values in one operation. (iirc sometimes depending on query, row and column oriented reading can be combined, even on a doc values field)
Perhaps we can let decode reuse decodeBulk? I will add a TODO, we can look into this in a follow up.
There was a problem hiding this comment.
I'm a bit worried about the overhead of calling decodeBulk for each call to decode, since decodeBulk allocates a large buffer on each call. Also, we'd want to make use of limitDocNumForBlock within decodeBulk to avoid reading from docOffsets for each value within a block. decode originally read from docOffsets directly, but caching in limitDocNumForBlock provided a good speedup. It would be easy to add this to decodeBulk, but I think that is unnecessary as we expect bulk reads to be large.
|
This really improves the clarity of the code. I like it! I held off from approving, due to the |
This change iterates on top of elastic#138631 and attempts to simplify the bulk read logic and improves testing. It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder. Relates to elastic#138983
This change iterates on top of #138631 and attempts to simplify the bulk read logic and improves testing.
It is try to simplify the bulk reading by find to first matching block id and the last matching block id for the documents values need to be loaded. Then it just de-compress and streams the values and offset into buffers that then get appended to the block builder.
Relates to #138983