Skip to content

Shard bulk failures yield O(N) message sizes #157861

Open

Description

@DaveCTurner

When a shard bulk experiences a shard-level failure, we add the same exception instance to all its items:

private void handleShardFailure(BulkShardRequest bulkShardRequest, ProjectMetadata projectMetadata, Exception e) {
// create failures for all relevant requests
for (BulkItemRequest request : bulkShardRequest.items()) {
final String indexName = request.index();
DocWriteRequest<?> docWriteRequest = request.request();
IndexDocFailureStoreStatus failureStoreStatus = processFailure(request, projectMetadata, e);
addFailure(docWriteRequest, request.id(), indexName, e, failureStoreStatus);
}
}

There's no way to capture this cross-item sharing in the transport protocol today, so instead we end up transporting N copies of the same exception over the wire (e.g. from an ingest node back to a coordinating node). Exceptions can be ~10kiB or more in size, and shard bulks can include many thousands of items, so this can end up producing pretty enormous transport messages.

Relates #157859 which describes a much worse situation in which these message sizes become quadratic in the number of items. But I still think we should consider finding a way to capture shared exceptions more efficiently than what we do today.

Activity

  1. added
    :Distributed/CRUDA catch all label for issues around indexing, updating and getting a doc by id. Not search.
    on Aug 27, 2026
  2. elasticsearchmachine commented on Aug 27, 2026

    @elasticsearchmachine
    Collaborator

    Pinging @elastic/es-distributed (Team:Distributed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    :Distributed/CRUDA catch all label for issues around indexing, updating and getting a doc by id. Not search.>bugTeam:DistributedMeta label for distributed team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions