Skip to content

Failure-store failures yield O(N²) message sizes #157859

Closed

Description

@DaveCTurner

A _bulk request whose items all fail with the same shard-level exception, and whose failure-store redirect also fails, produces a BulkResponse whose transport and XContent encodings have sizes which are quadratic in the number of items. The transport encoding size is particularly problematic because Elasticsearch materializes the entire response on-heap if it needs to send it back from an ingest node to a different coordinating node.

We observed this in v9.4.3 (linked code below matches this version) but the mechanism has been around for a while and still exists in main. In the example, a shard bulk failed with ElasticsearchSecurityException E1, redirecting the work to the corresponding failure store, but the failure store write also failed with ElasticsearchSecurityException E2.

The problem comes because Elasticsearch records E1 as the failure reason for every item in the bulk ...

private void handleShardFailure(BulkShardRequest bulkShardRequest, ProjectMetadata projectMetadata, Exception e) {
// create failures for all relevant requests
for (BulkItemRequest request : bulkShardRequest.items()) {
final String indexName = request.index();
DocWriteRequest<?> docWriteRequest = request.request();
IndexDocFailureStoreStatus failureStoreStatus = processFailure(request, projectMetadata, e);
addFailure(docWriteRequest, request.id(), indexName, e, failureStoreStatus);
}
}

... and then adds E2 as a suppressed exception of E1 once for each item in the bulk ...

BulkItemResponse bulkItemResponse = responses.get(idx);
if (bulkItemResponse == null) {
BulkItemResponse.Failure failure = new BulkItemResponse.Failure(index, request.id(), exception, failureStoreStatus);
bulkItemResponse = BulkItemResponse.failure(idx, request.opType(), failure);
} else {
// Response already recorded. We should only be here if the existing response is a failure and
// we are encountering a new failure while redirecting.
assert bulkItemResponse.isFailed() : "Attempting to overwrite successful bulk item result with a failure";
bulkItemResponse.getFailure().getCause().addSuppressed(exception);
bulkItemResponse.getFailure().setFailureStoreStatus(failureStoreStatus);
}

... yielding N BulkItemResponse instances each pointing to E1, which now has a suppressed-exceptions array of length N, every item of which is pointing to E2:

flowchart TB
    BR["BulkResponse"]
    I0["BulkItemResponse[0]"]
    I1["BulkItemResponse[1]"]
    IN["BulkItemResponse[N-1]"]
    E1["E1"]
    S0["suppressed[0]"]
    S1["suppressed[1]"]
    SN["suppressed[N-1]"]
    E2["E2"]

    BR --> I0
    BR --> I1
    BR --> IN
    I0 --> E1
    I1 --> E1
    IN --> E1
    E1 --> S0
    E1 --> S1
    E1 --> SN
    S0 --> E2
    S1 --> E2
    SN --> E2
Loading

This in itself isn't so bad, but the problem comes if this sharing-heavy BulkResponse exists on an ingest node which is not the coordinating node for the operation: in this case we must transport it back to the coordinating node, and the transport protocol does not have any way to represent this sharing-heavy graph of exceptions without expanding it to a pure tree:

flowchart TB
    BR["BulkResponse"]
    I0["BulkItemResponse[0]"]
    I1["BulkItemResponse[1]"]
    IN["BulkItemResponse[N-1]"]
    E10["E1"]
    E11["E1"]
    E1N["E1"]
    E200["E2"]
    E201["E2"]
    E20N["E2"]
    E210["E2"]
    E211["E2"]
    E21N["E2"]
    E2N0["E2"]
    E2N1["E2"]
    E2NN["E2"]

    BR --> I0
    BR --> I1
    BR --> IN
    I0 --> E10
    I1 --> E11
    IN --> E1N
    E10 --> E200
    E10 --> E201
    E10 --> E20N
    E11 --> E210
    E11 --> E211
    E11 --> E21N
    E1N --> E2N0
    E1N --> E2N1
    E1N --> E2NN
Loading

In the example we observed there were 1600 items in the bulk, and each exception was ~10kiB on the wire, so the total size of this tree-shaped response was (1600+1600×1600)×10kiB = 24GiB.

This is not specific to ElasticsearchSecurityException: it applies to any exception that fails the whole shard bulk, redirecting to the failure store, which also fails.

Without a failure store, handleShardFailure still shares the same exception across all N items, so the sharing-free wire encoding is still pretty large, albeit linear. It only goes quadratic when the failure store is involved. I think the linear growth is still an issue and reported it at #157861.

Activity

  1. elasticsearchmachine commented on Aug 27, 2026

    @elasticsearchmachine
    Collaborator

    Pinging @elastic/es-storage-engine (Team:StorageEngine)

  2. kkrik-es commented on Aug 31, 2026

    @kkrik-es
    Member

    @gmarouli can you please triage? Thanks!

  3. gmarouli commented on Sep 11, 2026

    @gmarouli
    Contributor

    The issue here starts from adding the individual failure exceptions (E2s) to a single shared exception (E1).

    When we implemented it we thought that the exceptions would be unique, so adding the failure store exception as a suppressed exception would be an 1-1 exception relation; it would double the exceptions but it would still be O(N). Furthermore, this is not only a bug for performance reasons, but it's also plain wrong because it reports the failure store exception wrongly.

    Proposal
    I believe the only way forward is to decouple the failure exception from the initial exception. We can extend the failure bulk item response to have a dedicated field for the failure store exception when applicable. This way each response item will contain the initial and the failure exception once. This is a user facing change, although I do not believe it is breaking.

    Furthermore, since this is not encoded yet in the transport layer, we could explore ways of encoding it from exception -> response items, we can set the framework that could be reused then to address #157861.

    cc @martijnvg for another pair of eyes.

  4. DaveCTurner commented on Sep 11, 2026

    @DaveCTurner
    MemberAuthor

    ++ this seems like the right sort of direction. Rather than sprinkling exceptions across the response items directly as happens today, we need a structure which keeps track of the fact that some exceptions apply to whole subsets of items at once.

  5. DaveCTurner commented on Sep 11, 2026

    @DaveCTurner
    MemberAuthor

    This is a user facing change, although I do not believe it is breaking.

    I think we can keep the user-facing side of this unchanged (apart from deduplicating the suppressed failure-store exceptions, moving the O(N²) size down to O(N)). The bulk response is sent using chunked-encoding so we can render each BulkItemResponse as if it had its own exception just like we have today, even if the in-memory representation is more compact.

  6. gmarouli commented on Sep 11, 2026

    @gmarouli
    Contributor

    The bulk response is sent using chunked-encoding so we can render each BulkItemResponse as if it had its own exception just like we have today, even if the in-memory representation is more compact.

    @DaveCTurner you mean that we can add the failure exception as a suppressed exception only at the end when we serialise it to the user?

  7. DaveCTurner commented on Sep 11, 2026

    @DaveCTurner
    MemberAuthor

    Maybe that, or maybe when we catch the failure-store exception we can tell the difference between "the whole shard bulk failed with a single shared exception E1" and "each item failed with a different exception" so that adding E2 as a suppressed exception doesn't blow up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions