Repository navigation
Description
A _bulk request whose items all fail with the same shard-level exception, and whose failure-store redirect also fails, produces a BulkResponse whose transport and XContent encodings have sizes which are quadratic in the number of items. The transport encoding size is particularly problematic because Elasticsearch materializes the entire response on-heap if it needs to send it back from an ingest node to a different coordinating node.
We observed this in v9.4.3 (linked code below matches this version) but the mechanism has been around for a while and still exists in main. In the example, a shard bulk failed with ElasticsearchSecurityException E1, redirecting the work to the corresponding failure store, but the failure store write also failed with ElasticsearchSecurityException E2.
The problem comes because Elasticsearch records E1 as the failure reason for every item in the bulk ...
elasticsearch/server/src/main/java/org/elasticsearch/action/bulk/BulkOperation.java
Lines 560 to 568 in 45f6a06
| private void handleShardFailure(BulkShardRequest bulkShardRequest, ProjectMetadata projectMetadata, Exception e) { | |
| // create failures for all relevant requests | |
| for (BulkItemRequest request : bulkShardRequest.items()) { | |
| final String indexName = request.index(); | |
| DocWriteRequest<?> docWriteRequest = request.request(); | |
| IndexDocFailureStoreStatus failureStoreStatus = processFailure(request, projectMetadata, e); | |
| addFailure(docWriteRequest, request.id(), indexName, e, failureStoreStatus); | |
| } | |
| } |
... and then adds E2 as a suppressed exception of E1 once for each item in the bulk ...
elasticsearch/server/src/main/java/org/elasticsearch/action/bulk/BulkOperation.java
Lines 865 to 875 in 45f6a06
| BulkItemResponse bulkItemResponse = responses.get(idx); | |
| if (bulkItemResponse == null) { | |
| BulkItemResponse.Failure failure = new BulkItemResponse.Failure(index, request.id(), exception, failureStoreStatus); | |
| bulkItemResponse = BulkItemResponse.failure(idx, request.opType(), failure); | |
| } else { | |
| // Response already recorded. We should only be here if the existing response is a failure and | |
| // we are encountering a new failure while redirecting. | |
| assert bulkItemResponse.isFailed() : "Attempting to overwrite successful bulk item result with a failure"; | |
| bulkItemResponse.getFailure().getCause().addSuppressed(exception); | |
| bulkItemResponse.getFailure().setFailureStoreStatus(failureStoreStatus); | |
| } |
... yielding N BulkItemResponse instances each pointing to E1, which now has a suppressed-exceptions array of length N, every item of which is pointing to E2:
flowchart TB
BR["BulkResponse"]
I0["BulkItemResponse[0]"]
I1["BulkItemResponse[1]"]
IN["BulkItemResponse[N-1]"]
E1["E1"]
S0["suppressed[0]"]
S1["suppressed[1]"]
SN["suppressed[N-1]"]
E2["E2"]
BR --> I0
BR --> I1
BR --> IN
I0 --> E1
I1 --> E1
IN --> E1
E1 --> S0
E1 --> S1
E1 --> SN
S0 --> E2
S1 --> E2
SN --> E2
This in itself isn't so bad, but the problem comes if this sharing-heavy BulkResponse exists on an ingest node which is not the coordinating node for the operation: in this case we must transport it back to the coordinating node, and the transport protocol does not have any way to represent this sharing-heavy graph of exceptions without expanding it to a pure tree:
flowchart TB
BR["BulkResponse"]
I0["BulkItemResponse[0]"]
I1["BulkItemResponse[1]"]
IN["BulkItemResponse[N-1]"]
E10["E1"]
E11["E1"]
E1N["E1"]
E200["E2"]
E201["E2"]
E20N["E2"]
E210["E2"]
E211["E2"]
E21N["E2"]
E2N0["E2"]
E2N1["E2"]
E2NN["E2"]
BR --> I0
BR --> I1
BR --> IN
I0 --> E10
I1 --> E11
IN --> E1N
E10 --> E200
E10 --> E201
E10 --> E20N
E11 --> E210
E11 --> E211
E11 --> E21N
E1N --> E2N0
E1N --> E2N1
E1N --> E2NN
In the example we observed there were 1600 items in the bulk, and each exception was ~10kiB on the wire, so the total size of this tree-shaped response was (1600+1600×1600)×10kiB = 24GiB.
This is not specific to ElasticsearchSecurityException: it applies to any exception that fails the whole shard bulk, redirecting to the failure store, which also fails.
Without a failure store, handleShardFailure still shares the same exception across all N items, so the sharing-free wire encoding is still pretty large, albeit linear. It only goes quadratic when the failure store is involved. I think the linear growth is still an issue and reported it at #157861.
Activity
elasticsearchmachine commentedon Aug 27, 2026
Pinging @elastic/es-storage-engine (Team:StorageEngine)
kkrik-es commentedon Aug 31, 2026
@gmarouli can you please triage? Thanks!
gmarouli commentedon Sep 11, 2026
The issue here starts from adding the individual failure exceptions (E2s) to a single shared exception (E1).
When we implemented it we thought that the exceptions would be unique, so adding the failure store exception as a suppressed exception would be an 1-1 exception relation; it would double the exceptions but it would still be O(N). Furthermore, this is not only a bug for performance reasons, but it's also plain wrong because it reports the failure store exception wrongly.
Proposal
I believe the only way forward is to decouple the failure exception from the initial exception. We can extend the failure bulk item response to have a dedicated field for the failure store exception when applicable. This way each response item will contain the initial and the failure exception once. This is a user facing change, although I do not believe it is breaking.
Furthermore, since this is not encoded yet in the transport layer, we could explore ways of encoding it from
exception -> response items, we can set the framework that could be reused then to address #157861.cc @martijnvg for another pair of eyes.
DaveCTurner commentedon Sep 11, 2026
++ this seems like the right sort of direction. Rather than sprinkling exceptions across the response items directly as happens today, we need a structure which keeps track of the fact that some exceptions apply to whole subsets of items at once.
DaveCTurner commentedon Sep 11, 2026
I think we can keep the user-facing side of this unchanged (apart from deduplicating the suppressed failure-store exceptions, moving the O(N²) size down to O(N)). The bulk response is sent using chunked-encoding so we can render each
BulkItemResponseas if it had its own exception just like we have today, even if the in-memory representation is more compact.gmarouli commentedon Sep 11, 2026
@DaveCTurner you mean that we can add the failure exception as a suppressed exception only at the end when we serialise it to the user?
DaveCTurner commentedon Sep 11, 2026
Maybe that, or maybe when we catch the failure-store exception we can tell the difference between "the whole shard bulk failed with a single shared exception E1" and "each item failed with a different exception" so that adding E2 as a suppressed exception doesn't blow up.
Failure-store failures yield O(N²) message sizes [#157859] (#162017)
Failure-store failures yield O(N²) message sizes [#157859] (#162017) (#…
Merge remote-tracking branch 'origin/main' into cursor/f45db70b
[9.4] Failure-store failures yield O(N²) message sizes [#157859] (#16…
Failure-store failures yield O(N²) message sizes [#157859] (#162017) (#…