Skip to content

Fix JSON formatter embedding non-JSON body as raw value - #2326

Merged
kasmarian merged 5 commits into
zalando:mainfrom
Vinu2111:fix/json-formatter-invalid-body-embedding
Jul 27, 2026
Merged

kasmarian merged 5 commits into
zalando:mainfrom
Vinu2111:fix/json-formatter-invalid-body-embedding

Conversation

@Vinu2111

Copy link
Copy Markdown
Contributor

Fixes #2318

Problem

When a request body has a JSON content-type header but contains non-JSON content (e.g. Base64-encoded ciphertext, binary data, or truncated/malformed JSON), all four JSON formatters were embedding the raw body directly using @JsonRawValue/writeRawValue(), producing malformed JSON log output that breaks downstream log ingestion (Logstash, Elasticsearch). This also resolves the long-standing TODOcomment inJsonHttpLogFormatterJackson2.prepareBody()` regarding missing JSON validation.

Fix

Applied to all four formatters:

  • JsonHttpLogFormatter (Jackson 3)
  • FastJsonHttpLogFormatter (Jackson 3)
  • JsonHttpLogFormatterJackson2 (Jackson 2)
  • FastJsonHttpLogFormatterJackson2 (Jackson 2)

Each formatter now uses a two-stage validation approach:

  1. Fast path - check first non-whitespace character against valid JSON start characters ({ [ " digit - t f n). Obvious non-JSON content like ciphertext is rejected immediately at zero parse cost.
  2. Slow path - only if the fast check passes, attempt full JSON validation via readTree(). If parsing fails, fall back to logging the body as a safely quoted plain string.

Behaviour

  • Valid JSON body with JSON content-type → embedded raw as before, no regression
  • Non-JSON body with JSON content-type (ciphertext, binary) → logged as quoted string
  • Truncated/malformed JSON body → logged as quoted string
  • Non-JSON content-type → logged as quoted string, unchanged

Test changes

WriteBodyMaxSizeTest assertions updated to reflect corrected behaviour - truncated bodies produced by BodyMaxSizeFilter are now correctly logged as quoted strings rather than raw invalid JSON fragments.

}

// Slow path: attempt full JSON validation
try {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather use a streaming approach like this.

if (body == null || body.isEmpty()) {
return false;
}
final char firstChar = body.trim().charAt(0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might as well throw in the last char too?


private static class DefaultJsonFieldWriter implements JsonFieldWriter {

private static final JsonMapper MAPPER = new JsonMapper();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use input JsonMapper instead?

When a request body has a JSON content-type header but contains
non-JSON content (e.g. Base64-encoded ciphertext, binary data, or
truncated/malformed JSON), the formatters were embedding the raw
body directly using @JsonRawValue / writeRawValue(), producing
malformed JSON log output that breaks Logstash and Elasticsearch
ingestion.

Fix applied to all four formatters:
- JsonHttpLogFormatter (Jackson 3)
- FastJsonHttpLogFormatter (Jackson 3)
- JsonHttpLogFormatterJackson2 (Jackson 2)
- FastJsonHttpLogFormatterJackson2 (Jackson 2)

Each formatter now uses a two-stage validation approach:
1. Fast path: check first non-whitespace character against valid
   JSON start characters ({ [ " digit - t f n). Obvious non-JSON
   content like ciphertext is rejected immediately at zero parse cost.
2. Slow path: only if the fast check passes, attempt full JSON
   validation via readTree(). If parsing fails, fall back to logging
   the body as a safely quoted plain string.

This ensures the overall JSON log structure is always valid while
preserving existing behaviour for valid JSON bodies and keeping
the performance impact minimal for the common case.

Also resolves the long-standing TODO comment in
JsonHttpLogFormatterJackson2.prepareBody() regarding missing
JSON validation.

Test assertions in WriteBodyMaxSizeTest updated to reflect the
corrected behaviour: truncated bodies (produced by BodyMaxSizeFilter)
are now correctly logged as quoted strings rather than raw invalid JSON.

Fixes #2318
@Vinu2111

Vinu2111 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback @skjolber!

Streaming validation replaced readTree() with a JsonParser-based streaming approach across all four formatters, consuming tokens without building a tree object in memory
Last char check added last character validation to looksLikeJson() so obvious non-JSON content is rejected before the slow path
Injected mapper DefaultJsonFieldWriter now takes the JsonMapper via constructor instead of creating a static instance

Also updated two test method names (shouldEmbedInvalid* → shouldNotEmbedInvalid*) and their assertions to reflect that {"name":"Bob"}; is correctly now logged as a quoted string rather than embedded raw.

@prashantpiyush1111

Copy link
Copy Markdown

Fix looks good overall. The two-stage validation approach (fast char check + streaming JsonParser) is efficient — avoids building a full tree in memory. Covers all four formatters consistently.
One thing to flag: PR is currently missing a required label (bugfix) — that's causing the COMPR/pr-labels check to fail. Adding it should unblock the required checks.

@Vinu2111

Vinu2111 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

@kasmarian could you please add the bugfix label? I don't have permission to add labels as an external contributor, and it's currently blocking the COMPR/pr-labels check.

@skjolber

skjolber commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

@Vinu2111 there is a lot of duplicated code. Also see comment on issue.

@Vinu2111

Vinu2111 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

@skjolber thanks for the feedback. I've addressed your three inline comments switched to streaming parser, added last char check, and injected the mapper via constructor.
For the duplication- I plan to extract the shared validation logic into a single package-private JsonUtil helper per module (logbook-json and logbook-json-jackson2 separately, since they have different Jackson namespaces). All four formatters will call their module's helper instead of repeating the logic.
On opt-in/opt-out- looking at the existing codebase, no formatter currently carries a boolean flag, and the Fast formatters use @AllArgsConstructor which makes adding a constructor parameter a breaking change. Given this is a bugfix, I'd propose keeping validation always-on. Happy to add a flag if you prefer a different approach.
Could you confirm this direction before I write any code?

@kasmarian kasmarian added the bugfix Bug fixes and patches label Jun 9, 2026
@skjolber

skjolber commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Zero code duplication should be the goal.

Maybe the maintainers could weigh in here. Validating JSON payload is pure overhead when

  • produced locally
  • produced remotly:
    • after using a validating filter, and/or
    • after successful databinding
    • when trusting the other party (i.e. internal service)

The JSON validation step should at least be possible to toggle based on payload, i.e. as a feature and possibly per request/response too.

@kasmarian

Copy link
Copy Markdown
Collaborator

I second the performance concern from @skjolber here. Adding an additional json parsing to all payloads by default will have a performance penalty, where most of the clients may not be OK with. Also the challenge of invalid json payloads is something I'd expect to be exceptional, rather normal.
Having this feature would be good behind an opt-in toggle (disabled by default and explained in the README the potential performance penalty).

@AllArgsConstructor shouldn't be a problem in terms of backwards compatibility as long as you preserve current constructors and overload them with the feature toggle(s). The latter one will be used by Spring module that'll pass the state of the toggle(s).

Extract shared JsonUtil validation helper per module. Add opt-in
validateJsonBody flag (default: false) to all four formatters and
wire it through LogbookProperties for Spring Boot users.

Fixes #2318
@Vinu2111

Vinu2111 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@skjolber @kasmarian refactored based on your feedback. Extracted shared JsonUtil helper per module (zero duplication). Added opt-in validateJsonBody flag defaulting to false existing behavior unchanged. Wired through LogbookProperties for Spring Boot users. All 213 tests pass.

private final ObjectMapper mapper = new ObjectMapper();

@Test
void shouldAcceptValidJsonStartAndEnd() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you convert this to a parameterized test instead? This way it'll give a more readable assertion error when a concrete test variant fails instead of an assertion line.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same apply for the similar tests here

@Vinu2111

Vinu2111 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Converted the tests to @ParameterizedTest with @MethodSource across both modules, with a per-case .as(...) message so failures now point to the exact input that broke. Also added validate-json-body to the README config table and to spring-configuration-metadata.json so it's auto-discoverable in IDEs. All checks are green now, including update-toc.
Let me know if anything else needs adjusting!

Comment thread README.md Outdated
by [JoJan](https://commons.wikimedia.org/wiki/User:JoJan) is licensed under a
| `logbook.write.status-code-based` | Enables status-code-aware log levels. Responses with 2xx/3xx are logged at `TRACE`, 4xx at `WARN`, 5xx at `ERROR`. Requests are always logged at `TRACE`. | `false` |
| `logbook.write.max-body-size` | Truncates the body up to `max-body-size` characters and appends `...`. <br/> :warning: Logbook will still buffer the full body, if the request is eligible for logging, regardless of the `logbook.write.max-body-size` value | `-1` (disabled) |
| `logbook.write.validate-json-body` | Validates JSON bodies before embedding as raw JSON values. Adds a small parsing overhead per request/response. | `false` |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add here (and in the spring-configuration-metadata) half of a sentence on the behavior if the validation fails? (i.e. logged as quoted string)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done - added a clause on the failure behavior to the validate-json-body row in the README table. It now reads: "...If validation fails, the body is logged as a quoted string instead. Adds a small parsing overhead per request/response." Also applied the same clarification to spring-configuration-metadata.json for consistency.

Comment thread README.md Outdated
Logbook is ready to use out of the box for most common setups. Even for uncommon applications and technologies, it should be simple to implement the necessary interfaces to connect a
library/framework/etc. to it.
# Logbook: HTTP request and response logging

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we revert the pure whitespace changes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed - re-checked the full README diff and there are no pure whitespace-only changes remaining. The diff is now scoped to just the intentional content edit above.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Vinu2111 that's not true, have a look at the README.md file in the PR, it has
Lines changed: 1188 additions & 1187 deletions label.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, sorry about that, my local copy had gotten resaved with LF line endings, causing every line to show as changed. I've reset README.md to the upstream version and re-applied only the two intentional content edits. The README diff is now 3 +/- lines as expected. Thanks for flagging it!

@kasmarian

Copy link
Copy Markdown
Collaborator

👍

@kasmarian
kasmarian merged commit 310b6d9 into zalando:main Jul 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Bug fixes and patches

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Logbook JSON formatter produces invalid JSON when request body is non-JSON (binary or Base64-encoded ciphertext)

5 participants