Skip to content

feat: show confidence level on AI field marker - #9901

Merged
tomivirkki merged 18 commits into
mainfrom
feat/form-ai-marker-confidence
Aug 21, 2026
Merged

tomivirkki merged 18 commits into
mainfrom
feat/form-ai-marker-confidence

Conversation

@tomivirkki

Copy link
Copy Markdown
Member

Summary

  • A reviewer checking AI-filled values wants to see at a glance which ones the model itself was unsure about. With source tracking enabled, the model already reports a confidence level per filled value, and the AI field marker is the default place to show it.
  • The marker shows the level only when the model reported one: a value without a level means the model did not judge itself, not that it was unsure, so no indicator is shown for it. A refill without a source clears the indicator from the reused marker.
  • No new public API: the marker reflects the source data the controller already tracks.

Depends on #9900 and vaadin/web-components#12436.

Fixes https://github.com/orgs/vaadin/projects/103/views/3?pane=issue&itemId=226179925

🤖 Generated with Claude Code

tomivirkki and others added 12 commits August 18, 2026 14:49
Data model for AI source tracking, shared by all controllers: the
snippets the model read, their page and rectangle when the source has
them, and a confidence level.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opt-in via FormAIController.setSourceTrackingEnabled. When on, the
fill_form tool description asks the LLM to wrap each value it fills in
an envelope carrying the snippets it read, their locations, and a
confidence level. The reported source is available from
FieldValueChangeEvent.getFieldSource and
FormAIController.getFieldSource, can be restored after a reload with
restoreFieldSource, and lasts as long as the field holds the value it
was reported with. Bad source data is dropped and logged, never
blocking the value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Line-removal and operator mutation over the feature's changed lines
found untested behavior: rect and page boundary values, the page-region
location without a rect, object-shaped extracts, drop-path logging,
null-guard messages, stale-source cleanup on value revert, and
ValueSource extracts normalization.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A fill_form write carrying no source kept the field's earlier entry,
and when the written value matched the one that source was recorded
with, getFieldSource described the new write with a citation it never
had. Every successful write now replaces the source, clearing it when
none was reported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page number beyond the int range truncated to its low 32 bits in
asInt() and could pass the range check as a wrong page. Whole-number
floats (e.g. 2.0) are now accepted, consistent with how integer fields
convert values.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stale entries were dropped only when getFieldSource read them, so an
application consuming sources through the change event alone — or a
field edited and then removed from the form — accumulated entries for
the life of the controller. Behavior of the getters is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The enum said "source" where the wording the LLM sees says
"document"; a reader comparing the two saw different meanings for the
same level.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When source tracking is on and the model reported a confidence level
for a filled value, the automatic field marker now shows it via the
marker's confidence indicator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A fill that writes the value the field already had fires no change
event and is never re-marked, but it does replace the source — a marker
kept from an earlier turn kept showing a level the new source did not
include.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

^ Conflicts:
^	vaadin-ai-components-flow-parent/vaadin-ai-extensions-flow/src/main/java/com/vaadin/flow/component/ai/form/FormAIController.java
@tomivirkki
tomivirkki force-pushed the feat/form-ai-marker-confidence branch from e49c5ed to a8a5348 Compare August 18, 2026 12:52
Base automatically changed from feat/form-ai-source-tracking to main August 20, 2026 06:03
…onfidence

# Conflicts:
#	vaadin-ai-components-flow-parent/vaadin-ai-core-flow/src/main/java/com/vaadin/flow/component/ai/common/PageRegion.java
#	vaadin-ai-components-flow-parent/vaadin-ai-core-flow/src/main/java/com/vaadin/flow/component/ai/common/Rect.java
#	vaadin-ai-components-flow-parent/vaadin-ai-core-flow/src/main/java/com/vaadin/flow/component/ai/common/SourceExtract.java
#	vaadin-ai-components-flow-parent/vaadin-ai-extensions-flow/src/main/java/com/vaadin/flow/component/ai/form/FormAIController.java
#	vaadin-ai-components-flow-parent/vaadin-ai-extensions-flow/src/test/java/com/vaadin/flow/component/ai/form/SourceTrackingTest.java
@tomivirkki
tomivirkki requested a review from ugur-vaadin August 20, 2026 13:11
@tomivirkki

Copy link
Copy Markdown
Member Author

/code-review

@vaadin-review-bot vaadin-review-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the changes — left 1 comment and collected 1 finding below that could not be attached to the diff.

Finding
📝 The confidence indicator text cannot be localized from Flow
🧪 No test covers the new applyValue path setting a level on a reused marker

📝 The confidence indicator text cannot be localized from Flow
The web component renders the level as visible text taken from i18n.confidence.{low,medium,high}, and puts it into the field's aria-describedby. FieldMarkerI18n maps only message, revert, badgeLabel and badgeTooltip, so the deep merge in I18nMixin always falls back to the English defaults for the new text — a fully localized app still shows "High confidence".

This also makes two javadocs over-promise:

  • FieldMarkerI18n: "Use to localize the marker; any text left null falls back to the built-in English default."
  • setFieldMarkerEnabled… rather setFieldMarkerI18n: "Sets the texts shown by the AI field marker — the "AI" badge, its tooltip, and the popover with the revert control — replacing the built-in English defaults."
    FieldMarkerI18n.java:26 · documentation

tomivirkki and others added 3 commits August 20, 2026 16:30
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tomivirkki

Copy link
Copy Markdown
Member Author

The confidence indicator text cannot be localized from Flow

Added a nested FieldMarkerI18n.Confidence class with low/medium/high texts mapping to the web component's i18n.confidence object, and updated the over-promising javadocs

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FormFieldMarker.add(element, fieldMarkerI18n);
// Applied on every mark so a refill without a source clears the
// indicator a previous fill set on the reused marker.
FormFieldMarker.setConfidence(element, getFieldSource(field)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: we can maybe add a test for this. It seems like all the tests still pass without this line.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, test added

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tomivirkki
tomivirkki enabled auto-merge August 21, 2026 13:39
@tomivirkki
tomivirkki added this pull request to the merge queue Aug 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 21, 2026
@sonarqubecloud

Copy link
Copy Markdown

@tomivirkki
tomivirkki added this pull request to the merge queue Aug 21, 2026
Merged via the queue into main with commit a8b6a0a Aug 21, 2026
19 checks passed
@tomivirkki
tomivirkki deleted the feat/form-ai-marker-confidence branch August 21, 2026 14:28
@vaadin-bot

Copy link
Copy Markdown
Collaborator

This ticket/PR has been released with Vaadin 25.3.0-alpha9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants