Repository navigation
Docker: split an image reference in the grammar - #8590
Merged
Merged
Conversation
FROM's name:tag@digest was split by scanning the value's text, which had to know which colons and quotes were part of a registry port, a variable reference or a quoted string. An IMAGE_REF lexer mode, entered at FROM and left at AS or the end of the line, emits ':' and '@' as tokens only where they separate the parts, so the parser expresses the split directly and a colon anywhere else belongs to the part that holds it. The mode also makes AS a keyword only where a stage alias can appear, replacing the afterFrom flag and its setType() hack, and a dangling separator such as `FROM alpine:` now round-trips instead of failing to parse. ImageReferences stays for the reference the grammar cannot see: the value of the `--from` flag of COPY/ADD, which the lexer keeps as a single token, and a reference handed to withImageReference by a recipe.
timtebeek
force-pushed
the
tim/docker-image-reference-grammar
branch
from
August 21, 2026 17:51
25b3e4e to
37f0279
Compare
timtebeek
added a commit
to openrewrite/rewrite-migrate-java
that referenced
this pull request
Aug 21, 2026
openrewrite/rewrite#8590 splits an image reference in the grammar, so `FROM eclipse-temurin:` now parses rather than failing. Cover the shape that newly reaches the recipe.
timtebeek
added a commit
that referenced
this pull request
Aug 21, 2026
USER's user:group was split by scanning the value's literal text for a colon, a near-copy of the colon handling #8590 moved into the grammar for name:tag@digest. A USER_SPEC lexer mode, pushed by the USER keyword and popped at the end of the line, emits ':' as a token only where it separates the two, so the parser owns the split and the visitor reuses the same part-up-to-its-separator helper FROM uses. A dangling separator (USER app:) now round-trips instead of failing to parse, and a quoted user or group carries its quote style rather than its quotes.
timtebeek
added a commit
that referenced
this pull request
Aug 21, 2026
USER's user:group was split by scanning the value's literal text for a colon, a near-copy of the colon handling #8590 moved into the grammar for name:tag@digest. A USER_SPEC lexer mode, pushed by the USER keyword and popped at the end of the line, emits ':' as a token only where it separates the two, so the parser owns the split and the visitor reuses the same part-up-to-its-separator helper FROM uses. A dangling separator (USER app:) now round-trips instead of failing to parse, and a quoted user or group carries its quote style rather than its quotes.
timtebeek
added a commit
to openrewrite/rewrite-migrate-java
that referenced
this pull request
Aug 21, 2026
* Upgrade `ARG` default values used in Docker `FROM` instructions Follow up to #1212, which left any `FROM` built from a variable untouched. When the variable is a global `ARG` with a literal default, that default can be upgraded instead, so `ARG java_version=17` used as `FROM eclipse-temurin:${java_version}` becomes `ARG java_version=25`. * Do not bump `ARG` defaults shared with images we leave alone Three follow ups from review: - A `FROM` whose image we do not upgrade now vetoes the arguments feeding it, so `ARG VERSION=11` used by both `eclipse-temurin:${VERSION}` and `node:${VERSION}` is left alone rather than turning the latter into `node:25`. - Drop the digest pin when an argument holding a whole `name:tag` reference is upgraded, as the stale digest would keep resolving to the old image. - Upgrade quoted default values, keeping their quotes. The parser hands an `ARG` value to us as a single literal with the quotes still in its text and no quote style, so `ARG JAVA_VERSION="11"` never matched a version before. * Trim comments and javadoc that restate the code * Plan `ARG` upgrades before rewriting any `FROM` Rewriting a `FROM` as it was visited, and only withholding the matching `ARG` bump after the traversal, left half applied edits behind: a dropped digest pin or a rename to `eclipse-temurin` next to an argument still holding the old version. The whole file is now planned up front, and replayed until the set of withheld arguments stops growing, as withholding one argument can rule out the images that depend on it. Only the surviving plan is applied. Every give up path now withholds the arguments that `FROM` reads. * Read `ARG` default values through the new `Argument` accessors openrewrite/rewrite#8576 routes `ARG` values through the same path as every other argument, so quotes are modelled in `Literal.quoteStyle` and `$VAR` references become `EnvironmentVariable`. The local `QuotedText` that took the quotes off the literal text by hand is no longer needed, and reading a value is now `Argument.getText()`, which gives up on a reference the same way. * Leave a `FROM` with a dangling tag separator alone openrewrite/rewrite#8590 splits an image reference in the grammar, so `FROM eclipse-temurin:` now parses rather than failing. Cover the shape that newly reaches the recipe. * Do not bump `ARG` defaults shared with images we leave alone An argument holding only an image name carries no version, so renaming a deprecated image holds for every `FROM` that reads it, whatever tag each one carries. Withhold a version alone: ARG BASE=openjdk ARG BASE=eclipse-temurin FROM ${BASE}:11-jre -> FROM ${BASE}:25-jre FROM ${BASE}:latest FROM ${BASE}:latest A `FROM` whose image we can not resolve, or is not one we upgrade, still withholds both, as we know too little to rename either. Hoist the recipe into `defaults`, leaving the one test that varies the target version to override it. * Bump an `ARG` default shared with an image we leave alone An image we do not recognise may well be an internal one built on the same Java version, and if it shared a version argument before it should move along with it: ARG VERSION=11 ARG VERSION=25 FROM eclipse-temurin:${VERSION} -> FROM eclipse-temurin:${VERSION} FROM acme/base:${VERSION} FROM acme/base:${VERSION} Each `FROM` now stands on its own, so the repeated planning that let one withheld argument withhold the next is gone, and a visitor reads the file in one pass. Group the tests that assert no change into a nested class. * Upgrade a Java image pulled through a registry Mirroring stock images through an internal registry is ordinary, so read the repository past a registry rather than treating the whole reference as a name we do not know: ARG REGISTRY ARG REGISTRY FROM ${REGISTRY}/eclipse-temurin:11-jre -> FROM ${REGISTRY}/eclipse-temurin:25-jre FROM docker.io/openjdk:11-jre FROM docker.io/eclipse-temurin:25-jre Which leading segment is a registry follows Docker's own rule: one holding a `.` or a `:`, or `localhost`. Anything else belongs to the repository, so `azul/zulu-openjdk` still reads as a name and `mycompany/eclipse-temurin` is left alone. Also rewrite only the `ARG` declaration a default was read from, as a name may be declared more than once and the others carry no value to upgrade. * Read a registry through rewrite-docker's `ImageName` The rule for which leading segment of an image name is a registry now lives upstream, in `org.openrewrite.docker.trait.ImageName`, so read the registry and repository from it rather than keeping a second copy of the rule here. That also covers a segment upstream reads as a registry and the copy here did not: a repository name may not be uppercase, so `MyRegistry/openjdk` names a registry, not a repository. Reading the name through `getTextWithVariables()` lets one parse serve a spelt-out registry and a `${REGISTRY}` alike, which retires the second helper that recognised the latter by the shape of its contents. * Upgrade a `FROM` as it is visited Planning every `FROM` before rewriting any was there to withhold an argument another `FROM` ruled out. Nothing is withheld now, so each one can be upgraded where it is met, which retires the second traversal, the plan holder and the map of replacements keyed by id. Read the declaration an argument's default came from by its value rather than by its id, so the two maps become one.
timtebeek
added a commit
that referenced
this pull request
Aug 22, 2026
#8590 gave `FROM` an `IMAGE_REF` lexer mode, but `COPY --from=` never reached it: the lexer emits `--name=value` as a single `FLAG` token, so `COPY --from=host:5000/img:1.2` stayed one flat literal and `ImageReferences.split(List, Space)` had to re-implement the registry-port rule the lexer already knows. A `FROM_FLAG` token now pushes a `FLAG_IMAGE_REF` mode, which is `IMAGE_REF` ending at the whitespace before the paths rather than at `AS` or the end of the line, and `fromFlag : FROM_FLAG imageReference? FLAG_END?` runs the value through the same `imageReference` rule a `FROM` uses. `ImageReferences.split(List, Space)` is deleted; `split(String, Space)` stays for the reference a recipe supplies as text.
timtebeek
added a commit
that referenced
this pull request
Aug 22, 2026
* Docker: a COPY --from carries an image reference in the grammar #8590 gave `FROM` an `IMAGE_REF` lexer mode, but `COPY --from=` never reached it: the lexer emits `--name=value` as a single `FLAG` token, so `COPY --from=host:5000/img:1.2` stayed one flat literal and `ImageReferences.split(List, Space)` had to re-implement the registry-port rule the lexer already knows. A `FROM_FLAG` token now pushes a `FLAG_IMAGE_REF` mode, which is `IMAGE_REF` ending at the whitespace before the paths rather than at `AS` or the end of the line, and `fromFlag : FROM_FLAG imageReference? FLAG_END?` runs the value through the same `imageReference` rule a `FROM` uses. `ImageReferences.split(List, Space)` is deleted; `split(String, Space)` stays for the reference a recipe supplies as text. * Docker: move the flag predicate behind the '--' both flag rules share A semantic predicate reachable without consuming anything sits in the ATN start-state closure of its mode, and ANTLR refuses to cache a start state whose closure carried a semantic context. `decisionToDFA[DEFAULT_MODE].s0` therefore stayed null and every token re-ran a closure over all of the default mode's rules: lexing a 12,001 line Dockerfile holding no `--from` at all took 318ms rather than 3ms. Both flag rules already begin with '--', so the predicate can sit after it. Only a token starting with '-' pays an uncached edge, and lexing that same file is back to 5ms. * Docker: one home for the two forms of an image reference `ImageReferences` was narrowed to text, which left the trait layer holding a private copy of the same partition and a split-then-rejoin round trip through a representation it did not want. It now owns both directions -- parts to contents and contents back to parts -- so `DockerCopyFrom` is two thin calls, `DockerParserVisitor` shares one `imageReference` walk between `FROM` and a `COPY`'s `--from`, and a second consumer of a flag's reference has something to call. In the lexer, the reset of the flags scoped to a logical line is one method rather than four inline copies that had already drifted, the two image reference modes share the text fragment they had a copy of each of, and that fragment reuses `ESCAPE_SEQUENCE`, which it had been a fourth spelling of.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
name:tag@digest.What changed
FROM's image reference was split by scanning the value's text (ImageReferences.split+tagColonIndex), which had to decide for itself which colons were separators and which belonged to a registry port, a variable reference or a quoted string — Docker: model quoting and variable references in argument values #8576 had to teach it about quotes so thatFROM "alpine:${TAG}"would not split inside them.A new
IMAGE_REFlexer mode, pushed at theFROMkeyword and popped atASor at the end of the line, emits:and@as tokens only where they separate the parts. A colon inside a quoted string, inside a variable reference, or in a registry port (host:5000/img:tag, kept whole by a':' … '/'fragment) is never a separator token, so the parser owns the split:Two things fall out of the mode:
ASis a keyword only where a stage alias can appear, so theafterFromfield and itssetType(UNQUOTED_TEXT)hack are gone.FROM alpine:,FROM alpine@) round-trips instead of failingrequirePrintEqualsInput, because a present separator with an absent part is now an empty argument rather than a missing one.The rules that make up the reference reuse
textElementrather than repeating the element list, and the quoted-string, variable, flag and continuation rules are now fragments shared between the default mode andIMAGE_REFinstead of being written twice.ImageReferencesstays for the reference the grammar cannot see: the value of the--fromflag ofCOPY/ADD, which the lexer keeps as one token, and a reference handed towithImageReferenceby a recipe.Verification
Beyond the existing suite and five new
FromTestcases (registry port with and without a tag, quoted reference keeping its colon, dangling separator, line continuation before the separator), the FROM LSTs of a 357-file corpus (docker-library images, 382FROMinstructions) were dumped before and after: byte-identical, with the same syntax-error set and no round-trip mismatches.