Repository navigation
internal/gzip: ReadCloserLevel goroutine leaks r on error, causing pullLimiter deadlock via layout.WriteImage #2342
Description
Summary
internal/gzip.ReadCloserLevel starts a background goroutine that holds r io.ReadCloser. In several error paths that goroutine calls pw.CloseWithError(err) without first calling r.Close(). When r wraps a limitedReadCloser (i.e. a remote layer reader acquired via the pullLimiter introduced in #2271), the token slot is permanently leaked.
This is a distinct bug from the call-site leaks fixed by #2277 and #2308 — those fixes ensure callers close the reader; this bug means the goroutine itself fails to close its copy of r when the pipe reader is closed early by the caller.
Relationship to existing issues
- crane: hang on crane export #2276 / mutate: close layer readers during export #2277:
mutate.Extract/crane export— reader not closed at call site. Fixed. - crane pull hangs on multi-layer images (v0.21.6 regression, symmetric to #2276 — pkg/v1/tarball) #2309 / tarball: close layer readers during Write #2308:
tarball.MultiRefWrite/crane pull— reader not closed at call site. Fixed (merged, not yet released). - This issue:
internal/gzip.ReadCloserLevel— reader not closed inside the goroutine on error. Not yet fixed.
Trigger
layout/write.go's writeBlob exits early (without reading all data) when it detects the blob file already exists on disk:
func (l Path) writeBlob(hash v1.Hash, size int64, rc io.ReadCloser, ...) error {
defer rc.Close() // ← closes the pipe reader even if not fully consumed
...
if s, err := os.Stat(file); err == nil && !s.IsDir() && ... {
return nil // ← early exit
}
...
}When rc is the pr pipe reader returned by ReadCloserLevel, closing it early causes the goroutine's next bw.Flush() → pw.Write() to return io.ErrClosedPipe. The goroutine then hits this path:
// internal/gzip/zip.go
if err := bw.Flush(); err != nil {
return pw.CloseWithError(err) // r.Close() is never called → token leaked
}limitedReadCloser.Close() is never called, so the pullLimiter slot is permanently occupied.
Why this deadlocks in practice
layout.WriteImage (used by kaniko to save a build stage) calls img.Layers() and spawns one goroutine per layer. After go-containerregistry v0.21.6's change to mutate.image.Layers() (which looks up layers by compressed digest), multiple manifest entries that share the same digest — e.g. empty-tar snapshots from metadata-only Dockerfile commands — are all backed by the same layer object. Every goroutine therefore tries to write the same blob.
The first goroutine succeeds and renames the blob file. Every subsequent goroutine hits writeBlob's early-exit and leaks a token. With defaultJobs = 4, four leaked tokens fill the limiter channel and all remaining goroutines block forever in pullLimiter.acquire.
Affected code
// internal/gzip/zip.go — ReadCloserLevel goroutine
gw, err := gzip.NewWriterLevel(bw, level)
if err != nil {
// r.Close() missing → leak
return pw.CloseWithError(err)
}
if _, err := io.Copy(gw, r); err != nil {
defer r.Close() // ✓ this path is fine
defer gw.Close()
return pw.CloseWithError(err)
}
if err := gw.Close(); err != nil {
// r.Close() missing → leak
return pw.CloseWithError(err)
}
if err := bw.Flush(); err != nil {
// r.Close() missing → leak (most common trigger)
return pw.CloseWithError(err)
}Fix
--- a/internal/gzip/zip.go
+++ b/internal/gzip/zip.go
@@ -56,6 +56,7 @@ func ReadCloserLevel(r io.ReadCloser, level int) io.ReadCloser {
gw, err := gzip.NewWriterLevel(bw, level)
if err != nil {
+ r.Close()
return pw.CloseWithError(err)
}
@@ -68,10 +68,12 @@ func ReadCloserLevel(r io.ReadCloser, level int) io.ReadCloser {
// Close gzip writer to Flush it and write gzip trailers.
if err := gw.Close(); err != nil {
+ r.Close()
return pw.CloseWithError(err)
}
// Flush bufio writer to ensure we write out everything.
if err := bw.Flush(); err != nil {
+ r.Close()
return pw.CloseWithError(err)
}Environment
- go-containerregistry v0.21.6
- Triggered via kaniko v1.27.6 (
osscontainertools/kaniko#775) usinglayout.WriteImageon a cached stage
Activity
FrancisQuaisr commentedon Jun 9, 2026
The issue body proposes adding
r.Close()to each missing error path, but a more robust fix is to deferr.Close()at the top of the goroutine — the same approach the existingTODO(go1.14)comment was pointing toward. This way any future error path added to the function is automatically covered, rather than relying on every contributor remembering to add the manual call.The
pwclose requires a named error variable to distinguish success from failure in the defer:internal/gzip: fix goroutine leak in ReadCloserLevel
internal/gzip: fix goroutine leak in ReadCloserLevel (google#2347)