Repository navigation
Keep slow integration providers from taking every database connection - #2162
Merged
Merged
Conversation
Connecting an integration polls connections/complete, and each poll opened a transaction that waited on the provider lock while holding a pooled connection, behind a call that held the lock across a slow provider request. A few polls were enough to take the whole API pool, so sign-in and every other request timed out waiting for a connection until the holder's transaction expired. Provider-lock transactions now run one at a time per process and wait for their turn without a connection. A poll that cannot get the slot or the lock quickly reports the connection as still pending, and other connection actions that wait too long get a 503 to retry. Checkouts that wait for a connection now log db.pool.saturated so pool pressure is visible before requests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
[Medium impact] The PR appears safe to merge; both previous findings are addressed and no new actionable issues were found. SummaryThe PR makes provider-lock transactions share one slot per process, so waiting calls do not hold database connections.
|
… key pool waits Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When an integration provider stalled, the whole app could fail for everyone. Connecting an integration polls
connections/complete, and each poll opened a database transaction that waited on the per-connection provider lock while holding a pooled connection. The call holding that lock was itself waiting on the provider for up to a minute. A handful of overlapping polls took every connection in the API's small pool, so session lookups, sign-in and every other request waited the full 10s connect timeout and failed until the holding transaction expired. The provider was the trigger; holding connections while queued behind it is what turned it into an outage.What changed
connections/completetakes the provider lock with a non-blocking try, and waits at most 2s for the slot. If another call is busy with the same connection it returns the row as still pending, and the client's existing poll picks it up on the next try.db.pool.saturated(pool name, max, total, idle, waiting, wait time) when a checkout had to wait more than a second, at most once every 10s per pool, so pool pressure shows up in logs before requests start timing out.User-facing copy: the 503 carries the message "Try again in a moment." It is only returned when a connection action could not start because another provider call was still running; without a message the client would show a generic error, and there is nothing more specific the user can act on.
How it was tested
connections/completereturns the pending row without calling the provider when the lock is held elsewhere and when the slot is busy.connectionsrouter tests updated for the try-lock and passing; db and api unit suites, type checks and lint pass locally.🤖 Generated with Claude Code