Skip to content

Keep slow integration providers from taking every database connection - #2162

Merged
elie222 merged 2 commits into
mainfrom
fix/provider-lock-pool-starvation
Oct 25, 2026
Merged

elie222 merged 2 commits into
mainfrom
fix/provider-lock-pool-starvation

Conversation

@elie222

@elie222 elie222 commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Why

When an integration provider stalled, the whole app could fail for everyone. Connecting an integration polls connections/complete, and each poll opened a database transaction that waited on the per-connection provider lock while holding a pooled connection. The call holding that lock was itself waiting on the provider for up to a minute. A handful of overlapping polls took every connection in the API's small pool, so session lookups, sign-in and every other request waited the full 10s connect timeout and failed until the holding transaction expired. The provider was the trigger; holding connections while queued behind it is what turned it into an outage.

What changed

  • Transactions that hold the provider lock across a provider call (connection begin, complete, removal) run through a process-wide single slot. Callers waiting their turn wait in memory instead of on a pooled connection, so a slow provider pins at most one connection per process.
  • connections/complete takes the provider lock with a non-blocking try, and waits at most 2s for the slot. If another call is busy with the same connection it returns the row as still pending, and the client's existing poll picks it up on the next try.
  • Other connection actions that wait longer than the transaction timeout fail fast with a 503 instead of an opaque 500.
  • Every pool now logs db.pool.saturated (pool name, max, total, idle, waiting, wait time) when a checkout had to wait more than a second, at most once every 10s per pool, so pool pressure shows up in logs before requests start timing out.

User-facing copy: the 503 carries the message "Try again in a moment." It is only returned when a connection action could not start because another provider call was still running; without a message the client would show a generic error, and there is nothing more specific the user can act on.

How it was tested

  • New unit tests for the slot: it runs one transaction at a time, hands over to the next waiter, gives up after the wait without running the work, and frees the slot when the work fails.
  • New router tests: connections/complete returns the pending row without calling the provider when the lock is held elsewhere and when the slot is busy.
  • New test for the saturation warning: it reports a slow checkout with pool counts and rate-limits repeats.
  • Existing connections router tests updated for the try-lock and passing; db and api unit suites, type checks and lint pass locally.

🤖 Generated with Claude Code

Connecting an integration polls connections/complete, and each poll opened a
transaction that waited on the provider lock while holding a pooled connection,
behind a call that held the lock across a slow provider request. A few polls
were enough to take the whole API pool, so sign-in and every other request
timed out waiting for a connection until the holder's transaction expired.

Provider-lock transactions now run one at a time per process and wait for
their turn without a connection. A poll that cannot get the slot or the lock
quickly reports the connection as still pending, and other connection actions
that wait too long get a 503 to retry. Checkouts that wait for a connection now
log db.pool.saturated so pool pressure is visible before requests fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
rakazo-www Skipped Skipped Oct 25, 2026 1:45pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous findings are addressed and no new actionable issues were found.

Summary

The PR makes provider-lock transactions share one slot per process, so waiting calls do not hold database connections.

  • Provider calls share one slot before they open a database transaction.
  • Busy connection polls return the current row as pending.
  • Slow database checkouts produce rate-limited pool warnings.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Connection action] --> B[Wait in memory for provider slot]
  B -->|Slot available| C[Open database transaction]
  C --> D[Take provider lock]
  D --> E[Run work and release slot]
  B -->|Completion poll times out| F[Return current row]
  B -->|Other action times out| G[Return 503]
Loading

Reviews (2) · Last reviewed commit: "Cancel a begun authorization whose resul..." · Reviewed by Greptile

Comment thread apps/api/src/router.ts
Comment thread packages/db/src/client.ts
… key pool waits

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@elie222
elie222 merged commit 63e7d73 into main Oct 25, 2026
29 checks passed

This branch was previously deployed

1 inactive deployment
Preview — 21d23349 Deployed Oct 25, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant