Database connection pool exhaustion in async agents

By KYN AI Advisory Team — AI implementation specialists, Singapore

Connection pool exhaustion happens when more tasks want a database connection than the pool holds, and the extras wait. Async agents are prone to it because they start many tasks cheaply and each task can hold a connection while it waits on a slow model or API call. The result is timeouts and retries before any clear error. Fix it by sizing the pool against the server limit, releasing connections before slow external calls, capping concurrency, and alerting on wait time.

Why are async agents prone to it?

Async code makes concurrency cheap. A loop can launch hundreds of tasks in a line. A database connection is not cheap. The pool holds a fixed number, and the server holds a fixed number across all clients.

Agents add a second hazard. A typical step reads a record, calls a model for several seconds, then writes the result. If the connection stays checked out during the model call, it sits idle while other tasks queue for one. A few slow calls hold the whole pool.

What does the failure look like?

It rarely announces itself.

1. Latency rises first. Tasks wait for a connection before they run.
2. Timeouts appear. The pool gives up after its wait limit.
3. Retries multiply. Each retry joins the queue and makes the shortage worse.
4. Finally, errors surface, often unrelated to the cause: "timeout", "connection refused", "too many clients".

By then the log shows symptoms in several places and the root cause in none.

How do you size the pool?

Work from the server limit down.

• Server limit. PostgreSQL's default max_connections is typically 100, and may be lower depending on kernel settings.
• Share it. That total covers every service, worker, migration job and admin session. Count them.
• Pool per process. Divide the usable total by the number of processes that will run at the same time, including during a deploy when old and new copies overlap.
• Library defaults. SQLAlchemy's QueuePool defaults to pool_size 5, max_overflow 10 and pool_timeout 30 seconds. That is up to 15 connections per process, and a 30-second wait before it gives up. Ten worker processes at the default could ask for 150 against a server that allows 100.

Treat these as upper bounds to check against, not targets. Set the pool lower than the arithmetic allows and leave headroom for maintenance.

If many small agents run in separate processes, consider a connection pooler in front of the database, which multiplexes many client connections onto fewer server connections. Check your pooler's mode against how your code uses transactions before enabling it.

Which misconfigurations cause most trouble?

| Pattern | Effect | |---|---| | Pool per task instead of per process | Connections multiply with concurrency | | Connection held across a model or API call | Pool exhausted by idle holders | | Unbounded gather over many tasks | A burst larger than the pool | | Retry on timeout with no backoff | Queue grows, failure spreads | | Overflow sized without regard to server limit | The server refuses connections | | Sessions not closed on error paths | Slow leak, then sudden stall |

For asyncio code with SQLAlchemy's async engine or other clients, use one engine per process, open sessions in a scoped block so they close on error, and keep the transaction short.

What fixes the holding problem?

Release the connection before slow work.

1. Read what you need and close the session.
2. Call the model or API with no connection held.
3. Open a new short session to write the result.

For multi-step jobs, store progress in the database between steps, not in an open transaction.

Cap concurrency with a semaphore sized below the pool. If you allow 8 concurrent tasks that each need a connection, a pool of 10 leaves headroom. Add a queue in front of it so a burst waits in memory, not inside the pool.

What should you measure?

• Wait time to acquire a connection. The earliest signal. Alert when the median or 95th percentile moves.
• Checkout duration. How long each task holds a connection. Long holds point to work that should happen outside it.
• Saturation. Connections in use as a share of the pool, and as a share of the server limit.
• Timeouts and retries per minute.
• Idle in transaction. Sessions waiting after an open transaction. Query the database's activity view for them.

How should an agent behave under load?

• Back off with jitter on a pool timeout. Do not retry at once.
• Use a circuit breaker: after a set number of failures, pause new work for a short interval.
• Degrade by priority. Pause low-value batch jobs and keep the tasks a person is waiting for.
• Stop at a bounded retry count and hand the job to a named person.
• Never let a retry loop run without a limit.

Anything that sends a message, moves money, files or deletes should not retry blindly. Make those steps idempotent first, so a repeat cannot send twice.

A worked sizing example

The numbers below are illustrative. Use your own server limit and process counts.

Assume a Postgres server with a limit of 100 connections. Reserve 10 for migrations, admin sessions and monitoring. That leaves 90.

| Service | Processes | Pool per process | Peak total | |---|---|---|---| | Agent workers | 4 | 8 | 32 | | Web app | 3 | 10 | 30 | | Scheduled jobs | 2 | 5 | 10 | | Total | | | 72 |

72 fits inside 90, with 18 spare. Now check the deploy. A rolling deploy starts new copies before old ones stop, so for a short time the worker count doubles to 8 and the web count to 6. Peak becomes 64 + 60 + 10 = 134. That exceeds 90, and the new copies fail to start or the old ones starve.

Three ways out: lower the pool per process, deploy in smaller steps, or put a pooler in front of the database. Work out the deploy peak before the first incident, not after.

What does a safe agent step look like?

This is the pattern in outline.

1. Open a short session. Read the record and the fields needed. Close the session.
2. Call the model or external API with no connection held. Apply a timeout.
3. Open a new short session. Write the result with a status and a timestamp. Close it.
4. If step 3 fails, retry only the write, not the model call. Store the model output first so the call is not paid for twice.

Each step is a short, separate unit. If the process dies between steps, the stored status tells the next run where to resume. This also makes each step easy to test alone: a unit test can fake the model call and check that no connection is open while it runs.

How do you reproduce the problem before production does?

Test in a copy of the environment, with the pool set small.

1. Set the pool to 5 and the pool timeout to 5 seconds.
2. Launch 50 tasks that each hold a connection for 3 seconds, which stands in for a slow model call.
3. Watch for the wait-time rise, then timeouts, then retries.
4. Apply the fix, which is to release the connection before the slow call, and run the same test.

If the second run completes with no timeouts, the fix works for that code path. Repeat for each agent step that calls out to a slow service.

How do you diagnose a stall that is happening now?

• Check the database first. Query the activity view for sessions in state "idle in transaction", and note how long each has been waiting. Long ones point at code holding a transaction open.
• Count by application name. Set an application name per service, so you can see which service holds the most connections.
• Look at the pool metrics. Connections in use equal to pool size, with a non-zero queue, confirms exhaustion in the process.
• Find the oldest holder. The task that has held a connection longest is usually waiting on something external.
• Avoid restarting first. A restart clears the evidence. Take a snapshot of activity, then restart if you must. Save the output of the activity query to a file, with the time, so the post-incident review has evidence to work from, and share it with the team that owns the code.

What edge cases catch teams out?

• Health checks that use the pool. A health check that needs a connection fails when the pool is full, and the platform restarts a healthy process. Give health checks a separate path or a tiny dedicated pool.
• Long-running migrations. A migration that locks a table holds connections that others queue behind. Run migrations outside peak load.
• Streaming responses. A task that streams a model reply for a long time should not hold a connection for the stream.
• Cancelled tasks. When a task is cancelled, its session must still close. Use a context manager so cleanup runs on cancellation.
• Pooler transaction mode. Session-level features, such as prepared statements or session variables, may not survive a pooler in transaction mode. Test before you switch it on.

Where KYN fits

KYN builds operations and finance agents on cloud infrastructure in the client's own account. For a manufacturer, eleven systems became three on one AWS account, with RDS Postgres as the database, and a ledger of 3,309 reconciled rows with zero duplicates. Agents have their own restricted accounts, and every action is logged with the actor.

The first 14 days are a review: an inventory of your stack and interviews with the team. A full build ships in under 90 days. See infrastructure and operations.

Frequently asked questions

How do I know the pool is the problem?

Look for rising wait time to acquire a connection before errors appear, and check connections in use against pool size. If tasks are slow while the database is idle, the pool is the likely bottleneck.

Should I just raise the pool size?

Only after checking the server limit. A larger pool pushes the failure to the database, where it affects every service. Reduce hold time and concurrency first.

Do serverless or short-lived workers change this?

Yes. Each new instance opens its own pool, so connection counts can spike with scale. Use a pooler or a small per-instance pool.

Checked 30 October 2026.

Curious whether Marketing agents fits your business? Talk to KYN on WhatsApp — no forms, just a conversation.

Start Building →