Dev Notes Daily

Tracking down a connection pool leak in pgx

Spent most of Monday chasing a slow memory climb on the order service. Grafana showed the pod RSS creeping up ~4 MB/hour, which doesn't sound like much until you realize it compounds overnight and the OOM killer visits at 3 AM.

The symptoms

The pgxpool.Pool stats showed AcquiredConns growing monotonically. Every few minutes another connection would be acquired but never released. Eventually we'd hit MaxConns and all new queries would block until the acquire timeout fired.

The bug

The handler looked roughly like this:

func handleOrder(ctx context.Context, pool *pgxpool.Pool, req OrderRequest) error {
    if err := validate(req); err != nil {
        return fmt.Errorf("validation: %w", err) // <-- returns here
    }

    conn, err := pool.Acquire(ctx)
    if err != nil {
        return err
    }
    defer conn.Release() // this defer is fine

    // ... business logic ...
    return nil
}

Looks correct, right? The defer conn.Release() is right after Acquire. But the bug was in a different code path — a batch insert function called later:

func insertItems(ctx context.Context, pool *pgxpool.Pool, items []Item) error {
    conn, err := pool.Acquire(ctx)
    if err != nil {
        return err
    }

    if len(items) == 0 {
        return nil // <-- LEAK: conn never released
    }

    defer conn.Release()
    // ... batch insert ...
    return nil
}

When items was empty, we returned before the defer. The connection stayed acquired forever.

The fix

Move defer immediately after Acquire. Always. No exceptions. No matter how obvious it seems that the early return "can't happen in practice."

conn, err := pool.Acquire(ctx)
if err != nil {
    return err
}
defer conn.Release() // always the very next line

Takeaway

This is the kind of bug that a linter should catch. I looked into it and go vet doesn't flag this pattern. There might be a staticcheck rule for it, but I haven't found one. For now, the code review checklist has a new item: "Is defer immediately after the resource acquisition?"