Neon PostgreSQL Troubleshooting

Quick Reference — Diagnosis and resolution for common Neon PostgreSQL, Hyperdrive, Prisma, and Better Auth issues in the bloqr-backend stack.

Table of Contents

• Connection Architecture & Failure Points
• Connection Pool Exhaustion (Hyperdrive)
• Prisma Migration Errors
• Hyperdrive Timeout Handling
• Local Development Issues
• Neon Branching Issues
• Better Auth + Neon Issues
• Diagnostic Commands

Connection Architecture & Failure Points

Every database request passes through multiple layers, each with its own failure modes. The diagram below highlights where things typically go wrong:

Connection Pool Exhaustion (Hyperdrive)

Symptoms

• Requests hang for 10–30 seconds then fail
• Error: too many connections for role "..." in Worker logs
• Intermittent 500 errors under moderate load
• Neon dashboard shows connection count at or near limit

Root Causes

1. PrismaClient not disconnected — forgetting await prisma.$disconnect() keeps the Hyperdrive proxy socket open
2. Multiple PrismaClient instances per request — creating separate clients for auth and route handlers without sharing via middleware
3. Long-running transactions — holding connections during slow operations

Resolution

const prisma = createPrismaClient(c.env.HYPERDRIVE!.connectionString);
try {
    const user = await prisma.user.findUnique({ where: { id } });
    return c.json({ user });
} finally {
    await prisma.$disconnect();
}

// worker/middleware/prisma-middleware.ts handles this automatically
app.use('/api/*', prismaMiddleware());

// Both Better Auth and route handlers share the same PrismaClient
app.get('/api/users', async (c) => {
    const prisma = c.get('prisma'); // shared instance
    // ...
});

-- Run via Neon SQL Editor or psql
SELECT count(*) FROM pg_stat_activity WHERE datname = 'neondb';

-- See who's holding connections
SELECT pid, usename, application_name, state, query_start
FROM pg_stat_activity
WHERE datname = 'neondb'
ORDER BY query_start DESC;

Prisma Migration Errors

"prepared statement already exists"

Symptom: prisma migrate dev or prisma migrate deploy fails with pooling errors.

Cause: Migration commands use interactive transactions that conflict with connection pooling (PgBouncer in Neon's pooler endpoint).

Fix: Always use DIRECT_DATABASE_URL (the non-pooled endpoint) for migrations:

# .env.local — note: NO "-pooler" in the hostname
DIRECT_DATABASE_URL=postgresql://<user>:<password>@ep-xxx.c-13.us-east-1.aws.neon.tech/neondb?sslmode=require

The prisma.config.ts automatically prefers DIRECT_DATABASE_URL over DATABASE_URL.

"migration failed — column already exists"

Symptom: A migration tries to add a column that already exists in the database.

Cause: Schema was pushed with db push (which doesn't create migration files), then a migration was created that duplicates the change.

# Mark the problematic migration as already applied
deno run -A npm:prisma migrate resolve --applied <migration_name>

# Or reset the migration history (local dev only!)
deno task db:local:reset

"P3009: migrate found failed migrations"

Symptom: A previous migration failed partway, leaving the database in a dirty state.

# 1. Check which migration failed
deno run -A npm:prisma migrate status

# 2. Fix the underlying issue, then mark it resolved
deno run -A npm:prisma migrate resolve --rolled-back <migration_name>

# 3. Re-run migrations
deno task db:migrate

"Cannot find module 'prisma/generated/client.ts'"

Symptom: Import error after schema changes or fresh clone.

Fix: Always use the Deno task (not raw npx prisma generate):

deno task db:generate

This runs prisma generate followed by scripts/prisma-fix-imports.ts, which rewrites import paths for Deno compatibility.

Hyperdrive Timeout Handling

Request Timeout (Worker-Side)

Symptom: Routes that involve database queries return 524 (timeout) after ~30 seconds.

Cause: Neon's serverless compute may need a cold start (~1–5 seconds), and if the Worker is placed far from Neon, round-trip latency adds up.

1. Verify Smart Placement is enabled in wrangler.toml:

   [placement]
   mode = "smart"
   

2. Add query timeouts to prevent unbounded database calls:

   // Use Prisma's built-in timeout (in milliseconds)
   const user = await prisma.user.findUnique({
       where: { id },
   });
   // Note: Prisma driver adapter doesn't support statement_timeout natively;
   // set it at the PostgreSQL level if needed:
   // ALTER ROLE your_user SET statement_timeout = '10s';
   

3. Monitor Neon cold starts in the Neon dashboard → Monitoring → Compute activity

Stale Connections

Symptom: Queries intermittently fail with Error: Connection terminated unexpectedly or Error: This socket has been ended by the other party.

Cause: Neon scales compute to zero after 5 minutes of inactivity. If Hyperdrive holds a connection that Neon closed, the next query on that socket fails.

Fix: Hyperdrive handles reconnection automatically. If you see persistent issues:

# Verify Hyperdrive config
npx wrangler hyperdrive get <resource-id>

# Recreate the Hyperdrive config if needed
npx wrangler hyperdrive update <resource-id> \
    --origin-host=ep-xxx-pooler.c-13.us-east-1.aws.neon.tech \
    --origin-port=5432 \
    --database=neondb \
    --origin-user=<user> \
    --origin-password=<password>

SSL / Channel Binding Errors

Symptom: Error: channel binding is required but server did not offer it

Fix: Add channel_binding=require to the connection string (Neon's pooler requires it in some configurations):

postgresql://user:pass@...-pooler.c-13.us-east-1.aws.neon.tech/db?sslmode=require&channel_binding=require

Local Development Issues

"HYPERDRIVE is undefined" in Local Dev

Symptom: TypeError: Cannot read properties of undefined (reading 'connectionString')

Cause: .dev.vars is missing or has an empty Hyperdrive local override.

# .dev.vars (gitignored) — point at your personal Neon dev branch
CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE=postgresql://<user>:<password>@<branch-host>.neon.tech/<dbname>?sslmode=require

Create a personal dev branch at console.neon.tech → your project → Branches → New Branch. Use the Direct connection string (not pooled). See Local Dev Setup.

⚠️ Restart wrangler dev after changing .dev.vars — it only reads the file at startup.

"SSL required" / sslmode errors

Symptom: Error: SSL required or Error: PGGSSENCMODE / TLS handshake failure.

Cause: The connection string is missing ?sslmode=require (required by Neon).

Fix: Append ?sslmode=require to every Neon connection string in .dev.vars and .env.local:

# Replace <user>, <password>, <branch-host>, and <dbname> with your Neon branch values
CLOUDFLARE_HYPERDRIVE_LOCAL_CONNECTION_STRING_HYPERDRIVE=postgresql://<user>:<password>@<branch-host>.neon.tech/<dbname>?sslmode=require
DATABASE_URL="postgresql://<user>:<password>@<branch-host>.neon.tech/<dbname>?sslmode=require"
DIRECT_DATABASE_URL="postgresql://<user>:<password>@<branch-host>.neon.tech/<dbname>?sslmode=require"

"Cannot connect to server" / ETIMEDOUT

Symptom: Connections time out during wrangler dev or prisma migrate.

Cause: The Neon branch may be suspended or the connection string is incorrect.

# 1. Verify your Neon branch is active
#    https://console.neon.tech → your project → Branches → check branch status

# 2. Test the connection directly
#    Replace <user>, <password>, <branch-host>, and <dbname> with your Neon branch values
psql "postgresql://<user>:<password>@<branch-host>.neon.tech/<dbname>?sslmode=require" -c "SELECT 1 AS ok;"

# 3. Ensure your connection string uses the direct (non-pooler) hostname
#    ✅ ep-<name>.<region>.neon.tech          (direct — use for local dev)
#    ❌ ep-<name>-pooler.<region>.neon.tech   (pooler — only for production Hyperdrive)

Neon Branching Issues

Branch Creation Fails

Symptom: NeonApiService.createBranch() returns a 400 or 422 error.

// Using the NeonApiService
const neon = createNeonApiService({ apiKey: env.NEON_API_KEY });
const branches = await neon.listBranches('steep-rice-35228530');

// Find branches older than 7 days (excluding main)
const stale = branches.filter(b =>
    b.name !== 'main' &&
    new Date(b.created_at) < new Date(Date.now() - 7 * 24 * 60 * 60 * 1000)
);

for (const branch of stale) {
    await neon.deleteBranch('steep-rice-35228530', branch.id);
}

Branch Endpoint Not Ready

Symptom: Branch was created but connection fails with timeout.

Cause: Neon endpoints take a few seconds to become active after branch creation.

Fix: Poll the endpoint status before connecting:

const { branch } = await neon.createBranch(projectId, { name: 'preview/pr-42' });
const endpoints = await neon.listEndpoints(projectId);
const ep = endpoints.find(e => e.branch_id === branch.id);
// Wait for endpoint to become active (Neon Operations API handles this)

Better Auth + Neon Issues

Session Creation Fails (500 on /api/auth/sign-in)

Symptom: Login returns 500, Worker logs show a database error.

1. Is HYPERDRIVE configured?
   # Check wrangler.toml has [[hyperdrive]] section
   grep -A2 'hyperdrive' wrangler.toml
   

2. Is BETTER_AUTH_SECRET set?
   # Local dev: check .dev.vars
   grep 'BETTER_AUTH_SECRET' .dev.vars
   

3. Do the auth tables exist?
   -- Check via Neon SQL Editor
   SELECT table_name FROM information_schema.tables
   WHERE table_schema = 'public'
   AND table_name IN ('users', 'sessions', 'account', 'verification');
   

4. Run migrations if tables are missing:
   deno task db:migrate
   

Adapter Errors ("Cannot read property of undefined")

Symptom: prismaAdapter throws during Better Auth initialization.

Cause: The PrismaClient was created with an invalid connection string, or the Prisma schema doesn't match the database.

# 1. Regenerate the Prisma client
deno task db:generate

# 2. Verify schema matches the database
deno run -A npm:prisma db pull   # introspect live DB
deno run -A npm:prisma migrate status  # check drift

Token Validation Fails After Session Creation

Symptom: Sign-in succeeds but subsequent API calls return 401.

Cause: The session table's token column may not be indexed, or the session was created but the expiresAt is in the past.

-- Check session exists and is not expired
SELECT id, token, expires_at, created_at
FROM sessions
WHERE user_id = '<user-id>'
ORDER BY created_at DESC
LIMIT 5;

Better Auth + Clerk Fallback Conflict

Symptom: Authenticated requests intermittently fail when both providers are active.

Cause: A client sends a Clerk JWT, Better Auth rejects it (not its format), and the Clerk fallback path is disabled.

Fix: Either:

• Enable the fallback: ensure DISABLE_CLERK_FALLBACK is not set to true
• Or migrate the client to use Better Auth credentials

See Auth Chain Reference for the full authentication flow.

Diagnostic Commands

Quick Health Check

# Worker dev server running?
curl -s http://localhost:8787/health | jq .

# Database reachable? (via Prisma)
deno task db:studio  # opens GUI at http://localhost:5555

# Neon API reachable?
curl -s -H "Authorization: Bearer $NEON_API_KEY" \
    "https://console.neon.tech/api/v2/projects/steep-rice-35228530" | jq .name

Database Connectivity

# Test direct PostgreSQL connection to your Neon dev branch (requires psql)
psql "$DIRECT_DATABASE_URL" -c "SELECT 1 AS ok;"

# Check Prisma migration status against your Neon dev branch
deno run -A npm:prisma migrate status

Wrangler / Hyperdrive

# Verify Hyperdrive binding
npx wrangler hyperdrive get <resource-id>

# Check Worker environment
npx wrangler dev --log-level=debug

Further Reading

• Neon Setup — Production Neon configuration
• Local Dev Guide — Neon branching setup for local development
• Auth Chain Reference — Authentication flow details
• Better Auth + Prisma — Prisma adapter configuration
• Neon Documentation — Official Neon docs
• Cloudflare Hyperdrive — Hyperdrive troubleshooting

Live Troubleshooting Session — 2026-03-25

This section documents the full sequence of events from the live debugging session that occurred on 2026-03-25. It is captured here as a reference for future on-call engineers.

Sequence of Events

1. User noticed UI banners — "Degraded performance — v0.75.0" and "Data may be stale" — on every page visit, regardless of login state
2. Initial diagnosis — curl /api/health showed database.status: "down" with latency_ms: 0
3. Checked Hyperdrive dashboard — zero traffic. Neon dashboard showed migration activity (migrations had run direct via DIRECT_DATABASE_URL, not through Hyperdrive)
4. Ran wrangler hyperdrive get — confirmed "scheme": "postgres" in the Hyperdrive binding config
5. Found root cause — PrismaClientConfigSchema only accepted postgresql://; Hyperdrive returns postgres://
6. Applied schema fix — updated the regex to accept both schemes; deployed at v0.76.0
7. Database still down at v0.76.0 — latency_ms: 0 persisted; catch block was silently swallowing errors
8. Added error surfacing — new error_code and error_message fields in the health response catch block
9. Added smoke test endpoint — GET /api/health/db-smoke for detailed post-deploy diagnostics

What .hyperdrive.local Means (Important)

When you see hyperdrive_host: "<resource-id>.hyperdrive.local" in the health response, this is correct and expected for a deployed Cloudflare Worker. It is the Cloudflare-managed internal proxy socket address that Hyperdrive injects at runtime. It is NOT a sign of a misconfiguration.

The .hyperdrive.local host is Hyperdrive's local connection proxy. When your Worker runs env.HYPERDRIVE.connectionString, Cloudflare resolves this to a postgres://...hyperdrive.local/... URL that routes through the Hyperdrive proxy to your origin database. This is the architecture working as intended.

Bottom line: seeing .hyperdrive.local in hyperdrive_host means the binding IS connected to the proxy. The issue is always further down the stack.

Zero Hyperdrive Dashboard Traffic vs Non-Zero Neon Traffic

If the Hyperdrive dashboard shows zero queries but Neon shows activity, it means:

• Migrations ran directly via DIRECT_DATABASE_URL / DATABASE_URL (which use the real Neon pooler URL, bypassing Hyperdrive)
• Worker queries are failing at instantiation — the Prisma client is never created, so no queries ever reach Hyperdrive

This pattern is consistent with a Zod validation failure (latency_ms: 0) — the probe throws before opening any connection.

The latency_ms: 0 Instant Failure Pattern

wrangler tail as the First Step for Production Debugging

Always run wrangler tail first when diagnosing a production issue. It shows the actual exception thrown by your Worker in real time, including ZodError messages with field paths that pinpoint exactly which validation failed.

wrangler tail --format=pretty
# Look for: ZodError, connection refused, P2024, PROBE_TIMEOUT, etc.

Without wrangler tail, the health response's status: "down" is the only signal — with it, you get the full stack trace.

The Smoke Test Endpoint as the Canonical Post-Deploy Check

After every production deploy, run:

curl -s https://<your-worker>.workers.dev/api/health/db-smoke | jq .

This endpoint (GET /api/health/db-smoke) runs current_database(), version(), now(), and COUNT(*) on information_schema.tables to verify:

1. The Hyperdrive connection reaches Neon
2. The correct database is selected
3. The public schema has tables (migrations have run)
4. The query latency is reasonable

If it returns ok: false, the error field (redacted of any credentials) will identify the failure layer immediately — no wrangler tail required.

Note on Future Sentry Integration

The current hardening captures error_code and error_message in the health response. A future improvement is to emit these as Sentry events via withSentryWorker() so production failures are automatically alerted rather than requiring manual health checks. See the Sentry integration docs for details.