PostgreSQL · Data ingestion
Loading a million rows into Postgres
I loaded the same CSV five ways. Typed COPY took 3.91 seconds; a text-only staging table took 1.29.
The workshop.
These are the experiments behind my posts. Each one has a GitHub repository if you want to run it yourself.
Playing with PgBouncer
Where's the bottleneck?
One PgBouncer process used a full CPU core while Postgres sat at 22%. I added more processes to see how far it would scale.
Switch between the recorded results for 1, 2, 4, and 8 processes. Every run keeps the same 128-connection budget.
PgBouncer benchmark Recorded results
45,122queries / secondPostgreSQL CPU: 22 percent. 1 PgBouncer processes.
PgBouncer processes
PgBouncer fills one CPU core while PostgreSQL uses 22% of its host CPU.
128 database connections in every run.p99 8.19 ms
Vector search · Compression
TurboVec at 100 million vectors
The 4-bit index fit in 37.4 GB with 91.4% recall@10. Median search time was 5.67 seconds.
Python · JSON
Reading large JSON files in Python
Loading the whole array used gigabytes of memory. Reading one JSONL record at a time kept peak RSS around 25 MB.