Skip to content
10 min read

Load Testing to 100K Concurrent Users with Distributed JMeter on GCP

PerformanceJMeterGCPBash

When an OTT platform asks "can we survive launch night?", the only honest answer is a number - and you don't get real numbers from a laptop running JMeter.

The ceiling of a single generator

One well-tuned JMeter instance realistically sustains 2–5K threads before the *generator* becomes the bottleneck and your results measure your test rig instead of your platform. 100K users means a fleet.

Architecture: waves, not a monolith

Instead of JMeter's fragile master/slave RMI setup, each generator ran standalone with its own slice of test data, launched in waves:

wave-1: 20 VMs x 1,000 threads  =  20K users   (warm-up)wave-2: 30 VMs x 1,000 threads  =  50K total   (sustain)wave-3: 50 VMs x 1,000 threads  = 100K total   (peak)

Waves give you a controlled ramp and a clean answer to "at what load did latency degrade?"

Automated CSV slicing

Every virtual user needs unique credentials. Sharing one CSV across the fleet means duplicate sessions and garbage data. The fix is slicing at provision time:

#!/usr/bin/env bash# slice-csv.sh - give each generator a disjoint block of test usersTOTAL_VMS=$1VM_INDEX=$2          # 0-based, derived from instance nameUSERS_PER_VM=1000START=$(( VM_INDEX * USERS_PER_VM + 2 ))   # +2: skip header, 1-indexedEND=$(( START + USERS_PER_VM - 1 )){ head -n 1 users_master.csv;  sed -n "${START},${END}p" users_master.csv; } > /opt/jmeter/users.csv

Each VM derives its index from its instance name in the GCP startup script, slices its block, and starts JMeter in non-GUI mode. Zero manual coordination across 100 machines.

Lessons that survived contact with reality

  • Fix the JMX before scaling. A bug that costs you one bad run on 1 VM costs you a corrupted dataset on 100 VMs.
  • Non-GUI mode, always. The JMeter GUI is for building plans, never for running them.
  • Watch generator health. CPU above ~70% on a generator invalidates its numbers.
  • Preemptible/spot VMs cut the bill dramatically - a load test fleet is the perfect disposable workload.

The final report wasn't "it feels fast." It was: *the platform sustained 100K concurrent users with p95 latency inside SLA, and the first component to degrade was X at Y load.* That's a sentence a CTO can plan a launch around.