Anatomy of a Cryptominer Incident: From Exposed Port to Hardened Host

It started the way these things usually do: a dev VM's CPU pinned at 100% with nothing on the sprint board to explain it.

Detection

The first signal was a monitoring alert on sustained CPU. top inside the VM pointed at a process spawned from within a Postgres container - a database has no business maxing out every core.

# What's actually burning CPU?docker stats --no-stream# Inspect processes inside the suspect containerdocker top blnk-postgres# Who can reach this port from the internet?gcloud compute firewall-rules list \  --filter="allowed[].ports:5432" \  --format="table(name,sourceRanges.list(),allowed[].map().firewall_rule().list())"

The firewall query was the smoking gun: a rule allowing 0.0.0.0/0 on port 5432. The database had been exposed to the entire internet, and automated scanners found it long before anyone on the team did.

Containment

Priority one is stopping the bleeding, not forensics.

# 1. Kill the exposure at the network layergcloud compute firewall-rules delete allow-postgres-public --quiet# 2. Stop the compromised container (preserve it for analysis)docker stop blnk-postgres# 3. Snapshot the disk before touching anything elsegcloud compute disks snapshot api-dev-egawa-disk \  --snapshot-names=ir-egawa-$(date +%Y%m%d) --zone=us-central1-a

Order matters. Deleting the firewall rule first means the attacker loses access even if persistence survives a container restart.

Eradication and hardening

With the network closed, the cleanup:

• Recreated the Postgres container from a pinned, trusted image - never trust a compromised filesystem.
• Rotated every database credential. Assume anything readable was read.
• Created a dedicated application role and locked the default postgres superuser away from remote tooling.
• Audited crontabs, authorized_keys, and systemd units for persistence.

What actually went wrong

The miner is the symptom. The root cause was process: a "temporary" firewall rule created during debugging that nobody deleted. Three cheap controls would have caught it:

1. A CI check on firewall rules - anything with 0.0.0.0/0 outside ports 80/443 fails the pipeline.
2. SIEM coverage on dev - production-only monitoring is how attackers get free reign in the environment with the weakest passwords.
3. Databases on private IPs only - if an engineer needs access, that's what IAP tunnels and bastions are for.

Dev environments hold real credentials, real code, and real lateral-movement paths. Attackers know this. Monitor them like they matter - because they do.