Why: On 2026-05-21, test.unglue.it went into an OOM cascade (Apache killed → guest network stack hung) and sat impaired for 5 days before discovery, because:
- No CloudWatch alarms are configured on the EC2 instance (
StatusCheckFailed, CPU, memory — none)
disable_admin_emails: true on test (and dj42) per Ansible config — error mail to Eric is suppressed by design- SolarWinds monitors prod only; test/dj42 aren't watched
Diagnostic trail from the 2026-05-26 healthcheck:
| Layer |
State |
SystemStatus (hypervisor) |
ok |
InstanceStatus (guest reachability) |
failed since 2026-05-21T15:51Z |
| CPU 5/21 16:45 → 18:35 |
sustained ~58–60% (runaway) |
| CPU 5/21 18:40 → present |
flatline 0.36% (guest hung) |
| Console screenshot |
Out of memory: Killed process 13064 (apache2) UID:33 |
Console screenshot saved locally (/tmp/test-unglue-console.jpg). Can attach if useful.
Minimum fix (this issue)
Add a CloudWatch alarm to each non-prod EC2 in AWS account 439256357102 (Gluejar member, region us-east-1):
| Instance |
Tag |
Instance type |
i-0bd911f9ae13310b6 |
test-unglue-it |
t3.medium |
i-035d248ac4b18ef60 |
dj42-unglue-it |
t3.medium |
Alarm spec:
- Metric:
AWS/EC2 StatusCheckFailed_Instance
- Statistic: Maximum, period 60s
- Threshold:
> 0 for 15 minutes (15 consecutive periods, or 3× 5-min)
- Treat missing data:
breaching
- Action: SNS topic → raymond.yee@gmail.com (and optionally Eric)
Out of scope (separate work)
- Investigate why test OOMs (Apache config? Celery? memory leak? t3.medium undersized?). Different problem; alarm is the alarm.
- Disk + memory alarms (CloudWatch agent install required for memory; can be a follow-up)
- Whether
disable_admin_emails: true should be reconsidered on test/dj42 (tracked separately — see handoff queue)
- Prod (
i-038105b781b514601 / prod7_py3) already has SolarWinds coverage; in scope only if we want belt-and-suspenders
Implementation note
Probably a single boto3/awscli script or a small task in this playbook (monitoring/ role?). Open to whichever pattern fits the existing provisioning conventions — TBD with reviewer.
Provenance
Issue body drafted by Claude Code from a live healthcheck session; reviewed by Raymond before filing.
Why: On 2026-05-21,
test.unglue.itwent into an OOM cascade (Apache killed → guest network stack hung) and sat impaired for 5 days before discovery, because:StatusCheckFailed, CPU, memory — none)disable_admin_emails: trueon test (and dj42) per Ansible config — error mail to Eric is suppressed by designDiagnostic trail from the 2026-05-26 healthcheck:
SystemStatus(hypervisor)InstanceStatus(guest reachability)Out of memory: Killed process 13064 (apache2) UID:33Console screenshot saved locally (
/tmp/test-unglue-console.jpg). Can attach if useful.Minimum fix (this issue)
Add a CloudWatch alarm to each non-prod EC2 in AWS account
439256357102(Gluejar member, regionus-east-1):i-0bd911f9ae13310b6test-unglue-iti-035d248ac4b18ef60dj42-unglue-itAlarm spec:
AWS/EC2 StatusCheckFailed_Instance> 0for 15 minutes (15 consecutive periods, or 3× 5-min)breachingOut of scope (separate work)
disable_admin_emails: trueshould be reconsidered on test/dj42 (tracked separately — see handoff queue)i-038105b781b514601/prod7_py3) already has SolarWinds coverage; in scope only if we want belt-and-suspendersImplementation note
Probably a single
boto3/awscliscript or a small task in this playbook (monitoring/role?). Open to whichever pattern fits the existing provisioning conventions — TBD with reviewer.Provenance
Issue body drafted by Claude Code from a live healthcheck session; reviewed by Raymond before filing.