Stop Calling It Magic: A Grumpy SRE’s Guide to What Is Machine Learning It was 3 AM on a Tuesday in 2019. We had just deployed a “smart” auto-scaler for our edge nodes. The idea was simple: use a lightweight ML model to predict traffic spikes and spin up instances before the load hit. Instead, … Read more
text $ top -pid 49201 Processes: 612 total, 2 running, 610 sleeping, 3214 threads Load Avg: 2.41, 2.10, 1.95 CPU usage: 14.22% user, 8.11% sys, 77.66% idle SharedLibs: 412M resident, 51M data, 62M linkedit. MemRegions: 182934 total, 4102M resident, 112M private, 1204M shared. PhysMem: 16G used (3210M wired), 214M unused. PID COMMAND %CPU TIME #TH … Read more
text [2024-10-14 03:14:22,891] ERROR:worker:Process ‘Icarus-Inference-Engine-7’ terminated with signal 9 (SIGKILL) [2024-10-14 03:14:22,892] CRITICAL:kernel:Out of Memory (OOM) killer invoked. Victim: python3.11 (pid 4402) [2024-10-14 03:14:23,004] TRACEBACK: File “/app/inference/model_loader.py”, line 142, in predict features = feature_store.get_latest(user_id) File “/app/data/feature_store.py”, line 89, in get_latest df = pd.read_sql(query, engine) File “/usr/local/lib/python3.11/site-packages/pandas/io/sql.py”, line 1561, in read_sql return pandas_sql.read_query(sql, index_col=index_col, params=params) MemoryError: … Read more
This is not a “retrospective.” It is not a “learning opportunity.” It is a post-mortem of a preventable disaster that cost this company six figures in lost revenue and cost me ten years of my life expectancy. If I see one more “move fast and break things” sticker on a laptop in this office, I … Read more
text HANDOVER LOG: 2024-05-14_04:30_UTC SRE: J. Miller (Shift 1/2 – 48hr Continuous) STATUS: CRITICAL / DEGRADED INCIDENT: #LLM-RECURSION-STORM-09 [SYSTEM LOG START] 2024-05-14T03:12:01.442Z [ERROR] [llm-gateway-v2] openai.RateLimitError: Error code: 429 – {‘error’: {‘message’: ‘You exceeded your current quota, please check your plan and billing details.’, ‘type’: ‘insufficient_quota’, ‘param’: None, ‘code’: ‘insufficient_quota’}} 2024-05-14T03:12:01.445Z [WARN] [agent-executor] Agent loop detected. … Read more
SUMMARY OF LOSS The air in the data center didn’t just smell like ozone; it smelled like burnt plastic and the collective career aspirations of an entire IT department going up in smoke. Forty-eight hours. No sleep. Just the rhythmic, mocking blink of amber LEDs on a SAN that used to hold three petabytes of … Read more
The Bloat is a Choice: A Return to Semantic Sanity Next.js is a cancer. There, I said it. It is a bloated, over-engineered, glorified wrapper for people who are too terrified to look at a raw html file. We have spent thirty years refining the most resilient, accessible, and lightweight document format in human history, … Read more
text $ du -sh node_modules 428M node_modules $ top -p $(pgrep -d ‘,’ node) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12842 silas 20 0 1.244g 312.4m 45.2m S 4.2 1.9 0:18.42 node I have spent the last three decades working with the 8051, the AVR, and various ARM Cortex-M … Read more
I’m sitting here looking at a dashboard full of “microservices” consuming 64GB of RAM to serve a basic CRUD application that could have run on a 486 DX2 with 8MB of memory, and I’m being asked to explain what is docker to a generation of developers who think a “server” is an abstract concept living … Read more
text $ node –version v20.11.0 $ npm audit npm audit report cross-fetch <3.1.8 Severity: high Regular Expression Denial of Service – https://github.com/advisories/GHSA-7gc6-9fqp-82jj fix available via npm install [email protected] node_modules/cross-fetch 128 vulnerabilities (42 high, 12 critical) $ vite build … <— Last few GCs —> [14022:0x65c2e00] 15232 ms: Mark-sweep 2032.4 (2048.5) -> 2031.1 (2048.5) MB, 1120.4 … Read more