It’s 3:14 AM. The production hotfix just failed because a self-hosted agent ran out of disk space on its C: drive, and the cleanup script I wrote three months ago was silently killed by a Group Policy update no one told me about. If you’re reading this, you’re my replacement. I’m gone. I’ve traded my … Read more
node:internal/modules/cjs/loader:1147 throw err; ^ Error: Cannot find module ‘left-pad’ Require stack: – /opt/legacy-app/dist/server/index.js at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15) at Module._load (node:internal/modules/cjs/loader:985:27) at Module.require (node:internal/modules/cjs/loader:1235:19) at require (node:internal/modules/helpers:176:18) at Object. (/opt/legacy-app/dist/server/index.js:42:18) at Module._compile (node:internal/modules/cjs/loader:1356:14) at Module._extensions..js (node:internal/modules/cjs/loader:1414:10) at Module.load (node:internal/modules/cjs/loader:1208:32) at Module._load (node:internal/modules/cjs/loader:1024:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:11) { code: ‘MODULE_NOT_FOUND’, requireStack: [ ‘/opt/legacy-app/dist/server/index.js’ ] } [3:14 AM] … Read more
text Mar 14 03:14:02 node-prod-042 kubelet[1402]: E0314 03:14:02.124 node_container_manager_linux.go:612] “Failed to update control group” err=”failed to set config for cgroup v2: write /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823.slice/memory.max: invalid argument” Mar 14 03:14:05 node-prod-042 kubelet[1402]: I0314 03:14:05.441 status_manager.go:614] “Patch status for pod failed” pod=”default/api-gateway-7f8d9b” err=”rpc error: code = Unavailable desc = keepalive ping failed” Mar 14 03:14:08 control-plane-01 kube-apiserver[821]: E0314 … Read more
text [ 4829.102394] NVRM: GPU at PCI:0000:41:00: GPU-8d2f1a3b-5c6e-7d8f-9a0b-1c2d3e4f5a6b [ 4829.102398] NVRM: Xid (PCI:0000:41:00): 31, Ch 00000018, recovery procedure: 1 [ 4829.102401] NVRM: Xid (PCI:0000:41:00): 79, GPU has fallen off the bus. [ 4829.102405] NVRM: Error: GPU at 0000:41:00.0 has fallen off the bus. [ 4829.102410] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 … Read more
It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire kubernetes cluster was a graveyard of CrashLoopBackOffs. I’m writing this because if I don’t document this nightmare now, I’ll probably hallucinate that it was all a bad dream induced by too much caffeine … Read more
[ 402.193284] nvidia-uvm: Loaded the UVM driver in 8 module load mode. [ 402.201112] NVRM: GPU at PCI:0000:01:00: GPU-8f2a1c5b-3d4e-5f6a-7b8c-9d0e1f2a3b4c [ 405.881203] torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB (GPU 0; 80.00 GiB total capacity; 72.12 GiB already allocated; 1.88 GiB free; 74.00 GiB reserved in total by PyTorch) [ 405.881245] File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, … Read more
Stop Treating Your Docker Image Like a Virtual Machine It was 3:14 AM on a Tuesday when the first PagerDuty alert fired. I was the lead SRE for a fintech startup, and we were scaling our transaction processing engine to handle a sudden spike in volume. I had just “optimized” our CI/CD pipeline by caching … Read more
POST-MORTEM AND MANIFESTO: THE 72-HOUR DEATH MARCH Document ID: PM-2024-05-12-PROD-COLLAPSE Status: Bitter. Author: Senior SRE (On-call since Tuesday, currently vibrating from caffeine toxicity) I’m writing this because if I don’t, the “DevOps Transformation Committee” will spin this as a “learning opportunity” in their next slide deck. It wasn’t a learning opportunity. It was a preventable … Read more
Stop Hiring “AI Researchers” to Build Your Production Pipelines I once watched a 12GB BERT model trigger a cascading failure that wiped out a regional payment gateway in seventeen minutes. The “Machine Learning Engineer” on the team—a brilliant guy with a PhD who could derive backpropagation from first principles on a whiteboard—had pushed a new … Read more
React Native: The Bridge is Out and Your App is OOM-Killed It was 3:00 AM on a Tuesday in 2021. I was the lead SRE for a fintech startup that had just launched a high-yield savings feature. We built the mobile app in react native because the “velocity” was supposed to be unmatched. We hit … Read more