[2023-10-24 03:14:22.891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 64.22 GiB already allocated; 1.12 GiB free; 72.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF [2023-10-24 03:14:25.102] … Read more
Stop Buying Security Tools and Start Fixing Your SSH Config It was 2017. I was working at a fintech startup that was “disrupting” the lending space. We were moving fast, which is industry shorthand for “we didn’t have a staging environment that matched production.” I was tasked with spinning up a Redis instance for a … Read more
The rain is drumming against the corrugated tin roof of the shed. It’s a cold, rhythmic sound that reminds me why I left Palo Alto. Out here, if a fence post rots, the cows get out. There’s no “fail-fast” philosophy when you’re chasing a half-ton of beef through a muddy creek at 3:00 AM. There … Read more
text [ 259200.412834] Out of memory: Killed process 124092 (python3) total-vm:154234824kB, anon-rss:82341204kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:302144kB oom_score_adj:0 [ 259200.413102] oom_reaper: reaped process 124092 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [ 259200.415001] Kernel panic – not syncing: Fatal exception in interrupt [ 259200.415050] Kernel Offset: 0x24000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 259200.415110] —[ end Kernel panic … Read more
AWS is a Collection of Primitives, Not a Magic Wand I once cost a former employer $42,000 in a single weekend because I misunderstood how AWS NAT Gateways handle data processing. I was migrating a high-throughput logging cluster—think 50TB of JSON blobs a day—from a co-located data center into a VPC in us-east-1. I followed … Read more
INCIDENT LOG: 2024-05-20T03:04:12Z LOCATION: us-east-1 (Production Cluster ‘Ares-01’) STATUS: CRITICAL FAILURE 03:04:12Z – PagerDuty alert: KubePodNotReady (Namespace: checkout-prod). 03:05:45Z – Node ip-10-244-12-84.ec2.internal enters NotReady state. 03:06:10Z – Cascading failure detected. kube-dns latency spikes to 4500ms. 03:07:30Z – API Server unresponsive. kubectl commands timing out. 03:10:00Z – On-call engineer attempts manual recovery. $ kubectl get pods … Read more
TO: Engineering All FROM: Senior SRE (On-Call Rotation 4) DATE: 04:12 AM – Day 3 of Incident #8842 SUBJECT: The Corpse of the Production Cluster and the Myth of Automation It’s 4:12 AM. I haven’t slept since Tuesday. My vision is tunneling, and the blue light from my third monitor is starting to feel like … Read more
TECHNICAL AUDIT AND DECONSTRUCTION: REACT 18.3.1 AND THE COLLAPSE OF THE BARE METAL Subject: Survival Guide for Low-Level Devs Navigating the “React Docs” Author: Senior Systems Engineer (Firmware/RTOS/Assembly) Status: Highly Irritated / Forced Compliance SECTION 1.0: PRE-FLIGHT HARDWARE AUDIT AND THE ABSTRACTION TAX I spent twenty years writing C and Assembly for 8-bit microcontrollers where … Read more
I just ran docker images on the “Hello World” container you pushed to the registry. REPOSITORY TAG IMAGE ID CREATED SIZE hello-world-app latest f3a2b1c0d9e8 2 minutes ago 2.14GB Two. Gigabytes. For a Python script that prints “Hello World” and exits. In 1994, I was managing entire SunOS clusters on less disk space than you’ve wasted … Read more
Timestamp: 03:14:22 UTC. The cluster started bleeding, and it was all because someone thought they were too good for type hints. Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 299, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 210, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/v2/endpoints/processor.py", line 84, in process_data result = … Read more