Optimize Edge LLMs: Finetuning Llama.cpp Quantization

The Edge is Not Your Cloud Cluster [ 4.291032] oom-killer: gfp_mask=0x270cc(GFP_HIGHUSER|__GFP_NOWARN|__GFP_NOREMAP), order=0, oom_score_adj=0 [ 4.293112] CPU: 6 PID: 1421 Comm: llama-server Tainted: P O 5.15.0-1040-tegra #1 [ 4.293121] Hardware name: NVIDIA Jetson Orin Nano Developer Kit [ 4.301984] Node 0: the_net_mem: 3042456KB total, 12KB free, 3042444KB anon, 0KB file, 0KB shmem [ 4.303212] Out of … Read more

Debugging eBPF-based Network Policies with Cilium & Hubble

The 3:00 AM Black Hole: When “It Works on My Minikube” Meets Production At 03:14 local time, PagerDuty decided my sleep was offensive. The alarm wasn’t just ringing; it was screaming that the core billing service, payment-processor-v4-86c8d7b7f-9xq2m, had lost its ability to talk to the fraud-detection engine running in an adjacent namespace. Naturally, the dashboard … Read more

Go 1.22 Range Loop: Understand New Semantics & Races

The PagerDuty siren didn’t just ring; it tore a hole straight through my three hours of broken sleep at 3:14 AM. My terminal was already open before my second cup of stale cold brew hit the desk, glowering at me with a stack trace from git.acme-fintech.com/internal/trade-engine/v2/pricing/settlement.go:412 that made my stomach drop. Accounts were drifting. Not … Read more

eBPF-Based Distributed Tracing: Kernel 6.8+ Overhead Deep Dive

[ALERT] 2024-10-24T04:12:09.102Z – api-gateway-prod-7d4b9c – HighTailLatency: p99 latency on target endpoint /checkout jumped from 520µs to 2.1ms across 40 nodes. Error budget burning at 14x normal rate. My coffee was still warm, but the adrenaline had already burned a hole through my stomach. There was no deployment of application code. No sudden traffic surge from … Read more

eBPF-Based Distributed Tracing Overhead: High-Throughput Kernels

The Unseen Tax: eBPF-Based Distributed Tracing Overhead in High-Throughput Linux Kernels Three AM is not a time for philosophy. It is a time for stale coffee, glaring terminal windows, and the dull panic of a production bleedout. Last Tuesday, our p99 latency for `api.prod-us-east-2.mycompany.io/v2/payments` spiked from a predictable 45 milliseconds to a vertical wall of … Read more

Optimize AWS Lambda Cold Starts with SnapStart & Containers

[ERROR] 2024-10-27T03:14:22.819Z requestId: f8c3e8a1-b2c4-4e5f-9a1b-7c8d9e0f1a2b Task timed out after 10.00 seconds INIT_START Runtime Version: java17:v1.2.3 Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:a1b2c3… [WARN] com.amazonaws.services.lambda.runtime.diagnostics.Initializer – Initialization phase took 8412ms. Total billed duration: 10000ms. You paid for ten seconds of compute just to discover your container-based serverless function choked on its own dependency injection framework before processing a single incoming … Read more

Benchmarking Java 21: Virtual Threads Latency Deep Dive

Post-Mortem: Socket Exhauster on Node 04 java.lang.OutOfMemoryError: Thread creation failed: Operating system resource exhausted at java.base/java.lang.Thread.start(Thread.java:1539) at java.base/java.lang.VirtualThread.start(VirtualThread.java:307) at com.fintech.ledger.io.SocketWorkerPool.dispatch(SocketWorkerPool.java:88) at com.fintech.ledger.GatewayServer.lambda$acceptConnections$3(GatewayServer.java:142) at java.base/jdk.internal.vm.Continuation.enter(Continuation.java:340) … Caused by: java.io.UncheckedIOException: java.net.SocketException: Too many open files at com.fintech.ledger.io.NativeEpollDispatcher.register(NativeEpollDispatcher.java:211) It pinned. Hard. After three days of staring at raw kernel-bypass logs on Node 04, watching p99.9 latencies blow past … Read more

PostgreSQL 17 Logical: Failover Slots & Conflicts

Sweat dripping onto the mechanical keyboard. Pager blaring at 03:42 UTC. The SRE channel melting down with all-caps panic. That was the moment `ledger_db_prod_us-east-1` effectively committed suicide. We watched in absolute horror as the primary node flatlined due to an abrupt storage array partition. Autonomous failover scripts executed precisely as engineered. They promoted `pg17-primary-core.data.corp:5433` to … Read more

ebpfbased distributed tracing in Envoy: Fix Leaks & Overhead

The 03:14 AM OOM Sweep [Wed Oct 18 03:14:02 2023] oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=900 [Wed Oct 18 03:14:02 2023] CPU: 37 PID: 14892 Comm: envoy Kdump: nullptr Tainted: G OE 5.15.0-88-generic #98-Ubuntu [Wed Oct 18 03:14:02 2023] Mem-Total: 32901416kB, AnonPages: 29841904kB, Shmem: 11232kB, Slab: 412091kB [Wed Oct 18 03:14:02 2023] Active(anonymous): 18402912 Inactive: 11419024 Isolated:₀ … Read more