AI Systems · Inference engineering · Distributed systems · LLMOps · Infra engineering
My philosophy
Don't accept the abstraction.
Understand what the system actually does. Take it apart. Measure it. Build your own mental model from the evidence.
Hi, I am Abi! I am an AI Infrastructure Engineer building inference systems for data centers, neoclouds, and hyperscalers. I founded Joule, an inference power economics engine that ties physical GPU energy to token throughput and SLO goodput.
I build the systems, measure them, research them, write about them, and teach them. The path, the books, and the research sit on About.
I built this
Here is the code →- Joule
Founder. Joule is an inference power economics engine that ties physical GPU energy to token throughput and SLO goodput.
- gpuengineering.com
The public list behind the domain: CUDA, kernels, serving engines, multi-GPU systems. ~600 stars as awesome-gpu-engineering.
- RelayServe
Minimal LLM inference gateway for heterogeneous devices. OpenAI-compatible chat, batching, streaming, request IDs, /metrics. On PyPI as relayserve.
- smol-vllm
Educational paged-attention engine: KV cache, continuous batching, preemption, prefill vs decode. Built to be read, not shipped as production vLLM.
- fullstack-inferencing
vLLM on Lambda Cloud, FastAPI gateway, nginx, Prometheus, Grafana, optional Jaeger. Documents engine-flag limits, cost metrics, and Lambda image caveats.
I measured this
Here is what I found →On Writing: Investigations, Archive, ModelCraft, Field notes.
- Why GPU Utilization Is a Misleading Metric for LLM Inference: Queueing, KV Cache, and SLOs
GPU utilization is not goodput. Queueing, KV pressure, and why $97 vs $0.38 / MTok.
- Why KV Cache Needs a Directory: Locality, Leases, and Cache-Aware Routing
On one GPU, KV is an allocator. Across a cluster it is a phone book.
- Why the Same GPU Serves One Request in 3.3 Seconds and Another in 39: Request Shape, Prefill, and Decode
Same model, same card, same code. The request shape did it.
- One Inference Roadmap, Not Fifty Tools: Request Path Before CUDA
Path, phases, memory, queue, then kernels. Not a bookmark pile.
- Why Agent Workloads OOM at 62% GPU Utilization: KV Cache, Multi-Turn State, and a $27k Weekend
The demo had headroom. Saturday agents leased HBM. Seven of twelve replicas died.
- The Inference Engineer Owns SLOs, Not vLLM Flags: Why This Job Didn't Exist in 2024
The role owns TTFT, TPOT, KV pressure, and $/token when the demo becomes traffic.
From the cohorts
See the full reviews →“Real case studies on multi-tenancy clustering and scaling a cluster with good practice, from cluster level to kernel engineering.”
“This course doesn't stop at explaining how LLM inference works, it puts you inside the actual machinery. Every project runs against a GPU you actually rent and break.”
“This course was very helpful in giving me a broad, end-to-end understanding of the LLM inference space and connecting the key concepts.”
“It goes beyond theory and dives into the practical challenges of building and scaling LLM inference systems, from KV cache to admission control.”
“I would give this course 10 stars if I could. We built every part of the inference stack by hand, so we came out understanding the mechanics, not just the APIs.”
“Fantastic course for deep-diving into LLM systems. The final project covers the full inference lifecycle: load balancing, agentic systems, and observability.”
1:1 mentorship
Topmate · 5.0 →“I got some truly insightful suggestions on my career path during my 1:1 with Abi. I woke up the next morning with absolute clarity on what I need to tackle next.”
“I have had few calls through Topmate and she is legit the best I have talked to so far!”
“An insightful session on transitioning into the inference side of LLMs, and the gaps in my resume I need to fill.”
- Joule
An inference power economics engine that ties physical GPU energy to token throughput and SLO goodput.
- LLMOps, and GPU Engineering
LLMOps, written 2023-2024, finished December 2024, published 2025, now in English and four translations. GPU Engineering is almost finished, launch is end of 2026 or early 2027.