Projects

A from-scratch LLM inference engine — KV cache, paged attention, continuous batching.

A minimal, readable inference engine for transformer models built to understand — line by line — how production systems like vLLM and TensorRT-LLM actually work. Focused on the hot path: attention kernels, KV cache management, scheduling, and batching.

• Custom KV cache with paged memory allocation, inspired by vLLM PagedAttention.
• Continuous batching scheduler that interleaves prefill and decode across concurrent requests.
• CUDA and Triton kernels for fused attention; CPU fallback for local development.
• HTTP + streaming SSE API compatible with the OpenAI chat completions surface.
• Benchmark harness comparing throughput and TTFT against vLLM on the same GPU.