halfrost @halffrost

✨ Fiat Lux ✨ Go Bears 🐻💙💛 Alumni @UC Berkeley @CMU 🇺🇸 Bay Area 🧑🏻‍💻 Computer Science❤️github@halfrost github.com/halfrost California Joined July 2016

Tweets

359
Followers

7K
Following

323
Likes

190

halfrost @halffrost

3 weeks ago

I traced vLLM V1’s KV cache from GPU budget to physical slot: fixed-size blocks, prefix hashes, refcounts, preemption, and block tables. Part 6: github.com/halfrost/Halfr…

0 0 0 459 0

View Details

vLLM V1 has no global “prefill phase” or “decode phase.” Each step asks: how many tokens may each request process? Article 05 traces token budgets, RUNNING-before-WAITING, Chunked Prefill, prefix-aware admission & KV-pressure preemption. github.com/halfrost/Halfr… #vLLM

3 0 8 1K 3

View Details

halfrost @halffrost

3 months ago

vLLM’s heartbeat is EngineCore.step(): schedule → execute → abort barrier → commit Article 04 traces request states, overlapped batches, optimistic counters, aborts, cleanup & continuous batching—one tick, line by line. github.com/halfrost/Halfr… #vLLM

0 1 1 636 1

View Details

halfrost @halffrost

3 months ago

vLLM V1 is not “an API server calling a GPU.” It is a process topology: API server → ZMQ → EngineCore → shared-memory queues → GPU workers. Article 03 traces GIL isolation, TP/DP math, startup barriers, and worker-death propagation. github.com/halfrost/Halfr… #vLLM

2 0 5 713 6

View Details

halfrost @halffrost

3 months ago

vLLM has 3 front doors: LLM.generate(), vllm serve, and OpenAI-compatible HTTP. Follow the source and they converge: process_inputs → EngineCoreRequest → EngineCore. Article 02 traces offline, CLI, HTTP, SSE & cancellation. github.com/halfrost/Halfr… #vLLM

0 0 6 633 2

View Details

halfrost @halffrost

3 months ago

What happens between LLM.generate() and the first token returned by vLLM? I traced the complete V1 request path—from input processing and continuous-batch scheduling to GPU execution, sampling, and output processing. Part 1 of a source-code walkthrough: github.com/halfrost/Halfr…