A follow-up/sub-part to Part 3 of the LLM inference internals series. Part 3 built sampling-mode speculative decoding with KV caching on both the draft and verifier sides, and it worked correctly, but the speedup it delivered didn't match expectations set by the paper. This post is the record of chasing that gap: every hypothesis tested, which ones were wrong, and what the real answer turned out to be. The first full gamma sweep, on an open-ended, opinion-style prompt, came back like this: Mode











