Inference Economics: The Real Cost per Token

Inside this episode

1. 00:00:00Cold open: the invoice that ate the pilot
2. 00:02:21What a token really is, and why it costs anything
3. 00:04:24The waiting machine: memory bandwidth and the roofline
4. 00:07:48The KV cache: the notebook that grows with every word
5. 00:10:11Prefill and decode: one job, two personalities
6. 00:12:20Batching: putting more passengers on the bus
7. 00:14:57PagedAttention: the hotel that stopped reserving whole floors
8. 00:17:43Prefix caching: never read the same manual twice
9. 00:20:22Speculative decoding: the intern who guesses ahead
10. 00:23:50The economics of guessing: acceptance rates and when speculation backfires
11. 00:25:53Medusa and EAGLE: growing the intern inside the senior engineer
12. 00:27:56Chunked prefill: stopping the sprinter from blocking the hallway
13. 00:29:32Disaggregation: two factories instead of one
14. 00:32:15Mooncake: trading storage for compute
15. 00:34:37Shrinking the bytes: quantization as packing lighter
16. 00:36:54Mixture of experts: a big library with a small reading room
17. 00:39:01Splitting the model: tensor, pipeline and the cost of talking
18. 00:42:21Measuring honestly: percentiles, latency budgets and benchmark traps
19. 00:44:57The cost-per-token formula
20. 00:48:27The hidden costs: agents, retries and the long tail
21. 00:50:21The meeting
22. 00:52:31Epilogue: the economics of waiting