Geek Out Time: LLM Inference Exposed-How It Works And How to Make It Fast on Google Colab

In recent months, the race to run Large Language Models (LLMs) faster and cheaper has accelerated — and not just for training. Inference, the process of generating predictions from a trained model, is often the most expensive and time-critical phase, especially when deploying models at scale or experimenting with limited resources like Google Colab.

This geek out dives into what happens during LLM inference, breaking it down into practical, measurable steps and how to inspect, accelerate, and optimize inference, from quantization and batching to decoding strategies and attention mechanics — all on free-tier Colab with a compact open-source model: TinyLlama (1.1B).

Why TinyLlama? It strikes a balance between simplicity and depth — small enough to run on a Colab T4 GPU with 4-bit quantization, yet large enough to demonstrate key inference behaviors seen in production-scale LLMs.

How Does LLM Inference Work?

This is an excerpt — the full article continues on Medium.