Geek Out Time: Reducing Hallucination with DoLa in Google Colab (Microsoft Phi-2)

Is decoding just about the top layer — greedy, beam, temperature sampling, maybe top-p if you want variation? What if we could make a model reason better just by changing how we decode, not what we train?

That’s the idea behind DoLa (Decoding by Contrasting Layers) . It was introduced in late 2023 by Wang et al. as a way to get better reasoning and factual accuracy out of existing models by combining information from different layers during decoding.

What Is DoLa?

DoLa is a novel decoding strategy designed to improve the factual accuracy and reasoning ability of Large Language Models (LLMs) without the need for fine-tuning or external knowledge. The core idea is based on the observation that LLMs are prone to hallucinations because they often prioritize fluent, linguistically plausible outputs over factual correctness. DoLa addresses this by exploiting the hierarchical knowledge structure inherent in the transformer architecture:

This is an excerpt — the full article continues on Medium.