Geek Out Time: Gradient Boosted Trees(GBT) Using a Hallucination Risk Playground

Gradient Boosted Trees (GBT) are one of those models everyone uses, and u probably know the story: sequential trees, correcting residuals, learning rate, depth trade-offs. This week’s geek-out was about playing with GBT in a modern setting — a synthetic “LLM hallucination risk” dataset. Not because I needed a production hallucination detector. In practice, GBT rarely shows up for that problem. The field has moved toward LLM-as-judge, semantic entropy, and RAG grounding checks — approaches that are more auditable and defensible in regulated environments. But that wasn’t the point. The LLM framing was just a vehicle to construct a dataset with the kind of structure that makes boosting interesting: overlapping distributions, label noise, nonlinear boundaries that no single decision rule can cleanly separate. A modern-feeling playground for an old favourite.

Problem Setup: Turning LLM Behavior into Tabular Features

Imagine sampling the same LLM prompt five times. Well, u can use the real LLM, like DeepSeek, to generate. But here I simplify it…

From those five answers, we compute observable signals:

This is an excerpt — the full article continues on Medium.