Geek Out Time: Mixture of Experts(MoE) vs. CNN: A Google Colab Experiment

Introduction

I’ve been thinking about Mixture of Experts (MoE) models — the concept of dynamically routing data to specialized “expert” subnetworks represents a significant leap in efficiency and performance. Riding the DeepSeek R1 wave and inspired by other large-scale MoE-based models that have proven their effectiveness, I wanted to explore this approach firsthand. My experiments aimed to answer several key questions:

All of these experiments were conducted in Google Colab using a GPU runtime, making the process accessible for anyone interested in replicating or building upon these findings.

Experiment 1: Basic MoE vs. CNN on CIFAR-10

This is an excerpt — the full article continues on Medium.