Geek Out Time: Simulating High-Bandwidth and Cache-Like Memory on Google Colab’s T4 GPU

This week, we’re delving into the GPU memory hierarchies and exploring how different types of memory impact the performance of transformer models like GPT, BERT, and T5. While Google Colab’s T4 GPU doesn’t provide direct access to High Bandwidth Memory (HBM) or detailed cache controls, we can simulate aspects of these memory behaviors to understand their effects on computational tasks.

Before jumping into our experiments, let’s explore how CPUs, GPUs, and various types of memory contribute to the efficiency of transformer models.

This is an excerpt — the full article continues on Medium.