The $100 Speedrun: Mastering Transformer Architecture through the karpathy/nanochat Repo


The $100 Speedrun: Mastering Transformer Architecture through the karpathy/nanochat Repo

karpathy/nanochat

2026-02-03

karpathy/nanochat is a project by Andrej Karpathy (founding member of OpenAI and former Director of AI at Tesla) that essentially "open-sources the factory" of LLMs. It’s not just a chat app; it’s a complete, end-to-end pipeline that takes you from raw text all the way to a functioning ChatGPT-like UI for about $100 in GPU credits.

Here’s a breakdown of why this is a game-changer for us engineers.

In our day-to-day, we often use LLMs as "black boxes" via APIs. nanochat peels back the curtain.

Transparency over Abstraction
Unlike complex frameworks (like LangChain or Megatron), this is roughly 8,000 lines of clean, readable PyTorch code. You can actually step through the code to see how a model learns to talk.

The Full Stack
It covers the entire lifecycle

Tokenization
A custom BPE tokenizer built in Rust for speed.

Pretraining
Teaching the model "world knowledge" from massive datasets.

Supervised Fine-Tuning (SFT)
Training it to follow instructions and use the "Assistant" persona.

Inference & Web UI
A simple React/Python setup to actually chat with your creation.

Scaling Laws in Action
The "$100" tag isn't just a marketing hook. It’s a benchmark. Karpathy shows that by simply turning the "depth" dial of the model, you can scale from a "kindergartener" model to something surpassing GPT-2 capabilities.

The project is designed to be a "speedrun." You boot up a cloud GPU instance (ideally an 8x H100 node), and you run one script.

Clone the repo and install the minimal dependencies

git clone https://github.com/karpathy/nanochat.git
cd nanochat
pip install -r requirements.txt

The repo includes a script called speedrun.sh. This is the "magic button" that executes the entire pipeline—tokenization, training, and evaluation—automatically.

bash runs/speedrun.sh

Wait about 3 to 4 hours, and you'll have a model ready to chat.

To give you a taste of the code's simplicity, here is a conceptual snippet of how the training iteration looks in the repo. It's pure, unadulterated PyTorch

# Simplified snippet of the training logic
for step in range(max_steps):
    # Get a batch of data (input tokens 'x' and target tokens 'y')
    x, y = train_loader.next_batch()
    
    # Forward pass through the Transformer
    with torch.autocast(device_type='cuda', dtype=torch.bfloat16):
        logits, loss = model(x, y)
    
    # Backward pass
    loss.backward()
    
    # Optimization step
    optimizer.step()
    optimizer.zero_grad()
    
    if step % 100 == 0:
        print(f"Step {step}: loss {loss.item():.4f}")

If you've ever felt like AI is "magic," this repo is the cure. It treats LLM training as a systems engineering problem rather than a mystical science.

For about $100, you aren't just buying a model; you're buying the knowledge of how to build one from scratch. It’s the ultimate capstone project for any engineer moving into the AI space.


karpathy/nanochat