Here is my new series where I will embark on an interesting journey: building a Large Language Model (LLM) from scratch using a reverse engineering approach.

Usually to learn and figure out how to build an LLM, you will start with the mathematical foundations and then work your way upward to a functional model. For this I would do the opposite. I will start from a ready made working LLM and systematically deconstruct it. This allows me (and you) to immediately see the end goal and how each of the different component contributes to the final system.

Why This Approach?

Most machine learning videos start with theory and gradually build complexity. While this is makes sense theoretically, it can be challenging to maintain motivation when I don’t see practical results for weeks or months. By starting with a functional LLM and working backwards, I’ll:

  • See immediate results from day one
  • Understand the bigger picture before diving into details
  • Build practical skills alongside theoretical knowledge
  • Maintain motivation by working with real, functioning code

What I hope to Learn

By the end of this series, I’ll have:

  • Built my own LLM from scratch using PyTorch
  • Understood every component of the Transformer architecture
  • Implemented my own tokenizer and training pipeline
  • Created a working chatbot using my custom model
  • Gained insights into scaling and optimizing LLMs

Course Curriculum

Here’s the complete roadmap for my journey. I’ll be checking off completed lessons and adding links to videos as they’re released:

Module 1: Using a Pretrained LLM

  • Lesson 1.1: Introduction & Course Philosophy
  • Lesson 1.2: Using HuggingFace Transformers
    • Loading a pretrained model and generating text
    • Code: Load GPT2, LLaMA, or Mistral and interact with it
  • Lesson 1.3: Fine-tuning a Pretrained LLM
    • Transfer learning and domain adaptation
    • Code: Fine-tune on custom dataset using HuggingFace Trainer

Module 2: Assembling the Pieces

  • Lesson 2.1: Tokenizers and Embeddings
    • What tokenizers are, types, and how they work
    • Code: Train a Byte Pair Encoding (BPE) tokenizer with tokenizers library
  • Lesson 2.2: Building the Model Architecture
    • Assembling a Transformer model (decoder-only)
    • Code: Use PyTorch to implement transformer blocks; load weights
  • Lesson 2.3: Training Pipeline
    • Setting up data pipeline, training loops
    • Code: Train a small GPT model from scratch using WikiText
  • Lesson 2.4: Saving, Loading & Inference
    • Model checkpoints and serving
    • Code: Save model and tokenizer; write an inference script

Module 3: Internal Mechanics

  • Lesson 3.1: Attention Mechanism Explained
    • Scaled Dot-Product Attention, Causal Masking
    • Code: Implement multi-head self-attention from scratch
  • Lesson 3.2: Positional Embeddings
    • Absolute vs relative; sinusoidal
    • Code: Implement positional encodings and visualize them
  • Lesson 3.3: LayerNorm, Residuals, GELU
    • Deep dive into supporting components
    • Code: Reimplement and replace with your own versions

Module 4: From Tokens to Training

  • Lesson 4.1: Tokenization Algorithms
    • BPE, Unigram LM, WordPiece
    • Code: Implement a toy BPE tokenizer
  • Lesson 4.2: Dataset Construction
    • Cleaning, chunking, shuffling
    • Code: Prepare a dataset using datasets or raw corpus
  • Lesson 4.3: Loss Functions and Optimizers
    • Cross-entropy, label smoothing, AdamW
    • Code: Write your own training loop with proper loss handling

Module 5: Scaling Up

  • Lesson 5.1: Scaling Laws
    • Effects of data/model size on performance
    • Theory-focused; mini case studies
  • Lesson 5.2: Distributed Training
    • Data parallelism, model parallelism
    • Code: Simple torch.distributed example
  • Lesson 5.3: Quantization and Inference Optimization
    • Techniques to shrink and speed up LLMs
    • Code: Use bitsandbytes, ONNX, ggml

Module 6: Wrap-Up Project

  • Lesson 6.1: Build Your Own Chatbot
    • Use your model, tokenizer, and training code
    • Code: CLI chatbot or Streamlit app
  • Lesson 6.2: Evaluating LLMs
    • Perplexity, BLEU, human evals
    • Code: Script to evaluate model on test dataset
  • Lesson 6.3: Where to Go From Here
    • Resources, communities, research directions
    • Optional: Guest researcher interview

What’s Next?

In the next post, I’ll dive right in by setting up our environment and interacting with a pretrained LLM through the OpenAI API. I’ll build an AI-powered application in just a few lines of code!

Stay tuned for updates, and feel free to reach out with questions or suggestions as we embark on this exciting journey together.

This post will be updated as new lessons are released. Bookmark this page to track your progress through the curriculum!