Beyond APIs. Architect the Frontier.

Week 1: Tensors, Memory Layout, and Strides

Understanding how multi-dimensional data is stored in contiguous 1D memory blocks on RAM. We dive into Strides, Broad-casting rules, and why operations like .transpose() do not move data in memory but simply change the stride metadata.

# Lab 1: Implement a custom Tensor class with Strided Memory
class CustomTensor:
    def __init__(self, data, shape, strides):
        self.data = data # Flat 1D array in memory (e.g., C-contiguous)
        self.shape = shape
        self.strides = strides
        
    def index(self, indices):
        # Calculate flat index using strides mapping
        flat_idx = sum(i * s for i, s in zip(indices, self.strides))
        return self.data[flat_idx]
        
    def transpose(self, dim0, dim1):
        # O(1) time complexity! We just swap the strides!
        new_shape = list(self.shape)
        new_strides = list(self.strides)
        
        new_shape[dim0], new_shape[dim1] = new_shape[dim1], new_shape[dim0]
        new_strides[dim0], new_strides[dim1] = new_strides[dim1], new_strides[dim0]
        
        return CustomTensor(self.data, tuple(new_shape), tuple(new_strides))

Week 2: Multivariate Calculus & The Jacobian Matrix

Deriving partial derivatives for vector-valued functions. Understanding the Jacobian matrix and how the Chain Rule operates in high-dimensional deep learning space. We will calculate the derivative of the Softmax function by hand.

Week 3: Architecting a Custom Autograd Engine

You will construct a Directed Acyclic Graph (DAG) to track mathematical operations dynamically. You will implement a topological sort algorithm to traverse the graph backwards, applying the chain rule automatically. This is exactly how PyTorch .backward() works.

# Lab 3: Build a micro-grad engine (DAG + Topological Sort)
class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._prev = set(_children)

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')
        
        def _backward():
            # Apply Chain Rule
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out
        
    def backward(self):
        # Topological Sort to ensure gradients flow correctly
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        
        self.grad = 1.0 # Base case
        for node in reversed(topo):
            node._backward()

Week 4: CUDA Architecture & Streaming Multiprocessors

Introduction to physical GPU hardware. We analyze Streaming Multiprocessors (SMs), Warps, Threads, Blocks, and the difference between Global Memory and Shared Memory. You will write basic kernels to add matrices on the GPU using Numba and Triton.

Week 1: The Multi-Layer Perceptron (MLP) Geometry

Implementing forward and backward passes for dense networks. We analyze non-linear activation functions (ReLU, GeLU, Swish) and prove the Universal Approximation Theorem. We also explore the "Dying ReLU" problem mathematically.

Week 2: Information Theory & Loss Functions

Moving beyond Mean Squared Error (MSE). We dive into Information Theory: Entropy, Cross-Entropy Loss, Kullback-Leibler (KL) Divergence, and how they relate to Maximum Likelihood Estimation in classification tasks.

Week 3: Advanced Optimization (AdamW vs SGD)

Vanilla Gradient Descent gets stuck in saddle points in high-dimensional space. We explore Momentum (moving averages of gradients) and RMSProp (scaling learning rates by gradient variance). Finally, we implement AdamW, proving why decoupling weight decay is strictly superior to L2 regularization.

# Lab 7: Implement AdamW from scratch
def step_adamw(params, grads, m, v, t, lr=1e-3, beta1=0.9, beta2=0.999, eps=1e-8, weight_decay=0.01):
    t += 1
    for p, g in zip(params, grads):
        # Weight decay step (Decoupled from the gradient update!)
        p.data -= lr * weight_decay * p.data
        
        # Momentum calculations (First Moment)
        m[p] = beta1 * m.get(p, 0) + (1 - beta1) * g
        # Variance calculations (Second Moment)
        v[p] = beta2 * v.get(p, 0) + (1 - beta2) * (g ** 2)
        
        # Bias correction (vital for the first few steps)
        m_hat = m[p] / (1 - beta1 ** t)
        v_hat = v[p] / (1 - beta2 ** t)
        
        # Final update
        p.data -= lr * m_hat / (torch.sqrt(v_hat) + eps)

Week 4: Regularization & Initialization Strategies

Dropout physics (why scaling by 1/(1-p) is required during training). Batch Normalization vs Layer Normalization (and why Transformers strictly use LayerNorm). We mathematically derive Xavier and Kaiming initialization to prevent vanishing/exploding variance in forward passes.

Week 1: Convolutions and Feature Maps

Kernel math, stride, padding, and receptive fields. Building a ResNet block to solve vanishing gradients in deep vision networks.

Week 2: Recurrent Neural Networks (RNNs)

Modeling time-series data. Backpropagation Through Time (BPTT) and the mathematical proof of vanishing gradients.

Week 3: LSTMs and GRUs

Solving the vanishing gradient with gating mechanisms (Forget, Input, Output gates). Implementing an LSTM cell from scratch.

# Lab 11: LSTM Cell Math
class LSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        # combined weights for all 4 gates
        self.W = nn.Linear(input_size + hidden_size, hidden_size * 4)
        
    def forward(self, x, h_prev, c_prev):
        combined = torch.cat([x, h_prev], dim=1)
        gates = self.W(combined)
        
        # Split into gates
        i, f, o, g = gates.chunk(4, dim=1)
        
        i = torch.sigmoid(i) # Input gate
        f = torch.sigmoid(f) # Forget gate
        o = torch.sigmoid(o) # Output gate
        g = torch.tanh(g)    # Cell candidate
        
        c_next = f * c_prev + i * g
        h_next = o * torch.tanh(c_next)
        return h_next, c_next

Week 4: Seq2Seq & The Attention Bottleneck

Encoder-Decoder architectures. Bahdanau Attention vs Luong Attention. The realization that sequential processing is too slow for modern GPUs.

Week 1: Scaled Dot-Product Attention & Mathematical Stability

Deriving Q, K, and V matrices. We dive into why softmax temperature scaling is required. Without dividing by the square root of the embedding dimension (d_k), the dot products of large vectors grow exponentially, pushing the softmax function into regions of near-zero gradient (vanishing gradients). You will mathematically prove this phenomenon before writing the code.

Lab Implementation: Raw Attention Kernel

import torch
import torch.nn.functional as F
import math

def raw_scaled_attention(query, key, value, mask=None, dropout_p=0.0):
    # Ensure shapes: (batch, seq_len, dim)
    d_k = query.size(-1)
    
    # 1. MatMul: O(N^2) complexity bottleneck
    # query shape: (B, T, C), key transpose: (B, C, T) -> scores: (B, T, T)
    scores = torch.bmm(query, key.transpose(-2, -1)) / math.sqrt(d_k)
    
    # 2. Causal Masking (Critical for Autoregressive Generation)
    if mask is not None:
        # Fill masked positions with -inf so softmax outputs 0
        scores = scores.masked_fill(mask == 0, float('-inf'))
        
    # 3. Softmax
    attn_weights = F.softmax(scores, dim=-1)
    
    if dropout_p > 0.0:
        attn_weights = F.dropout(attn_weights, p=dropout_p)
        
    # 4. Context Vector
    context = torch.bmm(attn_weights, value)
    return context, attn_weights

Week 2: Multi-Head Attention (MHA) & Tensor Reshaping

Single-head attention averages out different representation subspaces. By splitting embeddings into multiple semantic "heads", the model can simultaneously attend to grammar, context, and sentiment. We utilize einops for complex tensor reshaping without dimension errors, preventing catastrophic bugs in the computation graph.

Lab Implementation: The MHA Module

import torch.nn as nn
import einops

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
        
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        # Fused projection for speed (W_q, W_k, W_v)
        self.qkv_proj = nn.Linear(d_model, 3 * d_model)
        self.out_proj = nn.Linear(d_model, d_model)
        
    def forward(self, x, mask=None):
        B, T, C = x.size()
        
        # 1. Fused projection
        qkv = self.qkv_proj(x) # (B, T, 3 * C)
        
        # 2. Split into Q, K, V
        q, k, v = qkv.split(self.d_model, dim=2)
        
        # 3. Reshape for Multi-Head: (B, T, C) -> (B, Heads, T, d_k)
        q = einops.rearrange(q, 'b t (h d) -> b h t d', h=self.num_heads)
        k = einops.rearrange(k, 'b t (h d) -> b h t d', h=self.num_heads)
        v = einops.rearrange(v, 'b t (h d) -> b h t d', h=self.num_heads)
        
        # 4. Einsum for Attention Math
        # (B, H, T, d_k) @ (B, H, d_k, T) -> (B, H, T, T)
        scores = torch.einsum('bhid,bhjd->bhij', q, k) / math.sqrt(self.d_k)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
            
        attn = F.softmax(scores, dim=-1)
        
        # (B, H, T, T) @ (B, H, T, d_k) -> (B, H, T, d_k)
        out = torch.einsum('bhij,bhjd->bhid', attn, v)
        
        # 5. Concatenate heads back to original shape
        out = einops.rearrange(out, 'b h t d -> b t (h d)')
        return self.out_proj(out)

Week 3: Positional Encoding (Absolute vs RoPE)

Transformers have no innate sense of order. We must inject positional information. We contrast Vaswani's Absolute Sinusoidal Encodings with Su's Rotary Positional Embeddings (RoPE). RoPE rotates query and key vectors in the complex plane, allowing the model to naturally decay attention for tokens that are far apart.

Lab Implementation: RoPE Matrix Injection

def apply_rotary_emb(xq, xk, freqs_cis):
    # Reshape xq to complex numbers: (..., d/2, 2) -> complex
    xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
    xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
    
    # Rotate by multiplying with frequencies (freqs_cis)
    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)
    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)
    
    return xq_out.type_as(xq), xk_out.type_as(xk)

Week 4: The Decoder-Only Pre-Training Pipeline

Assembling the GPT architecture. We implement Pre-LayerNorm vs Post-LayerNorm stability differences, SwiGLU feedforward networks (used in Llama 3), and construct a distributed training loop.

class TransformerBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.ln_1 = RMSNorm(config.d_model)
        self.attn = MultiHeadAttention(config.d_model, config.n_heads)
        self.ln_2 = RMSNorm(config.d_model)
        # SwiGLU replaces standard ReLU for better gradient flow
        self.mlp = SwiGLU(config.d_model, config.hidden_dim)
        
    def forward(self, x, mask):
        # Pre-Norm architecture prevents gradient explosion in deep layers
        x = x + self.attn(self.ln_1(x), mask)
        x = x + self.mlp(self.ln_2(x))
        return x

Week 1: PyTorch ZeRO & FSDP Math

Standard Distributed Data Parallel (DDP) replicates the entire model on every GPU, which is impossible for LLMs. We dissect the ZeRO (Zero Redundancy Optimizer) papers. You will implement ZeRO Stage 1 (Sharding Optimizer States), Stage 2 (Sharding Gradients), and Stage 3 (Fully Sharded Data Parallel - FSDP).

Lab Implementation: Configuring PyTorch FSDP

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy

# Wrap our Transformer Block from Month 4
my_auto_wrap_policy = functools.partial(
    transformer_auto_wrap_policy,
    transformer_layer_cls={TransformerBlock},
)

# PyTorch magically shards the model across the NCCL process group
fsdp_model = FSDP(
    model,
    auto_wrap_policy=my_auto_wrap_policy,
    sharding_strategy=ShardingStrategy.FULL_SHARD, # ZeRO-3
    device_id=torch.cuda.current_device(),
    mixed_precision=MixedPrecision(
        param_dtype=torch.bfloat16, 
        reduce_dtype=torch.bfloat16, 
        buffer_dtype=torch.bfloat16
    )
)

Week 2: Megatron-LM & 3D Parallelism

When a model is so large it doesn't fit on one GPU even after sharding (e.g., GPT-4). We implement NVIDIA's Megatron-LM techniques: Tensor Parallelism (TP) and Pipeline Parallelism (PP). We split the actual Matrix Multiplications across PCIe lanes.

# Concept: Column Parallel Linear Layer
# Y = X * [A1 | A2]
# GPU 0 computes Y1 = X * A1
# GPU 1 computes Y2 = X * A2
# Both GPUs perform an All-Gather operation to reconstruct Y over NVLink.

class ColumnParallelLinear(nn.Module):
    def __init__(self, in_features, out_features):
        super().__init__()
        # Allocate only half the weights to this specific GPU
        self.weight = nn.Parameter(torch.Tensor(out_features // world_size, in_features))
        
    def forward(self, input_):
        # 1. Local MatMul
        output_parallel = F.linear(input_, self.weight)
        # 2. Synchronize across NVLink
        output = tensor_parallel.gather_from_tensor_model_parallel_region(output_parallel)
        return output

Week 3: Extreme Batching & Activation Checkpointing

Training stability relies on massive batch sizes. We simulate 1024-batch sizes on small 8-GPU clusters using Gradient Accumulation. Furthermore, we implement Activation Checkpointing (Gradient Checkpointing) which deletes intermediate activations during the forward pass and re-computes them during the backward pass, saving 50% VRAM at the cost of 33% more compute time.

# Lab 19: Activation Checkpointing & Gradient Accumulation
from torch.utils.checkpoint import checkpoint

for i, (inputs, targets) in enumerate(dataloader):
    # Instead of model(inputs), we wrap it in checkpoint
    # PyTorch will delete the activations to save VRAM!
    outputs = checkpoint(model, inputs, use_reentrant=False)
    
    loss = loss_fn(outputs, targets) / accumulation_steps
    loss.backward() # During backward, it re-calculates the deleted activations
    
    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

Week 4: Fault Tolerance & Cluster Networking

When training runs for 60 days on 10,000 GPUs, hardware failures are guaranteed. We architect fault-tolerant training loops, asynchronous distributed checkpointing to S3, and analyze the topology of Infiniband networks vs Ethernet for NCCL communication operations (All-Reduce, Scatter, Gather).

Week 1: Parameter Efficient Fine-Tuning (LoRA)

The mathematics of Low-Rank Adaptation. Freezing the base model and injecting low-rank matrices (A and B) to drastically reduce trainable parameters.

Week 2: Quantization & QLoRA

Compressing 32-bit weights to 4-bit (NormalFloat4). Double quantization and paged optimizers to fine-tune a 70B model on consumer GPUs.

Week 3: Reinforcement Learning from Human Feedback (RLHF)

Training a Reward Model to judge outputs. Using Proximal Policy Optimization (PPO) to update the LLM based on the Reward Model's score.

Week 4: Direct Preference Optimization (DPO)

Bypassing the Reward Model entirely. Using mathematical derivation to optimize the LLM directly on chosen/rejected response pairs. Implementing ORPO (Odds Ratio Preference Optimization).

# Lab 24: DPO Loss Function Concept
def dpo_loss(pi_logps, ref_logps, chosen_idx, rejected_idx, beta=0.1):
    # pi is the model we are training, ref is the frozen base model
    pi_chosen_logps = pi_logps[chosen_idx]
    ref_chosen_logps = ref_logps[chosen_idx]
    
    pi_rejected_logps = pi_logps[rejected_idx]
    ref_rejected_logps = ref_logps[rejected_idx]
    
    # Calculate implicit reward difference
    chosen_rewards = beta * (pi_chosen_logps - ref_chosen_logps)
    rejected_rewards = beta * (pi_rejected_logps - ref_rejected_logps)
    
    # Apply sigmoid to maximize the margin between chosen and rejected
    loss = -F.logsigmoid(chosen_rewards - rejected_rewards).mean()
    return loss

Week 1: Sparse vs Dense Retrieval (Hybrid Search)

Combining Dense embeddings (OpenAI/BGE) with Sparse lexical search (BM25) to ensure keyword matching (IDs, names) isn't lost in vector space.

Week 2: Cross-Encoder Re-Ranking

Bi-Encoders encode query and document separately. Cross-Encoders process them together through self-attention, providing immense accuracy at the cost of speed. Implementing a Two-Stage pipeline.

Week 3: Chunking Strategies & Context Management

Recursive character splitting, Semantic chunking (using embeddings to find topic boundaries), and Parent-Document retrieval architectures.

Week 4: GraphRAG & Entity Extraction

Using LLMs to extract Nodes (Entities) and Edges (Relationships) from raw text. Storing in Neo4j and using Cypher queries to retrieve multi-hop reasoning chains.

Week 1: Variational Autoencoders (VAEs)

Compressing high-dimensional images into low-dimensional latent space. The Reparameterization Trick and KL Divergence loss in VAEs.

Week 2: Denoising Diffusion Probabilistic Models (DDPM)

The Forward Process (injecting Gaussian noise) and Reverse Process (U-Net predicting noise). Stochastic Differential Equations (SDEs) governing diffusion.

Week 3: Latent Diffusion & Stable Diffusion

Why pixel-space diffusion is too slow. Running diffusion inside the VAE latent space. Conditioning the U-Net with CLIP text embeddings via Cross-Attention.

Week 4: Audio Generation & Flow Matching

Transformers for Audio (MusicGen). Continuous Normalizing Flows and Flow Matching as the successor to Diffusion models.

Week 1: Function Calling & JSON Schema

Fine-tuning models to output deterministic JSON. Writing Python middleware to intercept JSON, execute the tool (e.g., API call), and return the result to the LLM context.

Week 2: ReAct (Reason + Act) Architectures

Implementing the Thought -> Action -> Observation loop. Building an agent from scratch without LangChain abstractions.

Week 3: Multi-Agent Debate & CrewAI

Hierarchical vs Sequential agent orchestration. Designing specialized agents (Coder, Reviewer, Tester) that autonomously converse and critique each other.

Week 4: Code Sandboxing & E2E Autonomous Dev

Executing LLM-generated code safely in isolated Docker containers. Feeding stack traces back to the LLM for autonomous self-healing and debugging.

Week 1: The KV Cache Fragmentation Problem

Autoregressive generation generates one token at a time. To prevent recalculating previous tokens, we cache their Keys and Values. However, standard HuggingFace implementations pre-allocate max_length chunks of memory. If a user only generates 10 tokens, 99% of that memory block is wasted (Internal Fragmentation). You will profile this waste using PyTorch Memory Profiler.

Week 2: PagedAttention & OS Virtual Memory for GPUs

Borrowing the concept of Virtual Memory and Pages from Operating Systems. We divide the KV cache into non-contiguous blocks (pages). vLLM dynamically allocates these blocks as tokens are generated, eliminating internal fragmentation and enabling Continuous Batching.

Lab Implementation: PagedAttention Block Table Routing

# Conceptual representation of vLLM's Block Table
class SequenceGroup:
    def __init__(self, request_id):
        self.request_id = request_id
        self.logical_token_blocks = [] # e.g., blocks 0, 1, 2
        
class BlockAllocator:
    def __init__(self, total_gpu_blocks):
        self.free_physical_blocks = list(range(total_gpu_blocks))
        self.block_table = {} # Maps Logical -> Physical
        
    def allocate_next_block(self, sequence):
        physical_block = self.free_physical_blocks.pop(0)
        logical_idx = len(sequence.logical_token_blocks)
        
        sequence.logical_token_blocks.append(logical_idx)
        self.block_table[(sequence.request_id, logical_idx)] = physical_block
        return physical_block
        
# During generation, the custom CUDA kernel reads the block_table 
# to gather the fragmented KV vectors across the GPU HBM!

Week 3: FlashAttention 2 & Triton Kernels

Reading from GPU HBM (High Bandwidth Memory) is slow. Computing in SRAM is fast. Standard attention writes the intermediate N x N attention matrix to HBM. FlashAttention fuses the operation via loop tiling to compute the softmax incrementally while keeping data in SRAM. You will write a custom Triton kernel to execute this.

Week 4: TensorRT-LLM & Execution Graphs

Models execute layer-by-layer. Compiling PyTorch models to optimized execution graphs using NVIDIA TensorRT. We explore Graph fusion (combining operations so intermediate tensors never leave registers) and layer-wise INT8 KV cache optimization.

Week 1: Containerization for CUDA Workloads

Dockerizing CUDA workloads using the NVIDIA Container Toolkit. Managing environment sizes, dependency conflicts, and building multi-stage dockerfiles to keep inference images under 5GB (excluding weights).

Week 2: Kubernetes (K8s) Fundamentals

Pods, Deployments, Services, and Ingress controllers. Writing YAML manifests to orchestrate microservices and exposing endpoints securely.

Week 3: K8s GPU Operators & KEDA Autoscaling

A standard CPU autoscaler looks at CPU usage. This fails for GPUs (which sit at 100% during batch inference). You will implement KEDA (Kubernetes Event-driven Autoscaling) to scale vLLM instances dynamically based on the length of incoming Prometheus API request queues.

# KEDA ScaledObject for vLLM
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-autoscaler
spec:
  scaleTargetRef:
    name: vllm-inference-deployment
  minReplicaCount: 1
  maxReplicaCount: 8 # Scale up to 8 GPU nodes
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server.monitoring.svc.cluster.local:9090
      metricName: vllm_request_queue_length
      threshold: '50' # Add a new GPU pod for every 50 requests in queue!

Week 4: Ray Distributed Workloads

Using Ray to manage distributed compute across heterogenous clusters. Deploying Ray Serve for dynamic model routing, allowing you to seamlessly route traffic between an ensemble of models (e.g., routing Python questions to a specialized coding model, and casual questions to a base model) without latency drops.