Generative AI Essentials

The shift from discriminative to generative models — what changed between 2017's transformer paper and today's foundation-model era.

The engineer-shaped case for understanding generative models from first principles, not just calling APIs.

From rule-based parsers to statistical methods to neural language models — the four decades that led to ChatGPT.

Tokenization, stemming, lemmatization, normalization. The unglamorous foundation under every text model.

From bag-of-words to word2vec to contextual embeddings. How text becomes math a model can manipulate.

What changed in 2017 (attention), 2018 (GPT-1/BERT), 2020 (GPT-3 scale), and 2022 (ChatGPT, productization).

Vanilla recurrent networks: sequential context, the gradient problem, why they fail past ~50 tokens.

Gated memory cells. How LSTMs and GRUs extended the useful context window from tens to hundreds of tokens.

Sequence-to-sequence: an encoder compresses input to a fixed vector; a decoder generates output token-by-token. Translation's first real shot.

The 2017 paper that rebuilt the field. Self-attention, positional encoding, parallel training, and why this killed RNNs for language.

Masked language modeling. How BERT became the encoder of choice for classification, retrieval, and ranking.

Causal language modeling at scale. The architectural choice that turned a language model into a general-purpose tool.

Iterative denoising as a generative process. The architecture under Stable Diffusion, DALL·E 2, and Sora.

From convolutional layers to vision transformers. How images became sequences and joined the transformer party.

Large, broadly-pretrained models that serve as starting points for many downstream tasks. The reusable substrate of modern AI.

Gradient descent, backpropagation, loss functions, and the optimization loop. The engine under every neural network.

Causal vs masked vs contrastive vs span-corruption. The objective you pick determines what the model is good at.

Supervised fine-tuning, RLHF, DPO, LoRA, prompt-tuning. How a pretrained model becomes a product.

Quantization, distillation, pruning, KV-cache reuse, speculative decoding. The serving-cost levers that decide unit economics.

Scaling laws, compute budgets, emergent capabilities, and the cost shape that determines who can train frontier models.

Perplexity, MMLU, HumanEval, helpfulness ratings, holistic evals. Why every benchmark is wrong and you still need them.

Text + image + audio in one model. CLIP, Flamingo, Gemini, GPT-4o — how cross-modal alignment actually works.

Templates, role / system prompts, few-shot, chain-of-thought, and the prompt patterns that survive contact with production.

Vector stores, chunking, hybrid retrieval, reranking, and the eval harness that tells you whether your RAG actually works.

Tool use, planning, memory, multi-step loops. What's hard about turning a language model into something that takes actions.

Summarization, translation, rewriting, structured extraction. The bread-and-butter applications and how they're served.

From prompt to pixels: CLIP-guided diffusion, latent diffusion, ControlNet, the prompt-to-output pipeline at production scale.

Neural TTS, voice cloning, prosody, the streaming-audio pipeline. What real-time voice products are actually doing.

Frame coherence, motion priors, and the compute shape that makes video generation orders-of-magnitude harder than images.

Raw-waveform vs spectrogram vs token-based audio models. How MusicLM, Suno, and Udio actually produce sound.

Where the field is heading in 2026: agents, reasoning, on-device, multimodality, and the compute wall everyone is staring at.

What to learn next, in what order, and how to keep up when the field reinvents itself every six months.

RLHF, constitutional AI, red-teaming, refusal training. The engineering practices behind not-shipping-something-harmful.

Why models confidently make things up, what causes it, what reduces it, and how to measure progress on a moving target.