CSCI 4845/5845 · Fall 2026 · Lecture Schedule
29 class meetings (Tue/Thu 11:00–12:15), Aug 20 – Dec 3, plus the registrar’s final-exam slot (Tue Dec 8, 12:00–1:50 PM) for project presentations. Key dates per the official SLU 2026–27 academic calendar: no classes Thu Oct 22 (Fall Break) and Thu Nov 26 (Thanksgiving); SLU mid-term exam week Oct 12–16; classes end Fri Dec 4; final exams Dec 7–11.
Readings: [SLP3 n] = Jurafsky & Martin, Speech and Language Processing 3rd ed. draft, chapter n per the 2025 draft; [E n] = Eisenstein chapter n. Papers are the primary reading where listed.
Unit 1 — Foundations & Classical NLP
| # | Date | Topic & key ideas | Reading | Out / Due |
|---|---|---|---|---|
| 1 | Thu Aug 20 | Welcome: What is NLP & the three eras. Why language is hard (ambiguity, compositionality, variation); history from ELIZA and rules → statistics → neural nets → LLMs; course map: one thread from bigrams to reasoning models; logistics. | SLP3 1; E 1 | HW0 out (setup, ungraded) |
| 2 | Tue Aug 25 | Words & text processing. Regular expressions; tokenization (word, sentence; BPE teaser); normalization, stemming/lemmatization; edit distance & dynamic programming; corpora, annotation, Zipf’s law. | SLP3 2 | |
| 3 | Thu Aug 27 | N-gram language models. Probability of a sentence; Markov assumption & MLE; smoothing (Laplace, interpolation, Kneser-Ney intuition); backoff; perplexity — the metric that will follow us all semester; sampling from an LM. | SLP3 3 | HW1 out |
| 4 | Tue Sep 1 | Text classification I: Naive Bayes. Bag of words; generative classification; NB training & inference; sentiment analysis case study; evaluation: precision/recall/F1, cross-validation, statistical significance. | SLP3 4 | |
| 5 | Thu Sep 3 | Text classification II: Logistic regression. Generative vs. discriminative; features & feature engineering; cross-entropy loss, gradient descent, regularization; softmax/multinomial LR — the bridge to neural nets. | SLP3 5 | |
| 6 | Tue Sep 8 | Vector semantics I: sparse vectors. Lexical semantics (synonymy, similarity, connotation); distributional hypothesis; term-document & term-term matrices; TF-IDF, PPMI; cosine similarity; connection to information retrieval. | SLP3 6.1–6.7 | |
| 7 | Thu Sep 10 | Vector semantics II: word embeddings. word2vec (skip-gram with negative sampling), GloVe, fastText; analogies & their limits; embedding evaluation; bias in embeddings (first visit to a recurring theme). | SLP3 6.8–end; word2vec paper (skim) | HW1 due; HW2 out |
| 8 | Tue Sep 15 | Sequence labeling I: POS tagging & HMMs. Parts of speech & tagsets; NER as BIO tagging; hidden Markov models; the Viterbi algorithm (worked example). | SLP3 17 + Appendix (HMM) | |
| 9 | Thu Sep 17 | Sequence labeling II: CRFs & NER. Discriminative sequence models (MEMM → linear-chain CRF at a usable level); feature templates; NER systems & span-level evaluation; where sequence labeling lives today. | SLP3 17 (CRF §§) | |
| 10 | Tue Sep 22 | Syntax & parsing in one day. Constituency: CFGs, treebanks, CKY; dependency grammar & transition-based parsing; why parsing shaped the field, and what of it survives inside LLMs. | SLP3 18–19 (skim) |
Unit 2 — Neural NLP & the Transformer
| # | Date | Topic & key ideas | Reading | Out / Due |
|---|---|---|---|---|
| 11 | Thu Sep 24 | Neural networks for NLP. Feedforward nets, backprop & computation graphs (refresher, self-contained); the Bengio neural LM — embeddings as learned features; PyTorch idioms for NLP. | SLP3 7 | HW2 due; HW3 out |
| 12 | Tue Sep 29 | RNNs & LSTMs. Recurrence & BPTT; vanishing gradients; LSTM/GRU gating; RNN language models; bidirectional RNNs for tagging; contextual embeddings (ELMo preview). | SLP3 8 | |
| 13 | Thu Oct 1 | Seq2seq, attention & machine translation. MT as NLP’s grand challenge: rule-based & statistical MT in brief; encoder–decoder; Bahdanau attention — the idea that ate the field; MT evaluation (BLEU, chrF, COMET, and their pitfalls). | SLP3 13 | |
| 14 | Tue Oct 6 | The Transformer I: self-attention. From recurrence to attention-only; queries/keys/values; scaled dot-product & multi-head attention; positional encodings. | SLP3 9; Attention Is All You Need; The Illustrated Transformer | |
| 15 | Thu Oct 8 | The Transformer II: full architecture & tokenization. Residuals & LayerNorm, FFN blocks, encoder vs. decoder stacks, causal masking, teacher forcing; subword tokenization (BPE, WordPiece, SentencePiece) and its strange consequences. | SLP3 9; Sennrich BPE paper | |
| — | Tue Oct 13 | Midterm review & catch-up. Worked problems spanning L1–15; buffer for any topic running behind; project team formation due. | — | HW3 due |
| — | Thu Oct 15 | MIDTERM EXAM (in class; SLU midterm week is Oct 12–16). Closed book, one handwritten sheet. Covers Lectures 1–15. | — |
Unit 3 — Pretraining & the LLM Era
| # | Date | Topic & key ideas | Reading | Out / Due |
|---|---|---|---|---|
| 16 | Tue Oct 20 | Pretraining I: BERT & the encoder family. Transfer learning in NLP; ELMo → BERT; masked language modeling; fine-tuning for classification/QA; RoBERTa, sentence embeddings; where encoders still win in 2026. | SLP3 11; BERT paper | Project teams finalized |
| — | Thu Oct 22 | No class — Fall Break (Oct 22–23). | ||
| 17 | Tue Oct 27 | Pretraining II: GPT & the decoder-only paradigm. GPT-1/2/3: language models as few-shot learners; T5 & encoder–decoder; decoding algorithms (greedy, beam, temperature, top-k/top-p); the pivot the whole field took. | SLP3 10; GPT-3 paper (skim) | |
| 18 | Thu Oct 29 | Building an LLM: data & scaling. Pretraining data pipelines (Common Crawl → filtered corpora), deduplication, contamination; scaling laws (Kaplan, Chinchilla); a feel for the compute (GPUs, distributed training); open-weight vs. closed models; emergent abilities & the “mirage” debate. | Chinchilla paper (skim); Emergent Abilities + A Mirage? | HW4 out; project proposal due Fri Oct 30 |
| 19 | Tue Nov 3 | Post-training: instruction tuning & RLHF. Base model ≠ assistant; SFT/instruction tuning (FLAN, self-instruct); reward models & PPO (InstructGPT); DPO; Constitutional AI/RLAIF; what alignment does and doesn’t buy you. | SLP3 12; InstructGPT; DPO paper | |
| 20 | Thu Nov 5 | Prompting & in-context learning. Zero/few-shot prompting; why ICL works (hypotheses); chain-of-thought & self-consistency; system prompts, structured output & JSON schemas; prompt injection (first look at LLM security). | CoT paper (Wei et al.); prompting guide | |
| 21 | Tue Nov 10 | Efficient LLMs: adaptation & inference. Full fine-tuning vs. PEFT; LoRA/QLoRA; quantization (8/4-bit); distillation & small models; inference economics: KV cache, batching, speculative decoding; running models locally. | LoRA paper; QLoRA (skim) | |
| 22 | Thu Nov 12 | Retrieval-augmented generation. Why RAG: knowledge cutoffs, hallucination, private data; dense retrieval & embedding models; chunking, vector indexes, reranking; evaluating RAG (faithfulness, relevance); long-context vs. RAG. | SLP3 14; RAG paper (Lewis et al.) | |
| 23 | Tue Nov 17 | Agents & tool use. Function calling; ReAct & planning loops; code agents; Model Context Protocol & tool ecosystems; multi-agent systems; failure modes: compounding errors, prompt injection revisited. | ReAct paper | |
| 24 | Thu Nov 19 | Evaluating LLMs. From perplexity to benchmarks (MMLU, GSM8K, HumanEval); instruction-following evals; human evaluation & arenas (Elo); LLM-as-judge and its biases; contamination & saturation; building the eval your application actually needs. | HELM or eval survey (skim) | HW4 due |
| 25 | Tue Nov 24 | Safety, bias & interpretability. Harms taxonomy: stereotypes, toxicity, misinformation, privacy/memorization; hallucination; jailbreaks; interpretability: probing, attention analysis, a taste of mechanistic interpretability (induction heads, feature/SAE results). | Stochastic Parrots + a response; one interp reading | Project milestone check-ins this week |
| — | Thu Nov 26 | No class — Thanksgiving (break Nov 25–27). | ||
| 26 | Tue Dec 1 | Frontier & open problems. Reasoning models & test-time compute; multimodal LMs (vision-language); speech (Whisper/TTS) and multilingual/low-resource NLP in brief; what’s still unsolved; the full arc: n-grams → Transformers → RLHF → reasoning; careers & where to go next. | selected 2025–26 readings | |
| — | Thu Dec 3 | Course wrap + presentation dry-runs. The full arc revisited; each team gets quick feedback on its near-final slides. | — | — |
| — | Tue Dec 8, 12:00–1:50 PM (registrar’s exam slot) | Project presentations, all teams (8 min + 2 min Q&A each). No written final exam. | — | Slides due Mon Dec 7, 11:59 AM; Final report due Fri Dec 11, 11:59 PM |
Recurring threads (called out across lectures)
- Language modeling: L3 (n-grams) → L11–12 (neural LMs) → L15 (Transformer LM) → L17 (GPT) → L26 (reasoning models).
- Evaluation: L4 (P/R/F1) → L13 (BLEU) → L24 (benchmarks, LLM-as-judge).
- Meaning representation: L6–7 (static vectors) → L12/L16 (contextual) → L22 (embeddings for retrieval).
- Bias & safety: L7 (embedding bias) → L19 (alignment) → L25 (harms & interpretability).
Topics deliberately compressed
Coreference, discourse, semantic role labeling, information extraction get pointers rather than lectures (SLP3 Part II covers them for interested students); speech gets ~20 minutes in L26 rather than a unit. A 29-session comprehensive course must spend its budget where the field’s center of mass is.