COSI 230B | Spring 2026

Natural Language Annotation for Machine Learning

Lecture Materials

Lecture Materials

Slides and resources for each lecture

Note: Lecture slides are released after each class. Check MOODLE for any supplementary materials.

Week 1: Introduction (Jan 12, 14)

Lecture 1: Course Introduction & The Annotation Landscape

Course overview, what is annotation, evolution from manual labeling to LLM-assisted workflows

PDF

Lecture 2: Annotation Fundamentals

Formal definition, types of NLP tasks, the MATTER cycle, quality vs. quantity

PDF

Week 2: When to Annotate (Jan 21)

Lecture 3: When to Annotate | Tools & Formats

Rule-based vs. ML approaches, decision framework, annotation tool landscape, data formats

PDF

Week 3: Corpus & Data (Jan 28)

Lecture 4: Corpus Selection & Data Sourcing

MAMA criteria, sampling strategies, licensing, synthetic data generation

PDF

Week 4: What Models Learn (Feb 2, 4)

Lectures 5 & 6: What Models Learn from Annotation

How annotations shape model behavior, data-centric AI, annotation artifacts

PDF

Week 5: Design Pipeline & IAA I (Feb 9, 11)

Lecture 7: The Annotation Design Pipeline

End-to-end annotation project design, task formalization, guidelines, workflow

PDF

Lecture 8: Inter-Annotator Agreement I

Why measure agreement, percent agreement, Cohen's Kappa, interpreting values

PDF

Week 7: IAA II & IAA III (Feb 23, 25)

Lecture 9: Inter-Annotator Agreement II

Fleiss' Kappa, Krippendorff's Alpha, weighted agreement measures

PDF

Lecture 10: Inter-Annotator Agreement III

Advanced IAA topics, guideline iteration, annotator feedback

PDF

Week 8: IAA in the LLM Era & Annotator Reliability (Mar 2, 4)

Lecture 11: IAA in the LLM Era

Human-LLM agreement, annotation agreement with large language models

PDF LLM

Lecture 12: Annotator Reliability

Modeling annotator reliability, label noise, multi-annotator learning

PDF

Week 10: Annotator Reliability II & Annotation Projects (Mar 16, 18)

Lecture 13: Annotator Reliability II

Dawid-Skene EM algorithm, modeling annotator reliability, inferring true labels from noisy annotations

PDF

Lecture 14: Annotation Projects

Historical overview of treebanks, semantic resources, crowdsourced datasets, and LLM alignment data

PDF

Week 11: Annotation Projects & Supervision Engineering (Mar 23, 25)

Lecture 15: Annotation Projects

Historical overview of treebanks, semantic resources, crowdsourced datasets, and LLM alignment data

PDF

Lecture 16: Supervision Engineering

Paradigm shift from labeling to behavior shaping, supervision contracts, taxonomy, and failure modes

PDF LLM

Week 12: Instruction Annotation (Mar 30)

Lecture 17: Instruction Annotation

Instruction-tuning datasets, task contract specification, template leakage, mixture engineering

PDF LLM

Week 13: Instruction Annotation & RLHF (Apr 13, 15)

Lecture 18: Instruction Annotation

Instruction-tuning datasets, task contract specification, template leakage, mixture engineering

PDF LLM

Lecture 19: RLHF & InstructGPT

Misalignment, the three-step RLHF pipeline (SFT, reward model, PPO), KL regularization, alignment vs.\ raw scale

PDF LLM

Week 14: Preference Annotation & Reasoning Tuning (Apr 20, 22)

Lecture 20: Preference Annotation & RLHF in Practice

Preferences as value judgments, RLHF pipelines, DPO, Constitutional AI, rater pool composition, biases

PDF LLM

Lecture 21: From RLHF to Reasoning Tuning

Why the field pivoted: credit assignment, chain-of-thought, ORM vs.\ PRM, GRPO, train-time and test-time scaling

PDF LLM

Week 15: Reasoning Annotation & The Future (Apr 27, 29)

Lecture 22: Reasoning Annotation: Process Supervision

Rationales vs.\ process supervision, PRM800K, Math-Shepherd, e-SNLI leakage, faithfulness, CoT controllability

PDF LLM

Lecture 23: The Future of Annotation: From Text to Agents

Beyond text: agent trajectories, tool-use traces, multi-modal annotation, the road ahead

Upcoming