COSI 230B | Spring 2026

Natural Language Annotation for Machine Learning

Lecture Materials

Lecture Materials

Slides and resources for each lecture

Note: Lecture slides are released after each class. Check MOODLE for any supplementary materials.

Week 1: Introduction (Jan 12, 14)

Lecture 1: Course Introduction & The Annotation Landscape

Course overview, what is annotation, evolution from manual labeling to LLM-assisted workflows

PDF

Lecture 2: Annotation Fundamentals

Formal definition, types of NLP tasks, the MATTER cycle, quality vs. quantity

PDF

Week 2: When to Annotate (Jan 21)

Lecture 3: When to Annotate | Tools & Formats

Rule-based vs. ML approaches, decision framework, annotation tool landscape, data formats

PDF

Week 3: Corpus & Data (Jan 28)

Lecture 4: Corpus Selection & Data Sourcing

MAMA criteria, sampling strategies, licensing, synthetic data generation

PDF

Week 4: What Models Learn (Feb 2, 4)

Lectures 5 & 6: What Models Learn from Annotation

How annotations shape model behavior, data-centric AI, annotation artifacts

PDF

Week 5: Design Pipeline & IAA I (Feb 9, 11)

Lecture 7: The Annotation Design Pipeline

End-to-end annotation project design, task formalization, guidelines, workflow

PDF

Lecture 8: Inter-Annotator Agreement I

Why measure agreement, percent agreement, Cohen's Kappa, interpreting values

PDF

Week 7: IAA II & IAA III (Feb 23, 25)

Lecture 9: Inter-Annotator Agreement II

Fleiss' Kappa, Krippendorff's Alpha, weighted agreement measures

PDF

Lecture 10: Inter-Annotator Agreement III

Advanced IAA topics, guideline iteration, annotator feedback

PDF

Week 8: IAA in the LLM Era & Annotator Reliability (Mar 2, 4)

Lecture 11: IAA in the LLM Era

Human-LLM agreement, annotation agreement with large language models

PDF LLM

Lecture 12: Annotator Reliability

Modeling annotator reliability, label noise, multi-annotator learning

PDF

Week 10: Annotator Reliability II & Annotation Projects (Mar 16, 18)

Lecture 13: Annotator Reliability II

Dawid-Skene EM algorithm, modeling annotator reliability, inferring true labels from noisy annotations

PDF

Lecture 14: Annotation Projects

Historical overview of treebanks, semantic resources, crowdsourced datasets, and LLM alignment data

PDF

Week 11: Annotation Projects & Supervision Engineering (Mar 23, 25)

Lecture 15: Annotation Projects

Historical overview of treebanks, semantic resources, crowdsourced datasets, and LLM alignment data

PDF

Lecture 16: Supervision Engineering

Paradigm shift from labeling to behavior shaping, supervision contracts, taxonomy, and failure modes

PDF LLM

Week 12: Instruction Annotation (Mar 30)

Lecture 17: Instruction Annotation

Instruction-tuning datasets, task contract specification, template leakage, mixture engineering

PDF LLM

Week 13: Preference Annotation & Reasoning (Apr 13, 15)

Lecture 18: Preference Annotation & RLHF

Preferences as value judgments, RLHF pipelines, DPO, Constitutional AI, rater pool composition

PDF LLM

Lecture 19: Reasoning & Process Supervision

Chain-of-thought rationales, process supervision, faithfulness risks, step-level correctness annotation

PDF LLM

Week 14: Synthetic Annotation & Active Learning (Apr 20, 22)

Lecture 20: Synthetic Annotation

LLM-generated labels, model collapse, filtering pipelines, provenance tracking for synthetic data

PDF LLM

Lecture 21: Active Learning & LLM Annotators

Active learning selection strategies, LLMs as annotators, human-LLM routing, feedback loops

PDF LLM

Week 15: Evaluation & Multilingual Annotation (Apr 27, 29)

Lecture 22: Evaluation Annotation

Evaluation as annotation, rubric design, LLM-as-judge biases, adversarial benchmark construction

PDF LLM

Lecture 23: Multilingual & Cultural Annotation

Translationese artifacts, native annotation, cultural adaptation, cross-language annotator comparability

PDF