CSCI 4845/5845 · Fall 2026 · Final Project

The project (30% of grade) is the course’s capstone: teams of 1–4 formulate an NLP question, run real experiments, and communicate the results like researchers. Presentations run on two days — Thu Dec 3 (in class) and Tue Dec 8, 12:00–1:50 PM (the registrar’s final-exam slot) — 8 min talk + 2 min Q&A per team on either day.

Detailed instructions for each deliverable — team declaration, proposal, milestone check-in, presentation, and final report — are posted on Canvas.

The Oct 13 project studio runs in class; the worksheet is handed out that day.

Tracks

Every team — 4845, 5845, or mixed — chooses either track.

• Research track: investigate a question — form a hypothesis, design experiments, compare against at least one baseline, and analyze what the results do and do not show. Must include a related-work section situating the question in the literature.
• Application track: build an NLP system for a real task with a serious evaluation — a labeled test set, at least two compared approaches, and error analysis. “We built a chatbot and it seems good” does not clear the bar; the evaluation is the project.

Team policy

• Teams of 1–4. Scope scales with team size: a 4-person project should attempt roughly twice what a 2-person project does (more conditions, more analysis, a larger eval set) — the rubric is the same, but “enough work” is judged per person.
• Solo projects are allowed and graded on the same rubric; a solo report may run shorter (4–6 pages instead of 6–8) with the same required sections.
• Teams declare themselves on Canvas by Tue Oct 6, 11:59 PM (sign-up opens Tue Sep 1) — see the team-declaration instructions. Students without a team on Oct 6 are matched at the start of the Oct 13 studio.
• Teams of 2+ include an individual contribution statement in the final report.

Timeline

No late days on project deliverables. Both presentation days use the same rubric; day assignment does not affect grading.

Rubrics (summary)

Proposal (5%): clear question (2), feasible plan with identified data (2), sensible evaluation design (1). Main purpose is early feedback — graded generously, redirected firmly.

Presentation (10%): problem motivation & clarity (3), technical soundness (3), results & honest analysis including failures (3), Q&A (1).

Report (15%): problem & related work (3), methods correctness & clarity (4), experiments + evaluation rigor (4), analysis & limitations (3), writing quality (1). Negative results honestly analyzed grade as well as positive ones.

Example project menu (teams may also propose their own)

Any item can be adapted to either track — the difference is whether the deliverable is an answered question (research) or an evaluated system (application). Scope any item up or down for your team size.

Research-track flavored

1. Scaling laws in miniature: train a family of tiny GPTs (1M–50M params) and fit a loss-vs-compute curve; where does the Chinchilla intuition show up at toy scale?
2. BERT vs. LLM few-shot: for 3 classification tasks, at what training-set size does fine-tuned DistilBERT beat few-shot prompting of a larger model? Cost curves included.
3. LLM-as-judge audit: measure position bias, length bias, and self-preference of an LLM judge against human ratings on a public preference dataset.
4. DPO at small scale: align a small open model on a public preference dataset; evaluate with win-rates vs. the SFT base.
5. Probing for linguistic structure: do modern open-weight LLM representations encode POS/dependency structure? Layer-wise probing study.
6. Tokenizer politics: measure tokenization efficiency across languages for major tokenizers and quantify the downstream cost/performance gap for non-English users.
7. Speculative decoding in practice: implement/benchmark draft-model speculative decoding; speedup vs. acceptance-rate analysis across task types.
8. Long-context vs. RAG: same document-QA task, stuff-the-context vs. retrieve; accuracy, cost, and “lost in the middle” effects.
9. In-context learning sensitivity: how much do example order, label balance, and prompt format move few-shot accuracy? A controlled study across tasks and model sizes.
10. Quantization degradation: which capabilities break first at 4-bit and below? Same model, same eval suite, decreasing precision — a capability-by-capability map.
11. Chain-of-thought faithfulness: does the stated reasoning actually drive the answer? Perturb the chain (truncate, corrupt a step) and measure whether answers follow.
12. AI-text detectors vs. non-native writers: run Lecture 3’s detector story as an experiment — measure perplexity-based detector false-positive rates on human text across writer populations.
13. Prompt injection attacks & defenses: build a tool-using agent, attack it with a suite of injections, measure which published defenses actually help.

Application-track flavored

14. RAG faithfulness: build RAG over a domain corpus (e.g., SLU policies, medical guidelines) and measure hallucination rate vs. a closed-book baseline with a human-labeled eval set.
15. Low-resource MT: fine-tune NLLB/small models for a low-resource language pair; compare against zero-shot LLM translation with chrF/COMET.
16. Clinical/legal NER: LLM extraction vs. fine-tuned encoder on a public biomedical/legal NER benchmark; where does each fail?
17. PII redaction tool: detect and redact names, addresses, and identifiers in a document collection; measure precision/recall against a hand-labeled test set, and the cost of each miss type.
18. Faithful summarizer: summarize long documents (meetings, lectures, filings) with a human-labeled faithfulness evaluation — every summary claim traced to a source span.
19. Short-answer autograder: grade student answers against a rubric; validate an LLM judge against instructor labels before trusting it at scale (Lecture 24’s protocol).
20. Hybrid search for a real corpus: BM25 vs. dense vs. hybrid retrieval on a corpus you care about, with a labeled query set; which queries does each retriever win?