CSCI 4845/5845 · Fall 2026 · Final Project
The project (30% of grade) is the course’s capstone: teams of 3–4 formulate an NLP question, run real experiments, and communicate the results like researchers. All presentations run in the registrar’s final-exam slot — Tue Dec 8, 12:00–1:50 PM — so team size is set to keep the schedule feasible: with 31 students expect 8–10 teams at ~10 minutes per team (8 min talk + 2 min Q&A), which fits the 110-minute slot with a small buffer.
Tracks
- Research / reproduction track (required for any team including a CSCI 5845 student): reproduce a published result on a smaller scale and extend it (new dataset, ablation, model size, language), or investigate an original question. Must include a related-work section and comparison against at least one baseline.
- Application track (open to CSCI 4845-only teams; teams with a 5845 student may propose one only with instructor approval and an unusually rigorous evaluation): build an NLP system for a real task with a serious evaluation — a labeled test set, at least two compared approaches, and error analysis. “We built a chatbot and it seems good” does not clear the bar; the evaluation is the project.
Timeline
| Date | Milestone | Weight |
|---|---|---|
| Tue Oct 13 | Teams formed (form on Canvas; instructor matches stragglers) | — |
| Fri Oct 30 | Proposal (1–2 pages): question, data, methods, evaluation plan, compute plan, division of labor | 5% |
| Week of Nov 2 | Proposal feedback returned (approve/redirect within one week) | — |
| Week of Nov 23 (Mon/Tue) | Milestone check-in: 15-minute meeting per team in office hours; data in hand, baseline running, preliminary numbers | required, ungraded gate |
| Mon Dec 7, 11:59 AM | Presentation slides due (so the session runs off one laptop, no transitions lost) | — |
| Tue Dec 8, 12:00–1:50 PM | Presentations (all teams, single session in the exam slot): 8 min talk + 2 min Q&A per team | 10% |
| Fri Dec 11, 11:59 PM | Final report: ACL format, 6–8 pages + references; individual contribution statement | 15% |
No late days on project deliverables.
Rubrics (summary)
Proposal (5%): clear question (2), feasible plan with identified data & compute (2), sensible evaluation design (1). Main purpose is early feedback — graded generously, redirected firmly.
Presentation (10%): problem motivation & clarity (3), technical soundness (3), results & honest analysis including failures (3), Q&A (1).
Report (15%): problem & related work (3), methods correctness & clarity (4), experiments + evaluation rigor (4), analysis & limitations (3), writing quality (1). Negative results honestly analyzed grade as well as positive ones.
Example project menu (teams may also propose their own)
- Scaling laws in miniature: train a family of tiny GPTs (1M–50M params) and fit a loss-vs-compute curve; where does the Chinchilla intuition show up at toy scale?
- BERT vs. LLM few-shot: for 3 classification tasks, at what training-set size does fine-tuned DistilBERT beat few-shot prompting of a frontier model? Cost curves included.
- RAG faithfulness: build RAG over a domain corpus (e.g., SLU policies, medical guidelines) and measure hallucination rate vs. a closed-book baseline with a human-labeled eval set.
- LLM-as-judge audit: measure position bias, length bias, and self-preference of an LLM judge against human ratings on a public preference dataset.
- Low-resource MT: fine-tune NLLB/small models for a low-resource language pair; compare against zero-shot frontier-model translation with chrF/COMET.
- Reproduce DPO at small scale: align a 1B model on a public preference dataset; evaluate with win-rates vs. the SFT base.
- Prompt injection attacks & defenses: build a tool-using agent, attack it with a suite of injections, measure which published defenses actually help.
- Probing for linguistic structure: do modern open-weight LLM representations encode POS/ dependency structure? Layer-wise probing study (classic papers exist to reproduce).
- Tokenizer politics: measure tokenization efficiency across languages for major tokenizers and quantify the downstream cost/performance gap for non-English users.
- Speculative decoding in practice: implement/benchmark draft-model speculative decoding; speedup vs. acceptance-rate analysis across task types.
- Long-context vs. RAG: same document QA task, stuff-the-context vs. retrieve; accuracy, cost, and “lost in the middle” effects.
- Clinical/legal NER with LLMs: GPT-style extraction vs. fine-tuned encoder on a public biomedical/legal NER benchmark; where does each fail?
Compute guidance
Projects must be feasible on Colab (+ optionally Colab Pro), course API credits, or a student’s own hardware — the proposal’s compute plan is checked for this. Reproduction-track teams should target “smaller-scale reproduction,” not paper-scale training runs.