COMS 4995-15: Designing LLM Agents (Columbia, Fall 2026)

COMS 4995-15 Designing LLM Agents is a 3-point Columbia University computer science topics course for Fall 2026, listed in Columbia's Directory of Classes under instructor Shreya Havaldar (38 of 40 seats filled as of October 29, 2026). The course site says she is an AI researcher at Bloomberg with a 2026 PhD from UPenn. The public site has slides, Python activity files with solutions and the full homework specs, organized in three modules. Module 1, Interacting with LLMs, covers what agents are, LLM architectures and training, how LLMs generate text (with a beam search activity), and prompting, reasoning and memory. Module 2, Building an LLM Agent, covers post-training and retrieval (with a reward model activity), the agentic harness and planning, tool calling, frameworks and MCP, then AI safety and agent evaluation. Module 3 covers vibe coding, industry guest lectures and project presentations. Homework 1 builds a data-cleaning pipeline, trains BPE from scratch and trains a small transformer classifier in PyTorch. Homework 2 implements greedy, top-k and top-p decoding on GPT-2, structured output from the Gemini API with self-consistency, embedding retrieval with recall@k against a keyword baseline, and buffer, summary and fact memory. Homework 3 combines these into a persona chatbot agent with retrieval, a style module, in-character guardrails, persistent conversations and a React, Vue or Svelte front end on FastAPI or Flask. No lecture videos are posted.

Topic

Agentic

Level

Intermediate

Format

Course

Cost

Free

Duration

13 weeks, Sep 14 to Dec 14, 2026: one 2.5-hour class per week, three homeworks and a final project

Provider

Columbia University (Shreya Havaldar)

Hands-on

Yes — code/exercises

Certificate

None

• ✓Train a byte-level BPE tokenizer from scratch and compare it against GPT-2, multilingual BERT and T5 tokenizers
• ✓Implement greedy decoding, temperature, top-k and top-p sampling with a repetition penalty on GPT-2
• ✓Extract structured output from the Gemini API using few-shot, chain-of-thought and self-consistency prompting, validated against a schema
• ✓Build an embedding retriever over chunked documents and measure recall@k against a TF-IDF keyword baseline
• ✓Write grounded answer prompts that cite sources and abstain when the corpus does not cover the question
• ✓Design conversation memory from a turn buffer, running summaries and extracted durable facts
• ✓Add guardrails that detect out-of-scope or post-cutoff questions and decline them in the agent's persona
• ✓Ship a full agent with persistent conversations behind a FastAPI or Flask backend and a React, Vue or Svelte front end

• •All three homework specs are public and detailed enough to work through without enrolling, down to point breakdowns and short-answer questions
• •Each lecture comes with a runnable Python activity, and most have a published solution file
• •Goes from tokenization and decoding internals up to an end-to-end agent, so learners see what sits underneath agent frameworks
• •Runs on free tiers: GPT-2 locally, the Gemini API free tier and fastembed for embeddings
• •Covers MCP, the agentic harness, planning and agent evaluation in the Fall 2026 schedule

This page was written from 4 sources, 1 on domains other than designing-agents.github.io.

1. 1.designing-agents.github.io — designing-agents.github.iovendor
2. 2.designing-agents.github.io — HW2vendor
3. 3.designing-agents.github.io — HW3vendor
4. 4.doc.sis.columbia.edu — W4995 20263 015