Improving Accuracy of LLM Applications
by DeepLearning.AI
A disciplined accuracy workflow — evaluate first, then prompt, then fine-tune — built around a text-to-SQL agent.
Overview
Taught by Sharon Zhou, Vice President of Artificial Intelligence at AMD and previously co-founder and CEO of Lamini, alongside Amit Sangani, Senior Director of Partner Engineering at Meta, the course runs 1 hour 51 minutes across seven lessons: Introduction; Overview; Create an SQL Agent; Create an Evaluation; Finetuning, PEFT and Memory Tuning; Generate Data and Finetune; and Conclusion, closing with a graded quiz. Four lessons carry runnable code. The through-line is a single application — converting natural language to SQL against a custom schema using Llama 3-8B — carried across every technique so improvements are directly comparable rather than anecdotal. The 23-minute evaluation lesson is the heart of it and comes deliberately before any fine-tuning. The later lessons cover parameter-efficient fine-tuning with LoRA and then Lamini Memory Tuning, a technique that trains large numbers of LoRA adapters and selects among them via a Mixture of Memory Experts at inference, optimising for zero error on specific facts the model must recall rather than for low average loss. Treat the headline claim carefully: independent write-ups note the widely-quoted 95% accuracy figure comes from unnamed Fortune 500 deployments without a published benchmark or dataset, so the mechanism is well documented but the number is not independently reproducible. Prerequisites are intermediate Python and LLM familiarity.
At a Glance
- Topic
- Fine-Tuning
- Level
- Intermediate
- Format
- Course
- Cost
- Free
- Duration
- 1 hour 51 minutes, self-paced (7 video lessons, 4 with code examples)
- Provider
- DeepLearning.AI
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Build a text-to-SQL agent on Llama 3-8B against a custom database schema
- ✓Construct an evaluation harness before tuning, so every later change is measurable
- ✓Sequence the accuracy workflow correctly: evaluate, then prompt, then self-reflect, then fine-tune
- ✓Apply parameter-efficient fine-tuning with LoRA and understand what it does and does not change
- ✓Generate synthetic training data for fine-tuning and judge whether it is good enough to use
- ✓Explain memory tuning and Mixture of Memory Experts as a hallucination-reduction technique
- ✓Decide when prompting has run out of headroom and fine-tuning is genuinely warranted
Highlights
- •Puts evaluation before fine-tuning — the correct order, and the one most teams get backwards
- •One application carried end to end, so each technique's effect is directly comparable
- •Taught by the founder of Lamini (now VP of AI at AMD) together with Meta's partner engineering lead
- •Covers memory tuning and Mixture of Memory Experts, which almost no other course at this length reaches
- •Uses an open-weights model (Llama 3-8B), so the techniques transfer to self-hosted deployments
Who It’s For
Best For
- ✓Engineers whose LLM feature is accurate enough to demo but not to ship
- ✓Teams building text-to-SQL or other structured-output applications over a custom schema
- ✓Practitioners deciding between prompt engineering, RAG and fine-tuning for an accuracy problem
- ✓ML engineers who need a defensible evaluation methodology to show stakeholders
Prerequisites
- •Intermediate Python, including comfort reading training and evaluation code
- •Familiarity with large language models and prompt engineering fundamentals
- •Basic SQL, since the worked application is a natural-language-to-SQL agent
FAQ
What is Improving Accuracy of LLM Applications?
Most teams reach for fine-tuning before they can measure anything. This course inverts that: you build a text-to-SQL agent on Llama 3-8B, construct an evaluation harness for it, and only then work through prompting, self-reflection, LoRA and memory tuning, measuring each step. It is the closest thing to a repeatable accuracy methodology available in under two hours.
Is Improving Accuracy of LLM Applications free?
Improving Accuracy of LLM Applications is free to access.
What level is Improving Accuracy of LLM Applications for?
Improving Accuracy of LLM Applications is aimed at a intermediate audience. Recommended background: Intermediate Python, including comfort reading training and evaluation code, Familiarity with large language models and prompt engineering fundamentals, Basic SQL, since the worked application is a natural-language-to-SQL agent.
How long does Improving Accuracy of LLM Applications take?
Expect roughly 1 hour 51 minutes, self-paced (7 video lessons, 4 with code examples). Most learners work through it at their own pace.
What will I learn from Improving Accuracy of LLM Applications?
You'll learn: Build a text-to-SQL agent on Llama 3-8B against a custom database schema; Construct an evaluation harness before tuning, so every later change is measurable; Sequence the accuracy workflow correctly: evaluate, then prompt, then self-reflect, then fine-tune; Apply parameter-efficient fine-tuning with LoRA and understand what it does and does not change; Generate synthetic training data for fine-tuning and judge whether it is good enough to use; Explain memory tuning and Mixture of Memory Experts as a hallucination-reduction technique; Decide when prompting has run out of headroom and fine-tuning is genuinely warranted.
Topics
Sources
This page was written from 3 sources, 2 on domains other than deeplearning.ai.