Reinforcement Learning From Human Feedback (DeepLearning.AI)

What it covers

This short course, taught by Nikita Namjoshi of Google Cloud, explains reinforcement learning from human feedback (RLHF), the main technique for tuning language models towards what people prefer. It has six video lessons and four code examples. You learn:

• why a model trained on internet text still needs human preference data;
• the two datasets RLHF uses: a preference dataset, where humans chose the better of two responses, and a prompt dataset;
• how a reward model learns to predict human preferences, and how the language model is then tuned against it;
• how to compare the tuned model with the base model, including side-by-side evaluation.

The code examples tune an open Llama 2 model using Google Cloud's open-source pipeline components. You can follow the concepts without running them.

Why it matters for AI data work

If you rate, rank or compare model answers, your judgements become the preference dataset this course describes. Seeing where your work goes changes how you do it: a careless choice between two responses teaches the reward model the wrong thing, and many careless choices push a model towards answers that look good but are wrong. Understanding the pipeline also helps you explain your work credibly in applications and later in data or ML roles.

How to take it

• If you are on the annotation path, watch the first two lessons for the concepts and skip the code.
• If you are on the AI trainer path, especially as a programmer, follow the code examples and note what fields the preference data contains.
• Afterwards, explain in five sentences, without notes, how one of your ratings ends up changing a model.

Free and paid

Enrolling is free. A paid DeepLearning.AI membership adds things such as an accomplishment certificate; the course content does not need it.

Best for: people doing or preparing for rating and preference work who want to understand what their judgements train, and anyone moving towards ML roles.