Cheat sheet · Earn independently
AI data and training work cheat sheet (Data annotation)
Opens with creditsThe data annotation edition: annotation types and tools, guideline reading, agreement metrics, platform comparison, scam red flags and getting paid, for people labelling images, video, text and audio. Platform facts are as of October 2026.
Curated by Tauseef Fayyaz9 chapters66 concepts120 glossary termsChapters
9 chapters · 66 concepts
1What AI data work isWhy AI models need people: labels for traditional models, demonstrations and preferences for chatbots, and human testing for quality and safety. Plus who pays and what the 2026 demand data says.8 concepts
- 1Labelled dataExamples with the correct answer attached, such as a photo marked "bus" or a review marked "negative".
- 2Demonstrations (SFT data)A prompt paired with the ideal answer, written by a person.
- 3Preference data and RLHFRecords of which of two or more answers a person preferred.
- 4Reward modelA model trained on human preference data to predict which answer people would prefer, giving each answer a score.
- 5Eval (evaluation)A set of test prompts plus a way to score the answers, by code or by people using a rubric, to measure how good a model is at something.
- 6Red teaming (safety testing)Deliberately trying to make a model misbehave, such as giving harmful instructions or leaking data, under project rules, and reporting what worked.
- 7The AI data supply chainAI labs hire data vendors (Scale AI, Appen, TELUS Digital, Toloka, Labelbox, Turing), which recruit workers through platforms such as Outlier, CrowdGen…
- 82026 demand evidenceUpwork In-Demand Skills 2026: AI data annotation and labelling demand up 154 percent year over year.
8 in this chapterOpen this chapter2The honest picturePay, project gaps, contractor status and wellbeing, as they really are, before you put in time.5 concepts
- 1Pay that variesPlatforms set rates by expertise, project difficulty and location, and show the rate before you accept a project.
- 2Effective hourly rateWhat you actually earned divided by all the hours you spent, including reading guidelines, unpaid tests and rework.
- 3Project-based workWork opens when a client needs data and closes when they have enough.
- 4Independent contractorYou work for the platform as a contractor, not an employee: no paid leave, health cover or pension, and you handle your own tax.
- 5Sensitive content workProjects reviewing harmful material, such as violence, hate or self-harm, so models learn to refuse or flag it.
5 in this chapterOpen this chapter- Scam red flagsThe scams aimed at people looking for AI data work. Real platforms are free to join, recruit on their own sites and pay through named services.7 conceptsOpens with credits
- Core skills every test checksReading a guideline and applying it exactly, handling edge cases, staying consistent, and how platforms measure quality.7 conceptsOpens with credits
- Annotation types and toolsThe task types annotation projects use, read as a table: what each one is, where it is used, and the tools to practise in.10 conceptsOpens with credits
- Getting acceptedHow qualification works, how to check whether a platform can accept and pay you, and how to present yourself honestly.6 conceptsOpens with credits
- Platform comparisonThe main platforms as their own pages describe them in October 2026. No pay rates: they vary by project and location and change often. Always read the official FAQ before applying.7 conceptsOpens with credits
- Track record and getting paidQuality scores, payment methods by country, records, taxes and registration in general terms, and confidentiality.8 conceptsOpens with credits
- Moving upFrom tasker to reviewer, lead or specialist, and from data work into evals, data analytics, machine learning or AI engineering.8 conceptsOpens with credits
- 10Glossary, A to ZEvery concept in the sheet and the smaller terms it uses, in one list.120 terms
About this sheet
Annotation, model rating and AI trainer roles.
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allBest practices
See all- Best Practices for Building LLM AppsHow to build LLM apps and agents that hold up in production: prompts, evals, prompt injection, tool permissions, cost, logging and privacy.
- Prompt Engineering Best PracticesHow to write prompts that work the same way on the hundredth input as on the first: clear instructions, structure, examples, reasoning, output control and testing.
- Best Practices for AI Coding AssistantsHow to get reliable code from AI coding assistants and agents: instruction files, task prompts, verification, review, security, dependencies and session hygiene.
Checklists
See all- RAG Production ChecklistA RAG production checklist: chunking, embeddings, hybrid search, reranking, per-user permissions, citations, evals, re-indexing and cost, with how to verify each check.
- LLM Evaluation ChecklistAn LLM evaluation checklist: success criteria, eval datasets, code and model graders, judge calibration, agent evals, safety tests and production monitoring, with how to verify each check.
- AI Feature Launch ChecklistAn AI feature launch checklist: use case fit, evals, UX for errors, disclosure, human override, abuse filters, rate limits, cost caps, prompt privacy, monitoring and rollback.
Project structures
See all- GenAI RAG app (FastAPI) project structureFor developers turning a "chat with your documents" demo into an app people can rely on. A FastAPI backend where every layer has a home: ingestion workers, retrieval, prompts as versioned files, model clients with fallbacks and cost caps, guards, evals that gate CI, and tracing, so a wrong answer can be traced to the step that caused it.
- AI agent with an MCP server project structureFor developers who want to give AI assistants safe access to their own data or API. A TypeScript MCP server built on the official SDK, organised by feature (tasks, projects) with each feature owning its tools, resources and prompts, served over stdio for local hosts and Streamable HTTP with OAuth for remote ones, plus a small agent and the evals that prove a model can use it.
- Machine learning project project structureFor students and analysts whose model works in a notebook and now needs to be reproducible, reviewable, served and watched. It starts from the Cookiecutter Data Science v2 layout (immutable raw data, numbered notebooks, a real Python package) and grows into versioned data with DVC, typed parameters, data validation with pandera, experiment tracking and a model registry with MLflow, a FastAPI prediction service, drift monitoring and, for a team, scheduled retraining, model cards, fairness checks and shadow deployment.
Career questions
See all- Which freelance skills is AI replacing, and what should I switch to?AI is squeezing basic, repetitive freelance work first: plain data entry, simple transcription and translation, generic articles, basic logos and simple product descriptions. Work that combines a skill with judgement, client context or systems is holding up better, including automation, AI integration, analytics, specialist writing and full projects that own a business outcome.
- Which AI tools do employers expect junior developers to know in 2026?Employers rarely require a specific brand. They expect you to use an AI coding assistant well (in the editor and in the terminal), to review and test what it produces, to call an LLM API in a small feature, and to explain when you would not use AI. Pick one assistant, learn it properly, and show judgement in your projects.
- How do I use AI to become a better engineer instead of becoming dependent on it?Decide the job AI does before you open it. Use it as a tutor, reviewer and sparring partner that explains and challenges, not a vending machine that hands you finished code. Attempt first, ask second, and always be able to explain and defend what you ship.
Work with me
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com