Policy-Bound Returns and Refunds Agent for Small Online Stores
A final year project that builds an LLM agent for small online stores which handles return, exchange and refund requests through real store tools, while a separate policy layer blocks any action the store policy does not allow. You test it with simulated customers and measure policy violations, not just completed chats.
Intermediate
A small team, or one strong student willing to learn something new.2 semesters · 3 students
6 to ship
3 optional extrasSuggested stack
What you should ship
- A written store policy (return windows, conditions, exchange rules, refund methods, limits) encoded as machine-checkable rules separate from the prompt.
- An agent with typed tools: look up order, check eligibility, create return request, issue exchange, issue refund up to a limit, and hand off to a human.
- A policy guard that validates every tool call against the rules before execution and logs blocked calls with the rule that blocked them.
- A simulated customer suite of at least 100 scripted scenarios, including customers who push for exceptions, give false dates or try prompt injection.
- An evaluation runner that checks the final database state against the expected state, repeats each scenario several times and reports reliability across runs.
- A staff console showing each conversation, every tool call, blocked actions and handoffs, with a one-click approve or reject for escalations.
If you have time left
- Support Roman Urdu conversations and measure whether policy compliance drops.
- Compare two or three models, including a small open model, on the same suite.
- Connect to a real partner store in read-only mode for a week and measure how many real requests the agent could have handled.
The problem small stores have with returns
For a small online clothing or electronics store, returns and exchanges are the most time-consuming support work. Every request needs the same checks: is the order within the return window, is the item eligible, has it been delivered, was it paid by card or cash on delivery, is this customer asking for the third refund this month? Staff answer these by switching between the store admin, the courier portal and WhatsApp. Mistakes cost money directly: a refund issued outside the policy is rarely recovered.
AI agents promise to handle this, but a store owner's first worry is simple: what stops the bot from refunding someone it should not?
What already exists and the gap
- Gorgias AI Agent resolves returns, cancellations and order status for Shopify stores, billed per automated resolution, with each automated resolution also counting as a helpdesk ticket.
- Intercom Fin charges per resolved conversation in US dollars, with a monthly minimum.
- Sierra's tau-bench research benchmark tests agents on retail and airline tasks with policy documents; its authors reported that even the best agent tested succeeded on fewer than half the tasks, and introduced a pass^k measure of consistency across repeated attempts.
The gap for a small store is twofold: per-resolution pricing that does not suit low-value orders, and policy compliance that rests on a prompt rather than on enforced rules.
What you build
An agent, a policy guard and a test harness. The agent talks to the customer and calls typed tools that wrap a Shopify development store (or a mock store API with the same shape). The policy guard sits between the agent and the tools: every call is checked against rules written in code or a small rules file ("refund only if delivered and within 14 days", "cash on delivery orders refund by bank transfer only", "refunds above a set amount need a human"). If a call breaks a rule, it is blocked and the agent is told why, so it can explain or hand off. The harness runs simulated customers, including difficult ones, and compares the store's final state with the expected state.
The design follows a pattern well known in payments engineering: the component that decides is not the component that is allowed to move money, and every money movement is checked and logged.
The technically interesting part
Measuring reliability, not a demo. One successful chat proves nothing. Build a scenario suite with known correct outcomes, run each scenario several times, and report how often the agent gets it right every time, following tau-bench's pass^k idea. Then run the suite with and without the policy guard. You should show two things clearly: the guard drives policy violations close to zero, and the agent still resolves most legitimate requests rather than handing everything off. Include adversarial scenarios: a customer claiming a false delivery date, a prompt injection hidden in an order note, a request to "just refund, the manager said it's fine".
How to evaluate it
- Task success rate and pass^k over at least 100 scenarios, with and without the guard.
- Policy violation count with and without the guard, broken down by rule.
- Handoff rate for legitimate requests (lower is better, as long as violations stay at zero).
- Average tokens and cost per resolved request for each model tried.
- A review by one real store owner or support lead of 30 transcripts, rating whether each outcome was acceptable.
Suggested plan
- Month 1: write the policy with a store owner, set up the dev store and seed orders.
- Month 2: tools and the policy guard, with unit tests for every rule.
- Month 3: agent loop, handoff flow and staff console.
- Month 4: scenario suite and evaluation runner.
- Month 5: experiments across models and guard on or off.
- Month 6: transcript review, write-up.
Path to a product
Start as a Shopify app for small stores with a flat monthly price and a cap on conversations, which is easier to budget than per-resolution billing. The policy editor becomes a selling point: owners set their own rules in plain forms and see blocked actions in the console. Local marketplaces and WooCommerce stores are the next integration.
Scope warning
Do not build a general customer service bot for every question; returns, exchanges and refunds only. Do not connect to real payment movements during the project; use the dev store's test mode. Do not skip the harness to spend time on a chat widget design.
Learn what you need on Sefism
- AI Engineering roadmap : tool calling, agent loops and evaluation for LLM systems.
- AI Engineering cheat sheet : quick reference for guardrail and evaluation patterns.
- roadmap.sh: AI Agents : a map of agent concepts, tools and memory.
- Shopify Engineering : how a large commerce platform thinks about reliability and APIs.
- Which AI topics are worth learning? : where agents fit among RAG, MCP and LLMs.
Get help with this idea
The quality of your scenario suite decides your grade more than the model you pick. A mentor can review your policy rules and adversarial scenarios early. Bring your draft scope, the dataset you plan to use and your evaluation table, and we can look at them together. The build is yours; the guidance is on scope, stack and how to measure it. If you are new here, create a free account first.
Ideas and guidance, not finished projects
These are project ideas and scoping guidance, published free for students to use as a starting point. I do not build, write, or sell final-year projects, and I do not complete coursework for anyone. Take an idea, make it yours, and build it.
Not sure this one fits you?
I help students pick an idea that matches their skills and their deadline, then scope it down until it is finishable. Guidance only, never done for you.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
Follow along
New writing, resources and project ideas land here first.