Multi-Step Agent Workflow Runner with Cost Tracking

The problem with chained model calls

Chaining several language model calls is easy to start and very hard to debug. A support-ticket triage flow might classify, retrieve, draft, check tone and summarise. When the final answer is wrong, the question is which of five steps went wrong, and a lot of tooling cannot answer it cleanly. Cost is similarly invisible until the monthly bill arrives, and a crashed worker can re-run steps that were already paid for.

Teams at startups and software houses building these features need two things: a runtime they can reason about, and a record of every step.

What already exists and the gap

• LangGraph models agent workflows as graphs, with checkpointing, and LangSmith provides tracing for them.
• Langfuse is an open-source tracing and observability platform for model applications.
• Temporal provides durable execution for long-running workflows in general, and is increasingly used for AI pipelines.
• n8n offers visual workflows with model steps for lower-code users.

The pieces exist, but usually in separate products: one for execution, another for tracing, another for cost. The gap this project fills is a small, readable runner where execution, durable state, tracing and cost attribution are one design, plus a measured answer to a question teams keep guessing about: when is a decomposed workflow worth its extra cost over one good prompt?

What you build

• Definition format: steps, inputs, outputs, dependencies and retry policy.
• Engine: a scheduler that runs ready steps, respects dependencies, retries failures and records state after each step.
• Workers: pull jobs from a queue, call models and tools, report results.
• Trace store and viewer: timeline, per-step cost, full prompt and response on click.
• Experiments: the same task as one prompt and as a workflow.

Why explicit beats autonomous here

An agent that decides its own steps is harder to debug, more expensive and less predictable. A defined graph with model calls at the nodes covers a large share of real business use cases and can actually be reasoned about. Make that argument in your report and back it with numbers.

The technically interesting part

Two hard problems.

Exactly-once effects on an at-least-once system. Workers crash, retries happen. A step that already succeeded must not be paid for or executed again on resume. Record step completion transactionally with its output, and make tool calls idempotent with step-level keys, the pattern durable execution systems like Temporal are built around.

Trace storage. Prompts and responses are large, runs are frequent, and a naive schema becomes unusable quickly. Keep a compact index (run, step, status, tokens, cost, timing) separate from payload storage, and measure query speed at scale.

How to evaluate it

• Decomposition study: for both workflows, accuracy on a labelled set of at least 50 inputs, cost per run and p95 latency, workflow versus single prompt. A concrete result such as "decomposition was more accurate at three times the cost" is exactly what your report should lead with.
• Crash test: kill workers mid-run 20 times; count repeated paid calls (target zero) and runs that finish correctly.
• Overhead: engine latency added per step.
• Trace performance: viewer load time and query time after 10,000 runs.
• Developer test: give three developers a broken workflow and time how long they take to find the failing step with your viewer.

Suggested plan

1. Weeks 1 to 4: definition format, simple sequential engine, trace schema.
2. Weeks 5 to 9: dependencies, parallel branches, retries, timeouts.
3. Weeks 10 to 14: durable state and crash recovery, measured.
4. Semester 2, weeks 1 to 5: trace viewer, cost attribution, storage at scale.
5. Weeks 6 to 10: the two workflows and the decomposition study.
6. Weeks 11 to 14: developer test, report and release.

Path to a product

The realistic path is open source first. A small, well-documented runner with good traces attracts developers who find larger frameworks heavy; publishing the decomposition study brings attention. Monetise later with a hosted trace store and team features, or use it as a portfolio centrepiece for AI platform and backend infrastructure roles, where this exact set of problems (queues, retries, idempotency, observability) is daily work.

Scope warning

• Do not write another framework abstraction layer with dozens of integrations.
• Do not build autonomous tool-choosing agents; explicit graphs only.
• Do not build user accounts, teams or billing.
• Execution, durability, tracing and cost accounting is already a full project.

Learn what you need on Sefism

• AI engineering roadmap : model calls, tools and evaluation in workflow form.
• System design roadmap : queues, retries and idempotency behind a reliable engine.
• roadmap.sh: AI Agents : the agent concepts you are deliberately making explicit.
• Temporal Documentation : how durable execution handles crashes and retries.
• Backend cheat sheet : APIs, queues and database patterns for the engine.

Get help with this idea

The risk is spending the year on framework features instead of durability, tracing and the decomposition study. Tauseef can help you draw the line, design the crash tests and plan an experiment whose result is worth publishing. If you are new here, create a free account first.

• Discuss this idea with a mentor
• Send a query to Tauseef
• Book a 1:1 session