Consumption guardrails

Consumption guardrails

Agent products have a failure mode normal SaaS does not: one bad loop or one enthusiastic customer can burn a month of margin in an afternoon. Guardrails are simple numbers, checked often, with a plan for each alert. This kit sets the numbers from your own usage, checks them, and writes the runbook.

Inputs

1. Usage in the shared schema (timestamp, customer, task, run_id, model, tokens..., cost_usd, outcome), as fresh as possible. Use llm-usage-normalizer for provider exports.

2. Config data/guardrails.json (start from examples/guardrails.json):

   {
     "warn_at": 0.8,
     "defaults": { "daily_cap_usd": 6, "monthly_cap_usd": 100, "run_cost_cap_usd": 3, "max_calls_per_run": 25 },
     "customers": { "cobalt-labs": { "monthly_cap_usd": 60, "daily_cap_usd": 4 } },
     "anomaly": { "multiplier": 3, "baseline_days": 7, "min_usd": 1 },
     "failure_rate": { "max": 0.4, "min_runs": 5 }
   }

Steps

1. Set limits from data, not guesses. Look at 30 days of usage: each customer's normal and heaviest day, the p95 cost and call count per run for each task, and what each customer pays. Propose:

   • Monthly cap: the spend at which that customer's gross margin hits the floor the user accepts (revenue x (1 - target margin)), or their plan's included usage.
   • Daily cap: about 3x a normal heavy day.
   • Cost per run: 3x the p95 of the most expensive task. Calls per run: 2x the p95 calls. Show the proposal as a table and write data/guardrails.json once the user agrees.

2. Check:

   node scripts/guardrails.mjs data/usage.csv --config data/guardrails.json --prices data/llm-prices.csv --out out/guardrails

   --as-of 2026-10-27 checks a past day (useful to test the rules on a known incident). --fail-on-alert exits with code 2 on any critical alert, for a scheduled job or CI step.

3. Report from out/guardrails.md: critical alerts first, with the customer, the rule, and the numbers. For runaway runs, list run IDs so engineering can trace them.
4. Write the runbook out/guardrails-runbook.md from templates/runbook.md: fill in the limits table with the agreed numbers and why, who owns each alert, and the action for each. Ask the user what the product should do at a cap (notify only, throttle, queue, or pause non-urgent tasks) and who can raise a cap.
5. Make it routine. Suggest running the check on a schedule after usage is exported (daily is a good start; hourly for high-volume products) and sending the markdown to the team channel. Do not set up schedules or notifications yourself; give the user the command.
6. Tune. After a week, look at alerts that fired: if one rule fires daily with no action, its threshold is wrong. Change the config, not the code.

Rules the script applies

Guardrails for this kit

• The script only reports. It never throttles, blocks, or changes anything in the product or the provider account.
• Do not lower a paying customer's limits or recommend cutting them off without the account owner in the loop.
• Costs depend on the price table. Make sure prices were checked against each provider's current pricing.

Consumption guardrails

Sets spend caps, forecasts, spike rules, and runaway-run limits for each customer of your AI product, checks your usage against them, and writes a runbook for what to do when each alert fires. Exit code 2 on critical alerts, so it drops into a cron job or CI step.

Who it is for: founders and engineers running agent products who have been surprised by a bill, or want to make sure they never are.

Example

Input: examples/usage.csv (October usage, including a day when one customer's research agent looped), examples/guardrails.json.

node scripts/guardrails.mjs examples/usage.csv --config examples/guardrails.json --prices examples/prices.csv --as-of 2026-10-27

Output (out/guardrails.md):

Plus out/guardrails-runbook.md, written with you from templates/runbook.md.

What it can access

• Runs node scripts/guardrails.mjs (Node built-ins only).
• Reads data/. Writes data/guardrails.json (your limits) and out/guardrails*.
• No network. It reports; it never blocks or throttles anything.

Setup

None to try it. To run it daily, schedule the command after your usage export. Check the example LLM prices against each provider's current pricing first.