Llm usage normalizer

LLM usage normalizer

Every provider counts tokens a little differently. The big trap: OpenAI-style exports count cached tokens inside input_tokens, while Anthropic reports uncached input separately. Add them up naively and you double count. This kit converts everything to one schema where input_tokens always means uncached input, then prices it from a table you control.

Output schema

timestamp,customer,task,run_id,provider,model,input_tokens,output_tokens,cache_read_tokens,cache_write_tokens,cost_usd,outcome

Expected input columns

The script finds columns by name (first match wins). Rename columns that differ, or pass --source.

Also read when present: customer, task (or workflow, feature, agent), run_id (or trace_id, session_id), outcome (or status), batch or service_tier (batch rows get --batch-discount, default 0.5), and cost_usd (kept as is).

Export formats change. If a column is missing or named differently, show the user the header and map it by hand rather than guessing.

Inputs

1. One or more exports in data/.
2. Price table data/llm-prices.csv. If missing, copy examples/prices.csv there. Columns: provider,model,input_per_mtok,output_per_mtok,cache_read_per_mtok,cache_write_per_mtok,checked_on,notes, plus optional cache_write_1h_per_mtok (defaults to 2x input). Model names match exactly or by prefix, so gpt-5-mini prices gpt-5-mini-2025-08-07.
3. Customer map data/customer-map.csv (key,customer,task) when exports only know API keys, projects, or workspaces. One key per customer is the cleanest setup; suggest it if they share keys.

Steps

1. Check prices first. Tell the user plainly: the example prices are a starting point and must be checked against each provider's current pricing page. Ask them to confirm or update data/llm-prices.csv and set checked_on. The script warns when prices were never checked or are older than 60 days.

2. Run:

   node scripts/normalize.mjs data/anthropic-usage.csv data/openai-usage.csv --prices data/llm-prices.csv --customer-map data/customer-map.csv --out out/usage-normalized

3. Read out/usage-normalized-report.md. Fix, then rerun:

   • Keys with no customer: ask who each key belongs to and add it to data/customer-map.csv (internal, staging, and eval keys map to internal).
   • Models with no price: add a row to the price table after the user confirms the price.
4. Hand off out/usage-normalized.csv to the next kit (agent-cost-dashboard, gross-margin-by-customer, usage-pricing-simulator, consumption-guardrails).

Limits to explain

• Provider reports are usually daily totals per model and key, not single calls. They have no run_id or outcome, so cost per successful outcome needs your app's own logs.
• If the provider bill and this total differ by more than a few percent, look for: wrong prices, batch or priority tiers, long-context pricing, tool or search fees, credits, and taxes. The script only prices tokens.

Guardrails

• Never put API keys or secrets in any file. Key IDs and names from usage reports are fine; key values are not. If an export contains full key values, stop and ask the user to remove that column.
• This kit reads files only. It never calls a provider API.
• Prices are the user's responsibility to confirm. Always say so next to any cost total.

LLM usage normalizer

Converts Anthropic and OpenAI usage exports, or your own app's token logs, into one usage CSV with customer, task, cached and uncached tokens, and cost. It handles the cached-token difference between providers, maps API keys or projects to customers, and tells you what it could not map or price.

Who it is for: engineers and finance folks at AI products who need one clean usage file before they can talk about cost per customer or margins.

Example

Input: examples/anthropic-usage.csv (daily rows per model and key), examples/openai-usage.csv (daily rows per model and project), examples/customer-map.csv, examples/prices.csv.

Output (out/usage-normalized-report.md):

| examples/anthropic-usage.csv | anthropic | 8 | $7.65 | api_key_id | uncached_input_tokens |
| examples/openai-usage.csv    | openai    | 4 | $3.93 | api_key_id | input_tokens          |
Total $11.59 across 12 rows.
Keys with no customer: key-staging (1 row)

And out/usage-normalized.csv in the shared schema used by every Agent Costs kit.

What it can access

• Runs node scripts/normalize.mjs (Node built-ins only).
• Reads exports in data/. Writes data/llm-prices.csv, data/customer-map.csv, and out/usage-normalized*.
• No network, no API keys. It never calls a provider.

Setup

Copy examples/prices.csv to data/llm-prices.csv and check every price against the provider's current pricing page (the example prices are a starting point, not a quote). Then export usage from each provider's console into data/.