Decide what to measure and build a metrics dashboard

Product analytics

Goal: a small set of honest numbers that tell you whether the product works, computed from data you already own, with no prompt, file, or form content leaving your servers. Instrument last, not first.

This kit covers product numbers. For consent, use @slashclone/cookie-consent-banner. For traffic, attribution, and search, use @slashclone/website-traffic. Do not rebuild either here.

Steps

1. Read before you ask. Read the roadmap or plan docs, the database schema or migrations, and any existing admin pages. List the tables that already record what people do: sign-ups, installs or projects created, publishes, invites, searches, comments. Ask the person what stage the product is in and what decision the numbers should inform. Do not guess.

2. Pick the numbers per stage. Three to six per stage, no more. Each one should change a decision. Typical picks:

   • Early (does anyone get value?): activation (did a new account do the core action), time to first value, kept after 7 days.
   • Creator side (do people make things?): creators publishing per week, items per creator, time from sign-up to first publish.
   • Loop (does the product feed itself?): share of new content that came from a given loop, time from "work done" to "published", remixes or derived items, comments per item.
   • Funnels: sign-up to first core action, sign-up to first publish, connect an integration to first use. One funnel per question.
3. Write each definition. Copy templates/METRICS.md to docs/METRICS.md and fill one block per metric: what it answers, the exact counting rule, the source table, who is excluded, the time window, and the decision it informs. A number with no written rule will be counted two ways by next month.
4. Exclude yourselves. Every people count leaves out house, staff, test, and seed accounts. Keep that list in one place (a flag on the account, or one shared filter) and apply it in every query. Also leave out an owner's own actions on their own items (installing or starring your own item is not demand).
5. One function per metric. Put each metric in one shared server function that takes a time range and returns the number. The dashboard, scheduled jobs, emails, and public stats all call it, so they always agree. Test each one with a small fixture that includes an excluded account.
6. Define "kept" from real usage. Retention needs a signal that the thing is still in use, not just that it exists. Pick one the product already sends: a daily ping from a client, an update check, a sign-in, an API call with the item. Each refreshes a last seen time on the row. Then: kept after N days means last seen >= created + N days. Write down which signals refresh it. If no signal exists, say so; do not report retention you cannot see.
7. Build the staff-only dashboard. One internal page, behind a staff check on the server (not only hidden in the menu). Show each metric with its definition one click away, a 30-day series, the funnels as step counts with drop-off, and a small table for the top items and zero-result searches. Compute on read for small data; move to a scheduled rollup table when queries get slow. Cache briefly. Never show personal data a staff member does not need.

8. Add a structured audit log. One table and one write function for security and product events. One schema:

   • kind (a dotted name like account.signed_in, from a fixed list),
   • actor (account id or null),
   • subject (what it acted on, by id),
   • ip_hash (an HMAC of the IP keyed with a server secret, never the raw IP),
   • data (small metadata only). The write function drops keys that look like secrets or content (token, password, body, prompt, content, file) and truncates long strings. Never write prompt text, file content, form input, or emails into it. Use it for events no other table records, like failed sign-ins and token changes.

9. Client analytics, only after consent. If the team wants a product analytics tool in the browser (for example an open source self-hosted one or a hosted product analytics service):

   • Load it with a dynamic import() from the consent callback, never at page load.
   • Pin the exact package version.
   • Read its project id from an env var; when unset, the loader does nothing. Same code in dev and production.
   • Send page URLs as origin plus path, with no query string or hash.
   • Turn off autocapture, session replay, and surveys unless the team has decided it needs them and masked every input.
   • Identify by internal id only, never email or name. A cookieless page traffic tool may not need consent; follow the consent kit for that decision.
10. Error reporting, scrubbed. If you add an error reporting service, keep it off when its DSN env var is unset. In the beforeSend hook, remove request bodies, cookies, headers, query strings, and user fields other than an id. Turn tracing and replay off unless you have a reason. Add a test that a scrubbed event has none of those fields.
11. Verify. Seed a few accounts, including an excluded one, run the dashboard, and check each number by hand against a direct query. Confirm that with no env ids set, no analytics or error script loads and no request leaves the browser. Write in the project's progress log what you measured, what you could not, and why.

Rules

• Decide the numbers before adding any event or tool.
• Prefer existing tables over new events. Add an event only for something no table records.
• Never store or send prompt content, file content, form input, secrets, or raw IPs.
• Every people count excludes house, staff, and test accounts.
• One shared function per metric. Pages and jobs never re-implement a count.
• Client analytics loads only after consent, and is off when its id is unset.
• Ask before adding any dependency. Check the current stable version, and prefer one released at least two weeks ago.
• Do not report a number whose signal you do not collect. Say it is missing instead.