Product Metrics That Matter (And the Ones That Lie)

I've built a lot of dashboards. At ARTime, we tracked dozens of metrics — daily active users, session length, page views, bounce rate, feature adoption, NPS, the list goes on. Most of them were useless. Not because the numbers were wrong, but because they didn't drive decisions. They were noise dressed up as signal. Here's the framework I use now to separate metrics that matter from metrics that just look good in a deck. A metric matters if it changes a decision. That's the only test. If you look at the metric and your next action doesn't change — you'd make the same decision whether it's up, down, or flat — it's a vanity metric. It exists to make people feel good, not to guide the product. This eliminates most metrics immediately.

• Page views: Up 30%? Great. What do you do differently? Usually nothing. Vanity.
• Total registered users: Includes everyone who signed up in 2019 and never returned. Vanity.
• Session length: Long sessions could mean engagement or could mean users are confused and can't find what they need. Ambiguous, therefore not actionable.
• App downloads: Doesn't tell you if anyone actually uses the app. Vanity.

The metrics that survive this test are fewer, less impressive-sounding, and far more useful. What percentage of new users reach the "aha moment" — the point where they've experienced the core value of the product? For Menyo Pro, the activation event is specific: a restaurant owner publishes a live menu within their first session. Not signs up. Not starts setup. Publishes a menu that guests can actually scan and order from.

// The activation funnel I track
const activationFunnel = {
  signUp: 100,          // 100 users create an account
  startMenuSetup: 72,   // 72 begin the menu scanner
  completeScan: 58,     // 58 get a successful extraction
  editMenu: 51,         // 51 make at least one edit
  publishMenu: 44,      // 44 publish — THIS is activation
};
// Activation rate: 44%

Why this metric matters: if activation drops, I know exactly where in the funnel to look. If it drops from 44% to 38%, I check which step lost the most users and fix that step. It's not a dashboard I glance at — it's a diagnostic tool. Not "retention rate" (a single number). The full curve: what percentage of users come back on day 1, day 7, day 30, day 90? The shape of the curve tells you more than any single number:

Day 1:  62%  ████████████████████████████▌
Day 7:  41%  ███████████████████▏
Day 30: 34%  ███████████████▊
Day 60: 31%  ██████████████▎
Day 90: 29%  █████████████▍

A healthy retention curve flattens out. It drops initially (people who tried once and left) and then stabilizes at a plateau. That plateau is your real user base. A retention curve that keeps declining — even slowly — means you have a leaky bucket. You can pour all the new users you want in the top; they'll drain out the bottom. Fix retention before growth, always. For multi-tenant SaaS, the unit economics per tenant tell you if the business works. This isn't just MRR divided by tenant count (though that's the starting point). It's the full picture:

interface TenantEconomics {
  mrr: number;                    // Monthly recurring revenue
  infrastructureCost: number;     // Database, compute, AI API costs attributable to this tenant
  supportTime: number;            // Estimated cost of support hours spent on this tenant
  netRevenue: number;             // mrr - costs
  paybackPeriod: number;          // How many months until CAC is recovered
}

Some tenants are profitable. Some cost more to serve than they pay. This isn't a failure — it's information. It tells you which customer segments to target and which to deprioritize. How long does it take for a new user to go from signup to experiencing value? This is the most under-tracked metric in product, and it's the one that predicts retention better than anything else. For Menyo Pro: the target is under 3 minutes from photo upload to live menu. Every second beyond that costs activation. I track the distribution, not the average, because the long tail (users who take 20 minutes) tells me where the friction is. I've seen teams build dashboards with 40 metrics on them. Nobody looks at them. They exist for the same reason a security blanket exists — to make people feel like something is being monitored. A useful dashboard has 3–5 metrics, all of which would trigger a specific action if they moved significantly. If a metric wouldn't change your behavior, it doesn't belong on the dashboard. Averages hide distributions. "Average session length is 8 minutes" is meaningless if the distribution is bimodal — 60% of sessions are 2 minutes (people who bounced) and 40% are 18 minutes (people who engaged). Always look at the distribution. Percentiles (P50, P90, P99) are more honest than averages. Histograms are more honest than percentiles. "At ARTime, we noticed that users who used the recommendation feature had 40% higher AOV." This is a true statement and a misleading one. Did the recommendations cause the higher AOV? Or did users who were already inclined to spend more also happen to use more features, including recommendations? You can't tell from correlation. You need a controlled experiment — a holdout group that doesn't see recommendations, compared to a group that does. That's how we got the real number: +18% AOV lift attributable to recommendations. The 40% correlation was mostly selection bias. Never make product decisions based on observational correlations. Run experiments. Use holdout groups. Be skeptical of your own data. The temptation is to instrument everything. Don't. Instrument the decision points.

1. Define your activation event. One specific action that represents experiencing core value.
2. Track the funnel to activation. Every step from signup to activation, with counts.
3. Track retention by cohort. Weekly cohorts, day-1 through day-90 retention.
4. Track revenue per tenant. At minimum, MRR and attributable costs.
5. Run experiments with holdout groups. For any feature you believe moves a metric, prove it.

That's it. Five things. You can build this with PostHog (what I use) or Mixpanel or a database and some SQL. The tool doesn't matter. The discipline does. The best metric I ever tracked was also the most embarrassing. At ARTime, I tracked "features shipped per sprint" as a productivity metric. It went up consistently. I felt productive. Then I tracked "features shipped per sprint that were used by more than 10% of users within 30 days." That number was 40%. Sixty percent of what I shipped was used by almost nobody. That metric changed everything. It stopped me from shipping features that felt important and started me shipping features that were important. It turned the roadmap from a wish list into a hypothesis-testing machine. The best metric is the one that makes you uncomfortable. If your metrics always tell you you're doing great, you're tracking the wrong things.