A practical study guide for AI Product Managers covering AI quality metrics, product metrics, and business metrics—with formulas and examples.
1. AI Metrics
AI metrics measure whether an AI system is accurate, reliable, fast, cost-efficient, and stable in production.
1.1 Accuracy
Accuracy measures how often the model produces the correct answer.
Formula:
Accuracy = Correct Predictions / Total Predictions × 100
Example: If an intent-classification model correctly classifies 920 out of 1,000 requests:
Accuracy = 920 / 1,000 × 100 = 92%
PM takeaway: Accuracy is useful for classification and prediction tasks, but it is not sufficient for generative AI, where answers can be partially correct, well-written, but factually wrong.
1.2 Hallucination Rate
A hallucination occurs when an AI model generates information that is unsupported, fabricated, or factually incorrect.
Formula:
Hallucination Rate = Hallucinated Responses / Evaluated Responses × 100
For a customer-support chatbot, if 40 of 1,000 evaluated responses contain unsupported claims:
Hallucination Rate = 40 / 1,000 × 100 = 4%
PM takeaway: Track hallucination by use case. A 4% rate might be unacceptable for medical or financial advice but tolerable for a low-risk brainstorming assistant.
1.3 Invalid Response Rate
Invalid response rate measures how often the model produces an output that fails the required format, schema, business rule, or task constraint.
Formula:
Invalid Response Rate = Invalid Outputs / Total Outputs × 100
Example: If an LLM must return valid JSON and 25 of 500 responses fail parsing:
Invalid Response Rate = 25 / 500 × 100 = 5%
PM takeaway: This is particularly important when AI output is consumed by software, APIs, workflows, or agents.
1.4 Latency
Latency is the time between sending a request and receiving an AI response.
Common measures include:
- Time to First Token (TTFT): how quickly generation begins.
- Time to Last Token (TTLT): how long the complete response takes.
- p50 latency: median response time.
- p95/p99 latency: tail latency experienced by slower requests.
PM takeaway: A highly accurate AI feature can still fail commercially if users perceive it as too slow.
1.5 Token Usage
Tokens are the units processed by an LLM. Token usage affects cost, latency, context-window consumption, and scalability.
Track:
- Input tokens
- Output tokens
- Total tokens
- Tokens per request
- Cost per request
- Cost per active user/task
A PM should optimize prompts, retrieved context, conversation history, and output length when token usage becomes a significant cost driver.
1.6 Model Drift
Model drift occurs when an AI system’s performance changes because the environment, data, user behavior, or underlying relationships have changed.
Data Drift
The distribution of incoming data changes.
Example: A fraud model trained on historical transaction patterns encounters a new payment behavior.
Concept Drift
The relationship between inputs and the target outcome changes.
Example: A recommendation model learns from old customer preferences, but customer preferences change substantially over time.
PM takeaway: Monitor drift continuously and define thresholds that trigger investigation, retraining, prompt changes, model replacement, or rollback.
2. Product Metrics
AI quality does not automatically mean product success. Product metrics tell us whether users actually adopt, use, and retain the product.
2.1 North Star Metric (NSM)
The North Star Metric represents the core value a product consistently delivers to customers.
Example: For a recruitment platform, a possible NSM could be successful candidate-job matches per month.
A strong NSM should connect user value with business growth.
PM takeaway: Avoid vanity metrics such as total registrations if they do not represent meaningful customer value.
2.2 DAU and MAU
- DAU: Daily Active Users
- MAU: Monthly Active Users
A common engagement indicator is:
DAU/MAU Ratio = DAU / MAU × 100
Example: 20,000 DAU and 100,000 MAU gives:
20,000 / 100,000 × 100 = 20%
A higher ratio generally indicates more frequent usage, although the appropriate benchmark depends heavily on the product category.
2.3 Engagement
Engagement measures the depth and frequency of meaningful interaction with a product.
Possible measures include:
- Sessions per user
- Session duration
- Key actions per session
- Features used
- Tasks completed
- AI interactions per active user
PM takeaway: Define engagement around the product’s value event rather than raw clicks or screen views.
2.4 Bounce Rate
Bounce rate measures the proportion of visits where users leave without taking the desired next action.
For a landing page:
Bounce Rate = Single-Engagement Sessions / Total Sessions × 100
Example: If 300 of 1,000 visits leave without meaningful interaction:
Bounce Rate = 30%
Interpret bounce rate in context. A high bounce rate is not necessarily bad for a page where the user’s goal can be completed immediately.
2.5 Activation Rate
Activation rate measures how many new users reach a predefined aha moment or meaningful first-value event.
Activation Rate = Activated New Users / New Users × 100
Example: If 400 of 1,000 new users complete the key setup and first successful task:
Activation Rate = 40%
For AI products, activation might mean completing the first successful AI-generated workflow rather than simply creating an account.
2.6 Retention Rate
Retention rate measures how many users continue using the product after a defined period.
A simple cohort formula is:
Retention Rate = Users Active at End of Period / Users in Original Cohort × 100
Example: 1,000 users sign up in January and 350 return in February:
Retention = 35%
Retention is usually more meaningful than acquisition volume for evaluating product-market fit.
2.7 Churn Rate
Churn rate measures the percentage of customers or users lost during a period.
For customers:
Customer Churn Rate = Customers Lost During Period / Customers at Start of Period × 100
Example: Starting with 2,000 customers and losing 80:
Churn Rate = 80 / 2,000 × 100 = 4%
For subscription businesses, reducing churn can have a major impact on LTV and revenue.
2.8 NPS
Net Promoter Score (NPS) measures customer willingness to recommend a product.
Customers answer a 0–10 question.
- Promoters: 9–10
- Passives: 7–8
- Detractors: 0–6
NPS = % Promoters − % Detractors
The result ranges from −100 to +100.
Example: 55% promoters and 15% detractors:
NPS = 55 − 15 = +40
3. Business Metrics
Business metrics connect product performance to revenue, acquisition economics, customer value, and profitability.
3.1 Revenue
Revenue is the money generated by selling products or services during a period.
For a simple subscription business:
Revenue = Number of Paying Customers × Average Revenue per Customer
Revenue is a top-line metric; it does not account for costs.
3.2 MRR and ARR
MRR (Monthly Recurring Revenue) measures predictable recurring revenue in a month.
MRR = Sum of Monthly Recurring Revenue from Active Subscriptions
ARR (Annual Recurring Revenue) annualizes recurring revenue:
ARR = MRR × 12
These are especially important for SaaS and subscription products.
3.3 ARPU
Average Revenue Per User (ARPU) measures average revenue generated per user during a period.
ARPU = Total Revenue / Average Number of Users
Example: ₹10 lakh revenue from 20,000 users:
ARPU = ₹50 per user
ARPU helps PMs evaluate monetization and pricing changes.
3.4 CAC
Customer Acquisition Cost (CAC) measures the average cost required to acquire one new customer.
CAC = Total Sales & Marketing Cost / Number of New Customers Acquired
Example: Spending ₹5 lakh to acquire 500 customers:
CAC = ₹1,000
When evaluating growth, compare CAC with customer value rather than looking at acquisition volume alone.
3.5 LTV
Customer Lifetime Value (LTV) estimates the revenue or contribution a customer generates over their relationship with the business.
A simplified subscription formula is:
LTV ≈ ARPU × Gross Margin % / Customer Churn Rate
Example: If monthly ARPU is ₹1,000, gross margin is 70%, and monthly churn is 5%:
LTV ≈ ₹1,000 × 70% / 5% = ₹14,000
LTV should be defined consistently—revenue LTV and contribution-margin LTV are not the same.
3.6 Profit Margin
Profit margin measures how much of revenue remains as profit after relevant costs.
Profit Margin = Profit / Revenue × 100
For net profit margin:
Net Profit Margin = Net Profit / Revenue × 100
Example: If a product generates ₹10 lakh revenue and ₹1.5 lakh net profit:
Net Profit Margin = 1.5 / 10 × 100 = 15%
PM takeaway: Growth without healthy unit economics can create an unprofitable product. When evaluating an AI feature, include inference/API costs, infrastructure, support, acquisition costs, and other variable costs in the economic model.
4. How AI Product Managers Should Connect the Metrics
A strong AI product dashboard should connect three layers:
AI quality → Product behavior → Business outcome
For example:
Lower Hallucination Rate
→ Higher Task Completion
→ Higher Retention
→ Lower Churn
→ Higher LTV
→ Higher Profit Margin
This prevents the team from optimizing an AI model in isolation.
Example: AI Customer-Support Copilot
| Layer | Metric | Example Target |
|---|---|---|
| AI | Accuracy | >95% |
| AI | Hallucination Rate | <2% |
| AI | Invalid Response Rate | <1% |
| AI | p95 Latency | <3 sec |
| AI | Cost / Request | <₹2 |
| Product | Activation Rate | >45% |
| Product | DAU/MAU | >25% |
| Product | Task Completion | >80% |
| Product | Retention | >40% |
| Business | CAC | ₹1,500 |
| Business | LTV | ₹12,000 |
| Business | Profit Margin | >15% |
The exact targets should be based on the product, user expectations, risk level, competitive context, and business model—not generic benchmarks.
5. AI PM Interview Framework
When asked “Which metrics would you track for an AI product?”, structure the answer in this order:
- AI quality: accuracy, hallucination, invalid outputs, latency, token usage, drift.
- User behavior: activation, engagement, DAU/MAU, task completion, retention, churn.
- Business: revenue, MRR/ARR, ARPU, CAC, LTV, profit margin.
- Guardrails: safety, privacy, bias/fairness, failure rate, escalation rate.
- Trade-offs: quality vs. latency, quality vs. cost, automation vs. human review.
The key is not to list metrics. Explain why each metric matters and how one layer affects the next.
Quick Revision Cheat Sheet
| Metric | What it tells you |
|---|---|
| Accuracy | How often predictions are correct |
| Hallucination Rate | How often AI generates unsupported information |
| Invalid Response Rate | How often output violates required format/rules |
| Latency | How fast the AI responds |
| Token Usage | How much LLM processing is consumed |
| Data Drift | Whether input data distribution has changed |
| Concept Drift | Whether the input-output relationship has changed |
| North Star Metric | Whether the product consistently delivers core value |
| DAU/MAU | Frequency of product usage |
| Engagement | Depth and frequency of meaningful usage |
| Bounce Rate | How often users leave without meaningful interaction |
| Activation | Whether users reach first meaningful value |
| Retention | Whether users continue using the product |
| Churn | How many customers/users are lost |
| NPS | Customer willingness to recommend |
| Revenue | Top-line money generated |
| MRR/ARR | Recurring revenue |
| ARPU | Average revenue per user |
| CAC | Cost to acquire a customer |
| LTV | Estimated customer value over the relationship |
| Profit Margin | Percentage of revenue retained as profit |