AI Product Manager Guide to LLM Cost Estimation: How to Calculate RAG Chatbot Costs
Building an AI product is not only about choosing the right model. For an AI Product Manager, one of the most important responsibilities is understanding whether the product can scale economically.
A RAG-based customer-support chatbot may look inexpensive when you consider the price of a single API call. But at thousands of queries per day, costs from LLM inference, embeddings, infrastructure, and human support can materially affect the business case.
This guide shows how an AI Product Manager can estimate these costs and evaluate whether an AI support agent can create measurable savings.
Why LLM Cost Estimation Matters for AI Product Managers
Traditional software products often have relatively predictable marginal costs. AI products are different because each interaction can consume tokens and trigger model inference.
Your cost can change based on:
- Number of user queries
- Input tokens per query
- Output tokens per query
- LLM input and output pricing
- RAG context retrieved for each request
- Embedding volume
- Number of documents indexed
- Frequency of document updates
- Human-agent costs
- Model selected for inference
Therefore, an AI PM should treat AI unit economics as a product metric, not simply an engineering concern.
A useful starting equation is:
Monthly AI Cost = Inference Cost + Embedding Cost + Infrastructure Cost + Other Variable AI Costs
For a simple RAG chatbot, inference and embedding costs are often the first two components to model.
Step 1: Estimate Your Monthly Query Volume
Start with demand.
Suppose your support chatbot receives:
- 5,000 queries per day
- 30 days per month
Then:
Monthly queries = 5,000 × 30 = 150,000 queries
This number becomes the foundation for your AI cost model.
An AI PM should also create scenarios rather than relying on one forecast:
| Scenario | Queries/Day | Queries/Month |
|---|---|---|
| Conservative | 2,500 | 75,000 |
| Base | 5,000 | 150,000 |
| High Growth | 10,000 | 300,000 |
This makes the business case more resilient if adoption differs from the original forecast.
Step 2: Calculate Tokens Per Request
Token consumption is usually divided into input tokens and output tokens.
Consider a chatbot request with:
- User query: 20 tokens
- System prompt: 250 tokens
- Retrieved PDFs/context: 1,800 tokens
- FAQs: 240 tokens
- Support tickets: 240 tokens
- Expected model output: 300 tokens
Total input:
20 + 250 + 1,800 + 240 + 240 = 2,550 input tokens
Total output:
300 output tokens
Therefore:
Total tokens per query = 2,550 + 300 = 2,850 tokens
At 150,000 monthly queries:
Monthly tokens = 2,850 × 150,000 = 427.5 million tokens
This calculation is important because seemingly small changes in retrieved context can significantly increase inference costs at scale.
Product insight
An AI PM should not optimize only for model quality. Prompt and context efficiency are also product economics levers.
Reducing unnecessary retrieved content from 1,800 tokens to 1,000 tokens, for example, can reduce input-token consumption across every request.
Step 3: Calculate LLM Inference Cost
LLM providers typically price input and output tokens separately.
The general formula is:
Input Cost = (Monthly Input Tokens ÷ 1,000,000) × Input Price
Output Cost = (Monthly Output Tokens ÷ 1,000,000) × Output Price
Monthly Inference Cost = Input Cost + Output Cost
Using the reference model’s base case for a model priced at:
- $4 per 1M input tokens
- $20 per 1M output tokens
Per query:
Input cost = 2,550 ÷ 1,000,000 × $4 = $0.0102
Output cost = 300 ÷ 1,000,000 × $20 = $0.006
Therefore:
Cost per query = $0.0162
At 150,000 queries per month:
Monthly inference cost = $2,430
This is the number an AI PM should take into the product P&L or unit-economics model.
Step 4: Estimate RAG Embedding Costs
Inference is only one part of a RAG product.
You also need embeddings for:
- Your knowledge base
- New user queries
Suppose the knowledge base contains:
- 10,000 support tickets × 150 words
- 50 FAQs × 200 words
- 20 PDFs × 5,000 words
Using an estimated 1.3 tokens per word:
Support tickets
1,500,000 words × 1.3 = 1,950,000 tokens
FAQs
10,000 words × 1.3 = 13,000 tokens
PDFs
100,000 words × 1.3 = 130,000 tokens
Total:
2,093,000 tokens
At an embedding price of $0.13 per 1M tokens:
Knowledge-base embedding cost ≈ $0.27
Now consider user-query embeddings.
20 tokens per query × 150,000 queries = 3,000,000 tokens/month
At $0.13 per 1M tokens:
Monthly query embedding cost ≈ $0.39
Total embedding cost:
≈ $0.66/month
The key product lesson is that embedding costs can be extremely small compared with inference costs in this particular scenario.
However, this depends on the embedding model, document volume, re-indexing frequency, and query volume.
Step 5: Compare AI Cost With Human Support
An AI product should not be evaluated in isolation. Compare it with the existing workflow.
Suppose:
- 5,000 queries/day
- One support agent handles 600 queries/month
- 150,000 queries/month
Required agents:
150,000 ÷ 18,000 = 8.33 agents
If each agent costs:
- Salary: $350/month
- Administrative cost: $100/month
Total:
$450 per agent/month
Human support cost:
8.33 × $450 = $3,750/month
Now compare this with the RAG chatbot:
- LLM inference: $2,430/month
- Embeddings: $0.66/month
Total modeled AI cost:
≈ $2,430.66/month
Potential monthly savings:
$3,750 − $2,430.66 = $1,319.34
Potential cost reduction:
≈ 35.2%
This creates a much stronger product case than simply saying, “AI will reduce support costs.”
AI PM Metrics You Should Track
Cost estimation should continue after launch.
An AI Product Manager should monitor metrics such as:
1. Cost per resolved query
Total AI Cost ÷ Successfully Resolved Queries
This is more meaningful than cost per API call.
2. Resolution rate
Percentage of customer issues resolved without human intervention.
3. Escalation rate
Percentage of conversations transferred to human agents.
4. Cost per successful resolution
A high-quality AI agent may cost more per request but still be economically superior if it resolves substantially more issues.
5. Token efficiency
Track input and output tokens per successful resolution.
6. AI containment
Measure the percentage of support volume handled entirely by AI.
How AI Product Managers Can Reduce LLM Costs
Cost optimization does not necessarily mean choosing the cheapest model.
Optimize retrieval
Retrieve only the information required to answer the question. Excessive RAG context increases input-token consumption.
Use model routing
Use a smaller model for simple questions and a more capable model for complex cases.
Improve prompts
Remove redundant instructions and unnecessary context from system prompts.
Cache repeated requests
Frequently requested information may not require a new expensive inference call every time.
Control output length
If a support response can be answered in 100 tokens, there is little product value in generating 500 tokens.
Measure before optimizing
Do not optimize blindly. Track cost per query, cost per resolution, quality, latency, and escalation rate together.
A Simple AI Product Cost Model
For an AI PM, a practical spreadsheet should contain these sections:
Demand
- Queries per day
- Days per month
- Monthly queries
Inference
- Input tokens/query
- Output tokens/query
- Input price per 1M tokens
- Output price per 1M tokens
- Cost/query
- Monthly inference cost
RAG
- Number of documents
- Words/document
- Estimated tokens
- Embedding price
- Initial indexing cost
- Query embedding cost
Business Case
- Human agents required
- Cost per agent
- Current support cost
- AI operating cost
- Monthly savings
- Percentage savings
This converts an AI idea into a measurable product business case.
Final Takeaway
For an AI Product Manager, LLM cost estimation is not simply an exercise in calculating API prices. It connects product usage, architecture, model selection, customer experience, and business outcomes.
In the example above, 150,000 monthly support queries produce an estimated RAG chatbot cost of about $2,430.66 per month, compared with approximately $3,750 for the modeled human-support operation. That translates to potential savings of roughly $1,319 per month, or 35.2%, before considering infrastructure and other operational costs.
The bigger lesson is this:
Build AI products around unit economics, not just model capabilities.
An AI PM who can answer both “Will users love this?” and “Will this scale profitably?” is much better positioned to make strong AI product decisions.