Context Engineering for AI Product Managers: RAG, Prompt Engineering and Fine-Tuning

Large language models (LLMs) are powerful, but an LLM alone does not automatically know your company’s data, workflows, product context, or preferred way of working.

Ask a general-purpose model about current news and it can often answer. Ask it which Jira tickets are blocking your team’s sprint, what is stored in your company’s Notion workspace, or which Stripe transactions occurred last month, and the answer is different: the model needs the right context.

This is where context engineering becomes an essential concept for AI Product Managers.

Context engineering is the practice of providing an LLM with the right information, instructions, tools, and constraints at the right time so it can produce a useful result.

What Is Context Engineering?

A useful way to think about context engineering is:

Give the model enough relevant information to complete the task—without overwhelming its context window.

Context engineering combines several elements:

• Instructions: What should the model do?
• Relevant data: What information does it need?
• Examples: What does a good output look like?
• Tools: What external systems can it access?
• Constraints: What should it avoid?
• Conversation state: What has already happened?
• Output requirements: What format should it return?

This is broader than prompt engineering.

Context Engineering vs. Prompt Engineering

Prompt engineering focuses primarily on designing instructions for an LLM.

Context engineering considers the entire information environment surrounding the model.

For example, an AI assistant for a product team might receive:

System instructions
+ User question
+ Relevant Jira tickets
+ Product documentation
+ Customer feedback
+ Recent sprint data
+ Available tools
+ Output format

The goal is not to send everything available. The goal is to send the right context.

Too little context can produce incorrect or generic answers. Too much irrelevant context can increase cost, reduce signal-to-noise ratio, and potentially degrade response quality.

---

Why Context Windows Matter

An LLM processes information within a context window, which is the amount of input and conversation information it can consider for a request.

A common mistake in AI product design is assuming:

“If the model can accept a lot of context, we should give it all our data.”

That is usually a poor product architecture.

Imagine a company has thousands of Notion documents containing:

• PRDs
• meeting notes
• engineering documentation
• customer research
• product strategies
• HR information
• marketing plans

If a user asks:

“What initiatives are our engineers currently working on?”

Sending the entire Notion database to the model is unnecessary.

A better architecture retrieves only the information relevant to the question and puts that information into the model’s context.

---

What Is RAG?

Retrieval-Augmented Generation (RAG) is an architecture that retrieves relevant information from an external knowledge source and provides it to an LLM before generating an answer.

At a high level:

User Question
      ↓
Retrieve Relevant Information
      ↓
Build Context
      ↓
Send Context + Question to LLM
      ↓
Generate Answer

RAG is particularly useful when an AI product needs access to private, domain-specific, or frequently changing information.

Examples include:

• An AI assistant over company documentation
• A support chatbot using product knowledge
• An analytics assistant querying business information
• An AI assistant over Jira and Confluence
• A financial assistant using internal transaction data

How RAG Works

A typical RAG system has two major stages.

1. Build the Knowledge Base

The system first processes the organization’s data.

Documents
   ↓
Chunking
   ↓
Embeddings
   ↓
Vector Database

Step 1: Collect Data

The source can contain:

• PDFs
• Documents
• Web pages
• Meeting transcripts
• Code
• Database records
• Product documentation

The data may be loaded in batches or continuously through APIs.

Step 2: Chunk the Data

Large documents are divided into smaller pieces called chunks.

For example:

10,000-word document
        ↓
500-token chunks
        ↓
500-token chunks
        ↓
500-token chunks

Chunking makes retrieval more precise and makes it easier to fit relevant information into the model’s context.

The ideal chunking strategy depends on the data.

For text, a system may use fixed-size chunks with overlap. For code, function- or class-level chunking may preserve more meaning.

Step 3: Generate Embeddings

Each chunk can be converted into a numerical representation called an embedding.

Embeddings represent semantic meaning in a vector space.

For example, these concepts are semantically related:

Engineer
Developer
Software Engineer

A good embedding model can represent their meanings as nearby points in vector space.

This allows the system to retrieve information based on meaning, rather than relying only on exact keyword matches.

Step 4: Store Embeddings

The embeddings and associated metadata are stored in a vector-capable database.

A simplified architecture looks like:

Document
   ↓
Chunk
   ↓
Embedding
   ↓
Vector Database

---

How RAG Answers a User Query

Suppose a product manager asks:

“What initiatives are our engineers working on?”

The system does not send the entire company knowledge base to the LLM.

Instead:

User Query
    ↓
Query Embedding
    ↓
Similarity Search
    ↓
Top Relevant Chunks
    ↓
Context Assembly
    ↓
LLM
    ↓
Answer

The user’s question is converted into an embedding and compared with stored document embeddings.

The system retrieves the most relevant chunks.

Those chunks are then placed into a prompt such as:

Use the following context to answer the question.

Context:
[Relevant engineering documents]

Question:
What initiatives are our engineers working on?

The LLM generates the final response using the retrieved information.

The AI PM Takeaway

You should understand decisions around:

• What data should be indexed?
• How frequently should it be updated?
• What chunking strategy is appropriate?
• How many results should be retrieved?
• How should irrelevant information be filtered?
• How do we evaluate retrieval quality?
• What happens when the system cannot find supporting information?

These decisions directly affect product quality.

---

Prompt Engineering for AI Product Managers

A strong prompt can improve the reliability and consistency of an AI feature.

A practical framework is:

1. Take a Step Back

Before writing the prompt, define:

• What problem are you solving?
• Who is the user?
• What information does the model need?
• What does success look like?

2. Define the Role

Give the model an appropriate role when it improves the task.

For example:

You are a senior product manager reviewing a new product proposal.

3. Define the Output

Be explicit about the expected result.

For example:

Return the analysis as a table with:
- Finding
- Evidence
- Risk
- Recommendation

4. Break Complex Tasks Into Steps

Instead of asking the model to perform everything simultaneously, structure the workflow.

For a PRD:

1. Identify user problems
2. Analyze evidence
3. Define product requirements
4. Propose success metrics
5. Identify risks
6. Produce the final PRD

5. Provide Examples

Examples help establish the expected structure, tone, and quality bar.

This is especially useful when building AI features that require consistent outputs.

6. Add Constraints

If the model repeatedly makes a particular mistake, explicitly prohibit it.

Examples:

Do not invent customer data.
Do not use unsupported claims.
Return only JSON.
Do not exceed 300 words.

7. Iterate

Treat prompts like product components:

Prompt
  ↓
Output
  ↓
Evaluate
  ↓
Identify failure
  ↓
Improve
  ↓
Test again

The best prompt is usually discovered through iteration rather than written perfectly on the first attempt.

---

What Is Fine-Tuning?

Fine-tuning involves adapting a pre-trained model using task- or domain-specific training examples.

Instead of changing the model’s knowledge through retrieval at inference time, fine-tuning changes how the model behaves for a particular task.

For example, a company might fine-tune a model to produce customer-support responses in a consistent style.

Fine-tuning can be useful when you need:

• Consistent tone or style
• Specialized task behavior
• Structured outputs
• Domain-specific patterns
• Better performance on a narrow task

It is important to distinguish fine-tuning from RAG.

RAG primarily addresses knowledge and information retrieval.

Fine-tuning primarily addresses model behavior and task performance.

---

RAG vs. Prompt Engineering vs. Fine-Tuning

A simple decision framework for AI PMs is:

Requirement	Preferred Approach
Better instructions	Prompt engineering
Need private or changing knowledge	RAG
Need consistent specialized behavior	Fine-tuning
Need current company information	RAG
Need a particular response style	Prompt engineering or fine-tuning
Need both knowledge and behavior changes	Combine approaches

Example

Suppose you are building an AI customer-support assistant.

Prompt engineering:

Tell it how to respond and what tone to use.

RAG:

Give it the latest product documentation, policies, and customer account information.

Fine-tuning:

Teach it a specialized response style or narrow task behavior when prompting alone is insufficient.

---

Context Engineering as a Product Management Skill

For AI Product Managers, context engineering is not just an engineering concept.

It is a product design problem.

You need to decide:

What should the AI know?

Define the relevant sources of truth.

What should it not know?

Avoid unnecessary or sensitive information.

When should information be retrieved?

Not every request requires the same data.

How much context should be supplied?

More context does not automatically mean better results.

What should happen when information is missing?

The AI should be able to say that it does not have enough evidence rather than confidently inventing an answer.

Where should the human remain in control?

High-impact workflows often benefit from human-in-the-loop design, where the AI proposes an output and the user reviews, edits, approves, or rejects it.

---

A Practical Mental Model for AI PMs

When designing an AI feature, ask these questions in order:

1. What is the user trying to accomplish?
                ↓
2. What information does the AI need?
                ↓
3. Is that information already available to the model?
                ↓
4. If not, can we retrieve it?
                ↓
5. Does the model need different behavior?
                ↓
6. What should the prompt and context contain?
                ↓
7. How will we evaluate the output?
                ↓
8. Where does the human review the result?

This mental model helps prevent a common AI product mistake: starting with the model instead of starting with the user problem.

---

Final Takeaway

If you are an AI Product Manager, you do not need to become an ML engineer to work effectively with LLM products. But you should understand the architecture behind them.

The three concepts to remember are:

• Prompt engineering → improve instructions and output behavior.
• RAG → provide relevant, external, and changing knowledge.
• Fine-tuning → adapt a model for specialized behavior or tasks.

Above all, remember the broader concept: context engineering is about giving the model the right context for the job.

The strongest AI products do not simply connect an LLM to an application. They carefully design the information, retrieval, prompts, tools, UX, evaluation, and human oversight surrounding the model.

That is where an AI Product Manager can create significant product value.