Latency and Cost: The Real Challenges of Shipping AI Products

Latency and Cost: The Real Challenges of Shipping AI Products

The candidates who prepare the most often perform the worst. In a recent Q3 debrief for a Senior PM role at a Tier-1 AI lab, I watched a candidate walk through a flawless theoretical framework for a RAG-based search product. He spoke about vector databases, embeddings, and chunking strategies with academic precision. Then, the hiring manager asked one question: “The token cost for this query is $0.12, and the p99 latency is 4.2 seconds; how do you make this commercially viable for a free-tier user?” The candidate froze. He had built a product that worked in a notebook, but he had not built a product that worked in a P&L. He was rejected not because he lacked technical knowledge, but because he lacked the judgment to realize that in AI, the infrastructure is the product.

Most PMs treat latency and cost as optimization tasks for the engineering team to handle after the MVP is shipped. This is a fatal mistake. In the LLM era, the cost of inference and the speed of response are not “performance metrics”—they are the primary constraints that define your product’s value proposition and business model. If your latency is too high, your UX is broken. If your cost per query is too high, your unit economics are negative. The problem isn’t your prompt engineering; it’s your failure to treat compute as a finite, expensive resource.

Why does LLM latency kill user retention in production?

Latency kills retention because AI products shift the user’s mental model from “searching” to “conversing,” and any delay over 2 seconds breaks the conversational flow. In a product review for a generative AI feature I led, we saw a sharp drop-off in engagement when the Time to First Token (TTFT) exceeded 1.8 seconds. Users didn’t just wait; they abandoned the session. The psychological friction of a spinning loader is exponentially higher in AI because the user is expecting a “thought process,” not a page load.

The failure here is usually a misunderstanding of the difference between total latency and perceived latency. The problem isn’t the 10-second total response time—it’s the 3-second silence before the first word appears. I once sat in a debrief where a PM argued that a high-quality, slow response was better than a fast, mediocre one. I overruled him. In the real world, a user will forgive a slightly hallucinated answer if it arrives instantly, but they will never forgive a perfect answer that arrives after they have already closed the tab.

The first counter-intuitive truth is that the solution is not always a faster model, but a better orchestration of the user experience. We stopped trying to shave milliseconds off the inference and instead implemented streaming responses and “optimistic UI” patterns. By streaming the tokens, we reduced the perceived latency from 4 seconds to 400 milliseconds. The judgment call was to move from a “Request-Response” architecture to a “Stream-and-Render” architecture. This is not a technical tweak; it is a product decision that changes how the user perceives the value of the product.

How do you calculate the real cost of shipping an AI feature?

The real cost of an AI feature is not the API call price, but the cumulative cost of the entire inference chain, including pre-processing, embedding lookups, and post-processing. Most PMs look at the GPT-4o price list and assume that is their cost. They forget about the cost of the vector database queries, the cost of the prompt caching, and the cost of the human-in-the-loop evaluation. In one instance, a team estimated their cost per query at $0.01, only to realize after launch that their RAG pipeline was pulling 15k tokens of context per request, driving the actual cost to $0.08.

The mistake is treating AI costs as a fixed OpEx rather than a variable cost that scales linearly with growth. In a traditional SaaS model, adding 10,000 users costs almost nothing in marginal infrastructure. In AI, adding 10,000 users can suddenly burn $50,000 a month in API credits. I have seen products with 100k MAU that were actually losing money on every single active user because the PM had not mapped the token consumption to the subscription price.

The second counter-intuitive truth is that the most expensive model is often the least efficient for the business. The goal is not to use the “best” model, but the “cheapest model that satisfies the quality threshold.” We implemented a “model routing” layer that sent 80% of simple queries to a distilled, smaller model (like GPT-4o-mini or Llama 3 8B) and only routed the complex 20% to the frontier model. This reduced our monthly spend by 65% without a statistically significant drop in user satisfaction scores. The judgment is: do not optimize for accuracy; optimize for the “Accuracy-to-Cost Ratio.”

What is the trade-off between model size and product viability?

The trade-off is a zero-sum game between intelligence, speed, and cost; you can only pick two. If you want high intelligence and high speed, you will pay an exorbitant cost. If you want low cost and high speed, you must sacrifice intelligence. In a high-stakes debrief for a legal-tech AI tool, the team insisted on using the largest available model to ensure 99% accuracy. The result was a p99 latency of 12 seconds and a cost per query that made the product unmarketable to small law firms.

The problem isn’t the model’s limitation—it’s the PM’s refusal to define “good enough.” Most PMs treat “accuracy” as a binary (correct or incorrect), but in production, accuracy is a spectrum. We shifted the goal from “perfect accuracy” to “acceptable accuracy with a correction loop.” By using a smaller model and adding a “Regenerate” button or a “Verify” step, we achieved the same end-result for the user while reducing the cost per interaction by 90%.

This is where the “Not X, but Y” principle applies: the goal is not to build a “smart” product, but to build a “reliable” product. A reliable product uses a cascade of models—a fast classifier to categorize the intent, a small model to handle the routine, and a large model only for the edge cases. This “Cascading Inference” pattern is the only way to scale an AI product without bankrupting the company. If you are shipping a single-model architecture for every single query, you are not building a product; you are running an expensive experiment.

When should you move from API-based models to self-hosted open-source?

You move to self-hosted open-source models when your volume reaches a point where the marginal cost of GPU orchestration is lower than the per-token cost of an API, or when data privacy becomes a hard constraint. For most startups, this transition happens around the 50k-100k daily active user mark. I remember a debate where the engineering lead wanted to move to Llama 3 on private servers on day one. I blocked it because the operational overhead of managing Kubernetes clusters and H100s would have slowed our shipping velocity from days to months.

The hidden complexity of self-hosting is not the model—it’s the “LLMOps” stack. You are no longer paying for a token; you are paying for electricity, hardware depreciation, and the salaries of three specialized ML engineers. The cost of a dedicated ML engineer ($220,000 - $350,000 total compensation) often outweighs the API savings for the first year of a product’s life. The judgment is: do not optimize for infrastructure costs until you have proven the product-market fit.

The third counter-intuitive truth is that “open source” is not “free.” The cost of fine-tuning a model to match the performance of a frontier model is an upfront investment in compute and human labeling that can easily reach $100,000 before the first user even sees the feature. The decision to self-host is not a technical choice; it is a financial hedge. You are trading variable costs (API) for fixed costs (Infrastructure + Talent). If your growth is unpredictable, stick to APIs. If your growth is stable and high-volume, move to your own silicon.

Preparation Checklist

• Map the entire inference chain: Identify every step from user input to final output, including embedding calls, retrieval steps, and post-processing filters.
• Define the “Quality Floor”: Establish the minimum acceptable accuracy level for each use case so you can use the smallest possible model for the majority of tasks.
• Calculate the “Unit Economics per Query”: Determine the exact cost of a single interaction (Input Tokens + Output Tokens + Vector Search) and compare it against the LTV of the user.
• Implement a Model Router: Design a logic layer that routes queries to different models based on complexity to optimize for the Accuracy-to-Cost Ratio.
• Establish a Latency Budget: Set a hard limit for Time to First Token (TTFT) and Total Response Time, then work backward to select the model and infrastructure.
• Work through a structured preparation system (the PM Interview Playbook covers the System Design and AI Product frameworks with real debrief examples to handle these cost/latency trade-offs).
• Build a “Degraded Mode” strategy: Define what the product does when the API is slow or down (e.g., switching to a faster, dumber model or showing a cached response).

Mistakes to Avoid

Mistake 1: The “Frontier Model Obsession”

• BAD: “We will use GPT-4o for everything because it is the most capable model and we want the best quality.”
• GOOD: “We will use GPT-4o-mini for intent classification and basic summaries, and route only complex legal analysis to GPT-4o to keep our cost per user under $0.05.”

Mistake 2: Ignoring the “Cold Start” and P99s

• BAD: “The average latency is 2 seconds, so the user experience is great.”
• GOOD: “The average is 2 seconds, but the p99 is 15 seconds. We need to implement streaming and a loading state to hide the tail latency for the 1% of users who are experiencing timeouts.”

Mistake 3: Treating Tokens as a Free Resource

• BAD: “We’ll just increase the context window to 128k tokens so the model has all the information it needs.”
• GOOD: “Increasing the context window increases both cost and latency linearly. We will implement a more aggressive RAG reranking step to feed the model only the top 5 most relevant chunks.”

FAQ

Do I need to be a machine learning engineer to manage AI costs? No. The PM’s role is not to tune hyperparameters, but to set the “Cost-per-Query” budget and the “Latency Ceiling.” Your job is to define the business constraints that the engineers must solve for.

Is streaming responses enough to solve latency problems? No. Streaming solves perceived latency (TTFT), but it does not solve total latency. If a user has to wait 30 seconds for a full answer, they will still churn. You must combine streaming with model distillation and prompt optimization.

Should I use a vector database or just a long context window? Use a vector database for scalability and cost. Long context windows are expensive and slow. RAG (Retrieval-Augmented Generation) is not just about accuracy; it is a cost-saving strategy to reduce the number of tokens sent to the model.

---

Ready to build a real interview prep system?

Get the full PM Interview Prep System →

The book is also available on Amazon Kindle.