Welcome to Grind Engineer, your guide to becoming a better software engineer!
No fluff. Pure engineering insights.
In partnership with
Making AI work day to day is becoming its own job. Hear from three people doing it, on demand. Watch now
What if ChatGPT recommends your competitor first?
Over 2,500 businesses already show up in ChatGPT, Perplexity, and Google. AutoSEO writes expert articles and earns backlinks while they sleep. No SEO learning curve. No agency. Set it up once and let it run.
Catch takes real action, so it only acts when it's certain — checking with you when unsure. Always on, never drops the ball. Safe for what matters most. Trust Catch at catchagent.ai.
You type "gluten free lasagna for 4." Uber Eats builds a checkout ready grocery cart in seconds. Not a list of suggestions. A full cart with the right brands, the right quantities, and real time prices from your local store.
That's Cart Assistant, Uber's newest agentic AI feature. And the engineering behind it is a masterclass in how to use LLMs in production without letting them run wild.
TL;DR: Uber's Cart Assistant uses an 8 stage state graph pipeline where LLMs handle ambiguity (interpreting recipes, judging relevance) while deterministic systems handle data (pricing, availability, cart construction). This hybrid approach is why it works in production where monolithic LLM calls would fail.
The Problem With Monolithic LLM Calls
Most teams building AI features make the same mistake. They shove everything into one giant prompt: "Here's the user's grocery list, here are the available products, figure out the right items, quantities, and prices, and build a cart."
That works in demos. It fails in production for 3 reasons:
The LLM hallucinates prices that don't match real inventory
Quantity math goes wrong ("12 eggs" becomes 12 cartons instead of one 12 count carton)
You can't debug which part of the reasoning broke
Uber's solution? Don't give the LLM one big job. Give it 8 small, focused jobs.
The 8 Stage State Graph
Cart Assistant uses a multi prompt state graph where each step owns exactly one piece of reasoning, data fetching, or validation. Think of it as a microservices architecture for AI reasoning.
The key insight is the split: LLMs handle ambiguity, deterministic systems handle data. Pricing isn't ambiguous. Availability isn't ambiguous. Schema validation isn't ambiguous. So why would you waste an LLM call on them?
Stage | Type | What It Does |
|---|---|---|
1. Cart Plan Generation | LLM | Interprets text/image input, classifies intent, produces search terms |
2. Candidate Retrieval | Deterministic | Queries store inventory, enriches with pricing and deals |
3. Semantic Relevance Judge | LLM | Picks the best candidates for each planned item |
4. Price/Deal Constraints | Deterministic | Applies sale preferences, eligibility filters, item limits |
5. Quantity Selection | LLM + Rules | Converts "2 cups flour" into purchasable units |
6. Guardrails | Both | Schema checks, prompt safety, product eligibility |
7. Cart Assembly | Deterministic | Constructs the final cart from per item results |
8. Content Refinement | LLM | Generates shopper facing text and recipe instructions |
💡 Key Insight: Uber doesn't use one LLM call to build a cart. They use 4 focused LLM calls interleaved with 4 deterministic steps. Each LLM only reasons about what requires reasoning. Everything else is code.
The Quantity Problem Nobody Talks About
This is where most grocery AI breaks. A user types "12 eggs." What does that mean?
If you ask an LLM without constraints, it might return 12 individual eggs (not sold that way), 12 cartons of eggs (way too many), or a 12 count carton (correct).
Uber's quantity selection stage uses a data trust hierarchy:
Product titles and descriptions (most trusted)
Structured fields from the catalog
Common grocery knowledge (least trusted, only as fallback)
The LLM evaluates quantity context and packaging details. Then a deterministic validator ensures the result maps to an actual purchasable unit. If the LLM says "2 cups of flour," the system does volume to weight conversion accounting for product form and density, then finds the right bag size.
This two step approach (LLM reasoning + deterministic validation) is a pattern you'll see more and more in production AI systems.
How Uber Made It Fast
A 20 item grocery list going through 8 stages sequentially would be painfully slow. One LLM call per item per stage means 160 serial API calls. Not viable.
Uber's fix: dynamic asynchronous execution paths. Items are processed concurrently in parallel batches. The latency bottleneck shifts from total item count to the slowest single item path.
So whether you request 5 items or 50, the response time stays roughly the same. That's the difference between a feature that demos well and one that survives the Uber Eats dinner rush.
The Two Layer Guardrail System
Stage 6 is where things get interesting from a safety perspective. Uber runs two types of checks simultaneously:
Deterministic guardrails validate outputs, check enum values, enforce price limits, verify quantity calculations, and confirm availability. These are fast, cheap, and never wrong.
LLM based guardrails assess whether requests are actually in domain, detect prompt injection attempts, and validate that generated content is faithful to the source data. These catch the edge cases that rules can't.
A user asking for "a cake shaped like a weapon" might pass deterministic checks (it's a bakery item) but fail the LLM safety check. Without both layers, you'd either block legitimate requests or let problematic ones through.
Why This Matters Beyond Groceries
The architecture Uber built here isn't grocery specific. It's a template for using LLMs in any production system where you need:
Accuracy over creativity. Cart Assistant doesn't need creative writing. It needs to get the right eggs at the right price. The hybrid approach ensures LLMs only run where judgment is needed.
Debuggability. When a cart comes out wrong, Uber's engineers can pinpoint which of the 8 stages failed. Was it the planner generating bad search terms? The relevance judge picking the wrong candidate? The quantity resolver miscounting? With a monolithic prompt, you'd have no idea.
Eval driven development. Uber tests Cart Assistant with two complementary layers: strict deterministic checks for objective correctness and an LLM as a judge calibrated against human ratings for semantic quality. Every change to the pipeline gets measured against both.
Key Engineering Takeaways
Split LLM and deterministic work explicitly. If something doesn't require reasoning (pricing, availability, schema validation), don't waste an LLM call on it. Code is faster, cheaper, and deterministic.
Process items in parallel, not sequentially. The state graph pattern lets you batch concurrent items. Latency scales with the slowest item, not the total count.
Double your guardrails. Deterministic checks catch rule violations. LLM checks catch semantic violations. You need both in any system where users interact with AI.
Sources:
See you in the next one!
Scortier, Signing Off!