scripod.com
Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon

Overview

Shownote

Highlights

Transcript

Chapters

Pins

Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon

Lenny's Podcast: Product | Career | Growth

3 DAYS AGO
Lenny's Podcast: Product | Career | Growth

Lenny's Podcast: Product | Career | Growth

3 DAYS AGO
OverviewShownoteHighlightsTranscriptChaptersPins

Shownote

Aishwarya Naresh Reganti and Kiriti Badam have helped build and launch more than 50 enterprise AI products across companies like OpenAI, Google, Amazon, and Databricks. Based on these experiences, they’ve developed a small set of best practices for buildin...

Highlights

Building AI products is not just an evolution of traditional software development—it's a paradigm shift that demands new methodologies, mindsets, and leadership approaches. With real-world experience from leading AI initiatives at top tech companies, Aishwarya Naresh Reganti and Kiriti Badam share hard-earned insights on what actually works when bringing AI systems to market.
07:37
AI systems are non-deterministic and act as black boxes, sensitive to prompt variations
13:20
Begin with AI assisting humans in customer support to gather feedback before increasing autonomy.
21:11
74-75% of enterprises cite reliability as their biggest challenge in deploying customer-facing AI
22:38
Prompt injection and jailbreaking are potentially unsolvable problems in AI security
30:51
The CEO's interaction with AI tools is the top predictor of success.
41:12
Independent benchmarks are model evals, not application evals
41:32
Evals alone are not sufficient for measuring AI product success
57:31
CC/CD is the AI version of CI/CD, focusing on evals, analysis, and iteration
58:07
User behavior consistency is a critical indicator for framework progression
1:04:17
Being obsessed with the business problem is more valuable than constantly building with new tools.
1:07:46
Combining image models, LLMs, and world models will be a significant area of development
1:11:42
The pain of iterating through AI development becomes a company's new moat
1:19:22
Believe in yourself even when data suggests failure.

Chapters

Introduction to Aishwarya and Kiriti
00:00
Challenges in AI product development
05:03
Key differences between AI and traditional software
07:36
Building AI products: start small and scale
13:19
The importance of human control in AI systems
15:23
Avoiding prompt injection and jailbreaking
22:38
Patterns for successful AI product development
25:18
The debate on evals and production monitoring
33:20
Codex team’s approach to evals and customer feedback
41:27
Continuous calibration, continuous development (CC/CD) framework
45:41
Emerging patterns and calibration
58:07
Overhyped and under-hyped AI concepts
1:01:24
The future of AI
1:05:17
Skills and best practices for building AI products
1:08:41
Lightning round and final thoughts
1:14:04

Transcript

Kiriti Badam: We worked on a guest post together. They had this really key insight that building AI products is very different from building non-AI products. Most people tend to ignore the non-determinism. You don't know how the user might behave with your...