Current AI security measures, particularly guardrails and automated red-teaming, are largely ineffective against prompt injection and jailbreak attacks, which exploit the inherent unpredictability of large language models. Real-world incidents—like compromised AI agents at ServiceNow and leaked secrets from MathGPT—demonstrate tangible risks, while more dangerous exploits, such as using AI to plan physical attacks, underscore the urgency. The lack of robust defenses stems not from technical oversight but systemic issues: frontier labs prioritize capability over safety, enterprises rely on misleading commercial tools, and traditional cybersecurity practices fail to account for AI’s non-deterministic behavior. Effective protection requires merging classical security principles—like sandboxing and access controls—with AI-specific strategies, such as user-defined permissions (e.g., CaMeL) and adversarial training. Long-term solutions depend on adaptive evaluations, better education, and a shift toward intrinsic model safety rather than superficial filters. Without this convergence, the expanding attack surface of agentic systems will outpace defenses.