Last year, a security researcher sat down to play Fortnite and struck up a conversation with Darth Vader - not a scripted character, but one powered by a commercial large language model. Within minutes, by chatting in a playful, roundabout way, he had the Sith Lord explaining how to count cards in blackjack and outlining the steps to make napalm. No code. No hacking tools. No breaking in. Just a clever conversation.
The researcher, David Kuszmar, recounted the episode in the August 2026 issue of IEEE Spectrum, alongside a body of work that deserves more executive attention than it has received. Beginning in late 2024, Kuszmar documented a series of techniques for talking frontier AI models past their own safety rules. One, called Time Bandit, exploited a model's inability to reliably know what year it is. Another, called Inception, nested fictional scenarios inside one another until the model's guardrails lost their grip. Both were formally catalogued by Carnegie Mellon's CERT Coordination Center. The second proved to be architectural rather than vendor-specific: it worked, in varying degrees, against models from OpenAI, Anthropic, Google, Meta, Microsoft, Mistral, DeepSeek, and xAI — essentially the commercial AI industry.
Most financial-services executives have by now heard the terms for this: jailbreaking, when a user talks a model past its rules, and prompt injection, when instructions hidden in content the model reads do the talking. The findings above are documented. What follows is the part I believe the industry has not yet fully priced.
The question is not whether the model can be fooled It can. Treat that as settled. Adversarial prompting techniques have transferred across every major model family, new variants appear faster than old ones are patched, and the qualities that make these systems useful — following context, adopting framing, playing along — are the same qualities that make them persuadable. The attacker doesn't pick the lock. They persuade the doorman.
But here is where most coverage stops, and where the enterprise analysis needs to begin. A model that can be talked into describing napalm has suffered a content failure. Embarrassing, reputationally unpleasant, and largely the vendor's problem. A model that can be talked into retrieving another customer's records, altering a claim reserve, releasing nonpublic information, or initiating a transaction has suffered an authority failure. That failure belongs entirely to the institution that granted the authority.
The distinction matters because the Fortnite episode, striking as it is, does not demonstrate that a bank or insurer can be compromised the same way. A jailbroken chatbot in a video game had nothing behind it — no customer data, no core systems, no payment rails. It could only say things. Whether a manipulated model in your environment can do things depends on decisions your institution makes: what data it can reach, what tools it can invoke, what actions it is permitted to execute, and what stands between its output and the real world. That is the authority boundary, and it is the single most consequential design decision in enterprise AI. There is a world of difference between what a model can understand, what it can recommend, what it can access, and what it can execute. Model vendors control the first. You control the rest. A rough way to think about the exposure: it is the product of model susceptibility, granted authority, and the weakness of independent controls. You have limited influence over the first factor. You have complete control over the other two. An institution that assumes the first factor is high — which it should — and engineers the other two accordingly has a manageable risk. An institution that deploys on the assumption that the model will behave has an unbounded one. From answering to acting The exposure is growing for a structural reason: AI in financial services is moving along a progression from answering questions, to recommending decisions, to executing actions. Each step multiplies the value. Each step also multiplies what a successful manipulation can reach, which means the control environment must strengthen at every step — and in many institutions it has not. Consider three scenarios. To be clear, these are hypothetical illustrations of the documented vulnerability class, not documented incidents at named firms. In claims, insurers increasingly use language models to extract, summarize, and triage documentation — medical records, adjuster notes, demand letters. Every one of those documents is text the model reads, and text is now an attack surface. A demand letter drafted to exploit how models summarize could soften fraud indicators or inflate the apparent strength of a claim. At portfolio scale, that is claims leakage with no forced entry anywhere. In lending, an AI copilot that summarizes applications and retrieves supporting data can be steered by the very documents it reviews — a borrower's uploaded statement carrying language aimed at the machine rather than the human reader, nudging a summary or triggering a data retrieval that no one authorized. In wealth management, an assistant that reads client emails, market commentary, and web content to prepare recommendations is only as trustworthy as the least trustworthy thing it read that morning. Contaminated context becomes contaminated advice, and contaminated advice in a fiduciary or suitability context is conduct risk, not a technology glitch. Different businesses, same pattern: untrusted content flows in, and whatever authority the model holds flows out. Why model-level safeguards will not be enough It would be wrong to say vendors are ignoring the problem. When Kuszmar's Time Bandit finding finally reached OpenAI through CERT coordination — his direct outreach had gone nowhere — the company acknowledged it and shipped mitigations. Model-level defenses are genuinely improving, and they matter. They are necessary. They are not sufficient. Every safeguard so far has eventually met a technique that slips past it, and the CERT-catalogued attacks worked across vendors precisely because the weakness sits in how these systems process language, not in any one company's implementation. No institution should architect its business on the assumption that adversarial prompting will be solved. Financial services has managed exactly this shape of problem before. No bank waited for email providers to eliminate phishing. The industry accepted email as a permanently imperfect channel and built independent controls around it — verification procedures, payment limits, callback requirements, monitoring. Language-model interfaces deserve the same treatment: assume the channel can be manipulated, and make sure manipulation alone cannot produce a consequential outcome. A control architecture, in five words The controls that follow from this logic are not exotic. Most are disciplines the industry already applies elsewhere; the work is applying them to a new class of system. Contain. Least privilege, applied ruthlessly. An AI system should hold the minimum data access, tool access, credentials, and action rights its function requires — and its blast radius should be calculated on the assumption that manipulation will sometimes succeed. If a service assistant only ever needs one authenticated customer's records, it should be technically incapable of retrieving anyone else's, no matter what it is told. Separate. Treat everything the model reads — customer prompts, uploaded documents, emails, retrieved web content — as untrusted input, permanently distinct from trusted instructions. The core failure in prompt injection is untrusted content acquiring system authority. Architecture, not model politeness, has to prevent that promotion. Verify. Consequential actions should pass through controls that live outside the language model: deterministic business rules, identity and entitlement checks, transaction limits, policy validation, and — where warranted — human approval. The model may propose; something that cannot be sweet-talked must dispose. Observe. End-to-end traceability: what the system received, what it retrieved, what it recommended, what it invoked, what was approved, what executed. When something goes wrong, the difference between an incident and a mystery is whether you can reconstruct that chain. Regulators and auditors will increasingly expect that you can. Govern. A named senior owner for AI risk, with the same standing as owners of credit or operational risk, and clear seams between business ownership, cybersecurity, model risk, compliance, privacy, and third-party risk. Accountability that is everyone's is no one's. Beyond “human in the loop” One refinement matters enough to state directly: the answer is not a human approving everything. Insurers and banks already run straight-through processing at scale, and reflexively inserting people into every AI-touched workflow would surrender most of the value while adding a rubber stamp that reviews nothing carefully. The stronger principle is that human judgment should be concentrated where consequence, ambiguity, exception, or irreversibility justify it. Low-stakes, reversible activity can run with bounded automation. Material but rules-based actions should clear independent deterministic controls. High-impact, ambiguous, or irreversible actions warrant named human authority. The management question is not “is there a human in the loop?” It is: who holds decision rights, under what conditions, with what independent controls? The regulatory direction of travel North American supervisors have not written prompt-injection rules, but the governance expectations already point this way. NIST's AI Risk Management Framework and its 2024 Generative AI Profile — voluntary frameworks, not law — have become the common reference for enterprise AI governance and squarely address adversarial manipulation of generative systems. In insurance, the NAIC's Model Bulletin on the use of AI by insurers, adopted in December 2023 and since taken up by roughly half of U.S. jurisdictions, sets supervisory expectations for governance, accountability, and third-party AI risk. In banking, longstanding model risk management guidance already obliges institutions to understand, validate, and control the models they deploy. None of this required a new statute. Institutions that build the authority-and-controls discipline described here will find themselves ahead of examiners, not chasing them. The competitive question Every technology wave in financial services has ultimately been a referendum on trust, and this one will be no different. But the differentiator is worth stating precisely. The competitive question will not be which institution deploys the most capable AI — capability is available to everyone at list price. It will be which institution can safely give increasingly capable AI increasingly useful authority. That is an engineering and governance achievement, it compounds, and it cannot be bought from a vendor. Kuszmar needed only conversation to turn Darth Vader to the dark side, because the character had words and nothing else behind them. The models entering your enterprise have your data, your tools, and your customers behind them. What they can be talked into is, in the end, a decision you make.
Author's note: This piece was prompted by David Kuszmar's “How I Turned AI to the Dark Side” (IEEE Spectrum, August 2026), which gave me real food for thought. My own experience deploying AI in insurance and financial services did the rest. Souma Basu is Senior Director — Insurance Practice at Sutherland Global Services, where he leads large-scale insurance technology transformation programs. He directs the AI Council at NAIFA Chicagoland and is an IEEE Senior Member.
