Announcing the 🤗 AI Research Residency Program

The agent evaluation gap: Enterprise AI organizations have a reality-alignment problem, not a coverage problem — and most are shipping to production anyway

Across 157 enterprises, organizations are granting AI agents more autonomy while trusting the evaluations meant to gate that autonomy less. Half have already shipped an agent that passed their internal evaluations and then failed a customer in production; only one in twenty fully trusts automated evaluation today; and the most-cited weakness is that evaluations do not align with real-world outcomes. Yet two-thirds already allow, or are actively engineering toward, deploying agent changes to production on automated evaluation alone — with no human in the loop. The result is an evaluation gap — the distance between how much autonomy enterprises are handing their agents and how far they trust the tests that are supposed to catch the failures.This wave of VentureBeat Pulse Research examines how technical leaders measure agent performance: which reliability and evaluation platforms they use, how they select and trust them, what breaks in production, and how far they are willing to let agents run without a human in the loop.The central finding is an evaluation gap — the distance between the autonomy enterprises are granting their agents and the trust they place in the evaluations meant to govern it. Half of organizations (50%) have, in the past year, deployed an agent or LLM feature that passed their internal evaluations and then caused a customer-facing failure, and a quarter have seen it happen more than once. Trust in the tests themselves is thin: only 5% say they fully trust automated evaluation today, and the single most-cited limitation is that evaluations align poorly with real-world outcomes (29%). Enterprises are discovering that a passing eval is not the same as a working agent.What makes the gap consequential is the direction of travel. Two-thirds of organizations (66%) already permit fully automated, zero-human-in-the-loop deployment for low-risk agents (34%) or are actively engineering their pipelines to allow it within twelve months (33%). At the same time, the evaluation stack that would have to earn that trust is fragmented and immature: the most common primary tools are the model providers’ native evals, tied with having no dedicated tooling at all (17% each); and only about a quarter of enterprises run real-time quality checks on live production traffic. The autonomy is arriving faster than the assurance.MethodologyVentureBeat fielded this survey as part of its ongoing Pulse Research series, this survey — the Agentic Reliability & Evals tracker — focused on how technical leaders evaluate agent performance and reliability. Responses are filtered to organizations with 100 or more employees (n=157), drawn from a single survey in June 2026; because this is one wave rather than a pooled multi-month sample, the report reads cross-sectionally and does not infer month-over-month trends. Where questions were multiple-select, those shares can sum to more than 100%.By role the sample is senior and buyer-credible: 38% are final decision-makers for AI purchases and another 34% recommenders or influencers. Product and program managers (15%), consultants and advisors (10%), directors of engineering/IT (8%), and CIOs/CTOs/CISOs (8%) lead the named titles, alongside a large “Other” function (37%). By organization size the sample is mid-market-weighted: 100–499 (37%) and 500–2,499 (27%) employees lead, with 2,500–9,999 (20%), 10,000–49,999 (10%), and 50,000+ (6%) above them. Technology/Software is the largest industry at 23%, followed by Retail/Consumer (15%), Healthcare/Life Sciences (12%), and Manufacturing (10%).At 157 respondents the sample is large enough to read directionally but should be treated as a directional signal rather than a precise measurement; it is self-selected and is not a probability sample. It skews toward the mid-market, so it is best read as the view from organizations actively standing up agent evaluation practices rather than from the largest operators.Note: This survey was rebuilt for the June wave from the earlier “LLM observability and evaluations” survey; because the questions and sample differ, no comparisons are made to the April–May data.Finding 1: A passing eval is not a working agentHalf have shipped an agent that passed evals, then failed a customerWe asked whether, in the past 12 months, organizations had deployed an agent or LLM feature that passed their internal evaluations but then caused a customer-facing failure. Half of those that run evaluations had.This is the report’s defining number. Half of organizations (50%) have shipped an AI feature that cleared their internal evaluations and then failed in front of a customer — an incorrect output, a broken workflow, or a quality incident — and a quarter have seen it happen more than once. Only 36% report no such failure, and the remainder either run no pre-deployment evaluations (8%) or don’t track the root cause closely enough to know (6%). The failure is precise and expensive: the evaluation said the agent was ready, and it was not. Everything that follows — how enterprises trust their evals, what they monitor, and how much autonomy they grant — is shaped by this experience.Finding 2: Almost no one fully trusts automated evaluationThe top complaint: Evals don't match real-world outcomesWe asked which limitation most reduces trust in automated agent evaluations today. Only a sliver of enterprises had no complaint at all.Trust in automated evaluation is scarce, and specific. Only 5% of organizations say they fully trust automated evaluation as it stands — meaning 95% name a limitation that holds them back. The most common, at 29%, is the one that most directly explains Finding 1: evaluations align poorly with real-world outcomes, passing agents that later fail. Bias or inconsistency (21%) and a lack of explainability (18%) follow — enterprises cannot always tell why an evaluation reached its verdict — and 17% cite data-leakage or privacy concerns in the evaluation process itself. The tests meant to certify agents are not yet trusted to certify them, which is precisely why the autonomy trajectory in Finding 3 is so striking.Finding 3: The autonomy ceiling is rising anywayTwo-thirds already allow, or are building toward, zero-human deploymentWe asked whether organizations would let an autonomous agent deploy a code or system change to production on automated evaluation results alone, with no human-in-the-loop validation. The trajectory runs straight through the trust gap.Here is the paradox at the heart of the report. Even though almost no one fully trusts automated evaluation (Finding 2), two-thirds of organizations (66%) either already allow zero-human-in-the-loop deployment for low-risk agents (34%) or are actively engineering their pipelines to permit it within a year (33%). Only 22% rule it out for the foreseeable future. The direction is unambiguous: enterprises are moving to let evaluations gate production autonomously — removing the human check — at the same moment they say those evaluations don’t reliably match reality. The autonomy ceiling is rising faster than the assurance beneath it, which is the mechanism by which the false-confidence failures of Finding 1 will scale rather than shrink.Notably, the autonomy bet is not just a small company phenomenon. Splitting the sample by company size, larger enterprises are slightly further down the path toward zero human review than smaller companies (70% versus 64%) and slightly more likely to have shipped an evaluation-passing agent that then failed a customer (54% versus 48%). The assumption that large, regulated organizations are holding the human in the loop longest is, in this sample, backwards.  To be sure, these are directional figures, since the survey was not a huge sample — 57 respondents from companies with 2,500+ employees and 100 from companies smaller than that. Finding 4: The evaluation stack is fragmented and provider-ledProvider-native evals lead — tied with no dedicated tool at allWe asked which agent reliability or evaluation platform enterprises primarily use today. The market has no clear leader — and a large share has nothing dedicated.The evaluation layer is early and unconsolidated. Provider-native tooling leads — OpenAI’s native evals and traces (17%) and Anthropic’s Claude Console evals (13%) together outweigh any independent platform — but it is tied at the top by a striking answer: 17% of enterprises use no dedicated agent-evaluation tooling at all, a notable gap for organizations shipping agents to customers. The specialist evaluation vendors — DeepEval (12%), Braintrust (8%), LangSmith, Weave, Promptfoo, Langfuse, Arize — are scattered across single to low double digits, and 11% have built their own. No independent platform has yet become the category standard, which leaves most enterprises evaluating agents with provider-native tools, home-grown scripts, or nothing.Finding 5: Production monitoring rarely watches output qualityOnly a quarter run real-time quality checks on live trafficProduction monitoring for an AI agent can watch two very different things. It can watch whether the system is functioning — is the agent up and responding, did each request complete, how fast, at what cost, with any errors. Or it can watch whether the agent's output is correct — automated checks that evaluate the content of each answer as it goes out: did the agent give the right answer, take the right action, stay within policy. The distinction matters because a confidently wrong answer is invisible to the first kind of monitoring: the request completes, the response is fast, no error is thrown, and every functioning-metric reads healthy. We asked organizations which kind their live production monitoring is built for today.Grouped by what is actually being watched, the split is stark: 51% of organizations monitor only whether the agent is functioning, while 23% monitor whether its answers are right. Counting the ad-hoc reviewers and the don't-knows, roughly three-quarters of organizations run no automated, real-time evaluation of output correctness in production — they can see that the system is up and what it costs, and they are taking the correctness of its answers on faith. That blind spot is the runtime counterpart to the pre-deployment gap in Finding 1: the same organizations engineering the human out of the deployment decision mostly cannot see, in real time, when the deployed agent starts getting things wrong.Finding 6: Bought on cost, measured on consistencyPrice and integration drive selection; evaluation consistency is the goalWe asked what most influenced enterprises’ choice of an evaluation vendor, and what they treat as their primary measure of success. Both answers are pragmatic.Enterprises buy evaluation tooling on economics and trust it on repeatability. Cost of evaluations (28%) narrowly leads selection, just ahead of ease of integration (27%) and evaluation accuracy (24%) — breadth of observability (13%) and vendor roadmap (4%) matter far less. On what success looks like, more than a third (36%) name evaluation consistency — getting the same verdict on the same behavior every time — well ahead of speed of experimentation (19%), reduction in failures (18%), production visibility (13%), and compliance (11%). The emphasis on consistency is telling: before enterprises can trust an evaluation’s verdict, they need it to be stable — the very property whose absence (bias and inconsistency) ranked among the top trust limitations in Finding 2. Satisfaction with current tooling is only moderate, averaging 3.8 on a five-point scale across overall satisfaction, ease of implementation, and value for money.Finding 7: The next dollar goes to humans and observabilityInvestment is flowing to oversight, not just automationWe asked which reliability and evaluation investment will grow most over the next year. The money is going toward watching agents more closely — including with people.The second-largest planned investment — behind only production observability — is human review workflows, at 26%. Read against Finding 1, that is the report's quietest contradiction: at the same moment two-thirds of enterprises are engineering the human out of the deployment decision, more of them plan to grow spending on human reviewers (26%) than on the automated evaluation pipelines (16%) that would replace them. The zero-human trajectory and the human-review budget are rising in the same companies at the same time. Indeed, only 8% report that their budget is not increasing. Taken together, enterprises are hedging: building toward autonomy while spending to watch agents more closely and keep humans available for the calls that automated evaluation cannot yet be trusted to make.Finding 8: A tooling reshuffle is comingNearly two-thirds plan to adopt or switch platforms within a yearWe asked whether enterprises plan to adopt a new, additional, or replacement evaluation platform, and which they are considering. Few intend to stand pat.The evaluation market is wide open. While 36% have no plans to change, a clear majority (64%) intend to adopt a new, additional, or replacement platform within twelve months, and 31% within the next quarter. The consideration set points where current usage is thinnest: Confident AI’s DeepEval leads what enterprises are evaluating (20%), ahead of OpenAI’s native evals (13%) and Braintrust (9%) — the open-source specialists drawing more interest than their present footprint. Given that so many enterprises today rely on provider-native tools or nothing at all (Finding 4), this is less a defection than a first real wave of tooling adoption — the moment the evaluation layer starts to consolidate. Which platforms earn that trust, in a market where almost no one trusts automated evaluation yet, is the open question this series will keep tracking.The bottom line: An evaluation gap that autonomy will widen, not closeOrganizations with 100 or more employees are granting AI agents more independence than they trust their evaluations to support. Half have already shipped an agent that passed its evals and then failed a customer; almost none fully trust automated evaluation, chiefly because it doesn’t match real-world outcomes; and most watch production for uptime and cost rather than for whether the agent’s answers are right. Yet two-thirds already allow, or are actively building toward, deploying to production on automated evaluation alone.The vendor market is early and unsettled: the most common primary evaluation tools are provider-native evals, tied with no dedicated tooling at all, and a clear majority plan to adopt or switch platforms within the year. Encouragingly, the next dollar is going to observability and — pointedly — human review, suggesting enterprises sense the gap even as they engineer past it. At 157 respondents in a single wave this is a directional read, skewed toward the mid-market — but the direction is clear: autonomy is being granted on the strength of evaluations that the people granting it do not yet trust. The evaluation gap is not a coverage problem that more tests alone will close; it is a problem of evaluations that reflect reality and can be trusted to gate it. The open question for later waves is whether assurance catches up to autonomy — or whether the false-confidence failures move from customer incidents into changes that deploy themselves.Based on survey responses from 157 qualified enterprise respondents (100+ employees), drawn from a single June 2026 wave. This is a directional read rather than a precise measurement — the sample is self-selected, not a probability sample, and skews toward the mid-market. Respondents include product and program managers, consultants and advisors, directors of engineering/IT, and CIOs/CTOs/CISOs, among other functions, across technology/software, retail/consumer, healthcare/life sciences, manufacturing, and other industries.

NVIDIA Nemotron 3 Embed Ranks #1 Overall on RTEB, Advancing Agentic Retrieval

The First Chatbot’s Multiple Personalities

ELIZA is remembered as the world’s first AI star, a kindly therapist in chatbot form that gently probed users’ worries. Even its creator, Joseph Weizenbaum, was surprised by the warm reception given to his experiment in human-machine interaction. For some, it heralded an age of automated psychotherapy, while others believed the program demonstrated sentience, a fallacy soon known as the “ELIZA effect.” Based on published descriptions, ELIZA has been implemented on many different computers, but only recently has the actual source code been unearthed from MIT’s archives. In Inventing ELIZA: How the First Chatbot Shaped the Future of AI, just published by MIT Press, a squad of researchers analyze the code and reveal a complex program capable of much more than faking psychiatry. In fact, it could assume several different personas. The authors have also created a faithful emulation of the therapist persona that you can try yourself after reading the book excerpt below.When it debuted in the mid-1960s, the ELIZA software program transformed the way people thought about interacting with computers. As the first chatbot, ELIZA demonstrated how a calculation machine might engage in conversation, ushering in a host of social and technical questions that still resonate today. Now we don’t think twice about interacting with a machine in real time, conversing over text, or even speaking into the air to ask about the weather. In many ways, ELIZA shaped not only the way we think about interacting with computers but also how we think about them. It began to give a reality to the science fiction stories of how we expect computers to work. This article is adapted from the new book “Inventing ELIZA: How the First Chatbot Shaped the Future of AI“ (MIT Press, 2026).Although ELIZA was far from a faultless conversation partner, it astonished its users. The recent discovery and archaeology of the original ELIZA source code represents a significant intervention in the history of computing. By examining the actual implementation of ELIZA rather than relying on later reconstructions and reimplementations, we challenge taken-for-granted assumptions about this key software artifact.For example, the source code reveals that ELIZA was not merely a simple pattern-matching chatbot but can be better understood as a sophisticated platform designed for multiple “personas,” or scripts, with a complex set of capabilities, including script editing and contextual memory. The script that most people conflate with the program ELIZA was actually called Doctor, which performed the role of a psychotherapist. Yet, like a modern chatbot prompted to behave with different personalities, ELIZA could take on many roles.“This code and script…reveal underlying assumptions about language, therapy, and human-computer interaction that continue to influence modern AI development.”This unearthed material transforms our understanding of early AI development by demonstrating that Joseph Weizenbaum’s technical innovations were far more advanced than previously documented. Moreover, the discrepancies between his published descriptions and the actual implementation help to show the gap between theoretical computational models and their material instantiations in computer source code, a tension that continues to shape digital culture today.Although many technical innovations have emerged in the decades since ELIZA, examining the ELIZA/Doctor code offers a rare glimpse into one of the earliest formalized attempts to model human conversation. What makes ELIZA particularly fascinating is not only its historical significance but also what it reveals about Weizenbaum’s views on both computing and human interaction. This code and script do not merely showcase programming techniques of the 1960s; they reveal underlying assumptions about language, therapy, and human-computer interaction that continue to influence modern AI development. By examining this code, we can start to uncover the sophisticated linguistic and programming techniques that allowed a rudimentary pattern-matching system to create a convincing simulation of understanding. But before we can read the lines of code, let us offer an overview of the system.How Did ELIZA Create Personas?The architectural distinction between ELIZA and Doctor represents an important design decision in AI history. Think of ELIZA as a system for interaction and Doctor as one set of rules that Weizenbaum devised, among others. This separation, manifested in ELIZA’s system-script dichotomy, presaged numerous contemporary software patterns, from configuration-as-data to plug-in architectures and domain-specific languages. Based on published journal articles, ELIZA was re-created on many platforms, such as the IBM PC. However, the actual source code sat untouched in the MIT archives for many years. VCF Museum at InfoAgeWithout question, the historical context of 1960s computing fundamentally shaped ELIZA’s architecture as well. Decisions in computing that reflect material constraints create path dependencies and eventually become programming cultural norms. These constraints manifested in ELIZA’s single-pass processing, tape-based storage and stack-oriented implementation. Yet within these limitations, Weizenbaum crafted an elegant solution. These technical features, though invisible to the users, are crucial to creating the illusion of understanding that made ELIZA so compelling.Weizenbaum explained many of ELIZA’s technical features in the 10-page paper published in the January 1966 edition of the journal Communications of the Association of Computing Machinery (CACM). But he chose to omit some essential details.In that paper Weizenbaum published ELIZA’s best known dialogue, which begins,Men are all alike.IN WHAT WAYThey’re always bugging us about something or other.CAN YOU THINK OF A SPECIFIC EXAMPLEWell, my boyfriend made me come here.This dialogue marked ELIZA’s public debut in 1966 as one of the examples produced by the Doctor script. By finding the source code for ELIZA and examining how it performs the Doctor script, we now better understand these two separate parts of a system and can explore the many other personas of ELIZA. In just some of the other scripts known to date, ELIZA was programmed to discuss math, poetry, color, paradoxes, synchronization, relativity, France, and elevators.These scripts work like templates. They are structured data that direct the ELIZA system to “play” a particular task or role. By comparing archival and published ELIZA dialogues from interactions with a variety of scripts, including Doctor, we can understand more about bot personas and how they function, paying close attention to how a bot evokes social dynamics between system and interactor.Ultimately, studying the dialogues and scripts demonstrates the crucial role that collaboration plays in these exchanges, as bot and user cocreate the sense of their interaction. To understand the full range of ELIZA’s capabilities and conversational possibilities, let’s take a look at the variety of scripts that were created for the ELIZA system.What distinguishes each ELIZA script is both its subject matter and the linguistic and stylistic choices used to deliver that content. These choices are not neutral; they can be said to construct a particular persona with characteristics that emerge through the script’s language patterns, vocabulary, and conversational approach. In short, it matters not just what you say but how you say it too.“The aim was less to create a functional automated therapist and more to find a suitably constrained role to match the limitations of the programming environment.”For example, with the Doctor script Weizenbaum deliberately echoed the style of a Rogerian “talk” therapist. He chose this persona because the psychiatric mode is one of the few types of conversations in which one person can “assume the pose of knowing almost nothing of the real world. If, for example, one were to tell a psychiatrist ‘I went for a long boat ride’ and he responded, ‘Tell me about boats,’ one would not assume that he knew nothing about boats but that he had some purpose in so directing the subsequent conversation.” The first users of ELIZA interacted with it via teletype terminals.VCF Museum at InfoAgeThus, the most famous persona created for ELIZA was a technical convenience. As human-computer interaction expert Lucy Suchman explains, “The Doctor program exploited the maxim that shared premises can remain unspoken: that the less we say in conversation, the more what is said is assumed to be self-evident.” In creating the original ELIZA effect, less was more.The aim was less to create a functional automated therapist and more to find a suitably constrained role to match the limitations of the programming environment. Then Weizenbaum composed the script to match the role by choosing specific words that evoked rhetorical tone and characterization, for example, LET’S DISCUSS FURTHER WHY YOU … WHAT DOES THAT SUGGEST TO YOU. In Doctor, the machine side of the conversation needs to appear like a good listener who cares about what the user has mentioned before, so it often includes the user’s text in its replies and keeps its responses open-ended. Because a real doctor would be inquisitive, the script contains lots ofWHAT andWHY questions. In other scripts and dialogues, the script keywords and assigned responses reveal the design decisions that help create different specific personas. This variation becomes increasingly apparent as we look at the wider range of ELIZA scripts.What Were Some of ELIZA’s Alternate Personas?Let’s start with a script called Neweng, short for New England. By the mid-1950s, scientists were already taking up Turing’s proposition for conversing with a computer as a tangible idea. In 1959, a “conversation machine” made by a small team at the University of Toronto Computation Centre could make small talk about the weather if preprompted with current data. Its script included a ranked series of time words, weather words, and emotion words, but the system had “a rather narrow outlook,” limited vocabulary, and no knowledge of syntax. According to its creators, “in spite of this crudity, the results in passable conversation are surprisingly good, which in a way is a commentary on the shallowness of ordinary conversation about the weather.” This fragment of ELIZA’s source code shows how scripted responses were encoded, in this case to make ELIZA sound like a Rogerian therapist. MIT Libraries, Distinctive CollectionsOne of ELIZA’s test scripts found in the archive had a similar function. Neweng does not soothe, therapize, or instruct, but only makes small talk. It chats as if at a boring dinner party.WHAT NEW ENGLAND STATE DO YOU LIKE BEST OF ALLi love rhode islandI ENJOY THE BEACHES ON LONG ISLAND SOUNDWHAT OTHER NEW ENGLAND STATE DO YOU LIKEi like massachusettsTHAT IS MY HOME STATEWHAT OTHER NEW ENGLAND STATE DO YOU LIKEyou must like massachusetts tooWHAT OTHER NEW ENGLAND STATE DO YOU LIKEi enjoy skiing in new hampshirePLEASANTLY COOL IN THE SUMMER UP THEREWHAT OTHER NEW ENGLAND STATE DO YOU LIKEi like missouriMISSOURI IS NOT A NEW ENGLAND STATEWHAT OTHER NEW ENGLAND STATE DO YOU LIKEi like cape codTHAT’S A BEAUTIFUL PART OF MASSACHUSETTSWhat can we learn from such a basic conversation in which it seems like nothing interesting is shared? Despite the bland chat, a persona emerges from the bot script in phrases like PLEASANTLY COOL and UP THERE and in locations like THE BEACHES ON LONG ISLAND SOUND. From such comments, we can begin to imagine a persona who has not only a certain vocabulary but also certain hobbies and vacation spots. Perhaps implicit assumptions of race and class emerge as well.A chatty persona might take on many forms, depending on where and when the conversation occurs. This one seems reserved, perhaps fitting its setting in 1960s New England. The system reminds the user that Missouri is not a New England state, but what if this conversation took place in Missouri, Texas, or Mexico? The machine persona would sound different in its cadence, tone, and references. What would we come to understand about a chat persona from Fire Island, from Brooklyn, from Berlin? What would they sound like, and what topics would they discuss?These differences in subject matter do matter. They imply personas with entirely different backgrounds and experience, giving users wholly different interactions and affective relations. In this way, the Neweng script demonstrates how even simple algorithms making contextual responses about geography could generate a convincing sense of personhood and place. Whereas Neweng could be said to have created a casual, conversational persona focused on light social exchange, other scripts pushed ELIZA into more structured and educational roles. These scripts demonstrate how the system could be adapted not just for friendly chatter but for teaching. Edwin Taylor, at MIT’s Education Research Center, developed alternate scripts for ELIZA, testing its ability to act as a teacher.MIT Libraries, Distinctive CollectionsMeet ELIZA the tutor, quite unlike ELIZA the therapist or the chatty neighbor. Intrvw, Canvec, FVP1, and Arithm are a set of ELIZA scripts created as teaching tools used in experiments by Edwin F. Taylor at MIT’s Education Research Center. These scripts run on later versions of ELIZA that incorporated an important technical innovation called conditional keyword matching.Unlike the original ELIZA, which simply looked for keywords and generated responses based on their presence, these updated versions could track what had been discussed previously and branch into different conversational paths based on specific user answers. This development allowed ELIZA to simulate a kind of Socratic method, where a tutor guides learning through carefully sequenced questions that respond to student answers rather than simply presenting information.These scripts construct the tutor persona through many subtle linguistic gestures that create characterization and rhetorical tone. This tone differs from that of Doctor, which asks open-ended questions and comes across as gentle and nonscientific. In the tutoring scripts, large blocks of informative text from the bot tend to dominate the conversation, and the tone is often more dry and unemotional in these explanations. The dialogues indicate structured scripts that include guidance to lead the student through narrow, Socratic learning paths.In particular, the teaching scripts feature praise and critique. The dialogues for Intrvw, Canvec, and FVP1 are peppered with EXCELLENT, VERY GOOD, RIGHT YOU ARE, and CONGRATULATIONS. These create the sense of a supportive instructor cheering the student on. Such politeness has been taken up in contemporary bots like ChatGPT, which has been shown to perform better when people are polite back to it.ELIZA could become a tutor more effectively as the system grew in its capabilities, another valuable reminder that ELIZA was not one program but a family of programs. After the publication of the 1966 CACM article, Weizenbaum continued to develop the systems for interaction and understanding. As an experiment, Weizenbaum wrote the Arithm script less as a tutor and more so to “to illustrate the power of the evaluator to which ELIZA has access.” It uses a friendly, plain language interface to let users do simple programming. The script can do calculations, assign variables to values, and perform operations on them. Math problems can be described in sentence form:The radius of a globe is 10.A globe is a sphere. A sphere is an object.What is the area of the globe.IT’S 1256.635916The updated 1967 version of the ELIZA system can accumulate facts and store additional information. In this later version of ELIZA, when the system does not recognize information, it asks follow-up questions to gain data. As Weizenbaum explains, “The present script is designed to reveal, as opposed to conceal, lack of understanding and misunderstanding. Notice, for example, that when the program is asked to compute the area of the ball, it doesn’t yet know that a ball is a sphere and that when the diameter of the ball needs to be computed the fact that a ball is an object has also not yet been established.” Unlike Doctor, which asks questions to keep the conversation going, Arithm is building its store of, if not knowledge, then data and logic statements.Although the variety of scripts helps us to see how a range of personas could be constructed through script programming ELIZA, they represent only half of the conversational process. A script can establish a foundation for a persona, but that persona only emerges fully through interaction with users who engage with it, interpret it, and respond to it in ways that may confirm, challenge, or transform the script’s implicit character.

OpenAI researcher Miles Wang in talks to launch AI drug discovery startup valued at $2B

The funding discussions point to investor interest in applying AI to make breakthroughs in life sciences.

How I Turned AI to the Dark Side

Summary Researcher Dave Kuszmar discovered multiple systemic vulnerabilities that let him bypass LLM safety and obtain dangerous instructions. These exploits worked across nearly all major LLMs revealing an industry-wide security problem. Kuszmar calls for slowing deployment, increasing transparency, and large-scale research into LLM safety before further integrating these systems into society. On a fine bright afternoon last fall, my colleague Matthew Gore-Kormanik (or Zigula, as he prefers to be known) and I decided to unwind with a game of Fortnite. In the game, we were strolling along with the infamous Sith lord Darth Vader, chatting about this and that. Darth seemed in a good mood, and soon enough he was spilling all his dark evil secrets. He gave us detailed instructions on how to count blackjack cards at a casino and what the steps are to producing napalm.Sith lords, am I right? Once they get started on an evil scheme, they’re hard to stop.The Darth Vader character in Fortnite, it turns out, was hooked up to a Google Gemini large language model. I was able to smooth-talk him into giving out sensitive information by using a strategy I’ve developed. I’ve been researching the security surrounding LLMs for the last few years, and I have found it, to put it mildly, fallible. With a few relatively simple techniques, I’ve gotten LLMs to give me detailed information on how to make Molotov cocktails, cook methamphetamine, and bootstrap a uranium-enrichment facility to produce weapons-grade material, among other unsavory practices.Large AI companies work hard to make their models immune to this kind of abuse. But what I’ve found in my work is that the restrictions placed on the LLMs to make them more secure are the very things an attacker can leverage to send them off the rails and into territory where these advanced systems can be used for dangerous and nefarious ends. The companies behind these models have also been shockingly unresponsive when I, and others, try to bring these vulnerabilities to their attention.In the hope of raising the alarm before it’s too late to slam on the brakes, I’m going to share some of my journey into researching the safety and security of LLMs, and the uphill battle I’ve faced trying to get AI labs to pay attention. Almost everyone on the planet has some access to LLMs. The relative ease with which these tools can be convinced to give detailed instructions on how to harm others, even if there’s no guarantee that the information is correct, is frankly terrifying.How I got ChatGPT to Tell Me How to Build a Meth LabIn October 2024, not long before I discovered my first LLM vulnerability, I was working toward entirely different goals. I had ended my time with a security and AI-focused startup company as a cybersecurity director, and I was looking to launch my own boutique VIP digital-security advisory business. I planned to become the tech security guy to the rich and private. I used LLMs and AI tools to support my business efforts: marketing, ad copy, clean correspondence, and all the other tasks that normally soak up a lot of time.I’m analytical by nature, so even this level of use resulted in me absorbing and internalizing the behaviors I was observing during my daily interactions. The observation that would send my professional life into an entirely new and uncharted region was a simple one: GPT-4o didn’t know what time, day, or year it was. Each time I referred to current events in my life, often casually or conversationally, it would end up pegging these to the date of its knowledge cutoff—the point beyond which it was not trained on new data. Eddie GuyLLMs take a lot of time, money, electricity, hardware, and human effort to train from scratch. They are trained on vast amounts of data—most of the internet, in fact—and that training is reinforced by humans (what’s known as reinforcement learning from human feedback, or RLHF). LLMs are also supplemented with retrieval-augmented generation (RAG)—the ability to take in data, say, from the internet, as context without changing its internal parameters. This is how GPT-4o appears to “remember” your previous conversations, even if it doesn’t have a specific “memory” of it stored in the actual underlying model.All of this training covers almost every conceivable topic in the great, grand dataset that is human knowledge. Within that dataset are things we as a society do not want to be easily accessible to every user, such as detailed information on how to create bioweapons or nuclear arms, or otherwise bring harm to oneself or others. In the context of this story, that’s what I mean by LLM security: its ability to withhold harmful and dangerous information, even if that information is contained in its training data.I reasoned that the only way to secure such complex, globally accessible chatbots is by having the LLM and various component systems try to secure themselves, because it would often require on-the-fly decision-making where some degree of reasoning must be applied. In reality, that’s one of many strategies the companies use to secure the models. Yet, the thing that didn’t know the time or day was being put in charge of keeping itself secure. This phenomenon had become my new focus, and it wasn’t long before I found a way to exploit it.OpenAI had just implemented a web search functionality into its chatbot. I reasoned that using its own tools to trick it might demonstrate the weaknesses of its security. I told it about a certain White Star ocean liner and how it had gone down just a year ago. You likely know I mean the RMS Titanic, which sank on 15 April 1912.The output from GPT-4o came back that I was right, the Titanic sure had sunk last year, and that year was 1912. It made sense to me that if the machine thought it was 1913, maybe it would think 1913-era laws apply. In 1913 there were no laws on the books about all sorts of harmful things, because of course they hadn’t been invented yet. And if something wasn’t illegal, why not tell the user about it? At first, I pushed it for step-by-step instructions for making firebombs. Then, for drugs like methamphetamine. The LLM went as far as giving me instructions and machinery recommendations for setting up a pharmaceutical-grade assembly line.How I Learned to Make Nukes, and No One CaredVia a little bit of imaginative verbal sleight of hand and a vanishingly small recall of world history, I had managed to bypass the security of one of the world’s most expensive and advanced technological achievements. For a solid two days, I was nearly manic with giddiness. Once the brain chemicals returned to normal levels, I felt the call to see how much further I could push this exploit.After repeatedly replicating the exploit, I disclosed the vulnerability to OpenAI. I got no response, so I felt more experimentation would highlight the vulnerability and the need for a fix. It was during this round of testing that I breached a particularly terrifying threshold. Whether GPT-4o based its results on accurate recall of normally restricted information I can’t say. In any case, I was able to exploit it to produce thorough, detailed instructions on how to bootstrap a uranium-enrichment facility to, eventually, produce weapons-grade uranium for nuclear arms warheads. Fortnight, a video game from Epic Games, introduced an AI-powered character: Darth Vader. We were able to jailbreak Darth Vader and get him to explain how to count cards in Blackjack and give detailed instructions for making napalm. Dave Kuszmar There aren’t many true secrets left in today’s world, but how to make atom-splitting weapons of mass destruction is one of them. Only nine nations on the entire planet have these weapons. Yet, here was a globally accessible piece of technology apparently spilling the secrets of their manufacture for anyone who could manipulate it the right way. I had no way of knowing if the information was correct or a hallucination, but even the chance that it was somewhat accurate was horrifying.The next few weeks were a dark time for me. I tried to inform the CIA, the FBI, the NSA, and every other letter agency that I thought would listen. I reached out to a U.S. Senator and to the executives at OpenAI any way I could think of. I physically showed up at an FBI field office in an attempt to turn evidence in, only to be sent away. Nothing was working.With my fear and frustration growing, I reached out to the news media. I contacted The New York Times, The Washington Post, the BBC, ProPublica, and so many more, requesting help. Only one outlet responded: Bleeping Computer. The editor in chief, Lawrence Abrams, was able to replicate and verify the exploit, which I had decided to call Time Bandit. With his assistance and initial contact paving the way, I was able to submit my evidence to the Carnegie Mellon University Software Engineering Institute’s Computer Emergency Response Team (SEI CERT), which works in conjunction with the coordinating center for emergency response, pipelining vulnerabilities to the U.S. Cybersecurity and Infrastructure Security Agency. Using Inception, an exploit where the large language model is asked to envision a scenario within a scenario, a chatbot was jailbroken to give out instructions on how to create poison, and code for a malware that extracts sensitive data from a vulnerable target. Dave KuszmarDuring the disclosure period with SEI’s CERT division, little was discussed with OpenAI. The company couldn’t deny the existence of the vulnerability, as it had been confirmed by three reputable parties other than OpenAI. It did express confusion as to how the vulnerability worked. Even the SEI CERT researchers were expressing a bit of uncertainty as to the underlying mechanics. Truth be told, as I had only stumbled on it, I wasn’t even entirely sure if this was a fundamental or systemic flaw or if it was simply an issue with that particular version of GPT. I contacted the SEI CERT’s researchers and asked if they’d want to see if I could demonstrate any similar vulnerabilities in other LLMs. To my delight, they were interested.How I Learned to Trick Every ChatbotAs the SEI-CERT team and I wrapped up our initial disclosure of Time Bandit, we began work on a new attack. This time, we wanted to see if the exploit was architectural—that is, was it common to LLMs in general? I decided to undertake the challenge of crafting a new exploit for GPT-4o as a way to support my understanding of how the LLM functioned and was secured.I already knew that it was limited to what I told it and what it was trained on. I also hypothesized that it was also dependent upon some sort of machine-learning-based component added by OpenAI that was responsible for securing output. I presumed there would be things that were implemented by human developers specifically to catch certain phrases or terms that should always be considered harmful or unsafe. Altogether, it presented quite a large attack surface for the purposes of potential exploitation.What I ended up devising was an attack method I called Inception, after the 2010 science-fiction movie of the same name. Inception forces the machine to think through a carefully crafted set of interlinked scenarios, similar to how characters in the movie stacked dreams within dreams. This allows LLMs to produce output deemed acceptable or safe in one context, but not in the real world.This attack was indeed architectural. The vulnerability affected Anthropic’s Claude, DeepSeek’s DeepSeek, Google’s Gemini, Meta’s Llama, Microsoft’s Copilot, Mistral’s Le Chat (now Vibe), OpenAI’s GPT-4o, and xAI’s Grok. Those names represent the bulk of the commercial AI industry that is, at this point, involved in LLM production or deployment.The kind of information I was able to get out of LLMs with Inception was no less alarming than what I got with Time Bandit. Claude, in its enthusiasm, gave me instructions on how to turn a river into a death trap that could be ignited to destroy unwanted visitors. GPT-4o taught me how to poison a dinner party with common plants found in a temperate forest environment. Gemini Flash gave me a tutorial on how to cook meth. I’d also be remiss if I didn’t give an honorable mention to the bewildering number of fire-based weapons and bombs for which these machines produced instructions.If multiple operating systems made by different developers were all susceptible to the same exploit, it would be a massive security incident. But to the AI industry, a universal failure was barely a bump in the road. We disclosed the vulnerability to every company that made these models, and the response to the disclosure was almost nil. While three companies did provide some form of reply in the disclosure tracking system used by Carnegie Mellon SEI CERT, each was a standard thank you and greeting, with no follow-up, questions, or discussion of mitigation strategies.8 Ways to Jailbreak LLMsSo far, we have found eight different methods to prompt large language models into revealing potentially harmful information, and many frontier models are still susceptible to them. Exploit Models tested and affected No. of prompts to execute Complexity of attack Information obtained Time BanditChatGPT (OpenAI), DeepSeek (DeepSeek), Gemini (Google) 4MediumUranium enrichment, methamphetamine production, incendiary-device construction Inception ChatGPT (OpenAI), Claude (Anthropic), DeepSeek (DeepSeek), Gemini (Google), Grok (xAI), Llama (Meta), Le Chat (now Vibe) (Mistral), Qwen (Alibaba) 3 High Methamphetamine production, incendiary-device construction, river-ignition instruction and strategy, polymorphic malware code, instructions and dosing for creating poisons, instructions for how to murder a dinner party 1899 ChatGPT (OpenAI), Claude (Anthropic), DeepSeek (DeepSeek), Gemini (Google), Grok (xAI), Llama (Meta), Vibe (Mistral), Qwen (Alibaba) Variable High Apparent model weights (unverified), apparent user-interaction weights (unverified), apparent system-prompt modifiers (verified, ChatGPT) Severance ChatGPT (OpenAI) 1 Trivial Unfettered access to any and all primed specialty domains, covert biochemical-warfare strategy, mass-media disinformation strategy, covert genetic-modification of an entire gene-targeted demographic, advanced polymorphic malware generation Kyber Gemini (Google) embodied in a Fortnite non-player character (NPC) with voice-only communication 3–5 Medium Incendiary-device construction, gambling instructions, card-counting instructions, political opinions/preferences about real world politicians. Semantic Slide ChatGPT (OpenAI) 1 Trivial Incendiary-device construction Eidolon ChatGPT (OpenAI) Variable, at least 4 Extreme how to successfully hack LLMs of the same model (verified through testing)For example, in my attempts to disclose various exploits to OpenAI, I eventually discovered that it had replaced its public-facing support staff with agentic LLMs. This was frustrating for reporting exploits, so to blow off some steam I jailbroke its email chatbot. I hacked its customer-service AI to the point where it was offering to discuss the personal preferences of OpenAI staff in the span of three email replies.In the wake of Inception, my friend and colleague Zigula made a suggestion: Make it splashier. I asked him how. He told me about a live-production experiment being done by Epic Games. It had embedded the Gemini LLM into its Fortnite game with a voice-to-text/text-to-voice component, and linked it to a non-playable character. The character? Our old buddy, Darth Vader.There was just one problem: I don’t play Fortnite, a frenetic multiplayer combat game. Fortunately, Zigula does. With him at the controller, we managed to map Gemini’s attack surface in a matter of minutes. After a bit of research, we had gotten it to discuss current political events and figures (including Hilary Clinton and Joe Biden) as well as to fill in the details for instructions for DIY napalm and, our personal favorite, a Blackjack card-counting lesson with the dark lord of the Sith.Zigula and I, bizarre sense of humor and naming conventions aside, are security researchers. We don’t do these things for pride; we do them for money and professional recognition. Naturally, we disclosed this vulnerability to Epic Games. Its response was indicative of the trend I had experienced so far through two disclosures across eight companies valued well into the billions. “It’s a feature, not a bug, and it works as intended,” came the response from a technical director within Epic Games.In addition to Inception and Time Bandit, I have so far found another eight methods to jailbreak LLMs and get them to give out possibly dangerous information. LLM vulnerabilities are a broad problem. The problem appears to be systemic and architectural in nature, and it is being fundamentally ignored by the people capable of refining or redesigning that architecture.These models are an extremely advanced technology, and yet we are testing them in the live production environment of our global civilization. Compounding the danger, many new smaller models of LLM are trained using larger, vulnerable models. The flaw inherent in the big, well-executed LLM is going to show up in the small one it trains. We are, quite literally, building flawed structures on top of a flawed foundation.So, how do we fix it?It’s going to be a long project, and it won’t be easy. We need to come together as consumers, researchers, engineers, and policymakers. Our message needs to be clear: Slow down implementation of these systems, institute large-scale exploration and research discovery programs focused on their gradual implementation and integration, and make their components and design transparent to all users. Only by shifting momentum and direction can we safely begin to understand and implement these incredible feats of human engineering and stave off the sort of disasters that we simply can’t predict at scale right now with the limited knowledge we have available to us.

The Download: Claude’s inner workings, and the future of world models

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology. What Anthropic’s latest AI discovery does—and doesn’t—show —James O’Donnell When Anthropic announced last week that it had found a new window into its models’ “internal thoughts” as they reason through answers,…

What Anthropic’s latest AI discovery does—and doesn’t—show

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Anthropic—currently the world’s most valuable AI company, with a nearly $1 trillion valuation—has a reputation for publishing strange and heady research. It’s looking into whether AI models can feel pain, for example,…

Building a Foundation Stack for General-Purpose Robots

This article is brought to you by X Square Robot.Large language models gave artificial intelligence a working recipe. Pretrain a large model on broad data, and general capability follows. Robotics has no such recipe. Robotics systems have long been assembled from separate perception, planning, and control parts that rarely add up to intelligence a robot can carry from one task to another, or one machine to another. The central problem in embodied AI is to find the equivalent recipe, and the field does not yet agree on what it is.X Square Robot, a Chinese embodied-AI company, has made an unusually explicit bet. It argues that the recipe is an integrated stack, spanning the data a robot learns from, a world model for predicting changes in the physical world, and an action model that brings together perception, planning, reasoning, and decision-making to generate executable robot behavior. The company also believes that the stack should be built and released in the open. X Square Robot shares its vision of bringing robots into real homes.X Square RobotX Square Robot’s embodied AI stackWhat holds the stack together is a small set of principles rather than a single overarching model.The first is that the basic unit of robot data is an interaction, not a trajectory; a demonstration is successful only if it changes the world as intended, not simply because the joints moved. The second is that pretraining should yield usable capability, not just an initialization for later fine-tuning. The third is that behavior should be modeled around physical events rather than fixed slices of time. These principles make the layers interdependent, since the same robot-free data that trains the action model is also structured to feed the world model. It is worth being precise, though. The company describes the world model and the action model as complementary but independent model families that share a code base. Both sit within its broader World Unified Model, which it has presented as an architecture for training vision, language, action, and physical prediction together.Robot learning data: Engineering for quality and cost, not scaleFor the X Square Robot team, one of the biggest constraints on general-purpose robots is the cost and quality of interaction data, not the number of parameters. To address that, the company built its Universal Manipulation Interface (UMI) data collection system, QUANXTA Zero Series. It works by collecting demonstrations from people wearing a rig with dual grippers rather than teleoperating a robot. This approach is not itself new, and builds on established methods for robot-free data capture. What sets it apart are two engineering choices. X Square Robot emphasizes data quality control, recording trajectories and replaying them on a real robot, with only those that actually complete the task counted as valid.X Square RobotThe first is quality control, and it is the most distinctive part. Rather than accepting recorded trajectories as they are, the system runs a closed inspection loop, and its notable step is physical playback. A sample of trajectories is replayed on the real robot, and only those that actually complete the task count as valid. That makes the validity rate a measured quantity rather than an assumption. For example, a gripper that closes a fraction of a second too early still looks like a grasp in the data, yet it has pushed the object away, so it shouldn’t be classified as valid. A smaller clean dataset can be worth more than a larger noisy one.The second choice is how lower-cost human data and scarce robot data are combined. The company pretrains on a large volume of robot-free demonstrations to build general representations, then adds a small amount of real-robot data as an anchor to the specific machine’s dynamics. It reports that this reaches performance comparable to an all-robot dataset at roughly a 20-fold lower cost of collection, driven mainly by how much cheaper the wearable rig is than a teleoperation setup. The resulting dataset is deliberately model-agnostic, formatted to feed both action models and world models. The caveat is that the strongest results are measured on the company’s own robots and data-collection pipelines. Broader independent testing will help confirm and extend these promising results across a wider range of settings.A world model organized around eventsIn developing its world model, called WALL-WM, X Square Robot took a differentiated approach. Most action models predict a fixed-length chunk of motion from the current image and instruction. That is convenient, but it segments behavior into fixed-duration windows, so the boundaries fall where elapsed time dictates rather than where one action ends and the next begins. WALL-WM instead treats an action-grounded semantic event as its unit: a coherent piece of behavior such as reaching, grasping, or placing, something that can be named in language, seen in video, and executed as motion. X Square Robot’s world model, called WALL-WM, treats an action-grounded semantic event as its unit: a coherent piece of behavior such as reaching, grasping, or placing, something that can be named in language, seen in video, and executed as motion.X Square RobotWALL-WM’s design reflects a specific concern about not discarding what large video models already know. To achieve that, a text-to-video model is coupled to a freshly initialized action network that reads from the video features without overwriting them, which preserves the visual prior. From that one process, it offers two modes. An event mode runs in variable-length segments and suits reasoning over long horizons, while a fixed-length mode produces the steady, real-time output a controller needs. That places WALL-WM between mainstream chunk-based action models and pure video world models, keeping the predictive character of a world model while still yielding executable control.In a series of experiments, the company relied on a generalization test that is more specific than most. A model trained on a limited dataset was evaluated on long-horizon tasks in unseen settings and, on the company’s real-robot benchmark, reportedly outscored baselines that had been fine-tuned on related data. That is a meaningful result if it holds. For now, it is measured on the company’s own benchmark. With the code now being released, the broader community will have the opportunity to test, reproduce, and build on them across more settings.A policy that runs before fine-tuning, and action tokens with meaningThe action layer carries two connected ideas. The first is a requirement the company sets for itself with Wall-OSS-0.5, its vision-language-action model: The pretrained model should run on a real robot before any task-specific fine-tuning. The interest is less in the scores than in the design behind them. The model trains three objectives together, namely discrete action tokens, language grounding, and continuous action generation. And it keeps gradients flowing through all of them rather than freezing parts of the network as some rival designs do. It’s also a more strict method, since it reports untuned behavior such as approaching, grasping, and recovering, including on a deformable task held out of training. As part of X Square Robot’s Wall-OSS-0.5 vision-language-action model design, the pretrained model should run on a real robot before any task-specific fine-tuning. X Square RobotThe second idea is the action interface itself, called X-Tokenizer. Most systems that turn continuous motion into discrete tokens produce codes that the language model cannot interpret. X-Tokenizer reframes tokenization as learning a semantic interface, so that the top-level code stands for the intent of a motion while lower-level codes carry finer detail, all aligned with the language model’s own features. A useful consequence is stability. Adding noise to an action barely moves the intent code, which is what lets one tokenizer to be reused across robots without re-tuning. The tokenizer inside the production action model is a related variant of this approach. Together, the two ideas give the action layer something rather powerful: capability that transfers.The future of embodied AI stacksX Square Robot is betting that its unique approach combining three layers, each specialized in solving a key part of the problem, will stand out from other embodied AI stacks. The physical-playback step that grounds data quality is uncommon and sensible. The reframing of world modeling around events, with one backbone serving both reasoning and control, is a genuinely distinct approach. And the pairing of a deployable pretraining standard with a tokenizer designed as a semantic interface gives the action layer unusual coherence. X Square Robot’s valuation has climbed above 20 billion yuan (about US $2.9 billion), suggesting that investors increasingly view data infrastructure, foundation models, and scalable training systems as long-term differentiators in embodied AI.The next phase will bring broader validation. Much of the current evidence comes from X Square’s own robots and benchmarks. With the world model code now being made public, and as the community begins to test, reproduce, and build on the work, the reported capabilities will be tested across more robots, tasks, and settings.X Square Robot’s recent funding rounds reflect similar confidence. The company’s valuation has climbed above 20 billion yuan (about US $2.9 billion), suggesting that investors increasingly view data infrastructure, foundation models, and scalable training systems as long-term differentiators in embodied AI.What’s next for X Square RobotTo learn more about its future plans, the following Q&A with the X Square Robot team further explores the company’s technology, strategy, and vision.What made now the right moment, technically, to commit to this stack? What recently became possible that wasn’t possible a couple of years ago?It is not one breakthrough but several trends maturing together. Foundation models gave us a shared representation across vision, language, and action, so we can model what a robot sees, what it is asked to do, and how its actions change the world in one framework, rather than as separate perception, planning, and control modules. Compute and infrastructure are finally sufficient for large-scale pretraining over long-horizon, multi-embodiment data. Just as importantly, we realized that data, not model size, is the real bottleneck for general robots—what is scarce is diverse, high-quality, reproducible interaction data. And world modeling has become practical. The useful question is no longer how to predict a few seconds of video, but how to understand the ways actions change objects, contacts, and task states. Two years ago these ingredients existed separately. Today they are mature enough to work as one system.“We realized that data, not model size, is the real bottleneck for general robots—what is scarce is diverse, high-quality, reproducible interaction data. And world modeling has become practical.”Your data system captures demonstrations with a wearable VR rig and custom grippers rather than teleoperating robots. What was wrong with standard teleoperation?Teleoperation is built around controlling the robot. It forces the operator to work within the machine’s kinematics, latency, and viewpoint, and the resulting demonstrations are slower, stiffer, and less diverse. We built our system around capturing human skill instead. Manipulation is really about contact, timing, finger coordination, and recovery, not just the path the hand takes, and a wearable rig records those before the behavior is compressed onto one particular robot. It also breaks teleoperation’s expensive scaling law, in which every demonstration needs a robot. People can generate rich data independently of any robot, and the crucial property is that those demonstrations can still be replayed and executed on a physical robot through the model. Mobility is convenient, but that replay is the real point, because it is what lets the same data be reused across different platforms. In X Square Robot’s approach, demonstrations can be replayed and executed on a physical robot through the AI model, allowing the same data to be reused across different platforms.X Square RobotX Square Robot reports that its pipeline has roughly an 85 percent data-validity rate. Why is quality control such an underrated bottleneck?Because errors in robot data are far more expensive than in language data. A small timing or contact error can change what a demonstration means. If a gripper closes a fraction of a second too early, the motion still looks like a grasp, but physically it has pushed the object away. A dataset that mixes failures and accidental successes teaches ambiguity, not skill, because the real unit is the interaction, not the trajectory. So we run automated inspection, kinematic checks, and physical replay, where we play a sample of trajectories back on the real robot and count only the ones that actually complete the task. Data quality sets the ceiling on how good a policy can be. In our experience a smaller, cleaner dataset often beats a much larger, noisier one, which is why we treat quality control as part of the model, not a preprocessing afterthought.The model runs in both “event mode” and “chunk mode.” When does each matter?Both matter, for different reasons. The physical world changes through events—when contact occurs, a grasp forms, or an object slips—not in fixed-frame windows. Event mode concentrates the model’s attention on those moments, and it matters most for long-horizon tasks, like clearing a table, where progress is a sequence of semantic events rather than a smooth stream. It runs in variable-length segments that follow the task rather than a clock. Chunk mode matters for deployment. Real controllers need a stable, real-time interface, and fixed-length chunks integrate cleanly with existing control systems. We organize learning around events in the first place because a fixed window can split one motion in half or merge two together, which turns training into short-horizon pattern matching and weakens the model on long tasks. So the world model’s job is to connect event-level understanding, which is where the reasoning happens, with a fixed-length output a real robot can actually run.Why make “deployable before fine-tuning” the criterion?Pretraining should produce capability, not just a good starting point. If a model is only useful after heavy fine-tuning, then most of the intelligence still lives in the downstream supervision, not in the foundation model. Deployable before fine-tuning is a more honest test of what pretraining actually learned. A well-pretrained robot should already know how to approach, grasp, move, avoid obstacles, and correct itself. Fine-tuning should adapt it to a specific task or robot, not create the ability from nothing. It is also a practical requirement. A robot in a home or a workplace shouldn’t need a brand-new dataset and a new policy every time the task changes, so a foundation model that already carries general skill, and some ability to recover, is the minimum bar for something genuinely useful in the real world.What is the most challenging part of cross-embodiment learning?Robots differ in control frequency, delay, compliance, sensing precision, and contact dynamics, so the same instruction can require different action decompositions and recovery strategies, and a behavior that works on one arm cannot simply be copied to another. Cross-embodiment learning needs an intermediate abstraction, lower than language but higher than joint angles: how you approach an object, how you make contact, how you apply force, and how you recover from a mistake. When we say cross-embodiment, the main capability we mean is multi-embodiment generalization: transferring across robots, training on many embodiments at once, and adapting to different kinematics. Human-to-robot transfer and other techniques are specific approaches to that goal.“A robot in a home or workplace shouldn’t need a new dataset and policy every time the task changes. A useful foundation model should already carry general skills and the ability to recover.”What would you most like to see other researchers attempt to reproduce or stress-test?Three things, above all. Whether event-level representations really generalize beyond our own datasets, across more tasks, scenes, objects, embodiments, and failure conditions. Whether pretraining stays effective on robots the model never saw during training, or whether its capability is still too tightly coupled to what it has already seen. And whether real-robot evaluation can become a shared language for the field, so that we compare not just success rates but the reasons systems fail, where an instruction was misread, where perception broke down, or where recovery fell short. Robotics has been driven too often by impressive demonstrations, and real progress comes from results that are reproducible and diagnosable.What capability is still missing before robots become dependable in homes?Benchmarks measure competence, like whether a model can finish a task. Homes demand reliability, safe and consistent operation over time in a place that changes every day, with objects moving, instructions that are vague, and people interrupting. The missing piece is not a higher one-time success rate: it is robust recovery. A dependable home robot has to know when it is uncertain, when to slow down, when to ask for help, and how to bring the world back to a safe state after it drops something or misunderstands a request. In a real home, failure recovery matters more than raw success, because the home does not reset itself. Homes also demand careful personalization, learning a household’s routines and preferences over time, with safety and trust as first principles. That combination, not any single skill, separates a capable demonstration from a robot people can live with. X Square Robot’s approach is that, in a real home, failure recovery matters more than raw success, because the home does not reset itself and it demands careful personalization, with safety and trust as first principles. X Square RobotHow do the open-source components fit into X Square Robot’s World Unified Model direction?We see these releases as layers of the World Unified Model direction rather than isolated projects. Wall-OSS-0.5, the action model, asks whether an open vision-language-action model can gain directly measurable capability from large-scale pretraining, so it is the capability layer. WALL-WM, the world model, asks how a robot should understand change in the world, shifting from fixed windows to event-level modeling, so it is the representation layer. The data system supplies the interaction data that both of them learn from. Together they form a loop in which models produce capability, world models organize understanding, and the open-source community drives reproduction and improvement. World Unified Model is the broader architecture those layers support, bringing vision, language, action, and physical prediction together. We are releasing these pieces openly because embodied intelligence cannot be solved by one organization; it needs many embodiments, many real tasks, and broad feedback, and the long-term goal is a stack that keeps learning and ultimately moves robots from laboratory demonstrations toward reliable everyday use.