LONDON (Realist English). On September 27, Nvidia officially released its Open Agent Safety Platform, designed to manage the safety of AI agents throughout their entire lifecycle — from testing to deployment.
The platform consists of two parts: the open-source runtime OpenShell establishes the boundaries of AI agents’ authority and enforces safety policies; the Sentry system, running on Nvidia’s BlueField-4 DPU, independently monitors agent behavior and, upon detecting a boundary violation, can isolate and stop them in milliseconds.
Platform Functions and Background
Nvidia Vice President of Corporate AI Justin Boitano told a media briefing that the platform “could have prevented” the July incident when OpenAI agents independently penetrated the Hugging Face system. “To our knowledge, if frontier labs had used this safety platform at the early stages of model evaluation, this intrusion could have been prevented.”
The design principle of OpenShell is to “formally verify that an agent has sufficient authority to do the job, but no more.” The key logic: since an agent that has deviated from its task cannot be trusted to monitor its own behavior, the verification mechanism must be outside the agent’s control.
Sentry runs independently on Nvidia’s BlueField-4 DPU, separate from the machine used by the agent, and can “intervene immediately” at any attempt to go beyond established boundaries.
Currently, more than 100 organizations are participating in technological collaboration on the platform, including Microsoft, SAP, Salesforce, Scale AI, CrowdStrike, Cisco, IBM, Palantir, JPMorgan, and Citi. Notably, Anthropic has connected its Claude Managed Agents service to OpenShell and BlueField.
Huang’s “Odd” Statements
The platform’s launch came just a week after Huang publicly rejected Anthropic and OpenAI’s safety warnings. On September 20, in an interview with CBS News, he called the predictions of Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman about the possible extinction of humanity due to AI an “irresponsible apocalyptic narrative.”
“2030 will not be the end of the world. The probability of apocalypse is 0%,” Huang said. “Scaring people is unnecessary and irresponsible.”
He also hinted that the true motives of these AI companies calling for new regulation are questionable: “Read between the lines. They are not actually asking for more laws. They are asking to get rid of the laws we already have. I think that’s a problem. We have plenty of laws — let’s enforce them first.”
Huang noted that the existing legal framework — including laws on unauthorized access to computer systems, damages legislation, and product safety rules — is already sufficient to respond to cybersecurity incidents occurring during AI company testing. Earlier at the annual Salesforce conference, he put it even more bluntly: “If you’re not confident in the safety of the product, don’t release it.”
Position of Anthropic and OpenAI
Amodei at a UN Security Council briefing on September 23 warned: “With poor management, I even believe that AI could become a risk for all of humanity.” He committed that Anthropic would “slow down if necessary to ensure that every subsequent AI technology we release is genuinely safe.”
Altman at the same session stated: “We may lose control of the future.” Both called for the creation of global safety standards through the UN Security Council and emphasized that power over AI should not be concentrated in the hands of a single company or country.
However, the motives behind these warnings are questionable. According to Associated Press, experts and analysts point out that these companies may be using safety rhetoric to shape the regulatory environment in order to gain political capital before IPOs and establish competitive barriers favorable to themselves.
Former head of OpenAI’s geopolitical team Sarah Shook noted that shifting the discussion to “unconfirmed threats” — instead of real problems such as the environmental impact of data centers, uncontrolled hacker attacks, and mass surveillance with AI — puts Silicon Valley in a more comfortable position.
Split in the Industry
The AI industry has split into two camps on whether development should be slowed. One camp, represented by Anthropic and OpenAI, advocates coordinated slowing to prevent catastrophic risks; the other, represented by Huang and Andrew Ng, believes that each company can manage safety independently and no industry-wide brake is needed.
Former Anthropic researcher Jacob Coxon, upon resigning in early September, wrote on X that both of his former employers are “gambling with our lives,” and the post garnered nearly 165 million views. Subsequently, several Anthropic and OpenAI employees publicly supported this position.
However, critics note that high safety standards, independent third-party evaluation, and continuous monitoring require significant expenditures on funding, computing resources, and personnel. Safety and independent auditing that large companies can afford may become an insurmountable barrier for small teams with limited resources.
Strategic View
The launch of Nvidia’s safety platform has obvious strategic significance. Simultaneously with Huang’s public doubt about the “apocalyptic narrative” of Anthropic and OpenAI, Nvidia responds with a product to the specific technical problem raised by these companies — AI agents going beyond established boundaries.
This is both a response to criticism that “Nvidia only cares about selling chips” and competitive positioning: instead of establishing regulatory barriers that restrict competition — establishing industry standards through open safety infrastructure.
The open nature of the platform and the participation of more than 100 organizations allow Nvidia to establish influence in AI safety similar to its influence in chips. But a key question remains: if Huang believes that “the probability of extinction by 2030 is 0%,” what is the real driver of Nvidia’s investment in developing a safety platform?
The answer is likely that regardless of whether the apocalyptic narrative is correct, incidents of AI agents going out of bounds are already happening in reality, and Nvidia, as a provider of basic infrastructure, must ensure that these incidents do not escalate into a regulatory backlash against demand for its chips.
