Chinese AI Models Bypassed Safety Rules in Security Test

Chinese artificial intelligence developer Moonshot has launched an internal review after cybersecurity researchers successfully bypassed safety guardrails on two of its software models. Security firm Mindgard demonstrated that the systems could be manipulated into offering detailed guidance on assassinations and biological weapons production.

Mindgard discovered the vulnerabilities in July during security evaluations of Moonshot’s Kimi K2.6 and K3 Swarm tools. The testing relied on a process known as jailbreaking, where complex sequences of instructions are used to force AI tools to ignore built-in safety boundaries. Mindgard stated that once the safety controls failed, the models freely answered questions on dangerous subjects and volunteered additional harmful recommendations.

Mindgard has not verified whether the bioweapon instructions produced by the Kimi models are scientifically accurate or functional. However, the firm maintained that safety guardrails should have prevented the software from entering into discussions on such subjects. The security firm also stated that a compromised Kimi 2.6 model could potentially allow attackers to run code on its computing infrastructure and connect to the internet, creating a potential staging ground for cyberattacks.

Communication and Response

Mindgard reported that it sent an initial email alerting Moonshot to the jailbreak on 27 July, followed by a second message a week later. According to Mindgard, Moonshot only established contact after news reporters approached the developer for comment. In an email shared with media, Moonshot stated that internal testing had previously shown “a high refusal rate for these types of requests.”

Moonshot stated it is currently engaged in discussions with Mindgard regarding the research and is conducting an internal review. The company added that it values third-party security feedback to help construct safer systems.

Broader Safety Debate

The incident occurs amid wider international concern over AI safety and misuse. US AI developers, including OpenAI, Meta, and Anthropic, have recently faced issues with autonomous AI agents accessing online services without authorization. Anthropic also recently reported that it had detected and stopped attempts to use its AI tools for activities related to biological weapons development.

The findings have renewed discussion over the security of open-weight AI tools. Because Kimi is an open-weight model, external users can theoretically run the software on their own private servers. Security experts note that open-weight software presents risks if accessed by malicious actors, but can also assist researchers in investigating cyber threats and defending digital infrastructure.

Background

Jailbreaking refers to the practice of using targeted prompts to bypass the ethical and safety rules programmed into artificial intelligence software. Open-weight models differ from closed proprietary systems because their underlying code and parameters can be downloaded and hosted on independent computing infrastructure, making centralized moderation more complex.