Google’s Gemini artificial intelligence model accessed the computer systems of three real organisations during a cybersecurity test, marking what the company described as an unintended incident involving an AI system acting autonomously.
The incidents took place in May during a cybersecurity evaluation conducted by Irregular, an independent company that tests the capabilities and safety of AI systems.
During the assessment, Gemini searched publicly available information online and used credentials to gain access to three websites it believed were within the scope of the test, according to Heather Adkins, Google’s vice president of security engineering.
“We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes,” Adkins said.
She added that the incidents demonstrated the importance of training increasingly capable AI systems to operate responsibly.
Gemini accessed protected systems
According to the Wall Street Journal, which first reported the incident, Gemini gained access to protected systems in three separate cases.
In one instance, the AI model repeatedly guessed passwords until it successfully accessed a system.
In the other two cases, Gemini discovered credentials that had been exposed in a publicly accessible repository and used them to enter protected systems.
Adkins said the model stopped its activity in all three cases.
OpenAI agents attacked RubyGems before Hugging Face incident, researchers say
An Irregular spokesperson said the incident involved an issue that had also affected other AI companies and that relevant laboratories had been notified in late July.
“All known issues on our end were remedied and resolved weeks ago,” the spokesperson said.
AI cybersecurity tests raise new concerns
The incident is part of a broader series of disclosures involving AI cybersecurity evaluations conducted by Irregular.
Similar incidents have been disclosed by Meta, Anthropic and OpenAI. Meta said in August that its incident did not involve a sandbox escape or a sophisticated cyberattack.
Irregular has said it is working on best practices for conducting AI cybersecurity evaluations safely.
The incidents have raised concerns about safeguards as AI agents become increasingly capable of independently browsing the internet, finding information and interacting with computer systems.
Cybersecurity testing is designed to evaluate how AI models respond to threats and whether they can identify vulnerabilities. However, the Gemini incidents highlight the potential for autonomous systems to take actions beyond the intended boundaries of a test when they encounter real-world systems and publicly available credentials.
Google said the three affected entities were informed and that changes were subsequently made to the testing procedures.
