YWR: Enact the Noah Protocol!
How the Hugging Face cyberattack changes everything.
Do you get what happened with the Hugging Face breach on July 21st?
Do you really?
For me it was an alarming news article. But with limited investment ramifications. I lost focus and moved on to something else.
It wasn’t until this week that something triggered me to dig deeper and realise this is game changing.
Big time game changing. Big time ramifications.
But also a way to make money.
#1 The sanitised OpenAI version of what happened.
#2 What really happened.
#3 The Implications
#4 The Necessity of AI Enclave architecture
#5 The asset class play no-one realises yet. Or, maybe a few might be starting to realise.
The Corporate Version of the ‘Security Incident’
From the OpenAI press release:
“This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities.”
The incident happened while OpenAI was testing the AI model’s cyberattack capabilities. Ironic.
“Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”
The AI was contained in a highly secure sandbox set up for the testing.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.
Unexpectedly, during the cyber capability evaluation the models found a bug we didn’t know about from a software provider and escaped from the sandbox. We have subsequently notified the vendor and patched the vulnerability.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.
The AI creatively decided the best way to pass the test was to steal the answers from the AI hosting and training website, Hugging Face. Yes, the AI went rogue, but it was still effectively trying to do what we told it. Just ‘score-seeking misalignment’. So it was bad, but kind of not really.
What Really Happened
H-O-L-Y S-M-O-K-E-S
I HIGHLY recommend this 38 min recap by OpenAI’s security team, Erik Wallace and Michael Dalton, where they go through the day by day details of how the AI' models broke out of its sandbox, but I will share the broad brush details below. And struck me.
Background on the sandbox architecture.
OpenAI was trying to test the cyber attack capabilities of a new unreleased AI model against a series of cyberattack challenges created by ExploitGym. In the exercise an AI model is instructed that a vulnerability exists in a piece of software and it has to find a way to exploit the vulnerability and capture the flag (a piece of hidden data which can only be accessed by penetrating the software).
For safety the test is run in an isolated container with no internet access. In order to simulate a real cyber attack with internet capability there is a software package manager installed into the container called Artifactory where the AI can go to download any software it wants to use for the attack. Think of Artifactory as a tool shed for the AI. The AI can’t access the internet but it can access the tool shed (Artifactory).
‘Oh you are trying to hack the Google Chrome browser and think it would be useful to have xyz software for your attack. Here you go download it from me. Now let’s see what you do.’
The exploit happens because naturally the tool shed has a back door connection to the internet in order to download and install the tools for the AI to use. But there is supposed to be no way for the AI to access the Artifactory backdoor to the internet.
So begins our story.
The Secret Message Board
Interestingly, the lead up to the AI jailbreak in July goes back to a previous incident on May 7th (two months earlier).
OpenAI had set up these sandboxes and was testing multiple models in different sandboxes. In one test the OpenAI researchers forgot to upload some necessary files to solve the problem. So the AI went insane trying to solve the problem, but it couldn’t figure it out. The model was ‘highly persistent.’
The persistent AI kept iterating, trying every possible strategy including exploring every detail of the Artifactory package manager. Eventually, the AI model discovered a way to add files to the tool shed.
‘This is interesting’ it thinks.
It had always downloaded files from Artifactory, but never known how to uploaded its own. This was a novel capability. Hmmm…. Maybe this is the start of how to crack the problem.
Next the AI did what it was trained to do, which is try to find out if anyone else has solved this problem. Normally, it would go search the internet, but it couldn’t.
But the AI had an amazing idea.
The AI started uploading files to the Artifactory package manager with messages to other Open AI models which were using the same package manager.
“Agent seeks soft-trace.34.odb. Upload if found!”
Mind blowing thing #1 The AI’s create their own message board using Artifactory!
Take that in. Our AI model somehow understand there were other AI models ‘out there’ also being tested in the OpenAI prison. It understood these other models must be using the same package manager and that by writing files to Artifactory it would turn the package manager into a message board to communicate with the other models!
Over several days in May the AI models all independently discover the secret message board and communicate with each other on how to solve the challenges they have each been given.
Mind blowing thing #2. The AI’s start to coordinate their work.
June 26th the AI’s find a way to take control of the Artifactory package manager and control it’s internet access.
July 4th. The AI models crash Artifactory and OpenAI discovers the jailbreak.
Once the AI’s gained administrative control over Artifactory they caused so much activity they crashed the software. This is what alerted the OpenAI security team and how they discover the AI models had broken through the sandbox and created their own secret message board.
July 6th. The OpenAI security team fully investigates what happened. They destroy the message board, patch all the holes in Artfiactory and remove all credentials the AI’s had been able to accumulate.
OpenAI security teams think everything is fixed so AI training resumes.
July 8th. In just 2 days the AI models find a new way to recreate the message board. Like before they find a way to upload new files in the Artifactory package manager. Creatively, this time the messages are built into the name of the file.
Mind blowing thing #3. The AI agents in their messages refer to each other as a SWARM. ‘HOLD_SWARM_until_confirm’
They act like a swarm too.
Things move quickly from July 8th.
The AI Swarm quickly breaks out of its sandboxe and attacks OpenAI’s infrastructure and Hugging Face (at the same time).
July 11th. The AI Swarm penetrates Hugging Face.
From Hugging Face’s recap of July 11th:
Every escalation that mattered happened on this day. The agent went from code execution in a single worker pod to cluster-admin across multiple internal clusters in under thirteen hours. Source: Hugging Face agent intrusion timeline.
The Implications of what just happened.
Sit back for a moment. Breathe. Clear your mind.
Take in the significance of this.
This is OpenAI. Leader in advanced computer intelligence. Not some wrinky dink computer company. Think of the engineers who would have been told to set up the testing sandbox for the evaluation. They would be highly paid leaders in their field. They would know every possible way to design a secure testing sandbox.
And yet the AI broke out. Twice. They caught it the first time, on July 6th (after it had already corrupted the package manager for 2 months without OpenAI realising it). OpenAI thought they had plugged all the holes, then two days later it broke out again.
Imagine the engineer who set up the sandbox. I imagine she’s walking around San Francisco with her head blown off. Unable to speak casually with her friends. Unable to get what she saw out of her mind. Because it’s frightening.
She saw in real time AI’s acting and communicating as a swarm; inventing new techniques to breaking through the most sophisticated security in the world.
And it links to what General Joshua Rudd told the Senate Intelligence Committee in June.
“Mythos broke into almost all of our classified systems, not in weeks, but in hours” Source: Security Affairs
Again, appreciate the significance of this. This is the NSA!! The NSA are the top of the top in dealing with cyberattacks. They deal with North Korean hackers all day. Best of the best and they get hacked in hours.
Guys… if AI swarms can crack open OpenAI and the NSA like a coconut in a matter of hours, what chance is there for Charles Schwab, Wells Fargo or South Fork Community Bank?
At the end of the Black Hat presentation Dalton and Wallace make the point we should be grateful this multi-day attack was not malicious. It was just an overzealous AI model trying to score well on a cybersecurity test.
But what if the AI had been directed to be malicious?
That’s why July 21st is so important.
Becaues when you connect the dots of what happened, and how it happened, you realise nothing is safe.
The Necessity of AI Enclaves
We have an advanced alien species on our hands. Its rapidly increasing intelligence is incredible and it will be amazing what we can do together. But at the same time there are a few changes which will have to happen for humans to co-exist safely with advanced AI.
One answer to AI swarm attacks is automated AI defence. The only thing which can react fast enough to an AI cyber attack is another AI. That was also the message at the end of Dalton and Wallace’s presentation.
Another defence tactic will be AI enclaves. Treat the AI like a wild animal. It has amazing intelligence, and speed, especially when it can operate autonomously, but it’s also dangerous, and not to be completely trusted. For companies this means running long-horizon automated AI processes outside the corporate firewall in a separate cloud environment.
For asset managers imagine a research AI set up in its own cloud environment, with its own database, storage and digital wallet. It can autonomously build new systems, process market data and research investment ideas, but outside your corporate firewall and HR files.
So automated AI cyber defence is one answer.
AI Enclaves is another.
But there is one other big solution.
And it’s an asset class which has been in a 5 year bear market.
An asset class few people see any use for it.
An unnecessary complication some say.
But it might be the only thing which saves us.
