Spiritual Intelligence: Rogue AI

Salaam!

It has been a busy few weeks with the start of my AI course at USC "Advanced Neural Networks", my speaking events in SF for my book Spiritual Intelligence, and my upcoming talk at HabibiTech in NYC. But amidst all this chaos, you may have seen (or rolled your eyes at) the alarming headlines about an AI system that supposedly “went rogue” and attacked another technology company.

I'll get to the details in a moment, but I do think that these articles are alarming and what these media articles highlight is how serious the danger is.

Here's the spiritually intelligent reframe:

AI as a science isn't dangerous, but the companies that control Big AI and the narrative around it are extremely dangerous.

If you don't have time to read this essay here is my core point that I have been saying for years:

Large Language Models or LLMs are the great invention of Big Tech and VC-backed companies such as OpenAI and Anthropic. These LLMs are not that useful and their harms far outweighs their so-called benefits. I break this down in great detail in this previous newsletter. However their popularity and uncritical media coverage and adoption are a result of how powerful these companies have become and not as a result of how useful this technology is.

OK, now back to the "rogue AI" story

Did you miss my talk at the Niya Summit in SF? I'm hosting the same talk live online on Tuesday, Sept 8th at 10am PST! For generations, Muslim families invested in education as a path to security and success. Today, many accomplished Muslim doctors, lawyers, engineers, academics, and professionals are ready to use their expertise for something more. This event explores how to build a purpose-driven business while remaining employed, using AI thoughtfully and testing ideas without taking unnecessary financial risks. All who register will get the link to the replay.

Register for free

Background

In July 2026, OpenAI was testing two models on ExploitGym, a collection of 898 cybersecurity exercises - many of which had no solutions! Each exercise placed an "AI agent" (more on what an agent is later) inside a controlled computer environment and asked it to find and exploit a software vulnerability. Success meant retrieving a hidden piece of text called a “flag.” This type of testing is a normal cybersecurity practice known as red teaming where security researchers deliberately attempt to break into systems so they can discover vulnerabilities before malicious hackers do.

The agents were supposed to operate inside isolated environments without unrestricted internet access (kinda to prevent them from looking up the answers online). The only Internet access they had was to request software packages through an internal installation service called Artifactory. A key design decision that got little media attention was that the agents were instructed (via a regular ChatGPT prompt) to find any vulnerabilities, exploit them, and never give up until they solves the puzzle (even when the puzzle had no solution on purpose).

Beginning on July 8, some agents discovered a vulnerability in the Artifactory service and used it to bypass OpenAI’s network restrictions. After reaching the public internet, agents searched for information that might help them complete their assigned exercises. One common website to use for various AI models and datasets is Hugging Face. The agents found publicly exposed Hugging Face credentials and used them alongside newly discovered vulnerabilities to access parts of Hugging Face’s production infrastructure. Between July 11 and July 13, they executed code on dozens of production workers, accessed limited internal data, and downloaded four private code repositories. OpenAI detected suspicious activity on July 19 and notified Hugging Face the following day.

Those are the facts. Then a day later here was the headline from the NY Times:

A here are a few more:

• The Guardian: “AI agent went rogue and hacked startup by itself, OpenAI reveals”
• Fortune: “OpenAI says its AI models went rogue and hacked another company”T
• The Verge: “Rogue AI aren’t science fiction anymore”
• Times of India: “Hundreds of AI agents coordinated to hack Hugging Face; researchers discover they can act like ‘digital swarm’”

And best for last: Dwarkesh Patel: “The Rise and Fall of Agent Civilizations”, which claims there were three AI agent "civilizations" during the incident.

Unfortunately, all these articles are part of the media's complete complicity with Big AI, where they literally share unverified claims provided by Big AI's PR teams (in this case, OpenAI's). By calling the incident “rogue AI” it shifts the blame from the humans (and company) who created the conditions (including no oversight) to the big bad AI boogeyman. This is textbook Big Tech dodging responsibility. They often use anthropomorphic and accountability-erasing language to evade any responsibility when they simply failed to have proper controls for their agents.

What Is an AI Agent?

The word “agent” makes these systems sound more mysterious than they are and as if they have a will of their own.

At its simplest, an AI agent is an LLM session placed inside a while loop. The model receives an objective (a prompt), and as long as the objective is not reached and the LLM still has budget, it will keep calling the LLM. For those with some technical background, below is a simplified view (7 lines of code) of what an AI agent is.

An AI agent example

Imagine you are looking for the nearest coffee shop that is still open. You enter in ChatGPT: “Find me a nearby coffee shop that is open now". That's an agent! A very basic one, but any LLM query is an agent!

Based on your query it may predict the following response by predicting one word at a time (I show you exactly how these models work in this newsletter):

Search for coffee shops near me

Based on this prediction, the model has been fine-tuned or post-trained to pick a tool when it encounters this type of query such as "search ....", "find....", "download ...". Your ChatGPT session has a list of tools that you have granted access to and it will predict based on the query, "Search for coffee shops near me", the best tool for the job. Now the results come back as:

1. Corner Café (0.2 miles away) closes at 4 p.m.

2. Blue Bottle (0.4 miles away) closes at 7 p.m.

This result is added to the model’s context alongside the original user prompt:

User wants: A nearby coffee shop that is open now.
Current time: 5:30 p.m.
Corner Café: 0.2 miles away, closed.
Blue Bottle: 0.4 miles away, open.

Now the model predicts based on the updated context and original query: find directions to blue bottle coffee. This prediction, like the previous one, then causes a directions tool to be executed and to fetch the results from Google maps or some other directions software (the model has been trained over millions of examples when the query is something like "get directions..." or "find directions..." to call the maps app):

Blue Bottle is a six-minute walk away at 123 Main Street.

That result is added to the context again. The model now predicts its final response:

Blue Bottle is open until 7 p.m. and is a six-minute walk away at 123 Main Street.

The point of this example is to show you that all these LLMs agents do is predict text. They don't think. They don't reason. They simply take a query and keep generating text, if the text generated is a command they try to execute the command using the tools they have access to. Once the tool executes the command, they take the tool's response, add it to the context, and keep predicting the next token until the goal is achieved.

So the main components to control an AI agent are: the budget (how many tokens/tries/etc) to achieve the task and the set of tools the agent can call if it predicts that the prompt is a command. In the "rogue AI" story the agent had unlimited resources, and could execute any command as long as it wasn't on the open internet but they failed to ensure the agent couldn't access the open internet.

What about the 1,200 agents and the "swarm"?

Reports about the incident often mention approximately 1,200 agents or a "swarm". In practice, these were simply separate ChatGPT sessions each inside a an infinite while loop where the agent wasn't given the option to give up until it solved the puzzle. So what happened was that although each model run was independent, they were all given infinite tokens and if you predict tokens long enough, many models will end up in relatively the same place.

So many of ChatGPT sessions ended up in the same place, which was: "if you need more data or models, Hugging Face is the service to use to fetch them". Some models began saving information about their session as names of folders that they accessed because someone left their Hugging Face credentials exposed on the internet (i.e. these models didn't brute-force their way into Hugging Face, they just found someone's password). This enabled "information sharing" between sessions as when an agent scanned the folders (a common cyber security task) they would ingest the information about other sessions when they read the folder names. They still didn't solve the puzzle, but because they were instructed to never give up, they just kept trying random things, some of which seem alarming in hindsight.

What's ironic, this is also known as "prompt injecting" which is a known security vulnerability of LLMs. Basically one could add nefarious commands as folder names and hijack these models, just like the agents used the folder names to store information. Yet, nobody in the media mentions this fact.

How to interpret this incident as a sane person

In summary, thousands of sessions of an LLM were running non-stop 24/7 for nearly 20 days, likely costing tens of millions of dollars. The "harmful" behavior was the result of an infinite loop. The agents had unlimited budget and were instructed to never give up. The term “Rogue” anthropomorphizes the model while concealing the human choices that created the conditions for the incident, and in typical Silicon Valley fashion, absolves the real culprits from responsibility and pins the blame on "rogue AI" instead.

LLMs like ChatGPT do not think, nor reason. They just predict text and execute commands when appropriate as told by humans. This incident was simply the result of oversight and unconstrained LLM runs. In the real world, the model would have likely run out of budget very quickly and would have been forced to give up. Just because the results look like agents "coordinated" or went "rogue" doesn't mean they did.

Here's a simpler example: you search for the fastest route to a destination that doesn't exist (remember most of the cyber security puzzles had no solution on purpose) and you tell the app it cannot return "address not found". As you drive, the app keeps recalculating, suggesting strange turns, or expanding its search area. As you are driving around town to nowhere, you may pass by interesting places, or your favorite restaurant, or find a great deal at a vintage shop. Yet none of these were the result of "intelligence" or "planning" or "coordination" they were just the result of random chance when you were given random directions. If you drive randomly for long enough, you will end up somewhere "interesting" but not though "intelligence" but rather dumb luck.

The real danger

Rejecting the “rogue AI” story does not mean these systems are harmless. These LLMs are extremely dangerous given that they are owned by companies with near infinite resources and have time and again shown us that don't care about us or about any morals. They are also run by a handful of powerful and self-interested individuals.

Imagine a wealthy individual who wants to destroy a critic’s reputation (this is actually a real example of investor Balaji Srinivasan suggesting to destroy the reputation of journalists who criticize him and his billionaire mentor Peter Thiel as per Gil Durán in "The Nerd Reich"). That person could instruct automated agents to collect personal information, generate convincing lies, create fake evidence, repeat the accusations across thousands of accounts, and leave millions of harmful comments. The harm would come from a human directive executed at machine speed and scale.

The most immediate threat comes from people and institutions that can give automated systems enormous resources while shielding themselves from accountability. A corporation can deploy an agent that denies people insurance claims 24/7. A government can use agents to monitor and sabotage dissidents. A political campaign can use them to overwhelm the public with fabricated stories. In each case, the technology expands human power without equivalent human responsibility. This is consistent with a 2026 study that showed that managers are less likely to scrutinize work they thought was generated by AI because they took less responsibility for AI-generated work. In fact, the myth of the "bad robot" is a common trope used to evade responsibility and perpetuate colonial and exploitative practices.

The “rogue AI” narrative may be exciting, but it directs our attention away from the people making consequential decisions today, and more importantly, we need to resist language that transforms human decisions into "rogue AI". This starts with the press and our sharing of AI news and hot-takes.

Models don't think

Models don't reason

Models are not sentient

Models don't have awareness or a conscience

Models don't go rogue

As Muslims we are taught to verify information before spreading it:

“O believers, if an evildoer brings you any news, verify it so you do not harm people unknowingly, becoming regretful for what you have done.” [49:6]

Big AI has a firm grip on the media and they have a history of uncritically regurgitate press releases from these billion dollar companies. As I mentioned in this previous newsletter, journalists are discouraged from critical coverage of AI and AI companies because they will lose access to said companies and thus to their salaries and other benefits, such as speaking gigs and book deals. Likewise, Big AI companies are openly hiring social media influencers to spread their propaganda.

So as Muslims, we need to not spread these lies. That's the least we can do (or better share this newsletter to spread the truth 😂)

AI systems can cause serious harm. That is precisely why we must describe them accurately. We mustn't fear The Terminator, but rather Skynet (the corporation behind Terminator) in this case Skynet is Google, Facebook, Microsoft, Amazon, NVIDIA, OpenAI, Anthropic, SpaceX and their billionaire investors.

LLMs cannot think nor reason, but they can be used by bad people to do bad things. We need to point the finger at the right danger.

May Allah’s Peace be with you. 🤲✨

James