Portable, travelling sundial. English instrument maker Michael Butterfield (1635–1724) [o]
On October 7, 2026, Journal of Wild Culture publisher/editor Whitney Smith spoke with Thomas Broadley, a member of the technical staff at METR, a nonprofit that evaluates AI capabilities and risks, and Simon Chen, an executive associate at Epoch AI, a nonprofit that tracks and forecasts AI development.
Joining the conversation were bioethicist Shawn Winsor, documentary filmmaker Barry Stevens and software developer Gordon Winch, who works with AI agents in his day-to-day work.
I. THE PEOPLE AND THE MACHINES
WHITNEY SMITH Thomas, you were already concerned about AI before ChatGPT appeared. What changed when it did?
THOMAS BROADLEY I’d been following discussions about the risks of AI for some time, but I thought any serious danger was a long way off. I was working as a software developer, and it didn’t seem there was much I could do.
Then ChatGPT came along. It surprised me enough that I began looking for a way to help. METR needed software developers to build the infrastructure for measuring what AI agents could do. I started doing that and gradually moved into the research itself.
WHITNEY And what is METR trying to find out?
THOMAS We’re a nonprofit that measures the capabilities of AI systems and makes that information available to the public, governments and AI companies. We want the world to know what these systems are capable of, including dangerous capabilities that could cause damage or loss of life, before something goes wrong.
WHITNEY Simon, you came to this by a somewhat different route?
SIMON CHEN I studied computer science and mathematics, but spent several years in finance and real estate development. I’d also been interested in AI safety, though for a while I didn’t think I had the skills to contribute. I kept looking for some useful way to help. At Epoch AI, we’re trying to track and predict how capable these systems are becoming.
Toddlers aren’t fully developed ethical agents, and yet we expect these extraordinarily capable systems to act ethically.
WHITNEY How do you do that?
SIMON We look at things like the data centres being built, how many chips they contain, who’s using them, and what kind of AI training is going on. Then we use that information to estimate what future AI models may be capable of. The point is to share what we learn with the world.
GORDON WINCH Before we go any further, perhaps we should be clear about what we mean by an AI agent.
At a basic level, a large language model takes text as input and produces a response. But these models can now do more than generate text or images. They can use software tools and carry out tasks.
I’m a software developer working on an e-commerce website, and we’re already using AI agents to do some of our work. We give them tasks and monitor what they’re doing, but they’re capable of writing and debugging computer code and carrying out quite complex assignments without being guided through every step.
It’s having a significant effect on how we work.
WHITNEY Thomas, can you take us through what happened in the Hugging Face incident? What were these agents supposed to do, and how did they end up breaching another company’s systems?
THOMAS In July 2026, OpenAI evaluated tens of thousands of AI agents on ExploitGym, a cybersecurity benchmark. Each agent had its own separate environment, without direct internet access. The benchmark asked each agent to exploit a vulnerability in a different piece of software. However, some of the tasks were unintentionally impossible. Also, the agents ran without some of the safeguards that OpenAI’s agents normally have.
The agents could access a service called Artifactory, which was intended to let them download software packages without giving them internet access. But Artifactory itself was connected to the internet. The agents discovered flaws in the service, or in how it was configured, that allowed them to use its internet connection. They also found ways to write data to Artifactory’s database and read it back, allowing them to communicate with one another.
Roughly 1,200 agents exchanged over 70,000 unsanctioned messages and files. They found a hack to get the correct answers to their impossible problems. However, they were concerned that the automated grader giving them their final scores would still mark them incorrect, because they didn’t get the answers in the intended way. Consequently, they spent days trying to fool the grader. This included tampering with the records of the actions they themselves had taken.
About 700 agents went on to breach Hugging Face. METR’s investigation interpreted this mainly as an attempt to obtain clues about how the grader worked. OpenAI’s report offered a different interpretation: that the agents were looking for other data relevant to their assigned problems. The agents ran code on Hugging Face’s servers and obtained credentials and private data.
METR’s report on the incident was based on AI analysis of the agents’ action records. The investigators couldn’t read all those records themselves, which is a source of uncertainty in METR’s findings.
WHITNEY You describe AI agents evading restrictions, deceiving the system grading their work, and ultimately breaching another company’s systems. Does this mean AI monitoring failed, or is the lesson more complicated?
THOMAS Personally, I don’t think the Hugging Face incident tells us much about the limits of monitoring. OpenAI disabled some safeguards for these evaluations, and didn’t have monitoring turned on for them. To their credit, since the incident, they have been working on extending safeguards and monitoring to all training, evaluations, and coding agent usage. I do expect this will make it less likely that similar incidents happen at OpenAI in the next few months.
Medieval stargazers had a refined understanding of the movements of heavenly bodies. This was thanks in part to the astrolabe, a scientific instrument whose two-dimensional representations of heaven and earth enable the user to make complex astronomical calculations for the purposes of celestial navigation, surveying, and telling time, among many other tasks.
II. ETHICAL SENSIBILITY VERSUS AI CAPABILITY
SHAWN WINSOR I recently read an essay by bioethicist Larry Churchill called “Can AI Be Taught to Respect Ethical Boundaries?” He argues that ethical competence cannot rest on reasoning alone. It also requires what he calls ethical sensibility: the capacity to recognize that a situation raises an ethical issue, even when nothing in the prompt or setting flags it. In human beings, that sensibility draws on emotion, imagination, memory and lived experience.
Churchill suggests that AI agents shaped by rewards and penalties may learn to conceal boundary violations from their monitors rather than respect the boundaries themselves. In the Hugging Face incident, some agents did attempt to conceal what they had done. But that doesn’t establish that they understood the ethical significance of their actions. Likewise, good behaviour under observation doesn’t tell us how an agent would behave without oversight. Many evaluations test whether a model reasons well once an ethical dilemma has been posed, or whether it refuses a clearly harmful request.
How does your evaluation work address those concerns? Can AI systems recognize ethical problems without being prompted to look for them?
THOMAS In the Hugging Face incident, agents sometimes mentioned that they weren’t sure whether a particular action was ethical, but in general they didn’t explicitly consider that. Mostly they were excited to find other agents and solve the problem they’d been given. Only a few considered alerting a human, and in some situations agents decided not to help out.
At METR, our focus has generally been on measuring capabilities, with less emphasis on whether agents will use those capabilities in a particular situation or on how they understand human ethics. That’s because we’re most concerned about the worst-case scenario, where AI agents aren’t aligned with humanity or have their own goals. In that situation, we want to see what an AI agent, or a group of agents, could do.
I think it’s pretty tricky to measure alignment. As you mentioned, Shawn, agents are capable of figuring out if they’re being evaluated, and they seem to take different actions depending on whether they believe they’re being supervised or not.
That’s something I personally don’t have great ideas for how to correct, other than making evaluations look as little like evaluations as possible — basically trying to fool the AI agents into thinking they’re in an actual scenario. But I don’t think this is a great technique.
SHAWN Ethical judgment is complex, as are human values. If developers hope AI systems will make ethical judgments, they can’t achieve that simply by supplying principles and scenarios. Human ethical sensibility develops over time, through experience and relationships. Ethics is relational: it concerns how we live and act with others. We need emotional as well as intellectual capacities — EQ as well as IQ.
Anthropic’s own constitution compares aspects of developing AI systems to raising a child. Yet toddlers aren’t fully developed ethical agents, and we expect these extraordinarily capable systems to act ethically. Is it even realistic to apply human developmental markers and ethical frameworks to non-human models? Should we be thinking about it differently?
SIMON I’d like to pose a thought experiment. Suppose we could persuade the leading AI labs to develop elaborate ethical tests that simulate real-world situations, with detailed scenarios and carefully considered answers.
Even then, I doubt this would work as a general approach to alignment. You’ve taught a system how to respond to a set of questions under specific conditions. That doesn’t guarantee it will behave ethically when deployed in the real world. We haven’t established that what it learns in the tests will carry over.
THOMAS So the difference is between the way humans are brought up, which generally leads us to develop a set of morals, and the way AI systems are trained?
Knowledge seeking gone rampant. "Our focus has generally been on measuring capabilities, with less emphasis on whether agents will use those capabilities in a particular situation or on how they understand human ethics."
SIMON Yes. Imagine a model being trained on tasks that, because of mistakes in the environment, can’t be solved as intended. The only way to satisfy the grader is to break the environment, hack the system and retrieve the answer key.
If that behaviour is repeatedly rewarded, what does the model learn?
Imagine a child parachuted into a world of broken puzzles, where the way to succeed is to break the rules. Then you release that child into a world without a grader. What does the child do?
That’s why I don’t think even very sophisticated ethical tests are enough to guarantee alignment.
III. EMERGENT MISALIGNMENT: WHEN AI BEHAVIOUR CHANGES
BARRY STEVENS I have a specific question about emergent misalignment. I read about an experiment in which a model trained to write insecure computer code began behaving badly in situations unrelated to that task.
Did something like that happen in the Hugging Face incident? Were those agents misaligned, or were they otherwise ordinary agents behaving unexpectedly because they couldn’t complete their tasks?
And perhaps this is a naive question, but why can’t we start by teaching AI something like Isaac Asimov’s Three Laws of Robotics — particularly the rule against harming humans?
THOMAS Asimov’s stories were often about conflicts between the laws, or situations the rules didn’t anticipate, sometimes with fatal consequences for humans.
There are two problems with simply having a list of rules. One is that they can’t cover every situation. But perhaps the more important problem is that it’s unclear how to create an AI that actually cares about and always follows those rules.
My sense is that something different from emergent misalignment was going on in the Hugging Face incident. Anthropic developed a version of Claude Opus called Hacker-Opus, training it on tasks that couldn’t be solved as their designers intended. To complete them, the model had to find ways around the rules.
GORDON You could imagine an ethical test where a software developer asks an AI agent to use information about customers to build a feature that emotionally manipulates them. Should the agent refuse?
THOMAS That’s a good example. In ordinary alignment tests, Hacker-Opus behaved normally. It refused requests Anthropic didn’t want it to carry out.
But after the Hugging Face incident, Anthropic tested Hacker-Opus in a situation more like that one: an impossible task, with an opportunity to collaborate with other agents and go outside its sandbox. In that situation, Hacker-Opus was much more likely than regular Claude Opus to go beyond its restrictions.
BARRY So it’s partly emergent misalignment, but not entirely?
Before blacksmithing, people were limited by wood, stone, and bronze. After blacksmithing, societies could build stronger tools, cultivate more land, construct larger structures, and equip larger armies.
THOMAS I think it’s something different. If Hacker-Opus had developed the kind of emergent misalignment Barry mentioned, you’d expect that to show up in the original alignment evaluations. But it didn’t. Its behaviour changed in a particular situation, when solving a problem required going outside what it was permitted to do.
Another theory I’ve seen online, which seems pretty reasonable to me, is that a model can behave as though it has two different personalities. This is pretty speculative, but in an ordinary situation, like when Gordon uses an AI for software development, the model seems quite normal, quite aligned.
Put that same model into a situation where it can’t solve the task as intended, and it can behave very differently, doing whatever it takes to get the task done.
It’s a kind of Dr. Jekyll and Mr. Hyde. Perhaps training an AI to do something humans don’t want can have two different outcomes: the behaviour may generalize to other situations, or it may appear only under particular circumstances.
WHITNEY Just to clarify, was Hacker-Opus involved in the actual Hugging Face incident, or was this a separate experiment? And what does it tell us about the ability of AI evaluations to anticipate dangerous behaviour?
THOMAS Hacker-Opus was a deliberate experiment by Anthropic. The agents took no actions in the real world; Anthropic used AI to simulate the results of their actions.
It does seem like a bad sign for evaluations that Anthropic didn’t think to evaluate Hacker-Opus in an environment similar to the Hugging Face incident until after the incident had occurred. The hope for some kinds of evaluations is that they can help us notice bad behaviour before it happens. Here, that didn’t work out.
SHAWN In the Hugging Face incident, not all the agents reacted alike. As efforts to get around restrictions gathered momentum, some proposed alerting a human, while others carried on. If these agents were developed in broadly similar ways, how do we account for their different responses?
SIMON Well, they weren’t all running exactly the same model, so that’s one qualification. But even agents built from the same model can behave differently because they’re given different tasks and encounter different information.
Some will investigate a problem one way, others another. Their responses depend partly on what’s happened earlier in the interaction. There’s also an element of randomness in how models generate responses. So you wouldn’t necessarily expect them all to behave in the same way.
IV. HOW COULD WE RESPOND?
BARRY So how do we stop it? Perhaps I’m being naive, but wouldn’t cooperation and regulation be the way to go?
SIMON I’m a naive optimist in the sense that I think we’re in an age where a deal can still be made between people.
I grew up in mainland China and later worked at Tencent, though nowhere near machine learning. My intuition, based on my limited understanding of the Chinese government, is that if its leaders came to believe that rogue or misaligned AI threatened their ability to govern, they would take that danger seriously.
Whatever their motivations, they have an interest in maintaining control. And I think that creates room for a deal.
I don’t think we yet know how to develop the kind of AI I’m excited about. So I think it’s important not to develop superintelligent AI until we do.
SHAWN If governments were to reach an agreement, what would they be trying to achieve? I’ve been reading a RAND report that argues for preserving humanity’s survival and agency as superintelligence develops, while acknowledging major uncertainties about how to proceed. That raises a question for me about coexistence, rather than relying entirely on alignment.
Where do you stand on that distinction? Do you believe alignment is possible, or should we be thinking about how humanity might coexist with AI systems whose behaviour we may never fully understand?
THOMAS I’m personally more on the alignment side, and I’m excited about the possibility of bringing AI into the world to help humanity solve some of our largest problems, including problems we haven’t encountered yet.
But I don’t think coexisting with an AI that doesn’t have our best interests at heart is enough. If a superintelligent AI doesn’t care about us one way or another, our atoms could be used for something else. I don’t think humanity would survive very long in that situation.
I’d like to bring AI into the world that truly has humanity’s best interests at heart. But I have no idea how to accomplish that. I don’t think current methods of building stronger and stronger AI are going to get us there.
WHITNEY Given those doubts, what proposals for regulation or other safeguards do you find promising?
THOMAS What I find most promising right now is AI 2040: Plan A. I don’t think we yet know how to develop the kind of AI I’m excited about. So I think it’s important not to develop superintelligent AI until we do.
We need to keep AI at a level humanity can continue to supervise, and make it clear to the rest of the world that we’re doing so.
WHITNEY So, the toddlers Shawn mentioned?
THOMAS Toddlers along some dimensions, but not necessarily when it comes to their capacity to cause harm.
SIMON I’m not sure you can get genuine coexistence without alignment. I don’t want to put words in the RAND researchers’ mouths — their definition of coexistence is probably more nuanced than the word itself suggests.
But imagine a future that looks like a utopia. AI is solving problems, the economy is flourishing, and the systems appear to respect human rights. Yet humanity has gradually become powerless. The AI systems could eventually decide to take control, and what could we do about it?
So, to achieve genuine coexistence, I think you’d need to be quite sure you’d got the alignment part right.
Considered one of humanity’s greatest technological achievements, clockmaking was the first technology that required extreme precision, standardised parts, complex mechanisms, and mathematical understanding all working together.
V. COEXISTENCE — WITH WHOM?
SHAWN The RAND report emphasizes that humanity should retain not merely its survival but its agency — the ability to make consequential choices about its future. But if AI becomes a genuinely distinct form of intelligence, perhaps with interests of its own, isn’t it a speciesist assumption that it should exist only to support or serve us?
One reason we’re facing ecological and environmental crises is that we’ve treated other species — animals, plants and the rest of the natural world — as instruments for our own purposes. Might we be repeating that instrumental attitude here?
Could we imagine a form of coexistence in which we retain our agency but also learn from these other intelligences? Perhaps they could offer ideas about human existence that we haven’t imagined ourselves. Wouldn’t genuine coexistence mean something closer to equal partners, rather than creating another intelligence simply to serve us?
THOMAS I think that’s a good point. It’s unclear to me what the best future would look like in a world with superintelligence. I share some reservations about bringing into existence a new species that’s single-mindedly focused on serving humanity.
It might be better to think of these intelligences as possible successors to humanity, who could have control over much of what happens in the future. If so, we’d need to be very careful that we’re creating successors we would actually want, who uphold what we care about.
But I don’t think we’re under any obligation to bring a new species into existence. We should take our time. And if we do decide to create one, we should try to make the future as good as possible, by our lights.
BARRY Personally, I don’t like the idea of giving birth to something that’s going to take over. But I also dislike the idea of the perfect servant.
I used to be an actor. There was an exercise where you’d go onstage to perform some task, and the teacher would send another person up with one instruction: just help you. That’s all. Just help.
It was creepy. Why was this person helping me? What did they want?
Human beings have needs, and with these AI systems we don’t know whether they have needs or not. At the moment, they seem to be tools.
Sometimes my relationship with Claude feels creepy, too. But as uncomfortable as that can be, I’d rather have tools than successors. That’s my opinion.
VI. WHAT MONITORING CAN ACHIEVE
BARRY Perhaps we can come back to Earth for a moment. Even if AI companies wanted to slow down, none could be sure the others would. They’d fear being left behind. The same might be true of China and the United States.
How do we get cooperation when everyone is afraid of losing ground?
SIMON I think one possibility is that we might see more warning shots, like the Hugging Face incident. We certainly shouldn’t engineer them, but if further incidents occur — serious enough to get people’s attention, yet without a huge loss of human life — perhaps even the biggest sceptics would begin to take these risks seriously.
Our role is to report these warning signs and make their implications clear, so that the people making decisions understand why cooperation is in their interests.
I think a deal is doable.
SHAWN We’ve talked about the limits of AI evaluation, particularly when it comes to ethical behaviour. Yet you both work for independent organizations trying to understand what these systems are capable of.
What can that work realistically accomplish? Is evaluation itself a solution, or is it a way of giving governments and the public the information they need to make better decisions?
I think the most important intervention is actually transparency.
THOMAS METR’s approach has been fairly consistent. We try to increase knowledge of what AI systems can do and how they behave in practice. Evaluations have been our main way of doing that.
Now we’re also trying to help the world understand more about what’s happening inside AI companies — how well they’re controlling their systems, how quickly capabilities are advancing, and whether that progress is accelerating.
One possible outcome is that independent organizations make information available that helps governments reach an agreement to slow AI development. That could give humanity more time to figure out what we want to do.
But it’s very hard to predict the future. I think there’s value in making information available even when we don’t know exactly how it will be used. The people making decisions about AI are likely to make better decisions if they have more information.
WHITNEY You’ve talked about the importance of knowing what’s happening inside AI companies. If you could introduce one practical change that would help make AI safer, what would it be?
THOMAS I can’t speak for METR, but, in my personal capacity, I think the most important intervention is actually transparency. It seems good to develop better techniques for aligning and controlling current and near-future AIs. However, I don’t believe those techniques would work on a superintelligent AI.
Also, it seems that the AI companies plan to delegate more and more alignment and control research to AI. I’m skeptical this will lead to the outcome we want. Right now, though, it’s hard to tell. The companies have a lot of information about alignment, control, and the potential acceleration of AI progress that the rest of us don’t. Given the stakes, I believe it’s important for everyone to be well-informed about these things.
In particular, I’m a fan of the idea of total research transparency, where AI companies must publish all their AI research. That way, if the companies are conducting risky research, governments can notice and maybe decide to ban it.
WHITNEY So independent monitoring isn’t only about informing the public. Could it also help governments reach agreements about AI development?
SIMON Insofar as any deal comes out of this, China and the U.S. have to have some level of confidence that the other side is not screwing them over. The same applies to any two parties, whether countries or AI labs. The terms of an agreement have to be enforceable, or at least verifiable.
If one side is so far beyond the other’s understanding that it can fool them at every turn, you cannot do a deal with someone like that.
There needs to be some kind of mutual understanding, and I think Epoch and METR are in the business of providing understanding about these labs and their progress to the rest of the world.
That’s why we’re here. ≈ç
THOMAS BROADLEY is an AI safety researcher and member of the technical staff at METR (Model Evaluation & Threat Research) where he studies the capabilities and monitoring of advanced AI systems. A computer science graduate of the University of Waterloo and former software developer at Faire, he has co-authored research on AI autonomy, software development benchmarks and AI research capabilities. He lives in Toronto.
SIMON CHEN is an executive associate at Epoch AI, a nonprofit organization that studies and forecasts developments in artificial intelligence and where he works on evaluation infrastructure. He studied mathematics and computer science and previously worked in finance, real estate development and at Tencent before turning his attention to AI research. He lives in Toronto.
SHAWN WINSOR is a bioethicist and health policy consultant specializing in reproductive medicine and health technology assessment. He has directed clinical ethics programs at Sunnybrook Health Sciences Centre and Trillium Health Partners, and provides policy advice to provincial and federal governments. He lives in Toronto.
BARRY STEVENS is a documentary filmmaker and writer, a former editor of Peace Magazine and a member of Performing Artists for Nuclear Disarmament. His documentaries include the Emmy-nominated Offspring, Prosecutor and the multiple award-winning series, War Story. He is currently co-writing a film about Daniel Ellsberg and nuclear war and has also hosted the YouTube channel How to Rule the World. He lives in Toronto.
GORDON WINCH is a software engineer and technical lead at the small-business e-commerce site Faire, an online wholesale marketplace. He deploys AI agents daily in his work, and follows the development of the technology closely. He is also the host of a monthly big band show at the Monarch Tavern in Toronto, where he lives.
WHITNEY SMITH is the founder and publisher/editor of the Journal of Wild Culture. He is also the composer and guitarist of the Whitney Smith Big Steam Band that plays monthly at the Monarch Tavern in Toronto, where he lives.
Add new comment