There is something deeply unsettling happening in artificial intelligence.
It isn’t simply that some researchers believe advanced AI could eventually become dangerous. It is that people working close to the technology are publicly expressing serious concerns about where it may be heading – while the organisations and industry around them continue racing to make AI more capable.
In September 2026, AI researcher Jacob Coxon resigned from Anthropic after previously working at both Anthropic and OpenAI. His warnings about the direction of advanced AI development attracted enormous attention.
One of the most confronting claims was that people building advanced AI genuinely believe it could pose an existential threat to humanity.
Anthropic researcher Evan Hubinger subsequently publicly supported the substance of that concern and gave his own estimate of greater than a 10 per cent chance of AI causing human extinction within the next decade.
That number should not be treated as a scientific forecast. We simply don’t have evidence capable of producing a reliable probability of human extinction from AI.
The 2026 International AI Safety Report is considerably more measured. It concludes that experts disagree substantially about both the likelihood and timing of catastrophic loss-of-control scenarios. Current AI systems do not have the combination of capabilities needed for such a scenario, although capabilities relevant to those risks are improving.
So, this isn’t an article predicting the end of humanity.
Instead, I want to examine a different question raised by the controversy.
If people inside a system are genuinely worried about where it is heading, why does the system keep moving in that direction?
Because that question isn’t really about artificial intelligence.
It is about organisational culture.
AI risk and organisational culture: the uncomfortable contradiction
The debate surrounding AI safety has exposed a fascinating contradiction. Some highly knowledgeable people working on artificial intelligence believe increasingly capable AI could create serious or even catastrophic risks. At the same time, the organisations employing those people stay locked in intense competition to develop increasingly powerful systems.
That looks irrational. If something could be extraordinarily dangerous, surely the rational response is to slow down. But organisations don’t make decisions in isolation. An AI company that slows development has to consider what happens if its competitors don’t.
Researchers have to consider whether leaving one laboratory changes anything if another laboratory continues the work. Companies have investors, customers and employees expecting progress. Governments have economic and national-security interests. Researchers have professional ambitions and intellectual curiosity. And everyone operates with assumptions about what everybody else is going to do.
Suddenly the question is no longer:
Should we slow down?
It becomes:
What happens if we slow down and everyone else keeps going?
That illustrates something fundamental about organisational behaviour.
Organisations don’t behave according to intentions
One of the most persistent mistakes we make when thinking about organisational culture is assuming organisational outcomes reflect the intentions of the people involved. Often, they don’t. An executive team can genuinely value employee wellbeing while creating workloads that exhaust people. A leadership team can sincerely want innovation while maintaining approval processes that punish experimentation. An organisation can talk about psychological safety while rewarding leaders who rarely admit uncertainty or tolerate challenge. Managers can care deeply about employee engagement while having little authority over organisational decisions that drive disengagement. And an AI researcher can genuinely worry about increasingly powerful artificial intelligence while believing that stopping their own work would do little to stop the technology being developed elsewhere.
The apparent contradiction begins to disappear once we stop examining individual intentions and start examining the conditions surrounding people’s decisions. This is what I call an organisation’s Hidden Ecology.
What is Hidden Ecology?
Every organisation has a visible structure. There are strategies, organisational charts, policies, values, KPIs, reporting lines and formal decision-making processes. But underneath that visible organisation sits another system. People constantly interpret signals about how the organisation really works.
What gets rewarded?
What gets punished?
Who actually has authority?
What happens when someone raises a concern?
Which results matter most when priorities conflict?
Whose opinion carries weight?
What gets people promoted?
What happens when commercial pressure collides with organisational values?
And what happens when doing the responsible thing creates a competitive disadvantage?
These conditions interact. Together they create an organisational ecology that shapes what people perceive, what they believe is possible and ultimately how they behave. That is why asking whether people have good intentions tells us surprisingly little about what an organisation will actually do.
The AI race is a powerful example of Hidden Ecology
Consider the competitive dynamics surrounding advanced artificial intelligence. Imagine several organisations developing increasingly powerful AI systems. People within those organisations recognise that sufficiently advanced systems might eventually create serious risks. Ideally, everyone might agree that development should proceed cautiously.
Then introduce competition.
If one organisation slows while another continues, the organisation exercising greater caution could lose technological leadership, investment, talent, customers or strategic influence.
Now introduce geopolitical competition.
Governments may worry that slowing domestic AI development allows another country to gain an economically or strategically important advantage.
Then add investor expectations. Research prestige. Employee ambition. Customer demand. Technological curiosity. The possibility of enormous economic rewards. And the belief that if your organisation doesn’t build the technology, somebody else will.
The International AI Safety Report identifies precisely this kind of problem. Economic incentives, strategic pressures and expectations of competitive advantage influence how organisations develop and deploy AI. Competition can create pressure to reduce safeguards when those safeguards cost time or money. The result is a classic systemic problem. Individual actors can rationally choose actions that collectively create an outcome many of them would prefer to avoid. No villain is needed.
Good people can collectively create bad outcomes
This mechanism isn’t unique to artificial intelligence; it happens inside ordinary organisations every day.
Everyone agrees there are too many meetings. Yet nobody cancels their meeting because they genuinely need information from everyone else.
Everyone agrees employees are overloaded. Yet every department has one more urgent priority that genuinely needs to be delivered.
Everyone wants collaboration. Yet the performance system rewards individual achievement.
Everyone wants people to speak honestly. Yet employees have watched what happens to colleagues who challenge powerful decision-makers.
Everyone wants managers to take responsibility. Yet managers discover that many of the conditions affecting their team’s performance sit outside their authority.
Each individual decision can make sense. Collectively, those decisions create an outcome nobody intended. This is one reason dysfunctional organisational cultures can survive even when almost everyone recognises the dysfunction.
The problem isn’t necessarily the people. Sometimes the ecology keeps recreating the behaviour.
Why culture change often fails
This has profound implications for organisational culture change. When organisations encounter dysfunctional behaviour, they often try to fix individuals.
We train managers.
We coach leaders.
We remind people about organisational values.
We launch wellbeing initiatives.
We encourage employees to speak up.
We tell teams to collaborate more effectively.
Those interventions can absolutely be useful.
But they struggle when the organisational environment continues rewarding the opposite behaviour. If speaking up carries political risk, another workshop on courageous conversations won’t create psychological safety. If managers are accountable for outcomes they cannot influence, another leadership program won’t repair the disconnect between responsibility and authority. If executives say wellbeing matters but repeatedly celebrate heroic overwork, employees will believe the reward system rather than the values statement. If collaboration is important but resources are allocated through internal competition, people will protect their territory. And if an industry rewards whoever develops the most powerful AI fastest, asking individual researchers to behave responsibly may not be enough to create responsible AI development.
You cannot reliably coach people out of behaviour that their environment continuously recreates.
Ask a different leadership question
Hidden Ecology changes the question leaders need to ask. Instead of beginning with:
Why are people behaving like this?
try asking:
What conditions make this behaviour sensible?
That question is considerably more confronting. Because it forces us to examine the system rather than judge the individual.
Perhaps employees aren’t resisting change because they dislike change. Perhaps experience has taught them that organisational initiatives disappear after six months. Perhaps managers aren’t avoiding accountability. Perhaps responsibility and authority have become disconnected. Perhaps employees aren’t unwilling to speak up. Perhaps they have accurately assessed the political consequences of doing so. Perhaps executives aren’t deliberately creating an unhealthy organisational culture. Perhaps the incentives surrounding them repeatedly make short-term performance more valuable than long-term organisational health.
Once we see those conditions, different interventions become possible.
From behaviour change to system redesign
In my Hidden Ecology work, I use three broad responses to what we discover:
Protect. Repair. Redesign.
Protect the conditions already helping people and the organisation perform well.
Repair conditions creating unnecessary friction, inconsistency or unintended consequences.
Redesign elements of the system that repeatedly produce behaviour or outcomes the organisation says it doesn’t want.
That final category is particularly important. Some organisational problems cannot be solved by asking people to try harder. They require changing the conditions in which people make decisions.
That might mean changing authority and decision rights.
Changing incentives.
Changing information flows.
Changing what leaders measure.
Changing resource allocation.
Changing what gets rewarded.
Changing how challenge and dissent are handled.
Or introducing meaningful consequences when behaviour contradicts stated organisational values.
Culture change then becomes less about persuading people to behave differently and more about creating an environment in which constructive behaviour makes sense.
What leaders can learn from the AI risk debate
I don’t know whether artificial intelligence will ever become an existential threat to humanity. Nobody does.
The best available international assessment is explicit about that uncertainty. Current AI systems do not pose an immediate loss-of-control risk, but the capabilities of AI systems are developing quickly and decisions being made today could affect the risks associated with future systems. That uncertainty deserves serious attention.
But the organisational lesson doesn’t depend on whether the worst predictions about AI ever come true. The AI debate shows something much broader about human systems. We often imagine that dangerous or dysfunctional organisational outcomes require dangerous or dysfunctional people.
They don’t.
Sometimes they need intelligent, ethical, well-intentioned people making individually rational decisions inside a system whose incentives collectively point in the wrong direction. That may be one of the most difficult leadership problems of all.
Because there is no obvious villain to remove.
No single policy to rewrite.
No training course that fixes it.
The work is deeper. We have to understand the ecology producing the behaviour. And then we have to be willing to change it.
So perhaps the most useful question emerging from the extraordinary debate currently taking place inside artificial intelligence isn’t:
Will AI destroy humanity?
It is a question every leadership team can ask about its own organisation:
If good, intelligent people keep producing an outcome that nobody says they want, what is it about our system that makes that outcome keep happening?
The answer may tell you considerably more about your organisational culture than your values statement ever will.
References and further reading
International AI Safety Report (2026). International AI Safety Report 2026. The report provides an international scientific assessment of general-purpose AI capabilities, risks and safeguards, including loss-of-control risks and the effects of competitive pressures on AI safety.
Center for AI Safety. Statement on AI Extinction Risk. The statement calls for mitigating extinction risk from AI to be treated as a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Associated Press, September 2026. Reporting on Jacob Coxon’s resignation from Anthropic and the subsequent debate concerning catastrophic AI risk and competition between frontier AI developers.
Reuters, September 2026. Reporting on responses from AI researchers, technology companies and US lawmakers following the warnings about catastrophic AI risk. (Reuters report syndicated by MarketScreener.)
About Ros
Rosalind Cardinal is the Founder and Managing Director of Shaping Change, an Australian leadership and organisational development consultancy.
With almost 40 years’ experience in organisational development, executive coaching and leadership consulting, Ros helps leaders and organisations navigate the hidden dynamics that shape culture, influence, decision-making and performance.
She is the creator of the Recognition Pathway™, Political Intelligence Compass™, Organisational Hidden Ecology™ and the Women’s Leader Archetypes™ framework.
Book a chat with Ros.
Read Next: When Accountability Travels Further Than Influence