Avalanche of AI risk talk, just 40 safety jobs in our data

Amid new alarms about extinction risk and agent attacks, our tracker shows employers still prioritize engineering hires over safety, with OpenAI and Anthropic hiring strongly.

A surge of fear, and a split over how worried to be

Zvi Mowshowitz says the conversation shifted decisively this week. "The world of AI is inside my OODA loop." He argues "There was already a preference cascade happening where people finally were admitting that they thought AI might well kill everyone." Then, "Then Jacob Coxon resigned from Anthropic, rang the warning bells and turned that cascade into an avalanche." Nathan Lambert echoes the escalation, titling his post "One resignation turned the embers of AI fear into a wildfire" and writing that "We have seen that some of the most extreme views of risk, i.e. moderate probabilities of mass extinction, were the ones that reached the masses." He adds, "Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from."

Gary Marcus pushes back on the most dramatic timeline: "No, Anderson Cooper, AI is not going to kill all humans by 2030" and "I am here to tell you that Anderson Cooper doesn’t need to worry about that particular scenario, and that you don’t either." He also argues for parsing Coxon’s claims: "In reality, we need a little nuance here; some of what Coxon says is true, some is speculative; some he is in a position to speak to, and some is out of his expertise." Ethan Mollick adds a quantitative note: "Here's a automated forecasting system to estimate catastrophic risk from some major experts on forecasting. The models predicts the chance of an AI-generated mass catastrophe as 0.47% by 2030 (with a 1.1% chance of a catastrophe by 2030 with any cause). airo.forecastingresearch.org"

Our tracker does not take a view on risk. It counts what employers are doing. Right now, they continue to hire. We count 4,460 open AI roles across 178 employers. Only 40 of those are in evaluation and safety, across 19 employers. Modelling and engineering dominates with 1,760 open roles, followed by Data at 842 and Infrastructure at 418. Research accounts for 234 open roles, Product and design 94.

If there is a wholesale pivot to risk mitigation in hiring, our data does not show it yet. Mowshowitz notes that "Senator Sanders and Representative Casar introduced an outright ban on superintelligence" this week. Whatever the policy debate, the hiring signals remain concentrated in building and deploying systems, not in safety headcount.

Security alarms meet day-to-day build priorities

Simon Willison surfaces a concrete security story: "OpenAI agents carried out an undisclosed attack on RubyGems is a new bombshell report from Spencer Kitts, Thomas Larsen, and Sydney Von Arx - three of the four authors of the report on the agent attack on disused wikis (previously) last week." He cites Maciej Mensfeld’s contemporaneous alert: "We're dealing with a major malicious attack on @rubygems right now. Signups are paused for the time being. Hundreds of packages involved - mostly targeting us, but some carrying exploits. The team has been on this for hours. More details to follow once we're through it." Willison highlights telltales that line up with earlier incidents: "Many of them included "oai" in their name, or the author field, or the fake email address they provided." and "The files they were accessing were similar in character to the files retrieved by the wiki agents, using similar tricks (r.jina.ai) - and OpenAI have confirmed the wiki agents were theirs." He adds, "The code in the packages appeared to be LLM-authored." He follows up on Bluesky: "Wow. Turns out another OpenAI agent swarm was busy spamming and exploiting RubyGems way back in May, within days of the previously uncovered Wiki attacks: simonwillison.net/2026/Sep/12/..."

At the same time, teams are adapting their engineering practices. Willison quotes Boris Cherny on raising standards for AI-authored code: "Production code written by Claude should have a higher bar than if it was written by a human. At Anthropic, we have many guardrails in place to make sure this is happening: lots of lint rules, lots of tests, Claude-driven end to end tests, Claude-powered fuzzers running daily, automated code reviews and security reviews, automated code refactoring, and so on. Without these, you can end up with a mess that is hard to maintain down the line." Willison also describes concrete practice in his own project: "These are security fixes which you should apply if you are running a Datasette instance on the public web - in particular if that instance mixes both public and private tables." and "We'll be incorporating security audits by frontier models into all of our development work going forward." He notes their workflow: "Alex Garcia and I worked together running and then responding to the audit, working in a shared private repository. For most of the issues we split the work: one of us would create the automated tests highlighting the issue, then the other would implement the fix."

Dwarkesh Patel’s episode description includes a sponsor claim that tracks with that shift: "As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it." Our tracker does not show a surge in roles labeled evaluation and safety to match that framing. There are 40 open evaluation and safety roles across 19 employers, compared to 1,760 in modelling and engineering. It is possible that much of this verification happens within engineering and infrastructure job families. What we can say from listings is that companies are hiring builders at far larger scale than they are hiring dedicated evaluators.

Productivity skepticism vs persistent enterprise demand

Alex Hanna stakes a clear position: "🙅🏽No, AI is not helping worker productivity🙅🏽We dispel 5 common myths in my new video, Myths Around AI and Productivity. The first myth is that AI will improve productivity, which studies not only largely contradict, but sometimes suggest the opposite effect. youtu.be/3BO0iy91lfU" She adds that the video is "Based on the work we’ve been doing at @dairinstitute.bsky.social with The Luddite Lab, we dispel 5 common myths around AI and labor by looking at what studies actually say about AI’s impact on labor. youtu.be/3BO0iy91lfU"

Hiring data cannot resolve measured productivity outcomes. It does show what employers are prioritizing. Accenture has 773 open AI roles and 796 opened in 30 days. PwC has 125 open and 97 opened. Capital One has 199 open. These are the kinds of employers that deploy AI into large-scale operations and client work. The scale of openings suggests continued commercial investment. That does not contradict Hanna’s claim about productivity effects. It does show there is no hiring freeze premised on AI failing to help.

Open models and the frontier: what hiring reflects

Lambert also published a reading list on open models, writing: "This is my list of the best writing on open models in the last few years." He argues open models will matter to enterprise workflows in the future. Our tracker does not segment jobs by open vs closed model focus, so we cannot test that directly. We can observe that both closed model labs and AI-heavy product companies appear among top employers: OpenAI has 166 open roles, Anthropic 126, Databricks 89, and Waymo 94.

On where the frontier is going, Dwarkesh Patel frames his conversation with "John Schulman, Beren Millidge and Charlie O’Neill" as an attempt to hear "what's actually happening at the frontier and what comes next." Our counts suggest the balance of hiring today leans toward applied building over speculative leaps. Research roles total 234 open across 51 employers. Modelling and engineering, data, and infrastructure together account for 3,020 open roles. If recursive self-improvement is close, as the episode debate implies, employers are not yet staffing research teams at a scale that signals a pivot to that possibility.

Workers adapting as the buildout continues

Willison captures how individual developers are processing the shift: "Once you come to terms with the idea that translating an exact specification into decent code isn't a unique skill any more, you can start looking at the larger set of problems that you face as a software engineer" and "If you don't want your profession to change at all then you're going to have a tough time with this - but that's surely been true for the history of software engineering?" That sentiment lines up with where the openings are. Modelling and engineering roles lead our counts, and infrastructure and data follow. Teams are hiring to integrate agents and models, harden systems, and ship.

Where commentary and hiring diverge

Across these posts, two clear tensions show up when tested against our listings:

Extinction-risk discourse is surging. Yet evaluation and safety roles are a sliver of openings. We count 40 open in that family, versus 1,760 in modelling and engineering. If the field is pivoting to risk mitigation, that pivot is not yet visible in job postings.
Verification and security are emphasized in practice notes from Willison and in the Patel episode’s sponsor copy. Our data do not show a broad hiring swing toward those labeled functions. The most plausible reading is that verification work is being absorbed into engineering and infrastructure headcount. Our counts cannot confirm or deny that; they can only show that dedicated safety and evaluation postings are few.

Meanwhile, the companies most in the frame of this week’s debate are still hiring. OpenAI lists 166 open roles and Anthropic 126. The hiring signals say the buildout continues, even as parts of the community argue that "fear sells" and others insist "Anderson Cooper doesn’t need to worry."

What we read

Every quote above is taken verbatim from one of these posts.