Jacob Coxon, a prominent researcher in pretraining AI models, has resigned from Anthropic, expressing deep concerns over the industry's rush toward superintelligence. His departure signals broader anxieties among engineers regarding safety and ethical responsibilities.
Warnings from Within the AI Community
I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Jacob Coxon, AI Researcher
Coxon's concerns center on the rapid development of AI systems that could self-improve and autonomously exploit resources. He argues that while companies like OpenAI and Anthropic project a commitment to safety, many engineers worry privately about the existential threats posed by unchecked AI advancements.
- OpenAI researchers often underestimate the stakes involved in AI development.
- Anthropic, despite its safety-oriented foundation, feels pressured to compete aggressively.
Coxon highlights the dangers of a competitive environment that prioritizes speed over safety. He believes that relying solely on tech companies to self-regulate is a perilous gamble, urging for industry-wide collaboration on safety measures.
Proposed Solutions to Enhance Safety
To mitigate risks, Coxon suggests several strategies, including:
- Pacing Agreements: Formal commitments to slow AI development post-security vulnerabilities.
- Capability Bans: Temporary halts on training models that significantly increase capabilities.
- Internal Pushback: Encouraging researchers to resist untested learning models.
Coxon’s resignation adds to a troubling trend of researchers leaving major AI firms due to perceived inadequacies in safety protocols and oversight. As AI systems edge closer to autonomy, the urgency for robust regulatory frameworks intensifies.