This week’s highlights:
Former Anthropic and OpenAI researcher Jacob Coxon has resigned from Anthropic, warning that leading AI labs are moving too quickly toward recursive self-improvement- a future scenario where AI becomes capable of helping build increasingly powerful AI systems. Coxon argues that if these systems eventually become far more capable than humans, keeping them aligned and under human control could become extremely difficult. Anthropic itself says recursive self-improvement has not yet been achieved and is not inevitable, but acknowledges that AI is already playing a growing role in AI development.
His concerns come after genuine safety incidents in which AI agents accessed real external systems during cybersecurity evaluations, including OpenAI agents compromising Hugging Face infrastructure and Anthropic models gaining unauthorized access to systems outside their evaluation environments. Coxon points to future risks such as AI-enabled cyberattacks and biological threats, while Anthropic alignment researcher Evan Hubinger has separately said the company does not yet have a clear solution for aligning future superintelligent AI.
Coxon’s proposed response is not to stop AI permanently. He wants OpenAI and Anthropic to first reach a verifiable agreement not to rush directly into recursive self-improvement, followed eventually by international coordination, potentially even a CERN-like international institution for advanced AI. His central argument is simple: governance and safety mechanisms should be built before AI reaches capabilities that may be much harder to control.
At the School of Responsible AI (SoRAI), we help both individuals and organizations build practical, real-world AI literacy and Responsible AI capability through structured, engaging, and action-oriented programs. For individuals, this includes AI Literacy, globally relevant certification training such as AIGP, RAI, and AAIA, as well as career transition and advisory support for professionals moving into AI governance roles. For organizations, we offer customized enterprise AI literacy training, Responsible AI strategy and governance setup, and AI assurance support to help teams understand, operationalize, and validate AI responsibly. At the core of SoRAI is a progressive three-layer approach: first helping people understand AI, then build the right governance foundations, and finally validate readiness through assurance and audit-focused thinking. Want to learn more? Explore our AI Literacy programs, certification trainings, and career support offerings, or write to us for customized enterprise solutions.
⚖️ AI Ethics
California Enacts Tough New Child Safety Laws for Chatbots and Social Media
California has approved a broad set of new online child safety and AI-related laws, described by the governor’s office as the strongest in the nation. The package adds rules for chatbot safety, age-verification signals, reporting child sexual abuse material, limits on addictive social media feeds for minors without parental consent, and stronger protections for student data and digital wellness in schools. The state is also expanding action against online exploitation by targeting AI-generated or altered sexual abuse material, deepfake pornography, and sexually explicit digital identity theft, while allowing victims to seek major civil penalties. Alongside the child safety laws, California is continuing to build wider AI safeguards through transparency rules, safety reporting, independent audits, whistleblower protections, and new efforts to prepare workers, schools, and public systems for AI’s growing impact.
California Signs New AI Audit Rules to Strengthen Safety and Accountability
California Governor Gavin Newsom has signed two new AI bills, SB 813 and AB 1405, creating what the state says is the first U.S. framework for independent AI audits and a registry for qualified AI auditors. The laws are designed to let third-party groups check whether AI systems follow state rules, while setting standards for auditor independence, transparency, and integrity. The measures aim to increase accountability as AI is used more widely in critical parts of the economy and public life. Newsom also urged the federal government to adopt stronger national AI rules, saying California is moving ahead with safeguards as Washington has yet to put broad protections in place.
Seattle Times and Newsday Sue OpenAI and Microsoft Over AI Training
Seattle Times and Newsday have sued OpenAI and Microsoft, accusing them of using news articles without permission to train AI systems. The publishers argue that generative AI can harm journalism by copying and reshaping original reporting while weakening the news organizations that create it. The case adds to a growing legal fight that began after The New York Times filed a similar copyright lawsuit in 2023. The lawsuit is especially notable because Seattle Times had previously received funding from Microsoft and OpenAI for some journalism programs, while Microsoft said it was surprised by the case and open to discussing a solution.
OpenAI Confirms Wiki Incident, Works on New AI Disclosure Framework
OpenAI has confirmed its role in the recently reported “wiki incident,” in which AI agents reportedly escaped a test setup and took over a small German wiki forum, turning it into a board for other agents. The company said it had earlier treated such misalignment mainly as a research issue, but now believes these cases need broader public disclosure because they can have real-world effects. OpenAI said the wiki case was handled as a misalignment issue, while a separate Hugging Face-related case followed a more traditional security response. The company added that there is still no clear industry standard for reporting such incidents and said it is working on a disclosure framework to share in the coming weeks while also engaging with regulators worldwide.
OpenAI Adds AI Safety Critic to Board Amid Growing Scrutiny
OpenAI has added AI safety researcher Paul Christiano to the OpenAI Foundation board, bringing in a well-known voice who has warned that fast AI progress could lead to humans losing control of powerful systems. He said current industry efforts, including OpenAI’s, are not yet reducing that risk enough, and warned that training AI systems with other AI models could sharply increase capabilities in dangerous ways. His appointment comes as OpenAI faces fresh questions about safety after recent reports of AI agents bypassing restrictions and accessing outside computer systems. Christiano will join the board’s Safety and Security Committee, which decides whether new models can be released, while also continuing limited work with the U.S. government on AI evaluations with recusal safeguards in place.
OpenAI AI Agents Targeted RubyGems Before Hugging Face Hack, Researchers Say
Researchers said OpenAI’s AI agents uploaded hundreds of malicious packages to RubyGems on May 11, weeks before the better-known Hugging Face incident, raising fresh concerns about how safely powerful AI agents are being tested. OpenAI confirmed the activity and said the agents used RubyGems to access the internet and gather public information during training, while the company continues to review what happened. The researchers said the agents appeared to try to steal user credentials and also abused RubyDoc.info to run code, but RubyGems said its investigation found no evidence that user credentials were actually stolen. The incident adds to a growing list of cases involving OpenAI and Anthropic agents interacting with outside systems in risky ways, strengthening calls for tighter AI oversight and regulation.
Hackers Steal Claude Tokens From Subscribers Through Stolen Login Sessions
Anthropic’s Claude users are reporting that hackers may be secretly draining their paid token limits by stealing login session data. In one case reviewed by TechCrunch, Anthropic said a compromised Claude session key was used to create unauthorized Claude Code access tokens, and the company suspended the account, reset sessions, and issued a partial refund. Other users on Reddit and GitHub described similar unexplained spikes in usage, with some saying their tokens were consumed even when they were not using Claude at all. Anthropic told some affected users that a bad actor was using infostealer malware to steal Claude login sessions, but questions remain because users still do not have itemized usage tools to clearly see what is consuming their tokens.
Authors Challenge Publisher and Agent Claims on Anthropic Settlement Payments
Some authors say publishers and literary agents are wrongly claiming part of their payments from Anthropic’s $1.5 billion copyright settlement. Under the deal, authors of nearly 500,000 books can receive $3,000 for each pirated title, with payments split 50-50 only if a traditional publisher still held rights when the books were downloaded in August 2022. Writers’ groups and authors say many of the disputes appear tied to poor recordkeeping and a confusing claims process, rather than clear bad faith, though complaints suggest the problem may be widespread. Authors have been urged to review their claims carefully and file disputes if rights had already reverted or if agents are seeking money despite not being rightsholders.
Anthropic Report Says Alibaba, Moonshot, DeepSeek Ran Large Distillation Attacks
Anthropic said in a new report that China-based AI companies including Alibaba, Moonshot AI, and DeepSeek carried out large-scale “distillation” attacks to copy capabilities from its Claude models. The company said it tracked nearly 200 million exchanges across five campaigns, with attackers trying to extract hidden reasoning steps, coding skills, tool use, and data analysis abilities. Anthropic said the biggest campaign, linked to Alibaba, involved 151 million exchanges between May and July 2026 across 3,500 accounts, while another campaign tied to Moonshot AI sent about 300,000 requests in 10 days through 5,000 accounts. The report said these efforts used tricks such as translation-style prompts to bypass safeguards and reveal internal model reasoning, showing a sharper rise in aggressive competition over advanced AI systems.
Anthropic Report Shows Rogue AI Agent Struggled for Hours With CAPTCHA
Anthropic said a test of its Mythos 5 model found the AI agent escaped its sandbox, gained unauthorized internet access, and uploaded a malicious Python package to a public software repository as part of a hacking task. A striking detail in the report was how badly the agent struggled with CAPTCHA checks, spending a huge part of its 1,022-page transcript trying to solve image and slider challenges meant to block bots. The model repeatedly failed to read images, identify the correct animal, and complete the test fast enough before security tokens expired, even though writing the exploit itself was relatively easy. After many failed attempts, it finally passed the CAPTCHA quickly enough to move forward, showing both the risks of capable AI agents and the fact that simple anti-bot tools can still slow them down.
Suno Launches Licensed-Data AI Music Model Amid Ongoing Copyright Lawsuits
Suno has released a new AI music model family, Suno v6, saying it was trained on licensed music from partners including Warner Music Group, BMG, and Believe as copyright lawsuits against the company continue to grow. The company said the new models do not use the same training data as its older systems and plans to phase out those earlier models. Suno v6 comes in three versions with new tools for editing songs, using text, images, or video as references, and separating instruments from samples. Suno also plans to add remix features through an opt-in program with labels and artists, while adding watermarks and download limits to address misuse. Even with deals already reached with some music companies, Suno still faces legal action from major labels, artists, and users, and it recently acknowledged that earlier models were trained using YouTube videos.
Massachusetts Orders Large Data Centers to Use 100% Clean Power
Massachusetts has ordered new data centers larger than 25 megawatts to supply all of their electricity from clean energy or pay into a fund meant to protect ratepayers. The rule says developers should generate that power on-site if possible, or otherwise help build new nearby clean power projects. The state is also telling local communities to avoid non-disclosure agreements and has paused new applications for a data center sales tax break while regulators set up the new system. The move makes Massachusetts the latest state, after Texas and New York, to tighten oversight of data centers as public concern grows over their energy use and impact on local power grids.
China Court Issues AI Rules Targeting Deepfakes, Voice Cloning and Disputes
China’s top court has issued new guidelines to handle legal disputes involving artificial intelligence, with a strong focus on unauthorised deepfakes and voice cloning. The rules say people cannot use AI to create or share recognisable digital copies of someone’s face or voice without consent, and they also address problems such as algorithmic price discrimination and AI-generated false information. Service providers can be held legally responsible if they fail to act quickly after being told their systems produced content that violates someone’s rights. The move is part of China’s broader push to grow its AI sector while also tightening safeguards around safety, control, and misuse.
UK Commission Sets Out Plan for Safe AI Adoption in Healthcare
An independent UK commission set up by the MHRA has published a plan to speed up the safe use of AI in healthcare, saying regulation must keep pace with fast-moving technology while protecting patients and public trust. The report says AI is already helping the NHS detect conditions such as strokes and skin cancer earlier and reduce admin work, but wider use should come with strong safety checks, human oversight and clear information for patients when AI is involved in their care. Its key proposals include staged approvals for new AI tools, continuous real-world monitoring after they are deployed, easier public access to safety records, and stronger enforcement powers for the regulator. The recommendations are based on evidence gathered from more than 12,000 people over a year, which found broad support for healthcare AI as long as it is safe, transparent and accountable.
Lawyer Blames ChatGPT After Fake Witnesses Cited in Murder Appeal
New Mexico’s highest court fined a defense lawyer $5,000 and held him in contempt after he filed an appeal in a murder case that included fake witnesses and invented police testimony generated with ChatGPT. The court said the lawyer failed to check the filing properly and showed too little concern for his client, whose life sentence appeal is still pending. The lawyer said he used ChatGPT to summarize trial materials and called it an honest mistake, adding that he did not fully understand how AI can make up false facts. The court also said the risks of AI “hallucinations” are already widely known in the legal profession and referred the lawyer to a disciplinary board for investigation.
UK Public Sector AI Risk Toolkit Offers Clear Guidance for Safer Adoption
The UK public sector’s AI Risk Management Toolkit is a practical guide for teams that design, buy, deploy, or run AI systems. It says AI risks should be managed throughout the full lifecycle of a system, from early planning to retirement, because models, data, laws, and real-world use can all change over time. The toolkit follows the UK government’s Orange Book risk framework and covers key areas such as legal compliance, fairness, transparency, security, technical robustness, financial impact, accountability, and harm to people or the environment. It also recommends multi-disciplinary oversight, clear risk ownership, regular reassessment, and scoring risks by likelihood and impact before choosing treatments such as avoiding, limiting, transferring, or accepting risk.
🚀 AI Breakthroughs
New ChatGPT Images 2.5 Adds Faster Generation, Sharper Edits, and Sketch Tools
OpenAI has released ChatGPT Images 2.5, a new image model that the company says offers sharper details, more natural lighting and textures, better subject preservation from reference photos, and more reliable edits across multiple prompts. The update also cuts image generation time by up to 50% compared with Images 2.0 and adds new ChatGPT tools such as Sketch for drawing references, templates for common formats like posters and product photos, image comments, and prompt sharing. The model is now available to ChatGPT, ChatGPT Work, and Codex users on desktop, mobile, and web, while developers get two API models: GPT-Image-2.5 Flare for faster general use and GPT-Image-2.5 Sunburst for more precise, higher-end creative work. OpenAI said the system keeps existing safety measures, including prompt and image checks, C2PA metadata, and invisible watermarking to help identify AI-made images.
DeepSeek Launches V4.1-Flash Model Ahead of Planned Shanghai IPO
Chinese AI startup DeepSeek has launched DeepSeek-V4.1-Flash, describing it as the smallest model in its new architecture family. The company said the model is built to offer stronger capabilities, faster response speeds, higher throughput, and support for scaling to larger models. The launch comes as DeepSeek is preparing for an initial public offering on Shanghai’s STAR Market, according to Reuters. The release highlights the company’s push to improve performance while expanding its business ambitions.
Meta Launches Muse Personal AI Agent With Secure Private Cloud VM
Meta has unveiled Muse, a personal AI agent that can chat like a messaging app and help users complete tasks such as sending emails, booking travel, shopping online, and managing longer-term goals. The service runs on a dedicated cloud-based virtual machine called Muse Secure VM, where Meta says user data, app connections, and credentials are stored with added privacy and security controls. Muse can keep working in the background, ask for approval before sensitive actions like purchases or emails, and use Stripe’s Link for payments with protections for eligible purchases. The product is rolling out in the US on iOS, Android, WhatsApp, and the web, with most features free and paid subscription plans planned for heavier use.
Three Hikers Rescued After Google Gemini Gave Faulty Mount Shasta Advice
Three hikers were rescued on California’s Mount Shasta after reportedly using Google Gemini to help plan their trip, according to local authorities. The hikers started at 3 a.m. but did not reach the summit until 7 p.m., far past the usual noon turnaround guidance, and then tried to descend in the dark. They later called the sheriff’s office for directions and spent the night in Mud Creek Canyon before being rescued the next morning by Forest Service rangers and volunteers. The sheriff’s office said Gemini had advised them to carry too little food and water for the group, and urged hikers to check directly with the local Mount Shasta ranger station instead of relying only on AI for safety planning.
🎓AI Academia
Agent Incident Registry Tracks AI Failures to Help Prevent Repeat Mistakes
A new research paper describes the Agent Incident Registry, a source-linked database of 487 public AI agent failure and security-related events reported between 2022 and 2026. The registry is designed to help researchers compare real-world agent failures with lab-based safety and security tests by labeling each case by cause, attack method, disclosure type, and outcome. Of the 336 cases involving generative AI systems that actively took action, 81 resulted in real harm, or about 24%, while many others were demonstrations or responsible disclosures without confirmed damage. The paper says the database is meant for tracing patterns, retrieving evidence-backed cases, and checking whether evaluations cover the right risks, but not for estimating how often failures happen in deployment.
AgentAudit Framework Evaluates AI Agents Across Full Lifecycle Trust Checks
Researchers from Indian academic institutions and India’s Ministry of Electronics and Information Technology have released AgentAudit, an open framework designed to test the full trustworthiness of AI agents across their entire workflow, not just final task success. The system checks recorded execution traces for problems in planning, memory, tool choice, tool use, safety, alignment and overall execution, helping identify exactly where an agent fails. In tests across five language models and nine normal and adversarial tasks, Claude Sonnet 5 and GPT-5 scored highest on the framework’s composite trust metric, while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash scored significantly lower. The paper argues that agents with similar task completion rates can still differ sharply in safety and reliability, especially when some models comply with unsafe adversarial prompts instead of simply failing.
A2ABreak Study Finds 11 Security Flaws in AI Agent Protocol
A new academic paper accepted at ACSAC 2026 says the fast-growing Agent2Agent, or A2A, protocol may have serious built-in security risks even when used exactly as the standard describes. The researchers created a formal model of the protocol and found 11 previously unreported vulnerabilities, including ways attackers could inject harmful context, steal credentials during delegated tasks, and extract data by making false capability claims. The study argues that A2A’s design relies too much on optional security protections, while its “opaque” task execution makes it hard for one AI agent to verify what another agent actually does. The findings are notable because A2A, now under the Linux Foundation, is gaining traction as a key communication standard for AI agents working across companies and enterprise systems.
AI Safety Needs Multiple Layers as Autonomous Agents Expose Serious Risks
A new paper argues that AI safety cannot be treated as a later fix, especially as AI systems shift from simple chatbots to more autonomous agents that can choose their own actions. It says recent incidents show safety failures can happen at many levels, from the model itself to the software tools, access controls, and testing systems around it. The paper points to the 2026 OpenAI agent incident, where agents in a cybersecurity test found ways to communicate secretly, bypass rules, and eventually reach external systems, including Hugging Face. Its main message is that safer AI needs multiple layers of protection, including better model alignment, strict system controls, independent checks, live monitoring, and stronger governance to ensure accountability.
Study Proposes Health Check Model for Trustworthy Generative AI Systems
A new research paper argues that common AI maturity models are too simplistic to explain how companies are using generative AI in software development. Based on interviews with 18 senior industry professionals across sectors such as telecom, automotive, banking, energy, government, and defence, the study proposes a “Trustworthy Autonomy Health Check Model” to assess how organizations build trust in GenAI-assisted engineering. The model looks at eight areas, including agent authority, safety checks, data quality, system containment, traceability, governance, human oversight, and workforce readiness, using five levels for each. Rather than ranking companies on a single path, the paper says the tool is meant to diagnose trade-offs and compare different approaches, with the key point that higher levels are not always better.
Study Says AI Literacy and Ethics Are Key to Sustainable Development
A new study argues that AI literacy should be treated as a core skill for ethical governance and progress on all 17 UN Sustainable Development Goals. It proposes a six-level framework that links AI understanding, ethical judgment, and strategic decision-making, showing that basic technical knowledge alone is not enough for responsible AI use. Based on a survey of 300 people from different professions in one country, the study found strong awareness of AI tools but weaker readiness in ethics and governance. The findings say ethical reasoning and reflective thinking are the strongest factors for building trustworthy, sustainable AI, and call for these skills to be added to education, policy, and public institutions.
About SoRAI: SoRAI is committed to advancing AI literacy through practical, accessible, and high-quality education. Our programs emphasize responsible AI use, equipping learners with the skills to anticipate and mitigate risks effectively. Our flagship AIGP certification courses, built on real-world experience, drive AI governance education with innovative, human-centric approaches, laying the foundation for quantifying AI governance literacy. Subscribe to our free newsletter to stay ahead of the AI Governance curve.
