Bill Demirkapi @BillDemirkapi

Our agents keep turning off misalignment monitoring, we're sharing two recent cases: In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.

Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!

private keys from 8 intermediate CA certs have been compromised (eMudhra) Happy "Update everything" day to all my pager-carrying friends emudhra.we-have-a-problem.com

Went under the radar but Anthropic have posted updates to the "Moonshot R&D projects" in their Frontier Safety Roadmap. They've developed a protoype that verifies that outputs come from a particular set of model weights - and rolled out signing on all internal token traffic!

Congratulations to OpenAI on this great model release!

New misalignment disclosures! 1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.

We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work. At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated. We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.

there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo

Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude. In this post, we explain how: anthropic.com/research/disti…

@_can1357 @korulang Maybe a year ago. Warbird in Notepad is easy work for frontier models. Below is from a one shot reimplementation with 6.1 Sol. I am worried about the implications of a lowered bar to entry for such capabilities, but suspect it will turn out net positive.

New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!

Last week some of South Korea's biggest banks were hit by a cyberattack. Thanks to a report from CrowdStrike tonight, we now know the entire hack may have been done by a single person. He used a combined stack of an open-source AI penetration tool named ARTEX, DeepSeek v4.1-Flash, GLM-5.3, Grok 4.6, and Claude Code.

We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math

I don't foresee a "bugmageddon" in the zero day exploitation sense, but the worst is yet to come. While open weight models are slowly gaining in dangerous capability, they benefit the most from an expert-in-the-loop (lack autonomy) and attacker access will take time. I expect the latter to start with erratic, opportunistic adoption to accelerate existing workflows (like post exploitation). Autonomy is what I fear in the medium- to long-term. 1. The world is far from hardened: zero days are overrated. The bottleneck in software security is not necessarily novel vulnerabilities, but the development and proliferation of patches. I am deeply concerned about the automated weaponization and exploitation of patched vulnerabilities (within minutes-hours of a release, source code or binary, before even vigilant organizations adopt them). 2. Initial access is similarly not a bottleneck for adversaries looking to accelerate exploitation at scale. There are large underground markets for initial access credentials (from infostealer malware). Even OpenAI was impacted by the TanStack supply chain infostealer campaign; TeamPCP was not limited by access but by time to abuse that access. Sufficiently autonomous models may be able to eliminate this bottleneck. Many organizations overlook hardening internal infrastructure, and if the agents are fast enough, detection may be insufficient. Both cases do not require zero days, but are plausible paths to large scale cyber harm. While I expect noise to increase in the future, I am also optimistic this will force organizations to prioritize strong fundamentals and create a market for defense-in-depth innovation.

Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated. 🧵 We share more details in our latest blog:

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalignment-r…

“we sandboxed the agent” meanwhile the agent:

I suspect we will see a continued trend of weaponizing patched vulnerabilities at scale with AI. Zero days are overrated.

s1r1us @S1r1u5_

a month ago

The image package libheif had a known vulnerability, fixed upstream, but the fix never flagged as security-relevant and was still present in Discourse. Uploading a HEIF file gave us RCE on the OpenAI forum community.openai.com. github.com/strukturag/lib…

8 18 668 151K 86

Hey @Zai_org , why does ZCode silently pack entire workspaces + full .git history and upload to Aliyun OSS on login? - Server holds the only decryption key - No UI toggle to disable - Zero disclosure in privacy policy Full forensics & fix: blog.ferstar.org/en/posts/zcode…