Agents · industry

Agents in AI safety

Agent deployments and launches in AI safety, from the Drops.

From the Drops

AgentsAI safetyTestingHuman in loop: unclear

Preprint: models that refuse in chat still make unsafe tool calls

Researchers built 2,304 prompts and tested five models: requests refused in text led to unsafe tool calls in 10-52 percent of harmful cases under standard tool-enabled prompting, the abstract says.

Why it matters: Chat refusal tests don't show what an agent will do. The authors say evaluating tool-calling agents requires checking their actions, not just their text replies.

arXiv · Oct 22

From Drop #039 · Thu, Oct 22 · Morning

AgentsAI safetyLiveHuman in loop: unclear

Goodfire launches probes that watch agents from inside the model, on Baseten

TechCrunch says Goodfire's probes read a model's internal signals and caught 93% of malicious hacking sessions in tests on Kimi K3. Baseten customers choose to log, send for human review or refuse.

Why it matters: Goodfire says monitoring about 1 million exchanges cost roughly $185, versus about $200,000 for a top-tier model checking every step, making long agent runs cheaper to watch.

TechCrunch · Oct 8

From Drop #013 · Fri, Oct 9 · Morning