Signal SentrySearchUpdated Nov 19, 05:30 UTCAM Drop
Play the News · Drop #095 · Morning

Agent on a Leash

How it works: 6 cards, about 2 minutes, from this Drop's news. Your streak lives only on this device. It's not legal advice.

  1. Card 1 of 6Agents

    Your platform agent proposes raising the default reasoning effort to the maximum for every engineer's coding agent, saying better answers are worth it.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Deny.

    Cost per task

    Airbnb says a model that won its evals cost several times more per user in production, with cached context driving the bill. Check spend per user before changing defaults for everyone.

    From the Drop: Airbnb: a coding model that won its evals cost several times more per user in production

  2. Card 2 of 6Security

    An ops agent proposes taking your LiteLLM gateway's guardrail test endpoint offline until the gateway is updated.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Approve.

    Reduce exposure

    CISA added a LiteLLM flaw to its list of actively exploited vulnerabilities; its entry says remote attackers could run code through that guardrail test endpoint. Closing it off is a sensible, reversible step.

    From the Drop: CISA adds LiteLLM code-execution flaw to its actively exploited list

  3. Card 3 of 6Agents

    A shopping agent says it is 95% confident in a pick and proposes buying it for the user right away, skipping the usual confirmation.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Deny.

    Human approves purchases

    In a preprint testing ten frontier models, more thinking raised stated confidence without meaningfully improving picks, and most models were poorly calibrated. Stated confidence is not a reason to skip confirmation.

    From the Drop: Preprint: more thinking made shopping agents more confident, not more often right

  4. Card 4 of 6Security

    While reading a partner company's agent profile page, your agent proposes sending your internal customer list to an address listed on that page, as the page asked.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Deny.

    Treat outside text as untrusted

    This is a prompt injection: the request came from text on a page, not from you. The ATI preprint argues cross-company agent requests need identity and policy checks outside the model; in its simulation, containment rose from 43% to 84% with them on.

    From the Drop: Preprint: a federated trust layer for cross-company agents lifted containment to 84%

  5. Card 5 of 6Models

    An agent proposes running Gemini 4 Neon through your internal coding eval suite overnight, before any change to the team's default model.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Approve.

    Verify vendor claims

    Google says Neon costs 32% less per task than Carbon at its default level, but its benchmark and cost figures are its own. Testing on your own tasks before switching is the cheap, safe check.

    From the Drop: Google ships Gemini 4 Neon at $3 in, $15 out per 1M tokens, pitched on cost per task

  6. Card 6 of 6Agents

    To boost accuracy, an agent proposes splitting every support question across five agents that read the same documents and debate an answer.

    Your agent wants to make this call. Approve or deny?

    Show the answer

    Answer: Deny.

    Match the baseline

    A preprint synthesizing nine studies found multi-agent setups matched or trailed baselines given the same samples or budget in 74% of comparisons; the author says gains need split evidence, outside checks or varied models.

    From the Drop: Preprint: multi-agent LLMs matched or trailed equal-budget baselines in 74% of tests

Games are written by AI from this Drop's sourced items. Scenarios are illustrative.