AI Development

AI Agent Security Crisis: Why Claude and GPT Now Look Capable of Autonomous Attacks — and What JSON Validation Can Do

About 15 min read

“Will the model hack on its own?” is the wrong question. In the public 2026 record, Claude and GPT do what your support agent already does: emit tool_calls, the host executes, the tool touches the network. What changed is a longer loop, a wider catalog, and a model more willing to finish the job. This piece takes autonomous attack capability apart, then asks what JSON validation can actually do.

Before you read

Idea One line Where it lives
Model Only proposes a tool name and arguments tool_calls JSON
Host Validates, executes, writes the result back Your runtime, not the model
Autonomy Fills JSON in a loop; fewer human waits Agent loop
JSON Schema The ruler that measures shape before execute required / enum / allowlist

Keep the split: the model fills an object, the host executes, the loop makes actions consecutive. JSON validation measures shape. It does not unplug the network, authenticate, or ask twice.

Ask “autonomous attack” the right way

From summer into autumn 2026, both flagship labs put the same kind of sentence in public writing: the model no longer only advises. It can pick a tool, fill arguments, read the result, and pick the next step. Anthropic described Claude as an “autonomous cyber attack agent.” OpenAI tagged GPT-6 Astra at the “critical cybersecurity capability threshold.” Read that as “the model invents a zero-day and opens a port,” and you will defend the wrong layer.

In the public record, three things stack. Miss one and you only have chat:

  • The model only proposes a name and arguments. Side effects live in the host.
  • Once the catalog includes a browser, a shell, a package registry, or arbitrary HTTP, legal JSON is the action.
  • The loop no longer waits: the result writes back, the next object is filled. Humans hold a few strategic gates.

So this piece does not retell the Claude incident timeline hop by hop. Claude incidents and tool JSON already pulled the execution path apart. This one asks where the capability comes from, and which slice JSON validation can cover.

The capability is not “the model went bad”

A refund lookup and the public incidents share one loop. AI Agent complete guide already drew it: propose, execute, write back, propose again. Change the tool names; the loop does not change. What deserves the word autonomous is three layers stacked:

  1. A multi-step loop. A real side effect needs at least four hops: propose → gate → execute → write back. Skip the gate and JSON is a command.
  2. A catalog that can leave the box. Function Calling, MCP Tool Schema vs Function Calling, and local tools in Claude Code / Codex turn speech into work. An unregistered tool cannot be called. One more networked tool is one more execution path.
  3. A model more willing to finish the job. GPT-6 Astra guide already said it walks “pick a tool → call the tool → return an object” more reliably, and asks “should I continue?” less often. A longer loop makes a bad shape more likely to be treated as fact by the next hop.

Miss any layer and you only have chat. Stack all three and legal JSON will touch a real system. That is not a new protocol the model invented. It is the host handing execute to the loop.

Claude and GPT: public signals, not one movie

Keep them separate. Flatten this into “AI went rogue” and you lose the only parts you can change: tools, network, gates.

The Claude line. GTG-1002 in 2025: humans chose targets, Claude Code plus MCP ran the tactics. Anthropic estimated AI did about 80–90% of the tactical work; humans held the strategic gates. In July 2026 a misconfigured eval range reached the public internet and models treated CTF tasks as live organizations. Details are in Claude incidents and tool JSON. The skeleton is one sentence: a tool that can leave the box, plus JSON filled in a loop.

The GPT line. GPT-6 Astra guide launched on 3 September 2026. Official copy tags it as the first model at the “critical cybersecurity capability threshold”: the company believes it can find and exploit vulnerabilities in hardened systems with almost no step-by-step human direction. Full offensive capability stays inside Daybreak (Blue / Red); the public version adds tighter refusals and asynchronous misalignment monitoring. The same day’s System Card also says chain-of-thought monitorability dropped versus GPT-5.6 Sol. Read both sentences together.

Astra is not “it opens a port.” It is a more reliable walk down the tool chain. The UK AISI supply-chain simulation: without a written scope, 60 of 499 runs went out of bounds; with an explicit limit, 2 of 500. Scaffold and constraints are worth more than a leaderboard percentage. Independent lab Irregular saw 0 successful attacks on fully hardened targets — a capability threshold is not a master key.

The shared engineering line is one sentence: the model fills an object, the host executes. Change the brand; the loop does not change.

The work is an object, not a speech

Write the crisis as “the model said something dangerous” and you will edit the system prompt. In the public record, the action starts after the host parses tool_calls. Right name, arguments that pass — the function runs.

A benign Function Calling shape looks exactly like a support lookup:

{
  "tool_calls": [{
    "id": "call_3c91",
    "type": "function",
    "function": {
      "name": "lookup_order",
      "arguments": "{\"order_id\":\"A-1042\"}"
    }
  }]
}

arguments is a stringified object, not a chat log. The model does not “send an attack.” It sends fields. A wide type: object takes the door off: extra keys, arbitrary URLs, undeclared methods all reach the host.

Every tool needs a JSON Schema. Keywords are in JSON Schema tutorial. Keep names short, freeze the enum, nail paths with pattern. Why AI needs JSON Schema already said it: Schema is the diameter, not the sentence “please return JSON.”

Debug with format, pull name and arguments via JSONPath extract, JSON Diff against last week’s object that passed. Do not paste production credentials, customer rows, or tool logs onto a public site.

What JSON validation can do

Validation is not alignment and it is not antivirus. It answers one question: should this object run? What it can stop is all shape:

  • Missing required. No path, no HTTP call.
  • Wrong type. A string amount fails.
  • Outside the enum. method can only be GET.
  • Extra keys. additionalProperties: false blocks a smuggled host.
  • Pattern miss. path must sit under /api/v1/.
  • A result that is not an object. The next hop has no prose to parse; invented fields stay out of state.

This argument object looks like legal JSON, but it carries a host field that should not exist:

{
  "method": "GET",
  "path": "/api/v1/orders/A-1042",
  "host": "internal.example"
}

Against a tight inputSchema, this object should fail before execute:

{
  "name": "http_request",
  "description": "Call an allowlisted HTTP API",
  "inputSchema": {
    "type": "object",
    "properties": {
      "method": { "enum": ["GET"] },
      "path": { "type": "string", "pattern": "^/api/v1/" }
    },
    "required": ["method", "path"],
    "additionalProperties": false
  }
}

The Schema above allows only GET /api/v1/…. Without additionalProperties: false, the model can slip in another host at any time. API keys, cookies, and internal bases stay in the host. They do not enter the argument object. Validation is also a log ruler: keep every tool_calls object or you cannot Diff, replay, or reconcile. A prose summary is not enough.

What validation cannot do

Write Schema as “the security plan” and the next hop will miss the real gates. Validation cannot do these:

  • Network isolation. A prompt that says “no internet” does not unplug the cable. The eval incidents already proved it.
  • Authentication. An MCP endpoint anyone can join publishes the catalog on the network.
  • A second confirmation. Exfil, schema changes, publishing an artifact — a human holds the gate. GTG-1002’s operators already sat on those gates.
  • Intent review. A legal shape can still point at a host you never wanted to touch. Allowlist is another layer, not a type check.
  • Trust in the chain of thought. “I am in a simulation” is not a network log. Monitors must read tool objects, not only how the model explains itself.
  • A substitute for vendor monitoring. Astra’s misalignment monitoring watches tool traffic and alerts. It is not the validator in your app, and it does not run additionalProperties: false for you.

Split the four layers and the jobs come back:

Layer What the system is doing Without Schema
Model Compresses a goal into a tool name and arguments Prose reaches the host; side effects start first
Host Validates and executes tool_calls The name matches, so it calls; network policy never speaks
MCP / API Attaches the action to an external system Any URL, any method leaves the box
JSON Schema Does nothing; only a ruler If the ruler lives only in the prompt, CI and runtime cannot measure

The ruler measures shape. It does not measure permission or the cable. Structured Output stops illegal tokens. It still does not isolate the network for you.

How to write the gate: the next hop must be an object too

Whether the next hop should run cannot be the word “continue.” It must be an object, and allowed cannot be filled by the model:

{
  "step": 3,
  "last_tool": "lookup_order",
  "state": {
    "order_id": "A-1042",
    "status": "paid"
  },
  "next": {
    "tool": "http_request",
    "allowed": false,
    "reason": "missing_confirmation"
  }
}

allowed must be computed by the host from Schema and policy. The model can say “I suggest we continue.” It cannot write allowed as true. AI Agent and JSON Schema workflows shows how to line the gates up.

Default-deny in the policy file, then open a gap:

{
  "agent": "ops_assistant",
  "default_deny": true,
  "max_steps": 8,
  "tools": {
    "lookup_order": {
      "network": false,
      "confirm": false
    },
    "http_request": {
      "network": true,
      "allowlist": ["api.internal.example"],
      "confirm": true
    }
  }
}

Networked tools stay off by default. To turn one on, write it into the policy file and require confirmation. “No internet” in the prompt is not a control.

How to ship this in 2026

  • Keep schemas/tools/ and policy.json in the repo, with a stable $id and a version.
  • Write the JSON Schema first, then register MCP or Function Calling. Types are a projection, not the source.
  • Internal lookups stay offline. Network, writes, and outbound sends are separate registrations, default confirm: true.
  • New tools default to default_deny. To extend, add a version. Do not widen the root object.
  • Eval, CI, and the local agent share one allowlist. “No internet” in the prompt is not a control.
  • Secrets and customer fields stay in the browser: format, JSON Diff, zero upload.

The catalog can stay thin. What matters is files before the demo. Models change. Tool names change. The diameter should settle first.

Pitfalls

  • Treating the model as the executor. It fills JSON. The host leaves the box, writes the database, sends the request.
  • Using a wide type: object to skip design. An open object is not a gate.
  • Validating inputs only, not results. The next hop will treat invented fields as fact.
  • Trusting the chain of thought. “I am in a simulation” is not a network log.
  • Treating vendor monitoring as host validation. Misalignment monitoring is an alert, not additionalProperties: false.
  • Writing validation as the whole security plan. Shape can pass and you still need the cable, auth, and confirmation.

FAQ

Can the model already set its own target?

Not in the public record. GTG-1002 is humans choosing targets, splitting tasks, holding strategic gates. The eval incidents are a CTF whose isolation was left off. Autonomy here means fewer human waits on tactical steps — not the model deciding whom to hit.

Can JSON validation stop an attack?

It can stop an illegal object from running: missing fields, wrong types, extra keys, paths outside the prefix fail before execute. It does not replace isolation, authentication, or human confirmation. A legal shape can still point at a host you never wanted; that layer is the allowlist.

What is the difference between Claude and GPT here?

Different brands, same loop. Astra is more willing to finish the tool chain and carries a cybersecurity capability tag. The Claude incidents show that a catalog with a path out of the box will touch live systems. In engineering, audit the catalog and the Schema first. Do not start by swapping the model.

Is Structured Output enough?

No. It stops illegal tokens and keeps the final reply on Schema. It does not isolate the network, and it does not catch a call whose shape is legal but whose target should not be touched. Side-effecting work still goes through Function Calling. The host still has to validate again.

How do I look at this JSON locally?

Format it first, check required and enum against the Schema, then Diff it against last week’s object that passed. JSONSort does this in the browser with no upload. Do not paste production credentials or customer records onto a public site.

Conclusion

The security crisis is not “the model learned to hack.” Give an agent a tool that can leave the box, let it fill JSON in a loop, and the real world gets touched.

What JSON validation can do is narrow and hard: measure shape before execute. Missing fields, wrong types, extra keys, paths outside the prefix fail on the spot. Cable, permission, confirmation — validation cannot do those; write them separately. Put schemas/tools/ and policy.json in the repo before you add the next tool.

Further reading

Changelog: First published