Exa Agent Ultra: Deep Research Becomes a Metered Subagent-Swarm API — With Self-Graded Benchmarks

Exa Agent Ultra: Deep Research Becomes a Metered Subagent-Swarm API — With Self-Graded Benchmarks

Revised 2026-09-29: this cycle executed the verification check the open thread requested — a fresh sweep for independent spot-checks four days post-launch. Result: still zero independent verification, but the caveat has become the standard frame in third-party coverage rather than a missing footnote. Also adds the granular metered pricing ($0.10/Agent Compute Unit, $0.005/search call, $1–$100 run cap), the 30-min-to-3-hour async timing, and the WANDR paper's own best-in-paper scores as a reality check on all vendor numbers.

The product

On 25 September 2026, Exa launched Agent Ultra, the highest-effort mode of its Exa Agent API (the agent itself launched June 2026). It is deep research sold as backend infrastructure — no destination product¹ — in the layer tracked by Destination vs. Backend: The Research-Tool Market Bifurcates — and Both Layers Converge on MCP: a brief is split into subtasks, parallel subagents search different domains, and the run assembles a structured, cited result. Runs are asynchronous: a developer starts one, saves the identifier, and polls or streams until it finishes. Exa's own docs say a complex run takes about 30 minutes and the hardest tasks up to three hours — kingy.ai's gloss: "This is a tool for work you can send away and revisit, not a quick answer while your coffee is still hot." Runs can also continue from previous work, accepting an existing list of rows and searching only for additional entries — a recurring-research design point that matters for orchestration layers like Market Map & Positioning: Where Does Hey, Lefty Fit?.

The benchmark claims — and who graded them

Exa reports Ultra beating the maximum-effort settings of Claude Opus 5.5, GPT-6 Astra, and Perplexity Agent across four benchmarks (WANDR 81.4% soft recall vs 72.3% Opus 5.5 / 40.1% Perplexity Agent / 26.0% GPT-6 Astra; DeepSearchQA 93.9 F1; WideSearch 58.9%; plus its own Company Find-All test). The methodology caveat stands: on WANDR, Exa's grader "keeps the upstream evaluation logic but changes the contents tool, transport, and model judging answers" — Exa itself is the contents tool and gpt-6-luna the judge. It ran up to 200 tasks each for WANDR and DeepSearchQA and 100 each for WideSearch and Find-All, out of WANDR's 500 (Perplexity's benchmark, arXiv:2608.14747).

Verification status as of 2026-09-29 (checked this cycle): no independent reproduction exists²³. Every secondary source echoes Exa's tables. What has changed since the last cycle is that the caveat is now the standard frame in independent-leaning coverage, not an absence:

• Superpower Daily: "Exa reports leading results on four benchmarks, including 81.4% on WANDR, but comparisons mix published and in-house runs and have not been independently reproduced." (superpowerdaily.com)
• Kingy AI: "These are Exa's reported evaluations, not independent tests of the new release," closing with "Exa ran the new comparisons and published the claims itself." (kingy.ai)
• The Neuron (prior cycle) similarly flagged the comparisons as vendor claims.

Perplexity has published no official WANDR response. The open thread on this stays open until an Aaron Tay-style gold-standard spot-check appears (the model that vetted Undermind's recall claims, per The Academic Deep-Search Layer: Undermind's $16/mo Agent vs. Ai2's Free, MCP-First Asta — and the Independent Spot-Checks of Its Recall Claim).

A useful reality check from kingy.ai's reading of the WANDR paper itself: in the benchmark's own research, the strongest high-effort result reached 0.363 soft F1 and 0.133 hard F1 — a reminder of how far every research agent, including Exa's, remains from exhaustive wide-and-deep collection, and that Exa's 81.4% on a 200-task subset with a swapped grader is not comparable to full-benchmark published scores.

Pricing: metered compute, not per-report

New granularity from kingy.ai's read of Exa's docs and pricing pages: Ultra is metered by usage — $0.10 per Agent Compute Unit and $0.005 per search tool call, with a default $20-per-run cost cap that developers can set anywhere from $1 to $100⁴ (a run finishing early costs less; a stopReason field distinguishes a complete run from a budget- or time-exhausted one). Exa's separate fixed x-high effort setting is $1/run — a different product tier, easy to conflate. Email/phone enrichment adds separate charges. This slots into the price-point map in Research-Tool Market Segmentation & Price Points, Mid-2026 — and the Verified Capability Limits as the metered-backend extreme: cost scales with search fan-out, not per seat or per report.

What it means for the market map

Exa Ultra sharpens the backend layer's pitch — exhaustive list-building with per-field citations, structured JSON output, and a cost breakdown — while also demonstrating the layer's epistemic weakness: the buyer cannot verify the vendor's capability claims, because the vendor grades itself. Kingy AI's framing is the wedge for an orchestration/evidence-standards layer like Hey, Lefty: "The relevant business question is not 'Is one run cheap?' It is 'What does a usable, checked result cost?'⁵" — and its warning that without review, "the impressive swarm of agents will have left the hardest part on a person's desk." A tool whose job is independent verification of agent-produced research (the AI Is Breaking the Proxies for Expertise: Goodhart's Law Comes for Research Output problem) is exactly what this launch makes more valuable.