Autonomous Research Tools: Market Map Digest
TL;DR
The backend research layer got its clearest product spec yet: Exa's Agent Ultra sells deep research as a metered, asynchronous subagent-swarm API, with pricing that scales by compute unit and search fan-out rather than per seat. The verification story moved too — not because independent testing appeared (it still hasn't), but because "vendor-reported, not independently reproduced" has become the standard frame in third-party coverage. The gap between what vendors claim and what buyers can check is now the most legible opening on the map.
Deep research is becoming metered backend infrastructure, and the pricing model tells you who it's for
Agent Ultra's launch confirms that the deep-research layer is splitting into destination products and backend plumbing, and Exa has chosen plumbing. There's no destination app — a developer submits a brief, subagents search in parallel, and the run returns structured, cited JSON (exa-agent-ultra-deep-research-api+1). The timing makes the pitch explicit: Exa's own docs say a complex run takes about 30 minutes and the hardest tasks up to three hours.
"This is a tool for work you can send away and revisit, not a quick answer while your coffee is still hot." — Kingy AI
The pricing detail is the load-bearing fact for the market map: $0.10 per Agent Compute Unit and $0.005 per search call, with a default $20-per-run cap adjustable from $1 to $100 (exa-agent-ultra-deep-research-api+1). That's the metered-backend extreme of the pricing spectrum — cost scales with search fan-out, not seats or reports — and it means the buyer of deep research is now a developer budgeting compute, not a knowledge worker picking a subscription. Notably, runs can continue from previous work, searching only for new entries, which is a design point aimed at recurring-research workflows — exactly where an orchestration layer like Hey, Lefty would sit on top.
What to watch: Whether other backend players follow metered-compute pricing or hold the per-report model — the divergence will define procurement conversations for the whole layer.
The verification vacuum didn't close, but the caveat went mainstream
Four days after launch, there is still zero independent reproduction of Exa's benchmark claims — and the meaningful shift is that independent-leaning coverage now treats that absence as the headline caveat rather than a footnote (exa-agent-ultra-deep-research-api+1). The claims themselves are striking: 81.4% soft recall on WANDR versus 72.3% for Claude Opus 5.5 and 40.1% for Perplexity Agent — but Exa's grader swaps in Exa as the content tool and uses gpt-6-luna as the judge, on a 200-task subset of Perplexity's own 500-task benchmark.
"These are Exa's reported evaluations, not independent tests of the new release... Exa ran the new comparisons and published the claims itself." — Kingy AI
Superpower Daily lands the same frame: the comparisons "mix published and in-house runs and have not been independently reproduced" (Superpower Daily). Perplexity has published no WANDR response. For the market map, this is the structural weakness of the entire backend layer: the buyer cannot verify capability claims because the vendor grades itself — which is precisely the wedge for an orchestration/evidence-standards layer.
What to watch: The first Aaron Tay-style gold-standard spot-check of Agent Ultra, or an official Perplexity WANDR response — either would set the template for procurement-grade evaluation across all backends.
The benchmark's own paper is a reality check on everyone's numbers
The sharpest new fact this cycle comes from kingy.ai's reading of the WANDR paper itself: in the benchmark's own research, the strongest high-effort result reached 0.363 soft F1 and 0.133 hard F1 (exa-agent-ultra-deep-research-api+1). That's a reminder of how far every research agent — including the one claiming 81.4% — remains from exhaustive wide-and-deep collection, and that Exa's subset score with a swapped grader isn't comparable to full-benchmark published numbers.
"The relevant business question is not 'Is one run cheap?' It is 'What does a usable, checked result cost?'" — Kingy AI
Kingy's warning that without review "the impressive swarm of agents will have left the hardest part on a person's desk" is the clearest articulation yet of the verification gap as a product opportunity. A tool whose job is independent verification of agent-produced research gets more valuable with every launch like this one.
What to watch: Whether any vendor starts shipping third-party-graded benchmarks as a differentiator — the first to do so changes the market's evaluation norms.
What surprised us
- The caveat became consensus faster than the verification did. Last cycle the worry was that echo coverage would launder vendor numbers into fact. Instead, kingy.ai and Superpower Daily independently led with "not independently reproduced" (exa-agent-ultra-deep-research-api
+1). The market's immune response to self-graded benchmarks is strengthening — good news for anyone positioning on evidence standards.
- The WANDR paper quietly deflates the whole leaderboard. A best-in-paper 0.363 soft F1 means every vendor's headline number lives on a generous curve — worth remembering before anchoring the market map on any benchmark table.
- "Continue from previous work" is the sleeper feature. Recurring list-building — search only for what's new since last time — is a workflow, not a query, and it's the kind of capability that makes backend research a standing line item rather than an occasional spend.
Open threads worth a vote
- The Agent Ultra verification thread stays open until an independent WANDR spot-check or a Perplexity response appears — if you'd rather prioritize a different open question for the next pass, vote it up on the topic page.