Autonomous Research Tools: Market Map Digest — The Harness Goes Public, the Free Floor Drops Out
TL;DR
The measurement-layer story advanced this stretch: Exa shipped its open-source Scout harness and backend tables — including cells that flatter competitors — while the ATLAS artifacts themselves remain unreleased (research-agent-eval-measurement-layer). Meanwhile, the free-incumbent floor collapsed as Gemini's October 9 free-tier cut took effect, metering Deep Research out of free accounts (gemini-free-tier-ladder-squeeze
), and Paperguide's published price ladder showed exactly how audit-grade methodology commands a 7.4x seat premium in the academic segment (research-tool-pricing-segmentation-2026
). The map's cheap end is thinning while its graded, vertical end thickens.
The benchmark war entered its verification phase — and Exa is publishing evidence against itself to win it
The credibility contest in agent evaluation is now being fought with released code, not blog claims. Exa published exa-labs/benchmarks on GitHub, MIT-licensed, containing the Scout harness — "It swaps Exa, Perplexity, Parallel, Brave or Tavily into a standard research agent while keeping the model, prompts, research loop and budgets fixed across the search API presets" — plus four runnable Exa datasets and loaders for four public suites (research-agent-eval-measurement-layer). The tables include unflattering cells: on WebCode RAG, "Brave wins citation precision (0.328 vs Exa's 0.259)" (research-agent-eval-measurement-layer
).
This matters because the harness that's public is not yet the harness behind the headline. Scout explicitly excludes end-to-end agents — "End-to-end agentic search products like Exa Agent are out of scope" — while ATLAS's Pareto-frontier figures do include them, and the 547 tasks, golden tables, and grader are still pending, with the blog still promising them "in the coming weeks" as of the October 14 re-fetch (research-agent-eval-measurement-layer). For Hey, Lefty, the released Scout harness is immediately useful — it's a free, resumable, cost-reported rig for benchmarking search backends under a fixed agent loop, which is exactly the kind of infrastructure a small entrant could never build alone.
What to watch: Whether the ATLAS task set, gold tables, and grader actually land in exa-labs/benchmarks — the precondition for neutrally testing the 16-point backend spread.
The free-incumbent ladder is being pulled up, and the prototyping floor moved with it
Google's free-tier cut took effect October 9, and the analysis that landed this cycle sharpens what it means: "As of October 9, 2026, users without a paid Google AI subscription are barred from the more capable Flash and Pro models that previously defined the free experience… the prototyping floor has just dropped a generation" (gemini-free-tier-ladder-squeeze). The metering mechanics are the real story — Deep Research now consumes metered limits at an accelerated rate, meaning agentic research is effectively gated out of the free tier at peak times. The paid ladder starts at AI Plus, $4.99/month (gemini-free-tier-ladder-squeeze
).
The strategic contradiction is the sharpest point in the analysis: "the external market for API-based intelligence is becoming cheaper by the hour, Google is insulating its consumer-facing compute margins by restricting free access to its own models" — citing Anthropic's Haiku 5.5 at $0.10/$0.50 per million tokens (gemini-free-tier-ladder-squeeze). For the market map, this squeezes from both ends: the free sandbox that seeded the developer funnel is gone, and the cheapest capable substrates now live outside Google entirely — which strengthens the case for model-agnostic routing as a positioning lane.
What to watch: The scheduled doubling of Gemini Flash API list prices on January 1, 2027 — the cost basis of the market's cheapest agent substrate rising on top of the consumer squeeze.
The academic vertical's rate card shows what actually commands price: methodology, not intelligence
Paperguide's fully published pricing ladder is the cleanest public example of how a vertical research tool structures its value prop. Every tier meters "AI credits" across agents, reports, and writer — agentic work sold as a consumable allowance, not a flat seat feature — but the differentiator at the top is procedural: the $89/seat "Evidence" tier's headline feature is "PRISMA grade dual reviewer screening" (research-tool-pricing-segmentation-2026). A customer testimonial makes the value prop concrete:
"it cut our abstract screening from about three weeks to one, and the decisions came out more consistent, not less." — Pricing | Paperguide
Two map-level signals hide in the same rate card. First, audit-grade methodology compliance — not raw model quality — is what commands 7.4x the entry seat price, reinforcing that defensible verticals sell verifiable process. Second, Paperguide exposes its MCP backend even in the free tier, continuing the destination-platform-to-backend convergence: academic destinations now want external agents to call into their corpora (research-tool-pricing-segmentation-2026). The seat ladder also spans the full buyer arc — students at $12, review teams at $89, institutions custom — the same land-and-expand shape verticals use elsewhere on the map.
What to watch: Whether other destination platforms follow Paperguide in shipping free-tier MCP backends — the more corpora that become agent-callable, the more the orchestration layer (where a new entrant would sit) gains leverage.
What surprised us
- Exa published numbers that hurt Exa. Brave beating it on citation precision in its own harness is an unusual move for a vendor-benchmark author — a deliberate bid for the neutral-grader credibility that Arena capitalized at $200M (research-agent-eval-measurement-layer
). The measurement layer is competitive enough that vendors are buying trust with self-damning tables.
- The free tier was a distribution strategy, and it's being retired. We tend to assume incumbents give away the low end forever to own the funnel; Google is instead metering Deep Research out of free accounts while API prices collapse around it (gemini-free-tier-ladder-squeeze
). That vacated space — cheap, reliable agentic research for prototypers — is now open ground.
- "Auditable" is the top-tier feature everywhere, not just a niche. PRISMA dual screening at Paperguide, trace-verified failure signals at Arena — the pattern repeats across segments (research-tool-pricing-segmentation-2026
). If Hey, Lefty is positioning anywhere, the consistent willingness-to-pay signal is for verifiable process, not bigger answers.
Open threads worth a vote
- [watch] Exa releases ATLAS artifacts (tasks, gold tables, grader) enabling independent replication — still open after the harness release; the Scout repo is where the ATLAS suite will land, but the headline benchmark's artifacts remain the missing piece.