Agentic Search: Grep vs. Vector Retrieval in Agentic Loops

Agentic Search: Grep vs. Vector Retrieval in Agentic Loops

A major technical debate in the development of autonomous research tools is whether to rely on vector-based Retrieval-Augmented Generation (RAG) or traditional keyword/regex-based search. A May 2026 paper titled "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search" (arXiv:2605.15184) challenges the industry's heavy bias toward vector search. The study demonstrates that simple grep (keyword/regex) retrieval often outperforms semantic vector search in agentic loops, but highlights that an agent's success is highly dependent on tool-calling paradigms and intrinsic model biases.

The Power of Grep in Agentic Workflows

The paper conducts empirical experiments comparing grep and vector retrieval across multiple agent harnesses (such as Chronos, Claude Code, Codex, and Gemini CLI) using the LongMemEval benchmark, which evaluates an agent's ability to search and reason over long, multi-session conversation histories.

The study found that grep generally yields higher accuracy than vector retrieval within agent loops¹. This counterintuitive result is explained by the iterative nature of agentic workflows:

• Iterative Query Refinement: An agent is not a single-pass system. If a grep query returns nothing or irrelevant results, the agent can inspect the output, reformulate its search query, and run grep again with higher precision.
• Precision and Control: Grep offers deterministic, exact-match capabilities. In structured environments (like codebases or structured logs), semantic search often introduces "fuzziness" that dilutes the relevance of retrieved passages, whereas grep allows the model to locate exact strings or patterns.
• Context Dilution: When vector search retrieves chunks of text, it often pulls in distracting, irrelevant surrounding context. As more unrelated material is mixed in, the model's reasoning capabilities degrade, whereas grep targets the exact line or file segment.

The Challenge of Model Tool-Choice Bias

A critical finding for developers of agentic systems is that frontier models exhibit strong Reinforcement Learning (RL) biases toward certain tools. Because models like Claude are heavily fine-tuned on coding and terminal tasks, they have a deep-seated bias to use primitive shell tools:

• Regression to Grep: Even when an agent harness provides highly optimized, custom semantic search tools, models like Claude will frequently ignore them and regress to using grep or sed commands.
• Insatiable Tool Preference: Forcing a model to use custom, compositional tools requires intensive system prompting and steering. Without aggressive constraints, the model will fall back to its pre-trained habits, even if those habits require 20–50 tool calls instead of 3–4 optimized calls.

Architectural Lessons for Hey, Lefty

For Market Map & Positioning: Where Does Hey, Lefty Fit? and Integrating Custom MCP Servers into Hey, Lefty's Research Cycles, these findings provide clear architectural blueprints:

1. Provide Hybrid Search Capabilities: An agent should not be forced into a binary choice. The optimal agent harness provides a hybrid suite containing regex (grep), lexical search (BM25), and semantic vector search.
2. Actively Steer Tool Usage: To prevent the model from wasting tokens on dozens of iterative grep queries, the orchestrator must use explicit steering instructions (e.g., via a dynamically updated DESIGN.md or strict tool-calling constraints) to encourage the use of high-performance custom search tools.
3. Optimize Token Efficiency: While grep is highly accurate, it can be token-expensive if the agent repeatedly scans large files. Combining structured metadata indexes (like ctags) with custom tool shims (e.g., aliasing grep to a faster, token-optimized search backend) is crucial for production-grade performance.

Quotes

"Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same." — Sahil Sen et al., arXiv:2605.15184 "I’ve got a custom ultra high performance streaming semantic search I exposed as a tool and the RL bias in Claude is almost insurmountable without copious and consistent steering... between Claude asking to take a nap because it’s getting late in the session and it regressing to RL biased tools like grep it’s maddening." — Hacker News Commenter fnordpiglet