Rendered at 09:23:03 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
softwaredoug 14 hours ago [-]
In my research grep is fine if you don’t care about tokens and you have less than 100k files. The direct corpus interaction paper [1] shows a breakdown past this level. In my personal experience you get a bit better relevance than a BM25 search engine with grep plus an agent. But it requires you to eat tokens.
If you think grep is great, it’s because you’ve been social engineered to organize your content to be findable. We document why something is useful to an agent. We put it in a logical place.
Just organizing content is at least half of building search, agentic or not. It’s one reason Google is successful, we’re all trying to make our content findable by the search engine. It’s not all technology :)
> If you think grep is great, it’s because you’ve been social engineered to organize your content to be findable. ...
This is such a strange train of thought. How do did you get there?
softwaredoug 13 hours ago [-]
I'm not literally saying you were social engineered. I'm saying all the incentives are there for you to organize your content.
Incentives to make things findable is more important to search than any technology.
nh23423fefe 12 hours ago [-]
I read that as, "you've learned to insert weird entropy meta-breadcrumbs just for finding"
so if i just index and search then i can stop writing like that?
allan_s 11 hours ago [-]
Long before AI I remember asking people in code review to add comments specifically to make the code grep-able. Same for for privileging key value mapping to dynamic string concatenation.
piekvorst 10 hours ago [-]
The social engineering thing runs deep. For example, if you grep for “Key” method, chances are the type/class name would stand on the same line. This is the case in Go and, I think, in many other programming languages (ironically, not C).
Lines are a fundamental building block of text and it’s not unreasonable to optimize them.
)
contextfree 14 hours ago [-]
It seems ridiculous that, for example, Copilot running in Visual Studio working on a C# codebase finds stuff in code by grepping around instead of using the Roslyn-driven code symbol and semantic database built into Visual Studio. I'm guessing it's because the people they get to work on AI stuff are AI People who probably only write in Python
bee_rider 14 hours ago [-]
It is sort of funny when Copilot hasn’t been integrated with Microsoft’s stuff. But it does make some sense from a business point of view. Make it work with grep, it works everywhere.
inetknght 12 hours ago [-]
Microsoft was mostly on the Embrace step. They've reached the Extend step with Copilot. They'll eventually Extinguish grep.
It's best not to use Microsoft products.
contextfree 11 hours ago [-]
What does this even mean, how do you extinguish grep
SkitterKherpi 12 hours ago [-]
There's a lot more examples of grep usage than Visual code search in the training set.
pipeline_peak 5 hours ago [-]
Codex does this in VSCode as well.
Compilations break all the time and those symbols either become useless or it’s just quicker to use grep.
quinncom 16 hours ago [-]
Don’t presume this study has anything to do with programming. They measured an agent’s ability to search long conversations, not code.
> We evaluate on a 116-question representative subset of the LongMemEval benchmark (Wu et al., 2025), which tests an agent’s ability to answer questions over long conversations spanning multiple sessions.
schipperai 15 hours ago [-]
I get a sense that I was click-baited by article's title with the classic trope of "X is all you need". This research is a solid contribution, but is far from all we need to understand grep vs semantic search in agent retrieval.
pretty fast and neat project to search code interactively with a lot of optimizations on finding the right thing
boyter 9 hours ago [-]
Came here to post than and you already did. Thank you!
hmokiguess 18 hours ago [-]
Tangential, I have a hook that rewriters grep to rg but lately I wonder if this is actually wasteful as the model is so biased to grep, is there a way to shim/alias perhaps?
`gsc grep` is just an alias for `gsc rg`, mostly because agents are much more likely to reach for “grep” than “rg”.
It works pretty well, but it is not a perfect drop-in replacement. `grep` and `ripgrep` differ in a few details, especially around glob/wildcard behaviour and flags. What I found works is to not use `grep` in search examples, and have the CLI spit out an error message for the AI saying this is `ripgrep`, so it needs to use `ripgrep` syntax.
celrod 17 hours ago [-]
If performance is the concern, ugrep will get you most of the way there relative to gnu grep, and should be fully grep compatible in terms of syntax:
Many harnesses are doing this already, "Grep" is the tool name, ripgrep is the implementation
It depends on if it is using Grep the harness tool or Grep from the bash tool
hmokiguess 17 hours ago [-]
I see it using the Bash tool infrequently though sometimes Grep. I'm on Claude Code for now due to subscription lock-in, been contemplating moving to pi though
joelfried 17 hours ago [-]
My experience here (also Claude user) is that the model uses different tools in different contexts. I see rg more on frontend and grep more on backend work. I imagine it defaults to using the tool it has more learning around within the contexts it's reaching for and since for the most part it's 6 of one or half a dozen of the other you'll see environment specific usages for these tools in claude for now. I imagine eventually it'll standardize but we're early yet on such things.
If you'd told me a decade ago I'd finally learn some sed in 26 because I'd want to understand what the AI was doing I'd have told you you were crazy . . .
Analemma_ 16 hours ago [-]
Why do you have subscription lock-in? Even if you pay for a yearly subscription, Anthropic will refund you pro rata if you cancel early.
cyanydeez 17 hours ago [-]
I've been on a look out for any harness that properly secures a protocol to the LLM, but they're all just "here's some tools, hopefully you don't use bash for everything".
fwip 6 hours ago [-]
And they all do. I had to add special instructions to tell Claude Code to prefer its built-in read_file tool, rather than using `sed -n 180,210p` everywhere.
gbacon 17 hours ago [-]
This is a surprising result. With structured inputs like source code, I’d expect grep to outperform semantic search, but natural language’s errors and inconsistencies seem to leave so many cracks for information to fall through.
sdesol 17 hours ago [-]
This paper is based on quality so I don't think it should be that surprising if you take loops into consideration. What the agent finds in the first pass, can help if formulate the next grep if needed.
jeffchuber 17 hours ago [-]
If you are truly bitter-lesson pilled - give the agent all the tools and let it decide which to use.
- regex (grep)
- hybrid search (bm25+vector)
this X vs Y is uninteresting when the answer can be both.
pastel8739 17 hours ago [-]
That assumes that the agent knows which one is better. And to bake in which one is better via post-training would require a study like this to establish where each one works well
fnordpiglet 17 hours ago [-]
I’ve got a custom ultra high performance streaming semantic search I exposed as a tool and the RL bias in Claude is almost insurmountable without copious and consistent steering. Codex will follow instructions and use the tools I ask it to but for gods sake between Claude asking to take a nap because it’s getting late in the session and it regressing to RL biased tools like grep it’s maddening. When I can get it to use my compositional tools tool calls drop from like 20-50 to 3-4, but it’s almost impossible to steer.
mediaman 11 hours ago [-]
Anthropic is, I believe, fully pursuing the idea that you shouldn't use their model with anything but their own products. They don't care whether it generalizes.
I agree it's very frustrating to use with custom tools/harnesses that can speed up the process for domain specific purposes.
bachittle 17 hours ago [-]
Exactly this, and this tool called qmd is what I use for the hybrid search portion. It also uses local LLMs to provide summaries on your own markdown data too. My agents use both depending on what type of search they are doing, and both provide good results.
Both is usually the right answer, since you can use LLMs to do query expansion and effectively increase the recall performance of your retrieval algo
dominotw 17 hours ago [-]
it will only use tools it was trained on? what's the benfit of givig it all the tools.
jeffchuber 8 hours ago [-]
then you are not agi pilled
worthless-trash 17 hours ago [-]
I'm still disappointed that ai can't use ctags, its used for finding strings and patterns, its right there.
sdesol 17 hours ago [-]
> I'm still disappointed that ai can't use ctags,
What do you mean by this? Do you mean not automatically build the index?
worthless-trash 17 hours ago [-]
it inspects a project, finds the ctags files, then goes on to use grep.
worthless-trash 16 hours ago [-]
[flagged]
0xbadcafebee 7 hours ago [-]
> grep generally yields higher accuracy
And a lot more tokens, and slower speed. Yes you can get more accuracy if you suck tons more data into context.
But compare this to more advanced code agent methods like Tree Sitter, PageRank, LSP, that build semantic maps to provide more relevant context. Grep alone can't do that
piker 18 hours ago [-]
I recently watched the new Palantir + Kirkland & Ellis fund formation platform demo, and I was surprised to see how effective the union of structured data was in an agent harness. We're used to dealing with flat files and comparing here basic ways of searching, essentially, long strings, but using Palantir's "Ontology" graph framework, I think Kirkland is going to be able to achieve some exception and differentiating outcomes in legal tech. The whole idea assumes that they've got great structured data already, and perhaps that's the real valuable unknown, but giving an agent those tools is super powerful.
I wrote about it[1] and came away with a different view on both Palantir and the future of agentic workflows personally.
That was great, thanks for the write-up. It’s rare to get a peek into Palantir’s ontology-forward approach. I’ve certainly been curious.
> But it would make no sense to have an LLM regurgitate an existing form document token-by-token rather than call a piece of 1994 software like Hotdocs to populate some placeholders.
This is a real “oof”, isn’t it. Very difficult to understand what they were going for here. Perhaps they just assumed no one in the intended audience would pick it up. But it certainly is enough of a red flag that it made me go back to the top of your write-up for a re-read, thinking about their whole pipeline in much more sceptical terms.
piker 14 hours ago [-]
Really glad you liked it! Yes, I suspect it was just for show since a plain document popping onto the screen would just be jarring.
Edit: looks like you’re in London, too. Hit me up and let’s connect. My details are in the bio!
darkteflon 3 hours ago [-]
Yeah absolutely, will do! Looks like we have common interests. I’m in Tokyo though, actually. Rest of the team is in London.
SkyPuncher 14 hours ago [-]
Table 2 and 3 tell you basically all you need to know. When you use a harness that is tuned towards programing (Codex and Claude Code), grep wins. When you use a neutral harness, vector search wins.
So far every Grep vs RAG discussion I've seen conflates overlapping factors. The most common is simply that a company rebuilt their pipeline from scratch and fixed a bunch of problems. The worst is when they go from one-shot RAG to multi-step Grep and completely miss the fact that multi-step RAG would likely get them similar results.
At the end of the day, the most important thing is knowing the _product features_ your users care about and making sure that's represented in the pipeline.
ako 14 hours ago [-]
As far as i know Claude Code also uses LSP and tree-sitter to find things in your source code.
krzyk 13 hours ago [-]
If you install LSP. AFAIR their first versions used some kind of treee/structure for easy search, but they found out that grep was better/similar but with less complications (they now ship some kind of grep).
stephantul 16 hours ago [-]
This paper oversells on the title. Like, what is chronos, which embedding model was used, which reranker, how was the reranking done, why is chronos much better than claude code
> LongMemEval rewards recovering literal witnesses: exact dates, counts, preferences, and spans that often remain stable under tokenization.
Is this saying they chose a benchmark that is biased towards doing well against literal string matching, thus works well with grep, and then (gasp) showed that grep did well, finally declaring "grep is all you need"?
The examples in the benchmark's demo image(1) are all examples you could see grep doing well on. A conversation about bikes, then a query about bike(s) where "bike" is a common token hit. But not stuff like a conversation about a Beethoven sonata, then a question about classical music, where embedding based approach would shine.
Feels important, but I wish they also had compared against something like MeiliSearch or Algolia.
verdverm 18 hours ago [-]
100%, there's even Typesense, open source Algolia, which can do hybrid search and a number of other fancy things
I'm currently working on a markdown kb / search tool for my agents, in part built on TS
kwillets 17 hours ago [-]
I'm curious to see what patterns it's grepping.
sys_64738 18 hours ago [-]
Surely 'strings' would be even better?
greenavocado 17 hours ago [-]
This has been posted before, but a dead-simple pattern that helps enormously with steering the model to the right code area is a DESIGN.md that it creates, updates, and references periodically.
If you think grep is great, it’s because you’ve been social engineered to organize your content to be findable. We document why something is useful to an agent. We put it in a logical place.
Just organizing content is at least half of building search, agentic or not. It’s one reason Google is successful, we’re all trying to make our content findable by the search engine. It’s not all technology :)
1- https://arxiv.org/abs/2605.05242
This is such a strange train of thought. How do did you get there?
Incentives to make things findable is more important to search than any technology.
so if i just index and search then i can stop writing like that?
Lines are a fundamental building block of text and it’s not unreasonable to optimize them.
)
It's best not to use Microsoft products.
Compilations break all the time and those symbols either become useless or it’s just quicker to use grep.
> We evaluate on a 116-question representative subset of the LongMemEval benchmark (Wu et al., 2025), which tests an agent’s ability to answer questions over long conversations spanning multiple sessions.
grep’s design is surprisingly winning, exceeding expectations to this day.
pretty fast and neat project to search code interactively with a lot of optimizations on finding the right thing
https://github.com/gitsense/gsc-cli
`gsc grep` is just an alias for `gsc rg`, mostly because agents are much more likely to reach for “grep” than “rg”.
It works pretty well, but it is not a perfect drop-in replacement. `grep` and `ripgrep` differ in a few details, especially around glob/wildcard behaviour and flags. What I found works is to not use `grep` in search examples, and have the CLI spit out an error message for the AI saying this is `ripgrep`, so it needs to use `ripgrep` syntax.
https://github.com/Genivia/ugrep#aliases
Claude Code may ship with ugrep already.
It depends on if it is using Grep the harness tool or Grep from the bash tool
If you'd told me a decade ago I'd finally learn some sed in 26 because I'd want to understand what the AI was doing I'd have told you you were crazy . . .
- regex (grep) - hybrid search (bm25+vector)
this X vs Y is uninteresting when the answer can be both.
I agree it's very frustrating to use with custom tools/harnesses that can speed up the process for domain specific purposes.
https://github.com/tobi/qmd
What do you mean by this? Do you mean not automatically build the index?
And a lot more tokens, and slower speed. Yes you can get more accuracy if you suck tons more data into context.
But compare this to more advanced code agent methods like Tree Sitter, PageRank, LSP, that build semantic maps to provide more relevant context. Grep alone can't do that
I wrote about it[1] and came away with a different view on both Palantir and the future of agentic workflows personally.
[1] sorry, LinkedIn: https://www.linkedin.com/pulse/fund-managements-killer-app-d...
> But it would make no sense to have an LLM regurgitate an existing form document token-by-token rather than call a piece of 1994 software like Hotdocs to populate some placeholders.
This is a real “oof”, isn’t it. Very difficult to understand what they were going for here. Perhaps they just assumed no one in the intended audience would pick it up. But it certainly is enough of a red flag that it made me go back to the top of your write-up for a re-read, thinking about their whole pipeline in much more sceptical terms.
Edit: looks like you’re in London, too. Hit me up and let’s connect. My details are in the bio!
So far every Grep vs RAG discussion I've seen conflates overlapping factors. The most common is simply that a company rebuilt their pipeline from scratch and fixed a bunch of problems. The worst is when they go from one-shot RAG to multi-step Grep and completely miss the fact that multi-step RAG would likely get them similar results.
At the end of the day, the most important thing is knowing the _product features_ your users care about and making sure that's represented in the pipeline.
> LongMemEval rewards recovering literal witnesses: exact dates, counts, preferences, and spans that often remain stable under tokenization.
Is this saying they chose a benchmark that is biased towards doing well against literal string matching, thus works well with grep, and then (gasp) showed that grep did well, finally declaring "grep is all you need"?
The examples in the benchmark's demo image(1) are all examples you could see grep doing well on. A conversation about bikes, then a query about bike(s) where "bike" is a common token hit. But not stuff like a conversation about a Beethoven sonata, then a question about classical music, where embedding based approach would shine.
(1) https://github.com/xiaowu0162/LongMemEval/blob/main/assets/l...
I'm currently working on a markdown kb / search tool for my agents, in part built on TS