For three years, every RAG pipeline looked the same.
Chunk the document. Embed the chunks. Store them in a vector database. Search by similarity. Hope the top results contain the answer.
It works on a blog archive. It falls apart on a 200-page financial filing.
A new approach called Vectorless RAG is rewriting the playbook. No embeddings. No chunking. No vector database.
One public benchmark shows it hitting 98.7% accuracy on FinanceBench. Traditional vector RAG scores around 50% on the same test.
That is not a small gap. Here's what's going on.
Quick Refresher: How Traditional RAG Works
Retrieval-Augmented Generation lets an LLM answer questions about documents it was never trained on.
The classic pipeline has three steps:
- Index — split docs into chunks, embed each chunk, store in a vector DB.
- Retrieve — embed the user's query, find the most similar chunks.
- Generate — paste those chunks into the prompt, let the LLM write the answer.
Fast. Cheap. Simple. And quietly broken on the kind of documents that matter most.
Why Vector RAG Breaks on Real Documents
Vector search assumes semantic similarity equals relevance.
On long professional documents, those two are not the same thing.
Here are the five pain points that keep showing up:
1. Similarity is not relevance. A query expresses intent. A chunk expresses content. The closest vector is often the wrong answer.
2. Chunking destroys structure. Fixed-size chunks cut tables in half and strip sections away from their headings. The structure is the meaning.
3. Cross-references get lost. Page 12 says "see Appendix B." Appendix B is on page 87. Vector search has no way to follow that link.
4. No multi-step reasoning. Retrieval is one shot. If the answer needs two sections combined, the system can't plan a second hop.
5. No audit trail. "This chunk was retrieved because it was close in embedding space" is not an explanation. Regulated industries need real traceability.
Even advanced systems like Claude Code have moved away from vector search for code retrieval. The pattern is clear.
What Is Vectorless RAG?
Vectorless RAG replaces similarity search with LLM reasoning over a structured index.
The phrase was popularized by PageIndex, an open-source framework published in September 2025 by the team at VectifyAI.
The idea borrows from AlphaGo. Instead of searching every possible move, use a learned strategy to navigate intelligently.
Applied to documents, that means three things:
- No vector database. Retrieval uses document structure and LLM reasoning.
- No chunking. Documents are split along natural sections and headings.
- Human-like retrieval. The system scans a table of contents, picks the right chapter, then drills deeper.
The result is retrieval by reasoning, not matching.
How PageIndex Actually Works
PageIndex runs in two phases.
Phase 1: Build a Tree Index
Parse the document into its natural hierarchy. Store it as a tree.
Each node holds a title, a short summary, and a page range.
No embeddings. No vector store. No chunk boundaries slicing through tables.
The tree is persistent. You build it once per document.
Phase 2: Reason Over the Tree at Query Time
A user asks a question.
The system shows the tree's table of contents to an LLM and asks which node likely contains the answer.
The model picks a path, expands deeper, and repeats until it lands on the right section.
Only that section's text is passed to the final answer step.
Every step is traceable. You can see which nodes were visited and why the model chose them.
That is a real audit trail.
The 98.7% Result That Started the Hype
PageIndex powers a financial QA system called Mafin 2.5.
On FinanceBench, a standard benchmark for retrieval over SEC filings, Mafin 2.5 scored 98.7% accuracy.
Traditional vector RAG on the same task lands around 50%.
A 49-point jump is why the RAG community paid attention.
For structured professional documents, reasoning over hierarchy isn't a nice-to-have. It's a different class of system.
Vector vs Vectorless: Side by Side
Each approach has its sweet spot.
Vector RAG still wins on speed, scale, and cost.
Vectorless RAG wins on accuracy, traceability, and cross-reference handling.
The Honest Trade-Offs
Vectorless RAG is not free lunch. A few things get worse before they get better.
Cost. Each query fires multiple LLM calls to walk the tree. One developer benchmarked a 131-page World Bank report and got 137 tree nodes. That's 137 LLM calls just to build the index.
Across 50 documents, you're looking at ~7,000 LLM calls before answering a single question.
Latency. Vector search returns in milliseconds. Tree reasoning takes seconds.
Scale. Vector search can find a needle in ten million documents. Tree search can't. Vectorless shines when you already know which document holds the answer.
Not truly "vectorless." A Hacker News commenter had it right: PageIndex still approximates relevance. It just swaps vector approximation for LLM reasoning approximation. Both have costs.
Overkill for short docs. If your document is a blog post or support ticket, a tuned vector pipeline is fine.
When to Use Vectorless RAG
Reach for it when:
- You work with long, structured professional documents (annual reports, legal contracts, regulatory filings, technical manuals).
- Accuracy is the main constraint.
- Queries need multi-step reasoning or cross-reference following.
- You need a transparent audit trail for regulated industries.
- You're doing deep-dive analysis on a small number of critical documents.
Stick with vector RAG when:
- You need sub-second responses at high query volume.
- Your corpus is a big pile of short, unstructured content.
- 90% accuracy is good enough and cost matters more than precision.
- You're building a consumer-scale chatbot where speed wins.
Go hybrid when:
- You have both scale and structured documents.
- The common recipe: use vector search to narrow 10,000 docs down to the top 20, then hand those to a vectorless reasoner for the precise answer.
- Recall from vectors. Precision from reasoning. Best of both.
A Minimal Code Example
PageIndex is open source under MIT license.
Here's what a vectorless query looks like in Python:
from pageindex import PageIndexClient
import asyncio
client = PageIndexClient(api_key="YOUR_KEY")
async def main():
# 1. Upload a PDF and build the tree index
doc = await client.submit_document("annual_report_2025.pdf")
await client.wait_for_completion(doc.doc_id)
# 2. Ask a question. The SDK handles tree walking.
answer = await client.query(
doc_id=doc.doc_id,
query="What was the year-over-year revenue growth in Cloud?"
)
print(answer.response)
print("Nodes used:", answer.retrieved_nodes)
print("Reasoning:", answer.reasoning_trace)
asyncio.run(main())
That's the entire surface area.
No vector database. No embedding model. No chunk size to tune.
So, Is Vector RAG Actually Dead?
No. The headline makes for good clickbait and a bad conclusion.
Vector RAG isn't disappearing. It's getting a specialist sibling.
Vectorless RAG is what you call in when the documents are long, structured, and too important to get wrong.
Vector RAG is still the generalist for speed, scale, and flat corpora.
Most production systems in 2026 will end up hybrid: vectors for recall, reasoning for precision.
What's actually dead is the idea that every RAG pipeline should look the same.
FAQ
Is Vectorless RAG actually free of vectors? The retrieval step is. Some hybrid setups still use vectors upstream to narrow the search space before reasoning takes over.
Is PageIndex the only option? It's the most popular and has an open-source implementation. The broader family includes any system that retrieves by structure or reasoning instead of embeddings.
Does it need a big model? A capable reasoning model works best. Navigating a tree is a reasoning task, and smaller models struggle on complex docs.
How much more expensive is it? Roughly 10 to 100 times per query. For high-stakes, low-volume work that's fair. For high-volume consumer apps it usually isn't.
Why not just use long-context models? Stuffing 10M tokens into a prompt is slow (30 to 60 seconds), expensive ($1+ per query), and suffers from "lost in the middle" issues. Vectorless RAG sits between classic RAG and raw long-context.
Good fit for code search? Yes. Claude Code already moved away from vector search for exactly this reason.
Bottom Line
Vectorless RAG isn't hype.
On structured documents, reasoning over hierarchy beats similarity over chunks. The 98.7% FinanceBench score is the proof.
It's also not a silver bullet. It costs more, runs slower, and won't scale to millions of flat documents.
The smart play is to know both patterns and match the tool to the job.
Consumer chatbot on a blog archive? Keep your vector database.
Analyzing SEC filings, contracts, or clinical trial reports? Spin up PageIndex this week.
Your accuracy numbers will tell you everything you need to know.