Lease Search: Retrieval Experiment on 29 Commercial Lease Agreements (Dense vs BM25 vs Hybrid)

A search engine over 29 commercial lease agreements that returns the original contract clauses instead of an LLM answer. I left generation out on purpose: it hides the layer that actually breaks. The same question can be run through dense, BM25 and hybrid retrieval over one index, and an eval script with gold paragraph IDs measures where each mode fails. The demo shows one of those failures on purpose: a question that none of the three modes can answer, and the reason why.

Tech Stack

Features

Results

Challenges & Solutions

Challenge
Dense retrieval can't tell near-identical clauses apart. Every lease has the same default clause, so the embeddings collapse into roughly one point, and ranking is decided by noise.
Solution
Ran a controlled experiment: query the index with the exact text of the gold paragraph. It still ranked 2nd, with a spread of 0.01 across the top 5. That isolates the limit as architectural (a bi-encoder never sees the query), not a phrasing problem. So the fix is lexical signal, a cross-encoder or a narrower search space, not swapping the embedding model.
Challenge
Users ask by company name, but contracts say 'Tenant' and 'Landlord'. The name appears in 2 of 1,299 chunks, the header and the exhibits, so hybrid search scored the same 0.60 as BM25 alone.
Solution
Built a 5-question set that separates questions with a distinguishing term in the chunk from questions where the only distinguisher is the party name. The pattern was clean: the first kind passes, the second fails in every mode, because fusion can't help when both channels share the same blind spot. This is the question in the demo. The next experiment, with success criteria written down in advance, is contextual chunk headers that put the party name and section title into every chunk.
Challenge
The metric said 'hit' while the chunk couldn't answer the question. For 'monthly rent in month 20', BM25 put the right chunk first, but the rent table had lost its column headers.
Solution
Traced it with grep over the parsed index to a 40-character minimum-paragraph filter. It removes 43% of paragraphs, mostly page numbers and footers, but also the header 'Term $/SQ.FT Monthly Annually' (29 characters). I documented it, split the metrics into file@5 (right contract?) and chunk@1 (right paragraph?) so one score can't hide which layer fails, and specified the fix: filter by content instead of length, and glue table headers to their first row.

What I Learned