Paul Masurel on Tantivy: the tiny Rust engine that wouldn't stay small

The surprise challenger

For two decades, full-text search has felt like Lucene’s turf. Java, stability, and an ecosystem that built Elasticsearch and Solr — job done, right? Not quite. In 2017 Paul Masurel quietly published Tantivy, a compact Rust search library born from curiosity, annoyance, and a long flight from Tokyo to Paris. Within a few years it powered Quickwit, ParadeDB, LNX and other projects; it has been reported that it even sparked a performance collaboration with the Lucene team. Small, modular, and unapologetically pragmatic, Tantivy forced the industry to take Rust seriously for search.

Origins

Masurel’s story is, at heart, human: frustration turned into craft. He cut his teeth at Exalead and later worked on Lucene-based systems at Indeed, and he admits that not being on the core team fed a long‑fermented itch to build. The first Tantivy prototype was “a bit silly” and written in spare hours, yet it proved a point: you could reimplement search fundamentals in a modern systems language and learn a lot along the way. Who hasn’t rethought something simply because they couldn’t live with the status quo?

Design philosophy

Rust influenced the way Tantivy is organized, Masurel says, but the architecture follows Lucene’s proven model: a library that handles indexing, compression, and search, leaving distribution and orchestration to the embedding system. The priority was clear — small, modular, and owned by developers, not an opaque monolith. Batteries included, minimal. And an expected bonus: Masurel found himself far more productive and confident in Rust than he ever was in C++, which mattered when dealing with IO, concurrency, and the messy edges of real systems.

Impact and what’s next

Tantivy’s biggest win may be cultural as much as technical. It proved an alternative path: performant search without Java, and with an ownership model that appeals to teams building search as a product core. Masurel co‑founded Quickwit to build a cloud‑native log engine on Tantivy; Quickwit was acquired by Datadog in 2024, and Masurel now works on search at scale there. The tale is part reinvention, part collaboration — and a reminder that even “solved” problems can be nudged forward when someone brings new tools and new frustration to the table. What comes next? If recent trends are any guide, expect more polyglot infrastructure and more Rust under the hood of systems we thought were settled.

Sources: paradedb.com, Lobsters