Keeping search off the write path

There is a version of full-text search that looks correct on a whiteboard and fails the moment real people use it: save the record, index the record, return success. One handler, one transaction, no moving parts. It is the first thing most teams write, and it quietly makes the editor as slow as the search cluster — and as unreliable.

Why the obvious version hurts

In Mindora, our AI journaling platform, the write path is the product. Someone is mid-sentence. They expect the entry to save the way a text field saves: instantly, and every time.

Fold Elasticsearch indexing into that same request and two things follow.

• Latency becomes the sum, not the maximum. The user now waits for the database write and the index write. The slower of the two sets the floor, and search clusters are not optimised for single-document latency.
• Availability becomes the product, not the maximum. If the index is rebalancing, under GC pressure, or simply down, the save fails — even though the database was ready. A feature nobody asked for in that moment has taken the feature they did ask for offline.

The second point is the one that matters. Search being briefly unavailable is an inconvenience. Journaling being unavailable is a broken product.

What we did instead

The journal service publishes to an exchange. The search service consumes from it and indexes at its own pace. A save completes on the database write alone.

client → journal service → database        (the request ends here)
                        ↘ exchange → search service → Elasticsearch

That one arrow is the whole decision. The request path no longer contains Elasticsearch, so the editor's latency and uptime are the database's latency and uptime. Indexing lags by a moment.

Nobody notices. The person who just wrote an entry is not searching for it — they are writing the next sentence.

What it costs

This is a trade, not a free win, and it is worth being precise about the bill.

The index is eventually consistent. There is a window, usually well under a second, where an entry exists and is not yet findable. For a journal that is invisible. For something like an order status a user refreshes immediately, it would not be, and we would have made a different call.

There is a queue to operate. A broker is real infrastructure: it needs monitoring, dead-letter handling, and a decision about what happens to a message that fails to index repeatedly. The synchronous version has none of this. You are exchanging a correctness-and-latency problem for an operational one, and you should want that exchange before you make it.

Failure moves rather than disappears. An index that falls behind no longer breaks saves — it makes search stale, silently. That needs its own alert, because the symptom no longer arrives as an error the user reports.

The general shape

The rule we keep coming back to: anything that is not required for the write to be correct does not belong in the write.

Indexing, analytics, notification fan-out, webhook delivery, thumbnail generation, cache warming — none of these change whether the record is valid. Each one, left on the request path, donates its worst day to your most important endpoint.

Move them off. Accept the lag. Alert on the lag.

The same reasoning runs through the rest of the system: Mindora's AI layer falls through Gemini to Anthropic to HuggingFace and finally to deterministic heuristics, so a provider outage degrades mood detection rather than removing it. Different mechanism, same instinct — decide in advance which part is allowed to fail, and make sure it is never the part the user came for.

---

More on how these systems are put together in our engineering notes.