The Embedding Infrastructure Nobody Built (So We Did)#
AI Engineer Europe 2026 digest Program Core pick 8–10 April 2026
Filip Makraduli
- Why: Embedding serving is a quiet but ubiquitous piece of AI infrastructure, and existing engines (TEI, Infinity) have real operational gaps — one model per container, crashes under load, no story for multiple models contending for GPU memory.
- What: An embedding inference engine built around what the ecosystem lacked: models loaded at query time rather than deploy time, hot-swapping dozens of models on one GPU without restarts, LRU eviction on memory pressure instead of OOM crashes, and trying a new model without a rebuild. Includes profiler traces showing attention is only ~6% of the embedding workload — invalidating the author's own (and common) Flash-Attention-centric optimization advice.
- How: Query-time model lifecycle on shared GPU capacity with memory-pressure-driven eviction, grounded in actual profiling of the memory hierarchy rather than assumed bottlenecks.
- Where applicable: Any team serving multiple small models on shared GPU capacity — a general technique, not org-specific. It is the speaker's own product, so weigh it as vendor experience, but the profile-before-optimizing lesson is transferable.