Operating Distributed Inference Systems at Scale#

Speakers: Nishant Gupta, Naman Ahuja

• Why: The framing — inference as an infrastructure orchestration problem — matches the PM's view exactly, and the topic list (multi-tenant inference infrastructure, GPU scheduling, fault isolation, control loops) is squarely relevant.
• What: Lessons from building large-scale elastic compute and AI infrastructure: distributed inference architectures, GPU scheduling and elastic compute, multi-tenant inference infrastructure, caching/batching/latency optimization, reliability and fault isolation, observability and control loops, and cost/throughput/UX balancing (from abstract only). The abstract is a topic list with no concrete results, named systems, or affiliations — expect breadth over depth.
• How: Not specified at abstract level; likely an architectural-patterns survey.
• Where applicable: Large inference platform operators; attend if the Core inference talks leave gaps, but this is the least substantiated item in the set.