Skip to content

Fully Automated HW SKU Selection System to Optimize Apache Pinot's Cost-to-Serve at LinkedIn#

SREcon25 Americas digest Program Wildcard pick 25–27 March 2025

Recording

Jia Guo and Dino Occhialini, LinkedIn; Yifan (Sabrina) Zhao, Netflix — Wildcard

Play the recording: Fully Automated HW SKU Selection System to Optimize Apache Pinot's Cost-to-Serve at LinkedIn

  • Why: Not a networking or identity fit, but a genuinely novel operational capability — automatically recommending optimal hardware SKUs across a ~14,000-machine fleet based on live workload profiling — that touches the PM's "physical" interest from an unusual angle (hardware selection as an automated, continuous optimization problem rather than a one-time procurement decision).
  • What: A fully automated system that recommends optimal hardware SKU profiles for Apache Pinot's production OLAP database fleet (~14K machines) at LinkedIn, cutting cost-to-serve by nearly 50% across highly varied workload, cost, and dataset-size characteristics. (from abstract only)
  • How: Low-overhead profiling collects high-cardinality resource-usage data (CPU, memory, IO) from production clusters across multiple SKU profiles, feeding a cost-optimization algorithm that recommends SKU allocation with minimal operational overhead. (from abstract only)
  • Where applicable: Large heterogeneous-hardware fleets running variable workloads; the general pattern (continuous, profiling-driven hardware-fit optimization rather than static procurement) generalizes beyond Pinot/OLAP specifically.