Agentic Inference Optimization for RecSys ML models#

Shuyao Bi (Meta)

• Why: GPU inference optimization is core AI infrastructure work, and this replaces a documented, quantified expert bottleneck (~40 hours of manual fusion work per model) with an automated agent — a pattern likely to spread to every ML platform team.
• What: An automated optimization agent that detects kernel-fusion opportunities in PyTorch recommendation models, generates and validates Torch FX graph transformations, and converges iteratively — removing the manual per-model fusion effort that went stale as architectures changed (from abstract only).
• How: Fusion opportunities are detected from GPU traces plus FX graph structure; the agent then authors graph transformations (replacing operation sequences with fused kernels to cut launch overhead and host-device transfers) and validates them in a self-converging loop.
• Where applicable: PyTorch/Torch FX specific, aimed at production inference latency targets for recommender-class models. No headline numbers are given in the abstract, so treat impact as directional until the full write-up; the technique itself is broadly usable wherever FX graphs apply.