NEST: Network- and Memory-Aware Device Placement for Distributed Deep Learning#

Irene Wang, Vishnu Venkata, Arvind Krishnamurthy, Divya Mahajan (affiliations not listed in the program)

• Why: Device placement is where ML frameworks and the physical fabric actually meet. Most placement tools treat the network as an afterthought and memory as a post-hoc fix; NEST makes topology, allreduce latency, and memory feasibility first-class joint constraints — the co-design point a fabric-owning PM cares about.
• What: A placement framework that unifies model parallelism (tensor, expert, sequence, context — composed with pipeline, data, and ZeRO), explicit allreduce latencies over hierarchical or arbitrary network topologies, and per-device memory/compute profiles, solved via structured dynamic programming. Up to 2.43× higher throughput over SOTA baselines across diverse hardware and networks; open source (from abstract only).
• How: DP over operator graphs annotated with per-op parallelism configurations and measured network latencies, jointly optimizing co-location, communication cost, and memory feasibility rather than patching feasibility after placement.
• Where applicable: General technique, not tied to one operator's fabric — but its quality depends on accurate allreduce-latency and memory profiles for your specific network. Most useful as a foundation for co-designing parallelization strategies with datacenter interconnects; needs real fabric measurements as input.