Clock-Aligned Telemetry for AI Clusters: From BMC to GPU to Fleet#

Ahmad Byagowi (TBD), Lukas Landgraf (turbalance)

• Why: The observability gap this addresses — many dashboards, no common evidence model — sits exactly at the junction of the PM's telemetry and Kubernetes/eBPF interests, and clock alignment as the basis for cross-machine trust is an idea worth stealing.
• What: A lab-built reference architecture for clock-aligned AI telemetry: a source-bundle contract, live agents, a conservative benchmark ladder, a Redfish/BMC lane, and a lakehouse path that preserves provenance and never synthesizes unsupported metrics; links DCGM/NVML, host counters, Prometheus, Kubernetes, Redfish, eBPF/OTel evidence, and PTP/NTP clock state (from abstract only).
• How: Per-datacenter ingestion without global shared state feeds one Iceberg layer over cold and fresh stores; a six-level benchmark ladder (metric to global baseline), auto-discovery, credential-gated agents, 1s freshness, and capability labels — PTP/NTP state as a first-class signal changes what cross-machine comparison can be trusted.
• Where applicable: Prototyped on a heterogeneous lab fleet (not hyperscale production yet), with an open-source repo and a proposed OCP collaboration path — early-stage, which is exactly why it's a wildcard rather than core.