On-prem LLM cluster for network and security analyses#

Mischa Diehm - narrowin

• Why: A reference architecture for running AI over network data entirely on-prem — and, unusually, it takes the identity and security model seriously (scoped agent credentials, RBAC, MCP authorization, audit trails linking user → tool call → device change), which maps directly onto the PM's workload and agentic identity interests. It also includes real small-cluster physical networking detail.
• What: A four-layer sovereign stack: interaction (chat/web/API + a planning agent), AI (open-weight models — GLM, Qwen, Nemotron, DeepSeek — on a local GPU cluster served by vLLM behind an OpenAI-compatible API), integration (MCP servers exposing purpose-built tools plus skills, not raw data), and context (topology, configs, syslog, CMDB/IPAM, ticketing, or a single device via SSH/REST/SNMP). Evaluated on a multi-vendor IT/OT Containerlab lab network and agent-driven assessments of production data from real networks.
• How: Hardware is 4× NVIDIA DGX Spark (512 GB pooled memory for weights/KV cache when sharded across nodes) behind a MikroTik CRS804-4DDQ 400G switch — about 25K CHF. Notable networking finding: internal PCIe limits mean each Spark gets 2×200G, not 400G, so one 200G link carries data plane and management moves to a separate interface; the nodes are memory-rich but bandwidth-poor (~273 GB/s), so MoE models run best. The interesting operational lessons are in data fusion: combine state, signals, inventory, and intent (from a system of record like Infrahub), pair tools with skills instead of exposing raw data, and make every finding verifiable by linking it to source evidence. Security model covers RBAC per role/device/action, MCP Enterprise-Managed Authorization via the identity provider, scoped SSH/API identities for the agent, prompt injection (a malicious ticket can steer tool calls even with a local model), model-trust limits of open weights, audit trails, and human approval gates with rollback.
• Where applicable: Operators with privacy, sovereignty (CLOUD Act / AI Act), or cost constraints on shipping network data to cloud LLMs — particularly IT/OT and critical infrastructure. The honest trade-offs are stated: you need staff who can run it, the model market moves monthly, and without clear goals it becomes a time sink. Reproducible as a reference architecture; scale is deliberately small (desk-sized), not hyperscaler.