From Manual to Marathon: My Automation Journey at HEB#

Erich Crosswhite — H-E-B

• Why: The best near-miss postmortem in the program: a change that nearly removed DNS servers from the production config chain, what guardrail failed, and how the pipeline was hardened afterward.
• What: An IaC journey for a new Arista environment at a 400+ store retailer: AVD-based config generation, GitLab MRs with peer review, AWX webhooks, CloudVision (CVaaS) change controls, auto-generated Confluence docs, and — after the near-miss — Nautobot integration for access-port config that was subsequently removed, not because the product failed, but because the author's scripting lacked controls.
• How: The DNS incident: a Nautobot data cleanup (logical cables disconnected) propagated through CI/CD and the generated config dropped DNS servers; luck (execute_tasks: false in CloudVision) prevented an outage. The fix: build stage generating config from both main and feature branches, a validate stage diffing them plus Batfish verification, results reported directly into the merge request for review before deploy. Also includes a lab cautionary tale (a state: absent that wiped the lab fabric to ZTP) and future work toward Container Lab digital twins with ANTA validation.
• Where applicable: H-E-B/enterprise-specific and Arista-stack-specific, but the pipeline design (MR-integrated diff + Batfish gates, guardrails around SoT-driven generation) is adoptable by anyone. The core lesson — the SoT product was fine; the unguarded integration was the risk — generalizes completely.