Skip to content

Keynote: Duolingo's AI Slackbot: An Enterprise Assistant With 180+ MCP Tools - Aaron Wang, Software Engineer, Duolingo Inc#

MCP Dev Summit North America 2026 digest Program Core pick 2–3 April 2026

Abstract

Aaron Wang, Software Engineer, Duolingo Inc

  • Why: A rare production agentic-ops assistant with an honest failure-mode list — exactly the "agents for infrastructure operations" use case, including incident triage and debugging from Slack.
  • What: A Slackbot built on Claude's Agent SDK and MCP connecting 180+ tools across 15+ MCP servers (GitHub, Jenkins, Sentry, Grafana, PagerDuty, Honeycomb), with standardized server patterns (stdio/HTTP, consistent config schemas), YAML-based role-gated tool allowlists, and a parallel evaluation harness validating tool usage, anti-hallucination, and response quality. The real-world challenges section is the payload: testing in production without synthetic environments, keeping customer PII out of models, fast-answer vs good-answer tradeoffs, agents hallucinating instead of calling tools, and wrong parameters causing slow queries (from abstract only).
  • How: Standardized MCP server transports and config; RBAC via YAML tool allowlists by user role; scripted parallel evaluation of MCP tool behavior.
  • Where applicable: Broadly transferable to mid-size orgs with a SaaS observability/CI stack; the eval framework and failure taxonomy are general, the specific server set is Duolingo's.