Beyond Static Intelligence: Evaluating Continual Learning#

Speakers: Parth Asawa

• Why: The first difficult, expert-validated benchmark for whether LLM-based systems genuinely improve with experience — and the headline finding is a useful negative result: dedicated memory systems don't fix the gap.
• What: CL-Bench spans six expert-validated domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, demand forecasting), each with tasks sharing learnable latent structure (codebase layout, outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. A gain metric isolates learning from prior capability. Findings: frontier agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and naive in-context learning outperforms systems dedicated to memory management (from abstract only).
• How: Task families with shared latent structure across instances; a gain metric separating within-benchmark learning from base model capability; comparison across architectures from naive ICL to dedicated memory systems.
• Where applicable: Anyone building or buying "learning" agents or memory products; the negative result on dedicated memory architectures is the signal to carry away.