From Bits to Tokens: Knowledge-Driven Generative Communication of Multimodal Data#

Xingyu Chen (UC San Diego), Zihao Feng (USC), Wuqiong Zhao (UCSD), Jianrong Ding (CUHK), Ke Sun (Michigan), Xinyu Zhang (UCSD)

• Why: Not your domain — and that's the point. This is the most complete systems argument yet that "the network should transmit task-relevant knowledge, not bits," backed by a working prototype rather than a position paper. It signals where edge/IoT/real-time communication research is going as foundation models meet the network stack.
• What: Knowledge-Driven Communication (KDC): a framework transmitting semantic tokens rather than raw bitstreams. A full prototype achieves high downstream task accuracy with a fraction of the transmitted data and degrades gracefully under wireless loss, beating state-of-the-art codecs and learned baselines.
• How: A task-aware transmitter uses multimodal foundation models to abstract source data into tokenized embeddings and prioritize semantically critical content (zero-shot, no task-specific retraining); the receiver reconstructs task-relevant information from pretrained knowledge bases plus incrementally updated context.
• Where applicable: Wireless, lossy, bandwidth-constrained multimodal settings today (sensors, video analytics at the edge). Long-horizon relevance rather than near-term deployability for DC networking — but the tokens-not-bits framing will increasingly show up in adaptive video and agentic-traffic contexts.