Transformer

This paper presents a method for converting pretrained transformers into linear-time architectures by focusing on the efficiency of state update designs. By analyzing softmax attention through a first-order approximation, the authors prove that delta-style updates are superior for post hoc linearization.

This paper presents a method for converting pretrained transformers into linear-time architectures by focusing on the efficiency of state update designs. By analyzing softmax attention through a first-order approximation, the authors prove that delta-style updates are superior for post hoc linearization.

This study demonstrates that reinforcement learning post-training for LLMs is not a uniform process across all parameters but is concentrated in specific middle layers. By leveraging this discovery, researchers developed layer-aware training methods that outperform traditional full-parameter optimization.

This study demonstrates that reinforcement learning post-training for LLMs is not a uniform process across all parameters but is concentrated in specific middle layers. By leveraging this discovery, researchers developed layer-aware training methods that outperform traditional full-parameter optimization.

Wan-Streamer is a new foundation model that enables seamless, low-latency audio-visual interaction by processing interleaved text, audio, and video tokens within a single causal Transformer. By moving away from cascaded modular pipelines, it achieves sub-second end-to-end latency suitable for real-time digital humans and embodied assistants.

Wan-Streamer is a new foundation model that enables seamless, low-latency audio-visual interaction by processing interleaved text, audio, and video tokens within a single causal Transformer. By moving away from cascaded modular pipelines, it achieves sub-second end-to-end latency suitable for real-time digital humans and embodied assistants.