Transformer

This technical report explores how the placement of LoRA adapters within Transformer layers affects multi-hop reasoning capabilities. It concludes that targeting early layers is critical for preventing models from 'stopping thinking' too early during complex inference loops.

This article explores a method for creating deterministic arithmetic calculators by compiling algorithmic logic directly into transformer weights. By using the Torchwright compiler, the author bypasses the need for training and achieves exact multiplication, addition, and subtraction results.

This paper presents a method for converting pretrained transformers into linear-time architectures by focusing on the efficiency of state update designs. By analyzing softmax attention through a first-order approximation, the authors prove that delta-style updates are superior for post hoc linearization.

The article discusses why traditional neural networks fail to maintain physical conservation laws in dynamical systems. It proposes using differential geometry and symplectic structures to build architectures that inherently respect the physics of the problem.

This study demonstrates that reinforcement learning post-training for LLMs is not a uniform process across all parameters but is concentrated in specific middle layers. By leveraging this discovery, researchers developed layer-aware training methods that outperform traditional full-parameter optimization.

Wan-Streamer is a new foundation model that enables seamless, low-latency audio-visual interaction by processing interleaved text, audio, and video tokens within a single causal Transformer. By moving away from cascaded modular pipelines, it achieves sub-second end-to-end latency suitable for real-time digital humans and embodied assistants.