LoRA

This paper presents a method for converting pretrained transformers into linear-time architectures by focusing on the efficiency of state update designs. By analyzing softmax attention through a first-order approximation, the authors prove that delta-style updates are superior for post hoc linearization.

This paper presents a method for converting pretrained transformers into linear-time architectures by focusing on the efficiency of state update designs. By analyzing softmax attention through a first-order approximation, the authors prove that delta-style updates are superior for post hoc linearization.

The researchers present Program-as-Weights (PAW), a paradigm that shifts LLM usage from expensive per-input API calls to a 'compile-once, run-locally' model using neural adapters. This approach allows small, specialized models to outperform massive foundation models on specific fuzzy tasks while maintaining high efficiency and privacy.

This paper presents Distill to Detect (D2D), a technique for identifying stealthy, topic-specific biases in language models that evade standard detection. By distilling the distributional shift of a suspected model into a small prefix adapter, the method amplifies hidden signals until they become visible in generated text.

The researchers present Program-as-Weights (PAW), a paradigm that shifts LLM usage from expensive per-input API calls to a 'compile-once, run-locally' model using neural adapters. This approach allows small, specialized models to outperform massive foundation models on specific fuzzy tasks while maintaining high efficiency and privacy.

This paper presents Distill to Detect (D2D), a technique for identifying stealthy, topic-specific biases in language models that evade standard detection. By distilling the distributional shift of a suspected model into a small prefix adapter, the method amplifies hidden signals until they become visible in generated text.