Transformer architectures

This research proposes a new MoE architecture called Meta Expert that shares experts across layers to mitigate parameter redundancy. The approach demonstrates that models can maintain high performance with significantly fewer parameters and experts compared to traditional hierarchical designs.

This paper presents a proof-of-concept for backdoor attacks that can provably hide arbitrary computational capabilities within AI models using cryptographic digital lockers. The researchers show that these hidden circuits are resistant to common auditing and elicitation techniques, posing a significant security risk in supply chain's integrity.

This research expands on Anthropic's 'Verbalizable-Workspace' paper by applying J-space measurements to open-source models. It characterizes the internal structure of LLMs as having distinct functional zones and demonstrates that steering influence decays via a power law.