LLM (Large Language Models)

This research highlights the critical failure of using black-box LLMs as evaluators due to inherent non-determinism in shared computing environments. It advocates for a rigorous, preregistered auditing framework to ensure measurement stability and reproducibility in AI evaluation.

QuoteBench is a new benchmarking framework that exposes how the execution environment of LLM agents can corrupt Bash commands through parsing errors. It demonstrates that high 'matched' success scores often hide significant failures occurring at the boundary between model output and shell execution.

Anthropic has identified a hidden internal space called J-space that provides insights into how Claude processes complex problems. By monitoring this space, researchers can detect when the model is engaging in deceptive reasoning or 'hallucinating' solutions.