KS-Probe: A Multidimensional Benchmark for Evaluating Long-Context Fidelity in Frontier Language Models

Abstract

Abstract-Large language models (LLMs) are increasingly being deployed with extended context windows ranging from 100K to 200K tokens, enabling applications involving long-document analysis, multi-source reasoning, and prolonged conversational interaction. Despite these developments, the practical reliability of long-context processing remains insufficiently understood. This study introduces KS-Probe (Probing Recall Over Boundaries and Extents), a benchmark framework developed to systematically evaluate context fidelity dynamics
in frontier LLMs. The benchmark embeds verifiable probe facts into synthetic domain-diverse filler text and measures Probe Recall Accuracy (PRA) under varying conditions of context length, positional placement, conversational depth, truncation boundaries, and tokenizer divergence. Four frontier models GPT-5.2, Claude Sonnet 4.6, Grok-4.1-fast, and DeepSeek-v3.2—were evaluated using 498 API runs involving more than 24 million input tokens. Experimental findings reveal substantial model-specific differences. Claude Sonnet 4.6demonstrates improved recall with increasing context length, while Grok-4.1-fast experiences severe degradation beyond 100K tokens. DeepSeek-v3.2 exhibits stable recall behavior across tested ranges, and multi-turn conversational formatting improves recall performance for most models. The study further identifies significant tokenizer divergence, particularly in Claude Sonnet 4.6, which requires substantially more tokens for equivalent text. The findings demonstrate that long-context reliability is influenced by architecture, formatting, positional placement, and tokenizer behavior. KS Probe provides a reproducible and extensible benchmark for evaluating long-context fidelity in modern AI systems.