RYS-XLarge: Technical Deep Dive

LLM Neuroanatomy: A Technical Deep Dive into Layer Duplication for Leaderboard Domination

Executive Summary
David Noel Ng’s “RYS-XLarge” achieved the top position on the Hugging Face Open LLM Leaderboard in mid-2024 without any weight updates, merging, or fine-tuning. The technique involved taking a 72B-parameter base model and duplicating a specific block of seven consecutive middle layers, effectively inserting a second copy of the model’s “reasoning core” while preserving all original weights. This resulted in a model that outperformed heavily optimized fine-tunes and merges on the six-benchmark suite (IFEval, BBH, MATH Lvl 5, GPQA, MuSR, MMLU-PRO). The work introduces the concept of LLM Neuroanatomy—the hypothesis that early layers act as modality translators, late layers as output generators, and a narrow band of middle layers performs abstract symbolic reasoning in a language-agnostic latent space. The discovery was made on consumer gaming hardware (two high-end GPUs) using a custom “brain scanner” tool for layer ablation and activation analysis.

Technical Architecture

The core insight rests on the standard decoder-only Transformer architecture used by Llama-family models. In a typical 72B model (approximately 80–84 layers depending on exact variant), the forward pass can be abstracted as:

Input Tokens → Embedding → Layer 1 … Layer N → LM Head → Logits

Ng’s hypothesis, derived from two empirical clues, posits a functional specialization across depth:

1. Early layers (≈ layers 1–20): Act as input translators. They map diverse token distributions (English, code, Base64, Mandarin, etc.) into a shared abstract representation. Evidence: the model’s ability to reason correctly when the entire prompt is Base64-encoded, despite the tokenizer producing radically different subword IDs and positional encodings.

2. Middle layers (≈ layers 25–55): Perform pure reasoning. These layers operate in a highly abstract, language-agnostic latent space where symbolic manipulation, logical chaining, and multi-step planning occur. The author refers to this band as the “thinking core.”

3. Late layers (≈ layers 60–N): Function as output translators, mapping the abstract representation back into the desired output format (natural language, code, Base64, JSON, etc.).

The “Goliath Anomaly” provided the second clue. The 120B Goliath-120b model was constructed by alternating layers from two different fine-tuned Llama-2-70B models (Xwin and Euryale) with deliberate cross-connections (e.g., feeding layer 16 of Xwin into layer 8 of Euryale). Despite appearing architecturally “insane” from a conventional residual-stream perspective, the model performed surprisingly well. This suggested that the precise identity of layers is less critical than their functional role and that the residual stream is tolerant of significant structural surgery.

Building on these observations, Ng developed a homebrew mechanistic interpretability toolkit—“LLM Brain Scanner”—that allows rapid layer ablation, activation patching, and targeted duplication experiments. The scanner runs efficiently on two gaming GPUs by leveraging aggressive quantization (4-bit or 8-bit) and selective layer loading.

The winning modification was simple yet surgically precise: locate the seven-layer block that produced the strongest positive effect on reasoning benchmarks when duplicated, then insert a copy of that exact block immediately after the original. Mathematically, if the original layer indices are [l, l+1, …, l+6], the new architecture becomes:

… → Layer(l+5) → Layer(l+6) → [Duplicate Block: Layer(l) … Layer(l+6)] → Layer(l+7) → …

No weights are changed. The only modifications are:

• Updating the layer index mapping in the model configuration.
• Adjusting the residual stream connections at the splice points.
• Increasing the total layer count (typically from ~80 to ~87).

Because the duplicated block is identical to an existing segment, the model can be constructed in minutes using standard PyTorch/HF nn.Module surgery or by manipulating the model.layers list directly.

Performance Analysis

On the mid-2024 Hugging Face Open LLM Leaderboard (the harder “v2” version using IFEval, BBH, MATH Lvl 5, GPQA, MuSR, and MMLU-PRO), RYS-XLarge achieved the highest average score, surpassing models that had undergone extensive RLHF, preference optimization, and sophisticated merging strategies such as DARE, SLERP, and TIES.

While exact per-benchmark numbers are not fully enumerated in the source post, the author states that the duplicated model showed particularly large gains on:

• MATH Lvl 5 (heavy mathematical reasoning)
• GPQA (graduate-level science questions)
• MuSR (multi-step reasoning and puzzle solving)

These are precisely the tasks that should benefit most from an enlarged “thinking core.”

Benchmark Comparison Table (reconstructed from leaderboard context and author claims)

The key observation is that RYS-XLarge outperformed models with far more compute investment, demonstrating that architectural “neuroanatomy” tweaks can rival or exceed data-driven optimization for certain reasoning tasks.

Technical Implications

This work has profound implications for the open-source LLM ecosystem:

• Inference-time architecture search becomes a viable optimization axis. Instead of only scaling parameters or data, practitioners can treat layer ordering, duplication, and depth allocation as hyperparameters.
• Mechanistic interpretability gains a practical engineering application. Identifying the “reasoning band” in new architectures could become a standard step before deployment.
• Hardware efficiency: Duplicating a 7-layer block in a 72B model increases FLOPs by roughly 9%, but the performance gain appears to exceed what would be expected from simply adding random layers or increasing width. This suggests better scaling laws may exist when depth is allocated intelligently.
• Merging and model surgery: Techniques like layer stitching, duplication, and cross-model splicing may be more powerful than current merge methods (SLERP, DARE, TIES) for reasoning-heavy tasks.
• Interpretability research: The ability to run such experiments on consumer hardware (two gaming GPUs) lowers the barrier for independent researchers to contribute to mechanistic interpretability.

Limitations and Trade-offs

• Increased latency and memory: Adding layers increases both KV cache size and per-token compute. For the 72B → ~79B effective model, this is modest but non-zero.
• Diminishing returns: The author notes that duplicating too many layers or the wrong layers can degrade performance, suggesting there is an optimal “thinking core” size.
• Task specificity: Gains are strongest on reasoning benchmarks. The method may not improve (or could slightly hurt) creative writing, instruction following, or chat-style tasks.
• Generalization: Results were obtained on Llama-2/3-era 70–72B models. It remains to be seen how well the technique transfers to newer dense, MoE, or state-space architectures.
• Reproducibility: Exact layer indices for the optimal 7-layer block are not publicly released in the blog post, though the methodology is described in sufficient detail for skilled practitioners to replicate the discovery process.

Expert Perspective

Ng’s work is one of the most significant independent contributions to open LLM engineering in 2024. It demonstrates that even after massive pretraining and fine-tuning investments, there remain low-hanging architectural improvements that require no additional training data or compute—only insight into the internal computation graph.

The discovery reinforces the emerging view that LLMs are not monolithic “black boxes” but possess a surprisingly modular internal structure. The fact that a narrow band of middle layers can be duplicated to boost reasoning performance suggests these layers have converged on a general-purpose symbolic processor during pretraining.

From an engineering standpoint, this opens an exciting new research direction: differentiable architecture search at the layer level and functional specialization mapping for every major open model. Future leaderboards may see a split between “raw capability” and “architecturally optimized” categories.

The use of consumer gaming GPUs for this research is also noteworthy. It proves that high-impact LLM research is still possible outside well-funded labs, provided the researcher asks the right questions and builds the right tools.

Technical FAQ

How does layer duplication compare to traditional model merging techniques?

Layer duplication operates on a single base model and modifies depth rather than averaging or interpolating weights. Unlike DARE, TIES, or SLERP merges that attempt to combine capabilities from multiple fine-tunes, duplication amplifies an existing capability (abstract reasoning) by increasing the number of iterations through the critical computational subgraph. It is cheaper, faster, and in this case, more effective for reasoning benchmarks.

Can this technique be applied automatically without manual inspection?

In principle, yes. The “LLM Brain Scanner” approach can be automated using activation patching, causal tracing, or gradient-free layer importance metrics. A practical system could sweep over possible duplication windows and evaluate on a small validation set of hard reasoning problems. The author’s manual discovery process would likely be replaced by a search algorithm in future work.

What is the impact on inference speed and memory usage?

Duplicating seven layers in a 72B model increases total parameters by ~8–9% and forward-pass FLOPs by a similar factor. In practice, with continuous batching and paged attention, the throughput reduction is roughly proportional to the added depth. KV cache size grows linearly with layer count, which may be the bigger constraint on long-context deployments.

Is the technique backwards-compatible with existing inference engines?

Yes. The resulting model is still a standard Transformer with a modified config.num_hidden_layers. Most serving frameworks (vLLM, Hugging Face Text Generation Inference, llama.cpp) will run it without modification after the config and state_dict are updated. The only requirement is that the layer indices are renumbered contiguously.

References

• Original blog post: “LLM Neuroanatomy: How I Topped the AI Leaderboard Without Changing a Single Weight”
• Hugging Face Open LLM Leaderboard (v2 benchmark suite)
• Goliath-120B model card and construction details
• Related mechanistic interpretability literature on residual stream, layer specialization, and causal interventions

Sources

• David Noel Ng - LLM Neuroanatomy: How I Topped the AI Leaderboard Without Changing a Single Weight
• Hugging Face Open LLM Leaderboard
• Hugging Face Open LLM Leaderboard Organisation
• DeepLearning.AI - Hugging Face Overhauls Open LLM Leaderboard

All technical specifications, pricing, and benchmark data in this article are sourced directly from official announcements. Competitor comparisons use publicly available data at time of publication. We update our coverage as new information becomes available.