Technical Investigation: Why Fine-Tuning HimalayanGPT 0.5B Fails on the Hugging Face Stack (and Why Qwen2.5-3B Succeeded)

Authors: Dimanjan Dahal, Kushal Shrestha, Gemini Antigravity
Project: Food Restaurant Intent Classification Project
Date: 27 Jul 2026 | Document Version: 1.0
Models Evaluated: himalaya-ai/himalayagpt-0.5b vs Qwen/Qwen2.5-3B-Instruct
Target Domain: Multi-Intent Roman Nepali Restaurant Assistant

[!CRITICAL] Executive Summary & Verdict: Fine-tuning himalaya-ai/himalayagpt-0.5b with the industry-standard Hugging Face (HF) + PEFT + TRL + Unsloth pipeline failed at training step 0 with a CUDA device-side assert (Loss.cu assertion t >= 0 && t < n_classes). Root-cause analysis confirms that HimalayanGPT is built on Karpathy's custom Nanochat architecture rather than standard Transformers causal LM APIs. It introduces 8 auxiliary value_embeds tables, assigns an invalid out-of-bounds special token ID 32768, and uses ignore_index = -1 instead of the standard -100. Meanwhile, the exact same 14,000-sample Roman Nepali dataset trained cleanly on Qwen2.5-3B-Instruct, achieving 99.73% eval accuracy and 99.73% macro F1.

1. Project Objective & Selection Rationale#

The objective of this engineering initiative was to fine-tune a compact, Nepali-specialized model using 4-bit QLoRA on a validated multi-intent restaurant ordering dataset (train.json / validation.json), matching the production behavior already established with Qwen2.5-3B-Instruct.

Why HimalayanGPT Was Selected#

• Nepali Native Focus: Marketed by Himalaya AI specifically for high-fidelity Nepali language processing.
• Model Efficiency: Parameter size of ~524M (15 layers, hidden size 1024) compared to ~3.086B for Qwen2.5-3B.
• Deployment Footprint: Lower VRAM footprint to allow concurrent model serving on cost-effective 8 GB laptop GPUs (NVIDIA GeForce RTX 4070 Laptop GPU).
• Pipeline Reuse Hypothesis: Expected that standard Hugging Face ecosystem tooling (Transformers, PEFT, TRL SFTTrainer, BitsAndBytes NF4) could be ported seamlessly from Qwen to HimalayanGPT.

2. Baseline: The Proven Qwen2.5-3B Pipeline#

Before evaluating HimalayanGPT, the exact same multi-intent Roman Nepali dataset was trained on Qwen2.5-3B-Instruct using:

• Hugging Face Transformers
• PEFT (LoRA with rank 16, alpha 32)
• TRL (SFTTrainer with DataCollatorForCompletionOnlyLM)
• Unsloth (4-bit QLoRA acceleration)
• BitsAndBytes NF4 quantization

Qwen Validation Metrics#

• Smoke-Test Evaluation: Accuracy 0.6484 (64.84%), Macro F1 0.7328 after 1 epoch on limited subset.
• Full-Scale Training: Final validation accuracy 0.9973 (99.73%), Macro F1 0.9973.
• Inference Stability: Zero malformed predictions across thousands of test queries.

This conclusively proved that the dataset format, intent schema, multi-label combinations, and supervisory targets were mathematically and semantically sound.

3. HimalayanGPT Architecture Investigation#

Inspection of Hub weights and repository code revealed that himalaya-ai/himalayagpt-0.5b is not a native Hugging Face causal LM. It requires trust_remote_code=True and imports three proprietary modules:

• configuration_nanochat.NanochatConfig
• modeling_nanochat.NanochatForCausalLM
• tokenization_nanochat.NanochatTokenizer

+-----------------------------------------------------------------------------------+
|                           NANOCHAT CUSTOM ARCHITECTURE                            |
|                                                                                   |
|  NanochatTokenizer (Pickle-serialized Rust BPE encoder)                           |
|       |                                                                           |
|       +--> Token IDs: Valid vocab = [0 .. 32767] (Vocab Size = 32768)             |
|       +--> Special Tokens: Exposes BOS/EOS/PAD = 32768 (OUT OF BOUNDS!)           |
|                                                                                   |
|  NanochatForCausalLM                                                              |
|       |                                                                           |
|       +--> wte (Input Embeddings): [32768, 1024]                                  |
|       +--> value_embeds: 8 distinct tables [32768, 1024] on layers 0,2,4..14      |
|       +--> Forward Loss: F.cross_entropy(..., ignore_index=-1)                    |
|       +--> Optimizer in Official Repo: MuonAdamW (No HF Trainer, No PEFT)          |
+-----------------------------------------------------------------------------------+

Key architectural divergences from standard Hugging Face causal models:

1. Custom Decoder Stack: Derived from Karpathy's Nanochat research codebase (NanochatBackbone).
2. Proprietary Pickle Tokenizer: NanochatTokenizer wraps a pickle-serialized Rust BPE encoder (_enc), lacking standard SentencePiece or tiktoken Jinja template contracts.
3. Internal Value Embedding Tables: In addition to standard token embeddings (wte), Nanochat inserts 8 separate value_embeds tables feeding even-numbered transformer blocks (layers 0, 2, 4, ..., 14).
4. Bespoke Training Harness: The official repository relies on a custom training loop with a hybrid MuonAdamW optimizer (Muon for 2D weight matrices, AdamW for embeddings) and explicit ignore_index = -1.

4. Chronological Investigation Timeline#

The investigation unfolded in 10 sequential diagnostic steps:

5. Summary of Problems Encountered & Mitigations Attempted#

6. Deep Root Cause Analysis#

6.1 The Step-0 CUDA Device-Side Assert Anatomy#

In PyTorch CUDA cross-entropy computation:

// Loss.cu CUDA assertion:
CUDA_KERNEL_ASSERT(t >= 0 && t < n_classes);

For a vocabulary of size 32,768, valid class indices are 0 <= t <= 32767.

When TRL's SFTTrainer initialized, it detected that the tokenizer claimed its EOS/PAD token was 32768. TRL appended this token to mark sequence boundaries. During batch collating, target labels contained index 32768.

When F.cross_entropy evaluated the logits tensor [batch, seq_len, 32768] against the target tensor containing 32768, the CUDA kernel attempted to read beyond the logits tensor memory boundary, triggering an unrecoverable GPU device assertion and aborting the process.

6.2 The Eight-Table value_embeds Dilemma#

Standard Hugging Face models utilize a single input embedding matrix (model.get_input_embeddings()). Calling resize_token_embeddings(new_size) resizes this single table.

In NanochatForCausalLM, the model maintains:

• wte: Token embeddings [32768, 1024]
• value_embeds: A ModuleDict containing 8 separate embedding tables [32768, 1024] on layers 0, 2, 4, 6, 8, 10, 12, and 14.

Calling standard Hugging Face resize methods leaves all eight value_embeds tables at dimension 32,768. The moment a token with ID 32,768 passes into layer 0, the value embedding lookup triggers an index out-of-bounds error.

6.3 The Masking Collision: -1 vs -100#

Standard Hugging Face collators mask non-target prompt tokens using label = -100. Standard Hugging Face model forward implementations compute:

loss = F.cross_entropy(logits.view(-1, vocab_size), labels.view(-1), ignore_index=-100)

However, in modeling_nanochat.py (line 252), loss is explicitly hardcoded as:

loss = F.cross_entropy(logits.view(-1, vocab_size), labels.view(-1), ignore_index=-1)

Because -100 != -1, every prompt token that was supposed to be ignored is treated by PyTorch as an active class target. Because -100 < 0, this immediately trips the lower-bound assertion t >= 0 in Loss.cu.

7. Evidence Collected & Measured Model Properties#

7.1 Measured Parameters#

The following hardware and structural properties were empirically measured:

8. Comparative Matrix: Standard Models vs HimalayanGPT#

9. Alternative Approaches Evaluated#

Four engineering paths were evaluated following the root-cause diagnosis:

10. Root-Cause Severity Assessment & Risk Register#

Severity Assessment#

• Invalid Special Token ID 32768: Critical - Causes immediate CUDA crash at step 0.
• ignore_index Semantic Mismatch (-1 vs -100): Critical - Corrupts gradient computation if step 0 is bypassed.
• Unmanaged value_embeds Tables: High - Breaks hidden state representations when resizing embeddings.
• Lack of Gradient Checkpointing: Medium - Increases peak VRAM usage during training.
• LoRA Limited to c_proj: High - Drastically restricts model adaptation capacity.

Project Risk Register#

• Publishing Invalid Fine-Tuned Checkpoints: High likelihood if training is forced; mitigated by immediately halting the HF path.
• Silent Accuracy Degradation: High likelihood if ignore_index is incorrectly aligned; mitigated by adopting Qwen2.5-3B.
• Engineering Schedule Overrun: Mitigated by pivoting away from custom Nanochat reimplementation.

11. Strategic Recommendations & Conclusion#

1. Production Deployment: Standardize on Qwen2.5-3B-Instruct with 4-bit QLoRA as the production intent classifier for the Roman Nepali restaurant assistant. Its 99.73% accuracy and full HF ecosystem compatibility provide enterprise stability.
2. Himalaya AI Architecture Guidance: Discontinue attempts to fine-tune himalaya-ai/himalayagpt-0.5b via standard Hugging Face SFTTrainer or PEFT. If HimalayanGPT must be revisited, fork the official karpathy/nanochat repository and execute training using its native MuonAdamW loop.
3. Ecosystem Verification Rule: Prior to committing engineering resources to low-resource domain models, establish a standard pre-training audit checking: (1) native transformers support, (2) standard chat_template, (3) -100 loss masking semantics, and (4) gradient checkpointing compatibility.