DaraLM
A decoder-only Transformer for Khmer and English, written from scratch — data pipeline, tokenizer, pretraining, instruction tuning, evaluation, and serving.
Overview
DaraLM is an end-to-end language-model project for a low-resource language pair, built without reaching for a pretrained backbone. Everything from the attention implementation to the serving layer is in the repository: collect and clean the corpus, train a tokenizer, pretrain a base model, instruction-tune it, evaluate it on concrete tasks, then serve it behind an API.
The best checkpoint is a 33.4M-parameter model reaching 182.0 validation perplexity after 1,500 steps, and 110.7 on its SFT validation set after instruction tuning. A 149.2M model was also trained but stopped at 300 steps and is substantially undertrained — at 1,131.5 perplexity it performs worse than the smaller model, which is exactly what you would expect and is reported as such.
This is an educational research project, and the README says so first: generated facts are unreliable, language drifts, and no safety tuning was done. The interesting output is not a usable model — it is a documented account of what a small model at this compute budget can and cannot learn.
Highlights
- A Transformer implemented from the layers up: causal attention with rotary position embeddings, RMSNorm pre-normalization, GELU feed-forward, weight-tied LM head, GPT-style initialization.
- A Khmer/English data pipeline — Wikipedia and Common Crawl ingestion, Unicode normalization, markup and quality filtering, deduplication, and deterministic train/validation/test splits — producing a 16,077-document cleaned corpus.
- A SentencePiece tokenizer trained on that corpus, with fingerprint validation so a checkpoint can never be loaded against the wrong vocabulary.
- A training stack with mixed precision, gradient accumulation, learning-rate scheduling, and resumable checkpointing.
- Three trained checkpoints — 33.4M base, 33.4M instruction-tuned, 149.2M base — each with its own model card reporting validation perplexity and known failure modes.
- Task evaluations for language classification, grammar correction, and structured JSON extraction, plus KV-cached generation and a FastAPI service with a browser testing UI.
Design Notes
Negative results are the deliverable
Grammar correction is reported as a failed capability: scaling the training run 4x and 6x reduced both character and word error rate, and every model still did worse than returning the input unchanged. That is written up as a finding rather than quietly dropped from the model card, because 'more budget did not fix this' is the actual result of the experiment.
A high benchmark score that does not mean what it looks like
A frozen 50M backbone with a linear head hit 98.37% validation accuracy on Khmer/English classification. The model card says outright that the two scripts are trivially separable by Unicode range, so the number validates the classification path and demonstrates nothing about semantic understanding — reporting it without that caveat would be the misleading version.
Valid JSON overstates extraction quality
The structured-extraction task was evaluated under deliberate distribution shift, with real names, occupations and cities the model had never seen. It reliably produced well-formed JSON and reliably copied numeric ages, while failing on the unseen string fields — so schema validity alone was rejected as a metric in favor of per-field accuracy on novel vocabulary.
Tokenizer fingerprint validation
The single most confusing failure mode in a from-scratch pipeline is a checkpoint loaded against a tokenizer it wasn't trained with: nothing crashes, the output is just quietly garbage. A fingerprint is stored with the checkpoint and checked on load, turning a debugging afternoon into an immediate error.
KV-cached generation
Naive autoregressive sampling recomputes attention over the whole prefix at every step. Caching past keys and values makes generation cost proportional to the new token rather than the sequence length — necessary for the FastAPI serving path to answer interactively rather than in tens of seconds.
Bounded, reproducible data collection
Both the Wikipedia and Common Crawl fetchers take explicit per-language document caps and record source and licensing metadata alongside the raw records, with a dry-run preview before any merge. Corpus changes are additive and logged, so a perplexity number can always be traced back to the corpus that produced it — the merge notes state plainly that existing checkpoints predate the Common Crawl records and would need retraining to benefit from them.
Results
