Open Source · 2026 - Present

Orkhon

A complete, auditable, from-scratch LLM stack in PyTorch — a hand-written decoder-only Transformer (GQA, RoPE, RMSNorm, SwiGLU, KV-cache) with the full pipeline: byte-level BPE tokenizer, pretrain, SFT, DPO, GRPO/RLVR, eval, RAG/agent/tools, an OpenAI-compatible serve API, and safetensors export. Trains a chat-capable model end-to-end on your own machine (a ~1-minute MPS smoke run, or a 22M TinyStories model in ~43 min). Named after the Orkhon inscriptions — the oldest known Turkic writing.