```
├── .gitignore
├── LICENSE.md (200 tokens)
├── README.md (1400 tokens)
├── bdh.py (1000 tokens)
├── figs/
├── architecture.png
├── bdh_scaling.png
├── vocab.png
├── requirements.txt
├── train.py (700 tokens)
```
## /.gitignore
```gitignore path="/.gitignore"
input.txt
```
## /LICENSE.md
Copyright 2025 Pathway Technology, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
## /README.md
# BDH (Dragon Hatchling)
## **Bridging the Gap Between Transformers and the Brain**
**BDH (Dragon Hatchling)** is a biologically inspired large language model architecture that connects principles of deep learning with the foundations of neuroscience. Developed by researchers at [Pathway](https://pathway.com), BDH provides a theoretical and practical framework for understanding the emergence of reasoning and generalization in artificial systems.
This repository contains the official implementation from the paper:
> *A. Kosowski, P. Uznański, J. Chorowski, Z. Stamirowska, M. Bartoszkiewicz.*
> [_The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain_](https://doi.org/10.48550/arXiv.2509.26507), arXiv (2025).
## Overview
BDH represents a **scale-free, locally interacting network of neurons** capable of intrinsic reasoning dynamics. BDH scales like a Transformer on performance benchmarks—yet retains full interpretability and theoretical grounding in the fine-grained dynamics of neuron interactions.
**Key properties:**
- **Scale-free network topology** mimicking biological connectivity
- **Locally interacting neuron particles** with excitatory/inhibitory dynamics
- **Hebbian working memory** based on synaptic plasticity, displaying monosemanticity
- **GPU-friendly state-space formulation** for efficient implementation
- **Interpretable activations** that are sparse and positive
BDH formalizes a bridge between **neural computation and machine-based language understanding**. It shows how **macro reasoning behavior** in large AI models emerges from **micro-level neuron dynamics**, guided by principles of graph theory and local computation.
Empirically, BDH matches **GPT-2–scale Transformers** across language and translation tasks at equivalent parameter scales (10M–1B).
***
## Architecture
<img src="figs/architecture.png" width="600"/>
***
## Relation to Transformers
<img src="figs/vocab.png" width="600"/>
BDH and the Transformer share attention-inspired computation; however, BDH’s graph-based architecture makes its attention **emerge naturally from neuron-level interactions**, reflecting attention as seen in biological systems.
***
## Scaling Laws
<img src="figs/bdh_scaling.png" width="600"/>
BDH follows **Transformer-like scaling laws**, maintaining parameter efficiency while achieving interpretability at any scale.
***
## Latest research update: Sudoku Benchmark
Note: The Sudoku Extreme result refers to Pathway’s internal BDH implementation, not to the current open-source repository. This repository contains the implementation of the baseline variant as described in our [public paper](https://arxiv.org/abs/2509.26507) and does not reproduce the 97.4% benchmark result out of the box. See the dedicated Extreme Sudoku research blog post for additional benchmark context and the reported results.
On Sudoku Extreme, BDH reaches 97.4% accuracy across roughly 250,000 difficult puzzles, without chain-of-thought, solution backtracking, or external tool use, while leading LLMs struggle to perform on the benchmark at all.
Language is not enough for intelligence. Transformers process information token by token with limited internal state, which makes search-heavy, non-linguistic reasoning tasks like Sudoku awkward. BDH uses a larger latent reasoning space with intrinsic memory that supports learning and adaptation during use.
We believe that the future of AI will belong to systems that can reason natively across domains, that can hold multiple possibilities in a rich latent space, and that can converge on solutions without needing to verbalize every step. BDH is our answer to that challenge. It is designed to be a universal reasoning system that can speak our language without being trapped inside it. And yes, it solves Sudoku.
Read more: [Post-transformers: Sudoku Bench](https://pathway.com/research/beyond-transformers-sudoku-bench)
### Performance Comparison
| Model | Sudoku Extreme Accuracy | Relative Cost |
|------|------------------------|--------------|
| Pathway BDH | 97.4% | 10× lower, No chain-of-thought |
| Leading LLMs (O3-mini, DeepSeek R1, Claude 3.7 8K) | ~0% | High (chain-of-thought) |
*Table 1: Performance comparison on extreme Sudoku benchmarks (~250,000 difficult puzzles).*
*Source: Pathway internal data and https://arxiv.org/pdf/2506.21734 for the Leading LLMs’ accuracy score. Pathway’s approach reflects top-1 accuracy and does not rely on chain-of-thought nor solution backtracking.*
## Installation and Training
```bash
# install dependencies
pip install -r requirements.txt
# train BDH on a toy dataset
python train.py
```
<!--For visualization and interpretability analysis, explore the example notebooks in `notebooks/`.-->
## Learn and Discuss
- Watch the *SuperDataScience podcast* [▶️ *Dragon Hatchling: The Missing Link Between Transformers and the Brain*](https://www.youtube.com/watch?v=mfV44-mtg7c) (72 min.) featuring Adrian Kosowski in conversation with Jon Krohn, unpacking BDH’s neuron-level architecture and sparse reasoning dynamics.
- Read about BDH in
[*Forbes*](https://www.forbes.com/sites/victordey/2025/10/08/can-ai-learn-and-evolve-like-a-brain-pathways-bold-research-thinks-so/),
[*Semafor*](https://www.semafor.com/article/10/01/2025/new-ai-research-claims-to-be-getting-closer-to-modeling-human-brain),
[*The Turing Post*](https://www.turingpost.com/p/fod-121-300-million-to-start-a-big-promise-for-science#the-freshest-research-papers-catego),
[*Quantum Zeitgeist*](https://quantumzeitgeist.com/palo-alto-ai-firm-pathway-unveils-post-transformer-architecture-for-autonomous-ai/),
[*Golem*](https://www.golem.de/news/neue-ki-architektur-was-ist-baby-dragon-hatchling-2510-201047-2.html),
and elsewhere in the media.
- Discuss and share the BDH paper on:
[*Hugging Face Papers*](https://huggingface.co/papers/2509.26507),
[*Alphaxiv*](https://alphaxiv.org/abs/2509.26507),
and [*EmergentMind*](https://emergentmind.com/papers/2509.26507).
## Community Projects
- [adamskrodzki/bdh](https://github.com/adamskrodzki/bdh): dynamic vocabulary, stateful attention
- [mosure/burn_dragon_hatchling](https://github.com/mosure/burn_dragon_hatchling): Burn port
- [severian42/bdh](https://github.com/severian42/bdh): MLX port
- [Git-Faisal/bdh](https://github.com/Git-Faisal/bdh)
- [GrahLnn/bdh](https://github.com/GrahLnn/bdh)
## Acknowledgements
We thank Andrej Karpathy for the [nanoGPT](https://github.com/karpathy/nanoGPT/) code and the tiny Shapespeare dataset used in this demonstration.
BDH research stands at the intersection of **AI architecture**, **biological learning models**, and **theoretical computer science**—an effort to map the *equations of reasoning* between artificial and biological intelligence.
## /bdh.py
```py path="/bdh.py"
# Copyright 2025 Pathway Technology, Inc.
import dataclasses
import math
import torch
import torch.nn.functional as F
from torch import nn
@dataclasses.dataclass
class BDHConfig:
n_layer: int = 6
n_embd: int = 256
dropout: float = 0.1
n_head: int = 4
mlp_internal_dim_multiplier: int = 128
vocab_size: int = 256
def get_freqs(n, theta, dtype):
def quantize(t, q=2):
return (t / q).floor() * q
return (
1.0
/ (theta ** (quantize(torch.arange(0, n, 1, dtype=dtype)) / n))
/ (2 * math.pi)
)
class Attention(torch.nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
nh = config.n_head
D = config.n_embd
N = config.mlp_internal_dim_multiplier * D // nh
self.freqs = torch.nn.Buffer(
get_freqs(N, theta=2**16, dtype=torch.float32).view(1, 1, 1, N)
)
@staticmethod
def phases_cos_sin(phases):
phases = (phases % 1) * (2 * math.pi)
phases_cos = torch.cos(phases)
phases_sin = torch.sin(phases)
return phases_cos, phases_sin
@staticmethod
def rope(phases, v):
v_rot = torch.stack((-v[..., 1::2], v[..., ::2]), dim=-1).view(*v.size())
phases_cos, phases_sin = Attention.phases_cos_sin(phases)
return (v * phases_cos).to(v.dtype) + (v_rot * phases_sin).to(v.dtype)
def forward(self, Q, K, V):
assert self.freqs.dtype == torch.float32
assert K is Q
_, _, T, _ = Q.size()
r_phases = (
torch.arange(
0,
T,
device=self.freqs.device,
dtype=self.freqs.dtype,
).view(1, 1, -1, 1)
) * self.freqs
QR = self.rope(r_phases, Q)
KR = QR
# Current attention
scores = (QR @ KR.mT).tril(diagonal=-1)
return scores @ V
class BDH(nn.Module):
def __init__(self, config: BDHConfig):
super().__init__()
assert config.vocab_size is not None
self.config = config
nh = config.n_head
D = config.n_embd
N = config.mlp_internal_dim_multiplier * D // nh
self.decoder = nn.Parameter(torch.zeros((nh * N, D)).normal_(std=0.02))
self.encoder = nn.Parameter(torch.zeros((nh, D, N)).normal_(std=0.02))
self.attn = Attention(config)
self.ln = nn.LayerNorm(D, elementwise_affine=False, bias=False)
self.embed = nn.Embedding(config.vocab_size, D)
self.drop = nn.Dropout(config.dropout)
self.encoder_v = nn.Parameter(torch.zeros((nh, D, N)).normal_(std=0.02))
self.lm_head = nn.Parameter(
torch.zeros((D, config.vocab_size)).normal_(std=0.02)
)
self.apply(self._init_weights)
def _init_weights(self, module):
if isinstance(module, nn.Linear):
nn.init.normal_(module.weight, mean=0.0, std=0.02)
if module.bias is not None:
nn.init.zeros_(module.bias)
elif isinstance(module, nn.Embedding):
nn.init.normal_(module.weight, mean=0.0, std=0.02)
def forward(self, idx, targets=None):
C = self.config
B, T = idx.size()
D = C.n_embd
nh = C.n_head
N = D * C.mlp_internal_dim_multiplier // nh
x = self.embed(idx).unsqueeze(1)
# actually helps with training
x = self.ln(x) # B, 1, T, D
for level in range(C.n_layer):
x_latent = x @ self.encoder
x_sparse = F.relu(x_latent) # B, nh, T, N
yKV = self.attn(
Q=x_sparse,
K=x_sparse,
V=x,
)
yKV = self.ln(yKV)
y_latent = yKV @ self.encoder_v
y_sparse = F.relu(y_latent)
xy_sparse = x_sparse * y_sparse # B, nh, T, N
xy_sparse = self.drop(xy_sparse)
yMLP = (
xy_sparse.transpose(1, 2).reshape(B, 1, T, N * nh) @ self.decoder
) # B, 1, T, D
y = self.ln(yMLP)
x = self.ln(x + y)
logits = x.view(B, T, D) @ self.lm_head
loss = None
if targets is not None:
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
return logits, loss
@torch.no_grad()
def generate(
self,
idx: torch.Tensor,
max_new_tokens: int,
temperature: float = 1.0,
top_k: int | None = None,
) -> torch.Tensor:
for _ in range(max_new_tokens):
idx_cond = idx
logits, _ = self(idx_cond)
logits = logits[:, -1, :] / temperature
if top_k is not None:
values, _ = torch.topk(logits, min(top_k, logits.size(-1)))
logits[logits < values[:, [-1]]] = float("-inf")
probs = F.softmax(logits, dim=-1)
idx_next = torch.multinomial(probs, num_samples=1)
idx = torch.cat((idx, idx_next), dim=1)
return idx
```
## /figs/architecture.png
Binary file available at https://raw.githubusercontent.com/pathwaycom/bdh/refs/heads/main/figs/architecture.png
## /figs/bdh_scaling.png
Binary file available at https://raw.githubusercontent.com/pathwaycom/bdh/refs/heads/main/figs/bdh_scaling.png
## /figs/vocab.png
Binary file available at https://raw.githubusercontent.com/pathwaycom/bdh/refs/heads/main/figs/vocab.png
## /requirements.txt
torch
numpy
requests
## /train.py
```py path="/train.py"
# Copyright Pathway Technology, Inc.
import os
from contextlib import nullcontext
import bdh
import numpy as np
import requests
import torch
import torch.nn as nn
import torch.nn.functional as F
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# On a Mac you can also try
# device=torch.device('mps')
dtype = (
"bfloat16"
if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
else "float16"
) # 'float32', 'bfloat16', or 'float16', the latter will auto implement a GradScaler
ptdtype = {
"float32": torch.float32,
"bfloat16": torch.bfloat16,
"float16": torch.float16,
}[dtype]
ctx = (
torch.amp.autocast(device_type=device.type, dtype=ptdtype)
if "cuda" in device.type
else nullcontext()
)
scaler = torch.amp.GradScaler(device=device.type, enabled=(dtype == "float16"))
torch.manual_seed(1337)
torch.backends.cuda.matmul.allow_tf32 = True # allow tf32 on matmul
torch.backends.cudnn.allow_tf32 = True # allow tf32 on cudnn
print(f"Using device: {device} with dtype {dtype}")
# Configuration
BDH_CONFIG = bdh.BDHConfig()
BLOCK_SIZE = 512
BATCH_SIZE = 32
MAX_ITERS = 3000
LEARNING_RATE = 1e-3
WEIGHT_DECAY = 0.1
LOG_FREQ = 100
input_file_path = os.path.join(os.path.dirname(__file__), "input.txt")
# Fetch the tiny Shakespeare dataset
def fetch_data():
if not os.path.exists(input_file_path):
data_url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
with open(input_file_path, "w") as f:
f.write(requests.get(data_url).text)
def get_batch(split):
# treat the file as bytes
data = np.memmap(input_file_path, dtype=np.uint8, mode="r")
if split == "train":
data = data[: int(0.9 * len(data))]
else:
data = data[int(0.9 * len(data)) :]
ix = torch.randint(len(data) - BLOCK_SIZE, (BATCH_SIZE,))
x = torch.stack(
[torch.from_numpy((data[i : i + BLOCK_SIZE]).astype(np.int64)) for i in ix]
)
y = torch.stack(
[
torch.from_numpy((data[i + 1 : i + 1 + BLOCK_SIZE]).astype(np.int64))
for i in ix
]
)
if torch.cuda.is_available():
# pin arrays x,y, which allows us to move them to GPU asynchronously (non_blocking=True)
x, y = x.pin_memory().to(device, non_blocking=True), y.pin_memory().to(
device, non_blocking=True
)
else:
x, y = x.to(device), y.to(device)
return x, y
def eval(model):
model.eval()
if __name__ == "__main__":
fetch_data()
model = bdh.BDH(BDH_CONFIG).to(device)
model = torch.compile(model)
optimizer = torch.optim.AdamW(
model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY
)
x, y = get_batch("train")
loss_acc = 0
loss_steps = 0
for step in range(MAX_ITERS):
with ctx:
logits, loss = model(x, y)
x, y = get_batch("train")
loss_acc += loss
loss_steps += 1
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad()
if step % LOG_FREQ == 0:
print(f"Step: {step}/{MAX_ITERS} loss {loss_acc.item() / loss_steps:.3}")
loss_acc = 0
loss_steps = 0
print("Training done, now generating a sample ")
model.eval()
prompt = torch.tensor(
bytearray("To be or ", "utf-8"), dtype=torch.long, device=device
).unsqueeze(0)
ret = model.generate(prompt, max_new_tokens=100, top_k=3)
ret_decoded = bytes(ret.to(torch.uint8).to("cpu").squeeze(0)).decode(
errors="backslashreplace"
)
print(ret_decoded)
```
The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.