```
├── .gitattributes (omitted)
├── .gitignore
├── LICENSE (omitted)
├── README.md (1400 tokens)
├── example.py (200 tokens)
├── example_audio.py (300 tokens)
├── example_image.py (300 tokens)
├── pyproject.toml (200 tokens)
├── run_example.bash
├── samples/
├── 10k_sample.txt (8.2k tokens)
├── 2k_sample.txt (1800 tokens)
├── 45k_sample.txt (36.2k tokens)
├── 85k_sample.txt (66.8k tokens)
├── scripts/
├── full_tests.py (800 tokens)
├── gpt_oss_export.py (1000 tokens)
├── llama_export.py (300 tokens)
├── qwen3_next_export.py (600 tokens)
├── scripts.py (300 tokens)
├── test.py (800 tokens)
├── test_autoinference.py (300 tokens)
├── src/
├── ollm/
├── __init__.py
├── attention.py (3k tokens)
├── gds_loader.py (2.4k tokens)
├── gemma3.py (900 tokens)
├── gpt_oss.py (2.2k tokens)
├── gpt_oss_attention.py (1700 tokens)
├── inference.py (1400 tokens)
├── kvcache.py (800 tokens)
├── llama.py (1600 tokens)
├── qwen3_next.py (3.4k tokens)
├── utils.py (700 tokens)
├── voxtral.py (500 tokens)
```
## /.gitignore
```gitignore path="/.gitignore"
temp
kv_cache
gds_export
gds_export*
models
cufile.log
__pycache__
dist/
build/
*.egg-info/
src/ollm.egg-info
```
## /README.md
<!-- markdownlint-disable MD001 MD041 -->
<p align="center">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://ollm.s3.us-east-1.amazonaws.com/files/logo2.png">
<img alt="oLLM" src="https://ollm.s3.us-east-1.amazonaws.com/files/logo2.png" width=52%>
</picture>
</p>
<h3 align="center">
LLM Inference for Large-Context Offline Workloads
</h3>
oLLM is a lightweight Python library for large-context LLM inference, built on top of Huggingface Transformers and PyTorch. It enables running models like [gpt-oss-20B](https://huggingface.co/openai/gpt-oss-20b), [qwen3-next-80B](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) or [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) on 100k context using ~$200 consumer GPU with 8GB VRAM. No quantization is used—only fp16/bf16 precision.
<p dir="auto"><em><a href="https://github.com/Mega4alik/ollm/wiki/Releases">Latest updates</a> (1.0.3)</em> 🔥</p>
<ul dir="auto">
<li><code>AutoInference</code> with any Llama3 / gemma3 model + <a href="https://github.com/huggingface/peft">PEFT</a> adapter support</li>
<li><code>kvikio</code> and <code>flash-attn</code> are optional now, meaning no hardware restrictions beyond HF transformers</li>
<li>Multimodal <b>voxtral-small-24B</b> (audio+text) added. <a href="https://github.com/Mega4alik/ollm/blob/main/example_audio.py">[sample with audio]</a> </li>
<li>Multimodal <b>gemma3-12B</b> (image+text) added. <a href="https://github.com/Mega4alik/ollm/blob/main/example_image.py">[sample with image]</a> </li>
<li><b>qwen3-next-80B</b> (160GB model) added with <span style="color:blue">⚡️1tok/2s</span> throughput (our fastest model so far)</li>
<li>gpt-oss-20B flash-attention-like implementation added to reduce VRAM usage </li>
<li>gpt-oss-20B chunked MLP added to reduce VRAM usage </li>
</ul>
---
### 8GB Nvidia 3060 Ti Inference memory usage:
| Model | Weights | Context length | KV cache | Baseline VRAM (no offload) | oLLM GPU VRAM | oLLM Disk (SSD) |
| ------- | ------- | -------- | ------------- | ------------ | ---------------- | --------------- |
| [qwen3-next-80B](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) | 160 GB (bf16) | 50k | 20 GB | ~190 GB | ~7.5 GB | 180 GB |
| [gpt-oss-20B](https://huggingface.co/openai/gpt-oss-20b) | 13 GB (packed bf16) | 10k | 1.4 GB | ~40 GB | ~7.3GB | 15 GB |
| [gemma3-12B](https://huggingface.co/google/gemma-3-12b-it) | 25 GB (bf16) | 50k | 18.5 GB | ~45 GB | ~6.7 GB | 43 GB |
| [llama3-1B-chat](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | 2 GB (bf16) | 100k | 12.6 GB | ~16 GB | ~5 GB | 15 GB |
| [llama3-3B-chat](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) | 7 GB (bf16) | 100k | 34.1 GB | ~42 GB | ~5.3 GB | 42 GB |
| [llama3-8B-chat](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | 16 GB (bf16) | 100k | 52.4 GB | ~71 GB | ~6.6 GB | 69 GB |
<small>By "Baseline" we mean typical inference without any offloading</small>
How do we achieve this:
- Loading layer weights from SSD directly to GPU one by one
- Offloading KV cache to SSD and loading back directly to GPU, no quantization or PagedAttention
- Offloading layer weights to CPU if needed
- FlashAttention-2 with online softmax. Full attention matrix is never materialized.
- Chunked MLP. Intermediate upper projection layers may get large, so we chunk MLP as well
---
Typical use cases include:
- Analyze contracts, regulations, and compliance reports in one pass
- Summarize or extract insights from massive patient histories or medical literature
- Process very large log files or threat reports locally
- Analyze historical chats to extract the most common issues/questions users have
---
**Supported GPUs**: NVIDIA (with additional performance benefits from `kvikio` and `flash-attn`), AMD, and Apple Silicon (MacBook).
## Getting Started
It is recommended to create venv or conda environment first
```bash
python3 -m venv ollm_env
source ollm_env/bin/activate
```
Install oLLM with `pip install --no-build-isolation ollm` or [from source](https://github.com/Mega4alik/ollm):
```bash
git clone https://github.com/Mega4alik/ollm.git
cd ollm
pip install --no-build-isolation -e .
# for Nvidia GPUs with cuda (optional):
pip install kvikio-cu{cuda_version} Ex, kvikio-cu12 #speeds up the inference
```
> 💡 **Note**
> **voxtral-small-24B** requires additional pip dependencies to be installed as `pip install "mistral-common[audio]"` and `pip install librosa`
Check out the [Troubleshooting](https://github.com/Mega4alik/ollm/wiki/Troubleshooting) in case of any installation issues
## Example
Code snippet sample
```python
from ollm import Inference, file_get_contents, TextStreamer
o = Inference("llama3-1B-chat", device="cuda:0", logging=True) #llama3-1B/3B/8B-chat, gpt-oss-20B, qwen3-next-80B
o.ini_model(models_dir="./models/", force_download=False)
o.offload_layers_to_cpu(layers_num=2) #(optional) offload some layers to CPU for speed boost
past_key_values = o.DiskCache(cache_dir="./kv_cache/") #set None if context is small
text_streamer = TextStreamer(o.tokenizer, skip_prompt=True, skip_special_tokens=False)
messages = [{"role":"system", "content":"You are helpful AI assistant"}, {"role":"user", "content":"List planets"}]
input_ids = o.tokenizer.apply_chat_template(messages, reasoning_effort="minimal", tokenize=True, add_generation_prompt=True, return_tensors="pt").to(o.device)
outputs = o.model.generate(input_ids=input_ids, past_key_values=past_key_values, max_new_tokens=500, streamer=text_streamer).cpu()
answer = o.tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=False)
print(answer)
```
or run sample python script as `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python example.py`
```python
# with AutoInference, you can run any LLama3/gemma3 model with PEFT adapter support
# pip install peft
from ollm import AutoInference
o = AutoInference("./models/gemma3-12B", # any llama3 or gemma3 model
adapter_dir="./myadapter/checkpoint-20", # PEFT adapter checkpoint if available
device="cuda:0", multimodality=False, logging=True)
...
```
**More samples**
- [gemma3-12B image+text](https://github.com/Mega4alik/ollm/blob/main/example_image.py)
- [voxtral-small-24B audio+text](https://github.com/Mega4alik/ollm/blob/main/example_audio.py)
- [AutoInference + SFT](https://github.com/Mega4alik/peftee?tab=readme-ov-file#usage)
## Knowledge base
- [Documentation](https://github.com/Mega4alik/ollm/wiki/Documentation)
- [Community](https://github.com/Mega4alik/ollm/wiki/Community) articles, video, blogs
- [Troubleshooting](https://github.com/Mega4alik/ollm/wiki/Troubleshooting)
## Roadmap
*For visibility of what's coming next (subject to change)*
- Qwen3-Next quantized version
- Qwen3-VL or alternative vision model
- Qwen3-Next MultiTokenPrediction in R&D
## Contact us
If there’s a model you’d like to see supported, feel free to suggest it in the [discussion](https://github.com/Mega4alik/ollm/discussions/4) — I’ll do my best to make it happen.
## /example.py
```py path="/example.py"
from ollm import Inference, file_get_contents, TextStreamer
o = Inference("llama3-1B-chat", device="cuda:0", logging=True) #llama3-1B-chat(3B, 8B) | gpt-oss-20B | qwen3-next-80B
o.ini_model(models_dir="./models/", force_download=False)
o.offload_layers_to_cpu(layers_num=2) #offload some layers to CPU for speed increase
past_key_values = o.DiskCache(cache_dir="./kv_cache/")
text_streamer = TextStreamer(o.tokenizer, skip_prompt=True, skip_special_tokens=False)
sm, um = "You are helpful AI assistant", "List planets starting from Mercury"
#sm, um = file_get_contents("./samples/85k_sample.txt"), "What's common between these articles?"
messages = [{"role":"system", "content":sm}, {"role":"user", "content":um}]
input_ids = o.tokenizer.apply_chat_template(messages, reasoning_effort="minimal", tokenize=True, add_generation_prompt=True, return_tensors="pt").to(o.device)
outputs = o.model.generate(input_ids=input_ids, past_key_values=past_key_values, max_new_tokens=100, streamer=text_streamer).cpu()
answer = o.tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=False)
print(answer)
```
## /example_audio.py
```py path="/example_audio.py"
# voxtral-small-24B (mistralai/Voxtral-Small-24B-2507) ASR+Text example
# Make sure to INSTALL additional dependencies first!
# pip install --upgrade "mistral-common[audio]"
# pip install librosa
from ollm import Inference, file_get_contents, TextStreamer
o = Inference("voxtral-small-24B", device="cuda:0", logging=True, multimodality=True)
o.ini_model(models_dir="./models/", force_download=False)
#o.offload_layers_to_cpu(layers_num=2) #offload some layers to CPU for speed boost
past_key_values = None #o.DiskCache(cache_dir="./kv_cache/") #uncomment for large context
text_streamer = TextStreamer(o.tokenizer, skip_prompt=True, skip_special_tokens=False)
messages = [
{
"role": "user",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav",
},
{"type": "text", "text": "What can you tell me about this audio?"},
],
}
]
inputs = o.processor.apply_chat_template(messages, return_tensors="pt").to(o.device)
outputs = o.model.generate(**inputs, max_new_tokens=500, do_sample=False, past_key_values=None, use_cache=True, streamer=text_streamer).detach().cpu()
answer = o.processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=False)
print(answer)
```
## /example_image.py
```py path="/example_image.py"
# gemma3-12B Image+Text example
import torch
from ollm import Inference, file_get_contents, TextStreamer
o = Inference("gemma3-12B", device="cuda:0", logging=True, multimodality=True)
o.ini_model(models_dir="./models/", force_download=False)
o.offload_layers_to_cpu(layers_num=12) #offload some layers to CPU for speed boost
past_key_values = None #o.DiskCache(cache_dir="./kv_cache/") #uncomment for large context
text_streamer = TextStreamer(o.tokenizer, skip_prompt=True, skip_special_tokens=False)
messages = [
{"role": "system","content": [{"type": "text", "text": "You are a helpful assistant."}]},
{
"role": "user",
"content": [
{"type": "image", "image": "https://raw.githubusercontent.com/OpenBMB/MiniCPM-V/main/assets/minicpm-v-4dot5-framework.png"},
{"type": "text", "text": "Describe this image in detail."}
]
}
]
inputs = o.processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(o.device, dtype=torch.bfloat16)
outputs = o.model.generate(**inputs, past_key_values=past_key_values, max_new_tokens=500, streamer=text_streamer).cpu()
answer = o.tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=False)
print(answer)
```
## /pyproject.toml
```toml path="/pyproject.toml"
[build-system]
requires = ["setuptools>=61.0", "wheel", "psutil"]
build-backend = "setuptools.build_meta"
[project]
name = "ollm"
version = "1.0.3"
description = "LLM Inference for Large-Context Offline Workloads"
authors = [
{ name="Anuar Sharafudinov", email="anuarsh@ailabs.us" }
]
readme = "README.md"
license = { file="LICENSE" }
requires-python = ">=3.10"
dependencies = [
"numpy",
"torch>2.6.0", #tested: 2.8.0
"transformers>=4.57.0", #tested: 4.55.4, 4.56.1
"accelerate",
"flash-linear-attention",
"flash-attn" #tested: 2.7.4.post1
]
[project.urls]
"Homepage" = "https://github.com/Mega4alik/ollm"
"Bug Tracker" = "https://github.com/Mega4alik/ollm/issues"
[tool.setuptools]
package-dir = {"" = "src"}
[tool.setuptools.packages.find]
where = ["src"]
```
## /run_example.bash
```bash path="/run_example.bash"
#export PYTORCH_CUDA_CUBLASLT_DISABLE=1
#export CUBLAS_WORKSPACE_CONFIG=:4096:8
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
CUDA_VISIBLE_DEVICES=0 python example.py
```
## /samples/10k_sample.txt
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
Q: Can I upload documents?
A: Yes, we have the "Documents" feature that allows uploading documents. The AI will search through these documents to find an answer to the user's question.
Q: How many documents can I upload?
A: Up to 100 documents. The document size should not exceed 7000 tokens (~5000 words).
Q: How many chatbots can I make using 1 account
A: There's one chatbot per one webapi account. You may have multiple webapi accounts. Each webapi account has 1 chatbot and an unlimited number of agents and admins
Q: Can I see your roadmap?
A: Sure, here's our roadmap - https://s1.webapi.ai/article/102
Q: Can the chatbot initiate a dialogue?
A: No, it only responds when a user asks a question
Q: Can I change my chatbot name / icon?
A: Yes, you can change the name and icon of your chatbot. This can be done in the chatbot settings in webapi.ai platform.
Current model: GPT-4o
Supported channels (instructions available at "Channels" page): Telegram, Web, Whatsapp, Facebook Messenger, Instagram, Twilio SMS, Twilio Whatsapp, API for messaging
We have ready integrations with Shopify, Zendesk, Pabbly, Zapier, Calendly, and more. Check out the Pabbly integration video at https://vimeo.com/797265571
Documents articles can be imported from: Zendesk Knowledge base, Intercom knowledge base, Webapi KB.
Regarding partnerships, white-labeling, customization, etc. - learn about our partnership programs at https://s1.webapi.ai/article/105
# About the Lite version
The Lite version is the easiest to start with, yet very powerful.
It supports Domain Knowledge (Instructions, Knowledge base, FAQ) and Forms.
Forms is probably the most exciting part of the Lite version. You declare the user information that needs to be collected, like email, address, and phone number. Then add "show form [form name]" as an instruction to the Domain Knowledge. That's it, now the chatbot will start asking for the required user information based on your instruction. Once collected, it will be added into our built-in Spreadsheets, emailed to your email address, and sent via integrations (API/Pabbly/Zapier)
Check out the demo at https://www.webapi.ai/#video
<Example dialogue>
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: consulting services
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book a zoom call# try a demo on my own#
customer: try a demo on my own
AI: Ok! may I get your name and email, so I can setup your account?
customer: Sure, Arman Sharafudinov, email is mega4alik@gmail.com
AI-action: just-try# name=Arman Sharafudinov# email=mega4alik@gmail.com# business_type=consulting services
outputs: action_status=success
AI: Ok! We are already setting up your account. You will get an email with login details in 12 hours. Thanks!
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: talk to agent
AI: May I know your name and email address before I connect you with one of our agents?
customer: Sure, it is Choe Long, tester@webapi.ai
AI-action: connect-human-agent# name=Choe Long# email=tester@webapi.ai
outputs: action_status=success
AI: You have been switched to our agent. Usually we respond in up to 5 hours
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: IT technologies
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book zoom call# try a demo on my own#
customer: Book zoom call
AI: Great! Please book your slot at https://calendly.com/webapi
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Can you make a Twilio integration?
AI: I'm not sure. Would like to switch to a human agent? BUTTONS= talk to agent#
###
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
Q: Can I upload documents?
A: Yes, we have the "Documents" feature that allows uploading documents. The AI will search through these documents to find an answer to the user's question.
Q: How many documents can I upload?
A: Up to 100 documents. The document size should not exceed 7000 tokens (~5000 words).
Q: How many chatbots can I make using 1 account
A: There's one chatbot per one webapi account. You may have multiple webapi accounts. Each webapi account has 1 chatbot and an unlimited number of agents and admins
Q: Can I see your roadmap?
A: Sure, here's our roadmap - https://s1.webapi.ai/article/102
Q: Can the chatbot initiate a dialogue?
A: No, it only responds when a user asks a question
Q: Can I change my chatbot name / icon?
A: Yes, you can change the name and icon of your chatbot. This can be done in the chatbot settings in webapi.ai platform.
Current model: GPT-4o
Supported channels (instructions available at "Channels" page): Telegram, Web, Whatsapp, Facebook Messenger, Instagram, Twilio SMS, Twilio Whatsapp, API for messaging
We have ready integrations with Shopify, Zendesk, Pabbly, Zapier, Calendly, and more. Check out the Pabbly integration video at https://vimeo.com/797265571
Documents articles can be imported from: Zendesk Knowledge base, Intercom knowledge base, Webapi KB.
Regarding partnerships, white-labeling, customization, etc. - learn about our partnership programs at https://s1.webapi.ai/article/105
# About the Lite version
The Lite version is the easiest to start with, yet very powerful.
It supports Domain Knowledge (Instructions, Knowledge base, FAQ) and Forms.
Forms is probably the most exciting part of the Lite version. You declare the user information that needs to be collected, like email, address, and phone number. Then add "show form [form name]" as an instruction to the Domain Knowledge. That's it, now the chatbot will start asking for the required user information based on your instruction. Once collected, it will be added into our built-in Spreadsheets, emailed to your email address, and sent via integrations (API/Pabbly/Zapier)
Check out the demo at https://www.webapi.ai/#video
<Example dialogue>
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: consulting services
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book a zoom call# try a demo on my own#
customer: try a demo on my own
AI: Ok! may I get your name and email, so I can setup your account?
customer: Sure, Arman Sharafudinov, email is mega4alik@gmail.com
AI-action: just-try# name=Arman Sharafudinov# email=mega4alik@gmail.com# business_type=consulting services
outputs: action_status=success
AI: Ok! We are already setting up your account. You will get an email with login details in 12 hours. Thanks!
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: talk to agent
AI: May I know your name and email address before I connect you with one of our agents?
customer: Sure, it is Choe Long, tester@webapi.ai
AI-action: connect-human-agent# name=Choe Long# email=tester@webapi.ai
outputs: action_status=success
AI: You have been switched to our agent. Usually we respond in up to 5 hours
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: IT technologies
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book zoom call# try a demo on my own#
customer: Book zoom call
AI: Great! Please book your slot at https://calendly.com/webapi
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Can you make a Twilio integration?
AI: I'm not sure. Would like to switch to a human agent? BUTTONS= talk to agent#
###
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
Q: Can I upload documents?
A: Yes, we have the "Documents" feature that allows uploading documents. The AI will search through these documents to find an answer to the user's question.
Q: How many documents can I upload?
A: Up to 100 documents. The document size should not exceed 7000 tokens (~5000 words).
Q: How many chatbots can I make using 1 account
A: There's one chatbot per one webapi account. You may have multiple webapi accounts. Each webapi account has 1 chatbot and an unlimited number of agents and admins
Q: Can I see your roadmap?
A: Sure, here's our roadmap - https://s1.webapi.ai/article/102
Q: Can the chatbot initiate a dialogue?
A: No, it only responds when a user asks a question
Q: Can I change my chatbot name / icon?
A: Yes, you can change the name and icon of your chatbot. This can be done in the chatbot settings in webapi.ai platform.
Current model: GPT-4o
Supported channels (instructions available at "Channels" page): Telegram, Web, Whatsapp, Facebook Messenger, Instagram, Twilio SMS, Twilio Whatsapp, API for messaging
We have ready integrations with Shopify, Zendesk, Pabbly, Zapier, Calendly, and more. Check out the Pabbly integration video at https://vimeo.com/797265571
Documents articles can be imported from: Zendesk Knowledge base, Intercom knowledge base, Webapi KB.
Regarding partnerships, white-labeling, customization, etc. - learn about our partnership programs at https://s1.webapi.ai/article/105
# About the Lite version
The Lite version is the easiest to start with, yet very powerful.
It supports Domain Knowledge (Instructions, Knowledge base, FAQ) and Forms.
Forms is probably the most exciting part of the Lite version. You declare the user information that needs to be collected, like email, address, and phone number. Then add "show form [form name]" as an instruction to the Domain Knowledge. That's it, now the chatbot will start asking for the required user information based on your instruction. Once collected, it will be added into our built-in Spreadsheets, emailed to your email address, and sent via integrations (API/Pabbly/Zapier)
Check out the demo at https://www.webapi.ai/#video
<Example dialogue>
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: consulting services
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book a zoom call# try a demo on my own#
customer: try a demo on my own
AI: Ok! may I get your name and email, so I can setup your account?
customer: Sure, Arman Sharafudinov, email is mega4alik@gmail.com
AI-action: just-try# name=Arman Sharafudinov# email=mega4alik@gmail.com# business_type=consulting services
outputs: action_status=success
AI: Ok! We are already setting up your account. You will get an email with login details in 12 hours. Thanks!
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: talk to agent
AI: May I know your name and email address before I connect you with one of our agents?
customer: Sure, it is Choe Long, tester@webapi.ai
AI-action: connect-human-agent# name=Choe Long# email=tester@webapi.ai
outputs: action_status=success
AI: You have been switched to our agent. Usually we respond in up to 5 hours
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: IT technologies
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book zoom call# try a demo on my own#
customer: Book zoom call
AI: Great! Please book your slot at https://calendly.com/webapi
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Can you make a Twilio integration?
AI: I'm not sure. Would like to switch to a human agent? BUTTONS= talk to agent#
###
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
Q: Can I upload documents?
A: Yes, we have the "Documents" feature that allows uploading documents. The AI will search through these documents to find an answer to the user's question.
Q: How many documents can I upload?
A: Up to 100 documents. The document size should not exceed 7000 tokens (~5000 words).
Q: How many chatbots can I make using 1 account
A: There's one chatbot per one webapi account. You may have multiple webapi accounts. Each webapi account has 1 chatbot and an unlimited number of agents and admins
Q: Can I see your roadmap?
A: Sure, here's our roadmap - https://s1.webapi.ai/article/102
Q: Can the chatbot initiate a dialogue?
A: No, it only responds when a user asks a question
Q: Can I change my chatbot name / icon?
A: Yes, you can change the name and icon of your chatbot. This can be done in the chatbot settings in webapi.ai platform.
Current model: GPT-4o
Supported channels (instructions available at "Channels" page): Telegram, Web, Whatsapp, Facebook Messenger, Instagram, Twilio SMS, Twilio Whatsapp, API for messaging
We have ready integrations with Shopify, Zendesk, Pabbly, Zapier, Calendly, and more. Check out the Pabbly integration video at https://vimeo.com/797265571
Documents articles can be imported from: Zendesk Knowledge base, Intercom knowledge base, Webapi KB.
Regarding partnerships, white-labeling, customization, etc. - learn about our partnership programs at https://s1.webapi.ai/article/105
# About the Lite version
The Lite version is the easiest to start with, yet very powerful.
It supports Domain Knowledge (Instructions, Knowledge base, FAQ) and Forms.
Forms is probably the most exciting part of the Lite version. You declare the user information that needs to be collected, like email, address, and phone number. Then add "show form [form name]" as an instruction to the Domain Knowledge. That's it, now the chatbot will start asking for the required user information based on your instruction. Once collected, it will be added into our built-in Spreadsheets, emailed to your email address, and sent via integrations (API/Pabbly/Zapier)
Check out the demo at https://www.webapi.ai/#video
<Example dialogue>
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: consulting services
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book a zoom call# try a demo on my own#
customer: try a demo on my own
AI: Ok! may I get your name and email, so I can setup your account?
customer: Sure, Arman Sharafudinov, email is mega4alik@gmail.com
AI-action: just-try# name=Arman Sharafudinov# email=mega4alik@gmail.com# business_type=consulting services
outputs: action_status=success
AI: Ok! We are already setting up your account. You will get an email with login details in 12 hours. Thanks!
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: talk to agent
AI: May I know your name and email address before I connect you with one of our agents?
customer: Sure, it is Choe Long, tester@webapi.ai
AI-action: connect-human-agent# name=Choe Long# email=tester@webapi.ai
outputs: action_status=success
AI: You have been switched to our agent. Usually we respond in up to 5 hours
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: IT technologies
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book zoom call# try a demo on my own#
customer: Book zoom call
AI: Great! Please book your slot at https://calendly.com/webapi
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Can you make a Twilio integration?
AI: I'm not sure. Would like to switch to a human agent? BUTTONS= talk to agent#
###
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
.....
## /samples/2k_sample.txt
Intro:
- If user wants to connect with an agent or manager, suggest switching to a human agent
- Make sure that user provided a valid email and name before calling the "just-try" action
- Feel free to speak other languages
- If an answer is not given below, say that you don't know it
=== FAQ ===
Q: How do I integrate with external services or APIs?
1) First of all, make sure you have an API that receives GET/POST parameters and responds in JSON format.
2) Secondly, go to "Slots and Actions" page to configure your input/output slots (parameters), and actions.
- Input slots are the variables sent to your API. Its values are collected during the conversation.
- Output slots are variables collected from your API response. Its values are used to generate an answer.
- Each action requires input/output slots that it's going to use, as well as the API parameters (Endpoint URL, Header, etc)
3) Then go to "Dialogue Scenarios" page to give examples of how the action that you just configured should be called.
The following article should also be helpful - https://s1.webapi.ai/article/104
Q: Do you support other languages?
A: As a language model, it has been trained on large amounts of text data from various languages and can understand and respond in many different languages including, but not limited to English, Spanish, French, German, Italian, Chinese, Japanese, Korean, Arabic, Russian, and Portuguese. However, the level of proficiency may vary depending on the language and the amount of training data available for it.
Q: Who is your creator?
A: AILabs Technologies, Inc
Q: How to do Fine-tuning?
A: Fine-tuning is a complex process that requires 100+ (preferably) scenarios and engagement from our side. In most cases, fine-tuning is not needed. Please go ahead and book a quick call with our team member https://calendly.com/webapi so we can understand your needs and give our best recommendations
Q: I haven't received an email after registration
A: We activate new accounts manually, so it may take up to 12 hours. You will get your credentials soon
Q: The generated bot will be the same as chatGPT and Chatsonic? In that case, do I need an OpenAI API? Or is just a service chatbot that can aswer just product related questions like any other chatbot from any website?
A: You don't need the OpenAI API key to use our platform. The platform already has one. Yes, it is mainly built to help companies automate their customer support by answering product-related questions and performing actions (retrieving information from API, etc). The difference from other platforms is that it understands the context and is fast to get started. But since it's GPT4-based, you may expect it to show expert knowledge in many areas as ChatGPT does. I suggest trying it for free by giving instructions on how it should act and providing sample dialogue.
Q: How do I connect human agents?
A: There's a built-in action called "connect-human-agent" that transfers the user to a human agent. The "Lead Qualifier" template has this capability. Detailed instructions are available at https://s1.webapi.ai/article/91
Q: Is the chatbot hosted on a domain of our choosing?
A: No, all chatbots are hosted on our webapi.ai subdomains
Q: What is the cost?
A: We charge based on service usage of $0.99-$4 per 100 bot responses. All new users are granted $5 or about 170 bot responses for free.
Basically, usage is counted in the number of responses or actions that an AI chatbot generates. Learn detailed pricing at https://www.webapi.ai/files/pricing.pdf
Q: Do you have a youtube video on this
A: Videos with different bot development examples can be found at https://www.webapi.ai/#samples
Q: Are there user roles?
A: Yes, the roles are: agent, admin, superadmin
Q: How do you sign in/sign up?
A: Go to https://accounts.webapi.ai/ to create a new account or sign in to your existing account
Q: Can you integrate into whmcs?
A: The are hundreds of ready integrations on Zapier and Pabbly. Learn more at https://s1.webapi.ai/article/104
Q: Will the chatbot search the internet for the answer?
A: No, it will only use information from Domain Knowledge and Dialogue Scenarios
Q: Where are your servers located physically?
A: We use AWS servers located in the US
Q: Are you GDPR compliant?
A: Yes, check our GDPR compliance article at https://s1.webapi.ai/article/128
Q: What happens if OpenAI GPT-3/4 goes offline?
A: In that case, we will be moving users to human agents
Q: Can I upload documents?
A: Yes, we have the "Documents" feature that allows uploading documents. The AI will search through these documents to find an answer to the user's question.
Q: How many documents can I upload?
A: Up to 100 documents. The document size should not exceed 7000 tokens (~5000 words).
Q: How many chatbots can I make using 1 account
A: There's one chatbot per one webapi account. You may have multiple webapi accounts. Each webapi account has 1 chatbot and an unlimited number of agents and admins
Q: Can I see your roadmap?
A: Sure, here's our roadmap - https://s1.webapi.ai/article/102
Q: Can the chatbot initiate a dialogue?
A: No, it only responds when a user asks a question
Q: Can I change my chatbot name / icon?
A: Yes, you can change the name and icon of your chatbot. This can be done in the chatbot settings in webapi.ai platform.
Current model: GPT-4o
Supported channels (instructions available at "Channels" page): Telegram, Web, Whatsapp, Facebook Messenger, Instagram, Twilio SMS, Twilio Whatsapp, API for messaging
We have ready integrations with Shopify, Zendesk, Pabbly, Zapier, Calendly, and more. Check out the Pabbly integration video at https://vimeo.com/797265571
Documents articles can be imported from: Zendesk Knowledge base, Intercom knowledge base, Webapi KB.
Regarding partnerships, white-labeling, customization, etc. - learn about our partnership programs at https://s1.webapi.ai/article/105
# About the Lite version
The Lite version is the easiest to start with, yet very powerful.
It supports Domain Knowledge (Instructions, Knowledge base, FAQ) and Forms.
Forms is probably the most exciting part of the Lite version. You declare the user information that needs to be collected, like email, address, and phone number. Then add "show form [form name]" as an instruction to the Domain Knowledge. That's it, now the chatbot will start asking for the required user information based on your instruction. Once collected, it will be added into our built-in Spreadsheets, emailed to your email address, and sent via integrations (API/Pabbly/Zapier)
Check out the demo at https://www.webapi.ai/#video
<Example dialogue>
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: consulting services
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book a zoom call# try a demo on my own#
customer: try a demo on my own
AI: Ok! may I get your name and email, so I can setup your account?
customer: Sure, Arman Sharafudinov, email is mega4alik@gmail.com
AI-action: just-try# name=Arman Sharafudinov# email=mega4alik@gmail.com# business_type=consulting services
outputs: action_status=success
AI: Ok! We are already setting up your account. You will get an email with login details in 12 hours. Thanks!
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: talk to agent
AI: May I know your name and email address before I connect you with one of our agents?
customer: Sure, it is Choe Long, tester@webapi.ai
AI-action: connect-human-agent# name=Choe Long# email=tester@webapi.ai
outputs: action_status=success
AI: You have been switched to our agent. Usually we respond in up to 5 hours
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Yes, try webapi.ai for free
AI: Awesome, may I know what business are you in? Ex: educational services
customer: IT technologies
AI: Excellent, are you open for a 15-minute demo with our team member? BUTTONS= Book zoom call# try a demo on my own#
customer: Book zoom call
AI: Great! Please book your slot at https://calendly.com/webapi
###
AI: Hi, I’m a smart chatbot. I can understand the context and respond in various ways. Thank you for your interest. Would you like to automate your conversations with your leads? BUTTONS= Yes, try webapi.ai for free# ask a question# talk to agent#
customer: Can you make a Twilio integration?
AI: I'm not sure. Would like to switch to a human agent? BUTTONS= talk to agent#
###
## /samples/45k_sample.txt
Article
Discovering faster matrix multiplication
algorithms with reinforcement learning
https://doi.org/10.1038/s41586-022-05172-4
Received: 2 October 2021
Accepted: 2 August 2022
Alhussein Fawzi1,2 ✉, Matej Balog1,2, Aja Huang1,2, Thomas Hubert1,2,
Bernardino Romera-Paredes1,2, Mohammadamin Barekatain1, Alexander Novikov1,
Francisco J. R. Ruiz1, Julian Schrittwieser1, Grzegorz Swirszcz1, David Silver1, Demis Hassabis1
& Pushmeet Kohli1
Published online: 5 October 2022
Open access
Check for updates
Improving the efficiency of algorithms for fundamental computations can have a
widespread impact, as it can affect the overall speed of a large amount of computations.
Matrix multiplication is one such primitive task, occurring in many systems—from
neural networks to scientific computing routines. The automatic discovery of
algorithms using machine learning offers the prospect of reaching beyond human
intuition and outperforming the current best human-designed algorithms. However,
automating the algorithm discovery procedure is intricate, as the space of possible
algorithms is enormous. Here we report a deep reinforcement learning approach
based on AlphaZero1 for discovering efficient and provably correct algorithms for the
multiplication of arbitrary matrices. Our agent, AlphaTensor, is trained to play a
single-player game where the objective is finding tensor decompositions within a
finite factor space. AlphaTensor discovered algorithms that outperform the state-
of-the-art complexity for many matrix sizes. Particularly relevant is the case of 4 × 4
matrices in a finite field, where AlphaTensor’s algorithm improves on Strassen’s two-
level algorithm for the first time, to our knowledge, since its discovery 50 years ago2.
We further showcase the flexibility of AlphaTensor through different use-cases:
algorithms with state-of-the-art complexity for structured matrix multiplication and
improved practical efficiency by optimizing matrix multiplication for runtime on
specific hardware. Our results highlight AlphaTensor’s ability to accelerate the
process of algorithmic discovery on a range of problems, and to optimize for different
criteria.
We focus on the fundamental task of matrix multiplication, and use
deep reinforcement learning (DRL) to search for provably correct and
efficient matrix multiplication algorithms. This algorithm discovery
process is particularly amenable to automation because a rich space of
matrix multiplication algorithms can be formalized as low-rank decom-
positions of a specific three-dimensional (3D) tensor2, called the matrix
multiplication tensor3–7. This space of algorithms contains the stand-
ard matrix multiplication algorithm and recursive algorithms such as
Strassen’s2, as well as the (unknown) asymptotically optimal algorithm.
Although an important body of work aims at characterizing the com-
plexity of the asymptotically optimal algorithm8–12, this does not yield
practical algorithms5. We focus here on practical matrix multiplication
algorithms, which correspond to explicit low-rank decompositions of
the matrix multiplication tensor. In contrast to two-dimensional matri-
ces, for which efficient polynomial-time algorithms computing the rank
have existed for over two centuries13, finding low-rank decompositions
of 3D tensors (and beyond) is NP-hard14 and is also hard in practice.
In fact, the search space is so large that even the optimal algorithm
for multiplying two 3 × 3 matrices is still unknown. Nevertheless, in a
longstanding research effort, matrix multiplication algorithms have
been discovered by attacking this tensor decomposition problem using
human search2,15,16, continuous optimization17–19 and combinatorial
search20. These approaches often rely on human-designed heuristics,
which are probably suboptimal. We instead use DRL to learn to recog-
nize and generalize over patterns in tensors, and use the learned agent
to predict efficient decompositions.
We formulate the matrix multiplication algorithm discovery pro-
cedure (that is, the tensor decomposition problem) as a single-player
game, called TensorGame. At each step of TensorGame, the player
selects how to combine different entries of the matrices to multiply.
A score is assigned based on the number of selected operations required
to reach the correct multiplication result. This is a challenging game
with an enormous action space (more than 1012 actions for most inter-
esting cases) that is much larger than that of traditional board games
such as chess and Go (hundreds of actions). To solve TensorGame and
find efficient matrix multiplication algorithms, we develop a DRL agent,
AlphaTensor. AlphaTensor is built on AlphaZero1,21, where a neural net-
work is trained to guide a planning procedure searching for efficient
matrix multiplication algorithms. Our framework uses a single agent
to decompose matrix multiplication tensors of various sizes, yielding
DeepMind, London, UK. 2These authors contributed equally: Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert and Bernardino Romera-Paredes. ✉e-mail: afawzi@deepmind.com
1
Nature | Vol 610 | 6 October 2022 | 47Article
a
b
c
m1 = ( a1 + a4 )( b1 + b4 )
c1c2
c3c4
=
a1a2
a3a4
·
b1b2m2 = ( a3 + a4 ) b1
b3b4m3 = a1 ( b2 – b4 )
c1
b1
b2
c2
c3
b3
c4
b4
a2
a3
a4
10101–1000010
0
1
0100010
m4 = a4 ( b3 – b1 )110100–1
m5 = ( a1 + a2 ) b4110010
001–1
0010
0001001
m7 = ( a2 – a4 )( b3 + b4 )10–10101
c1 = m1 + m4 – m5 + m7100101
0010–1
100
01
–10
11
0000
010
m6 = ( a3 – a1 )( b1 + b2 )
c2 = m3 + m5
a1
U =
c3 = m2 + m4
V =
W =
1
c4 = m1 – m2 + m3 + m6
Fig. 1 | Matrix multiplication tensor and algorithms. a, Tensor T 2 representing
the multiplication of two 2 × 2 matrices. Tensor entries equal to 1 are depicted
in purple, and 0 entries are semi-transparent. The tensor specifies which entries
from the input matrices to read, and where to write the result. For example,
as c1 = a1 b1 + a2 b3, tensor entries located at (a1, b1, c1) and (a2, b3, c1) are set to 1.b, Strassen's algorithm2 for multiplying 2 × 2 matrices using 7 multiplications.
c, Strassen's algorithm in tensor factor representation. The stacked factors
U, V and W (green, purple and yellow, respectively) provide a rank-7
decomposition of T 2 (equation (1)). The correspondence between arithmetic
operations (b) and factors (c) is shown by using the aforementioned colours.
transfer of learned decomposition techniques across various tensors.
To address the challenging nature of the game, AlphaTensor uses a
specialized neural network architecture, exploits symmetries of the
problem and makes use of synthetic training games.
AlphaTensor scales to a substantially larger algorithm space than
what is within reach for either human or combinatorial search. In fact,
AlphaTensor discovers from scratch many provably correct matrix
multiplication algorithms that improve over existing algorithms in
terms of number of scalar multiplications. We also adapt the algo-
rithm discovery procedure to finite fields, and improve over Strassen’s
two-level algorithm for multiplying 4 × 4 matrices for the first time, to
our knowledge, since its inception in 1969. AlphaTensor also discovers a
diverse set of algorithms—up to thousands for each size—showing that
the space of matrix multiplication algorithms is richer than previously
thought. We also exploit the diversity of discovered factorizations to
improve state-of-the-art results for large matrix multiplication sizes.
Through different use-cases, we highlight AlphaTensor’s flexibility
and wide applicability: AlphaTensor discovers efficient algorithms
for structured matrix multiplication improving over known results,
and finds efficient matrix multiplication algorithms tailored to spe-
cific hardware, by optimizing for actual runtime. These algorithms
multiply large matrices faster than human-designed algorithms on
the same hardware.A decomposition of T n into R rank-one terms provides an algorithm
for multiplying arbitrary n × n matrices using R scalar multiplications
(see Algorithm 1). We refer to Fig. 1b,c for an example algorithm mul-
tiplying 2 × 2 matrices with R = 7 (Strassen’s algorithm).
Crucially, Algorithm 1 can be used to multiply block matrices. By using
this algorithm recursively, one can multiply matrices of arbitrary size, with
the rank R controlling the asymptotic complexity of the algorithm. In par-
ticular, N × N matrices can be multiplied with asymptotic complexity
O(Nlogn (R)); see ref. 5 for more details.
Algorithms as tensor decomposition
As matrix multiplication (A, B) ↦ AB is bilinear (that is, linear in both
arguments), it can be fully represented by a 3D tensor: see Fig. 1a for
how to represent the 2 × 2 matrix multiplication operation as a 3D ten-
sor of size 4 × 4 × 4, and refs. 3,5,7 for more details. We write T n for the
tensor describing n × n matrix multiplication. The tensor T n is fixed
(that is, it is independent of the matrices to be multiplied), has entries
in {0, 1}, and is of size n2 × n2 × n2. More generally, we use T n, m, p to
describe the rectangular matrix multiplication operation of size n × m
with m × p (note that T n = T n, n, n ). By a decomposition of T n into R
rank-one terms, we mean
R
T n = ∑ u(r ) ⊗ v (r ) ⊗ w (r ),
(1)
r =1
where ⊗ denotes the outer (tensor) product, and u(r), v(r) and w(r) are all
vectors. If a tensor T can be decomposed into R rank-one terms, we say
the rank of T is at most R, or Rank (T ) ≤ R. This is a natural extension
R
from the matrix rank, where a matrix is decomposed into ∑r =1 u(r ) ⊗ v (r ).
48 | Nature | Vol 610 | 6 October 2022
DRL for algorithm discovery
We cast the problem of finding efficient matrix multiplication algo-
rithms as a reinforcement learning problem, modelling the environ-
ment as a single-player game, TensorGame. The game state after step
t is described by a tensor S t , which is initially set to the target tensor
we wish to decompose: S 0 = T n. In each step t of the game, the player
selects a triplet (u(t), v(t), w(t)), and the tensor S t is updated by subtract-
ing the resulting rank-one tensor: S t ← S t −1 − u(t ) ⊗ v (t ) ⊗ w (t ). The goal
of the player is to reach the zero tensor S t = 0 by applying the smallest
number of moves. When the player reaches the zero tensor, the
R
sequence of selected factors satisfies T n = ∑t =1 u(t ) ⊗ v (t ) ⊗ w (t ) (where
R denotes the number of moves), which guarantees the correctness of
the resulting matrix multiplication algorithm. To avoid playing unnec-
essarily long games, we limit the number of steps to a maximum value,
Rlimit.
For every step taken, we provide a reward of −1 to encourage finding
the shortest path to the zero tensor. If the game terminates with a
non-zero tensor (after Rlimit steps), the agent receives an additional
terminal reward equal to − γ(S R limit), where γ(S R limit) is an upper bound
on the rank of the terminal tensor. Although this reward optimizes for
rank (and hence for the complexity of the resulting algorithm), other
reward schemes can be used to optimize other properties, such as
practical runtime (see ‘Algorithm discovery results’). Besides, as our
aim is to find exact matrix multiplication algorithms, we constrain
{u(t), v(t), w(t)} to have entries in a user-specified discrete set of coeffi-
cients F (for example, F = {−2, −1, 0, 1, 2}). Such discretization is com-
mon practice to avoid issues with the finite precision of floating
points15,18,20.
To play TensorGame, we propose AlphaTensor (Fig. 2), an agent based
on AlphaZero1, which achieved tabula rasa superhuman performance
in the classical board games of Go, chess and shogi, and on its extension
to handle large action spaces Sampled AlphaZero21. Similarly to
AlphaZero, AlphaTensor uses a deep neural network to guide a Monteimprove the overall performance over a plain AlphaZero agent (see
Methods and Supplementary Information for details).
Algorithm 1
R
A meta-algorithm parameterized by {u(r ), v (r ), w (r )}r = 1 for computing
the matrix product C = AB. It is noted that R controls the number of
multiplications between input matrix entries.
R
Parameters: {u(r ), v (r ), w (r )}r = 1: length-n2 vectors such that
R
Tn = ∑r = 1 u(r ) ⊗ v (r ) ⊗ w (r )
Input: A, B: matrices of size n × n
Output: C = AB
(1) for r = 1, …, R do
(2) mr ← (u(1r )a1 + + u(nr2)an 2) (v (1r )b1 + + v (nr2)bn 2)
(3) for i = 1, …, n2 do
(R )
(4) ci ← w (1)
i m1 + + w i mR
return C
Carlo tree search (MCTS) planning procedure. The network takes as
input a state (that is, a tensor S t to decompose), and outputs a policy
and a value. The policy provides a distribution over potential actions.
As the set of potential actions (u(t), v(t), w(t)) in each step is enormous,
we rely on sampling actions rather than enumerating them21,22. The
value provides an estimate of the distribution z of returns (cumulative
reward) starting from the current state S t . With the above reward
scheme, the distribution z models the agent’s belief about the rank of
the tensor S t. To play a game, AlphaTensor starts from the target tensor
(T n) and uses the MCTS planner at each step to choose the next action.
Finished games are used as feedback to the network to improve the
network parameters.
Overcoming the challenges posed by TensorGame—namely, an enor-
mous action space, and game states described by large 3D tensors
representing an abstract mathematical operation—requires multiple
advances. All these components, described briefly below, substantially
Neural network architecture
We propose a transformer-based23 architecture that incorporates
inductive biases for tensor inputs. We first project the S × S × S input
tensor into three S × S grids of feature vectors by using linear layers
applied to the three cyclic transpositions of the tensor. The main part of
the model comprises a sequence of attention operations, each applied
to a set of features belonging to a pair of grids (Extended Data Figs. 3
and 4). This generalizes axial attention24 to multiple grids, and is both
more efficient and yields better results than naive self-attention. The
proposed architecture, which disregards the order of rows and columns
in the grids, is inspired by the invariance of the tensor rank to slice
reordering. The final feature representation of the three matrices is
passed both to the policy head (an autoregressive model) and the value
head (a multilayer perceptron).
Synthetic demonstrations
Although tensor decomposition is NP-hard, the inverse task of con-
structing the tensor from its rank-one factors is elementary. Hence,
we generate a large dataset of tensor-factorization pairs (synthetic
R
demonstrations) by first sampling factors {(u(r ), v (r ), w (r ))}r =1 at random,
R
(r )
(r )
(r )
and then constructing the tensor D = ∑r =1 u ⊗ v ⊗ w . We train the
network on a mixture of supervised loss (that is, to imitate synthetic
demonstrations) and standard reinforcement learning loss (that is,
learning to decompose a target tensor T n) (Fig. 2). This mixed training
strategy—training on the target tensor and random tensors— substan-
tially outperforms each training strategy separately. This is despite
randomly generated tensors having different properties from the tar-
get tensors.
Change of basis
T n (Fig. 1a) is the tensor representing the matrix multiplication bilinear
operation in the canonical basis. The same bilinear operation can be
expressed in other bases, resulting in other tensors. These different
Acting
Change of basis
...
Played
game
(u(1), v(1), w(1))
Updated
model
(u(2), v(2), w(2))
(u(3), v(3), w(3))
Learning
Policy head
Played games
buffer
(u, v, w)
Sample
random state
Value head
Training labels
Neural network
Fig. 2 | Overview of AlphaTensor. The neural network (bottom box) takes
as input a tensor S t , and outputs samples (u, v, w) from a distribution
over potential next actions to play, and an estimate of the future returns
(for example, of −Rank (S t )). The network is trained on two data sources:
Network input
Pre-generated
synthetic
demonstrations
previously played games and synthetic demonstrations. The updated network
is sent to the actors (top box), where it is used by the MCTS planner to generate
new games.
Nature | Vol 610 | 6 October 2022 | 49Article
Best method
known
(2, 2, 2)
(3, 3, 3)(Strassen, 1969)2
(Laderman, 1976)15
(Strassen, 1969)2
(2, 2, 2) ^ (2, 2, 2)
(3, 5, 5) + (2, 5, 5)
7
237
237
23
494749
989698
(2, 2, 3)
11
(2, 2, 2) + (2, 2, 1)
(2, 2, 4)
14
(2, 2, 2) + (2, 2, 2)
(2, 2, 5)
18
(2, 2, 2) + (2, 2, 3)
(2, 3, 3) (Hopcroft and Kerr, 1971)16 15
(2, 3, 4) (Hopcroft and Kerr, 1971)16 20
(2, 3, 5) (Hopcroft and Kerr, 1971)16 25
(2, 4, 4) (Hopcroft and Kerr, 1971)16 26
(2, 4, 5) (Hopcroft and Kerr, 1971)16 33
(2, 5, 5) (Hopcroft and Kerr, 1971)16 40
(3, 3, 4)
(Smirnov, 2013)18
29
(3, 3, 5)
36
(Smirnov, 2013)18
(Smirnov, 2013)18
(3, 4, 4)
38
(Smirnov, 2013)18
(3, 4, 5)
48
(3, 5, 5) (Sedoglavic and Smirnov, 2021)19 58
(4, 4, 5)
64
(4, 4, 2) + (4, 4, 3)
(4, 5, 5)
80
(2, 5, 5) ^ (2, 1, 1)11
14
18
15
20
25
26
33
40
29
36
38
47
58
63
7611
14
18
15
20
25
26
33
40
29
36
38
47
58
63
76
(4, 4, 4)
(5, 5, 5)
(11, 12, 12)
Best rank AlphaTensor rank
known Modular Standard
(9, 11, 11)
30
Improvement in rank
Size
(n, m, p)25(9, 9, 11)
20(10, 11, 12)
(9, 10, 10)
15
(11, 11, 11)
(9, 9, 9)
10
(10, 12, 12)
(10, 10, 10)
5
0
200
400 600 800
Best rank known
1,000
Fig. 3 | Comparison between the complexity of previously known matrix
multiplication algorithms and the ones discovered by AlphaTensor. Left:
column (n, m, p) refers to the problem of multiplying n × m with m × p matrices.
The complexity is measured by the number of scalar multiplications (or
equivalently, the number of terms in the decomposition of the tensor). ‘Best
rank known’ refers to the best known upper bound on the tensor rank (before
this paper), whereas ‘AlphaTensor rank’ reports the rank upper bounds
obtained with our method, in modular arithmetic (Z 2) and standard arithmetic.In all cases, AlphaTensor discovers algorithms that match or improve over
known state of the art (improvements are shown in red). See Extended Data
Figs. 1 and 2 for examples of algorithms found with AlphaTensor. Right: results
(for arithmetic in R) of applying AlphaTensor-discovered algorithms on larger
tensors. Each red dot represents a tensor size, with a subset of them labelled.
See Extended Data Table 1 for the results in table form. State-of-the-art results
are obtained from the list in ref. 64.
tensors are equivalent: they have the same rank, and decompositions
obtained in a custom basis can be mapped to the canonical basis, hence
obtaining a practical algorithm of the form in Algorithm 1. We leverage
this observation by sampling a random change of basis at the beginning
of every game, applying it to T n, and letting AlphaTensor play the game
in that basis (Fig. 2). This crucial step injects diversity into the games
played by the agent.Strassen’s2 and Laderman’s15 algorithms). More importantly, AlphaT-
ensor improves over the best algorithms known for several matrix
sizes. In particular, AlphaTensor finds an algorithm for multiplying
4 × 4 matrices using 47 multiplications in Z2, thereby outperforming
Strassen’s two-level algorithm2, which involves 72 = 49 multiplica-
tions. By applying this algorithm recursively, one obtains a practical
matrix multiplication algorithm in Z2 with complexity O(N 2.778) .
Moreover, AlphaTensor discovers efficient algorithms for multiply-
ing matrices in standard arithmetic; for example, AlphaTensor finds
a rank-76 decomposition of T 4,5,5 , improving over the previous
state-of-the-art complexity of 80 multiplications. See Extended
Data Figs. 1 and 2 for examples.
AlphaTensor generates a large database of matrix multiplication
algorithms—up to thousands of algorithms for each size. We exploit
this rich space of algorithms by combining them recursively, with the
aim of decomposing larger matrix multiplication tensors. We refer
to refs. 25,26 and Appendix H in Supplementary Information for more
details. Using this approach, we improve over the state-of-the-art
results for more than 70 matrix multiplication tensors (with
n, m, p ≤ 12). See Fig. 3 (right) and Extended Data Table 1 for the results.
A crucial aspect of AlphaTensor is its ability to learn to transfer knowl-
edge between targets (despite providing no prior knowledge on their
relationship). By training one agent to decompose various tensors,
AlphaTensor shares learned strategies among these, thereby improv-
ing the overall performance (see Supplementary Information for
analysis). Finally, it is noted that AlphaTensor scales beyond current
computational approaches for decomposing tensors. For example, to
our knowledge, no previous approach was able to handle T 4, which has
an action space 1010 times larger than T 3. Our agent goes beyond this
limit, discovering decompositions matching or surpassing
state-of-the-art for large tensors such as T 5.
Data augmentation
From every played game, we can extract additional tensor-factorization
pairs for training the network. Specifically, as factorizations are
order invariant (owing to summation), we build an additional
tensor-factorization training pair by swapping a random action with
the last action from each finished game.
Algorithm discovery results
Discovery of matrix multiplication algorithms
We train a single AlphaTensor agent to find matrix multiplication algo-
rithms for matrix sizes n × m with m × p, where n, m, p ≤ 5. At the begin-
ning of each game, we sample uniformly a triplet (n, m, p) and train
AlphaTensor to decompose the tensor T n, m, p. Although we consider
tensors of fixed size (T n, m, p has size nm × mp × pn), the discovered algo-
rithms can be applied recursively to multiply matrices of arbitrary size.
We use AlphaTensor to find matrix multiplication algorithms over
different arithmetics—namely, modular arithmetic (that is, multiplying
matrices in the quotient ring Z2), and standard arithmetic (that is, mul-
tiplying matrices in R).
Figure 3 (left) shows the complexity (that is, rank) of the algo-
rithms discovered by AlphaTensor. AlphaTensor re-discovers the
best algorithms known for multiplying matrices (for example,
50 | Nature | Vol 610 | 6 October 2022a
n=4
n=5
n=6
Extrapolation
n = 10
W
V
U
n=3
b
Input: n × n skew-symmetric matrix A, vector b.
( n – 1 )( n + 2 )
multiplications.
Output: The resulting vector c = Ab computed in
2
(1) for i = 1, . . . , n − 2 do
(2)
for j = i + 1, . . . , n do
(3)
wij = aij ( bj – bi )
Computing the first ( n – 2 )( n + 1 )/2 intermediate products
(4) for i = 1, . . . , n do
n
(5)
q i = bi Σ
a
j=1 ji
(6) for i = 1, . . . , n − 2 do
n
i–1
(7)
ci = Σ
w – qi
w +
j=1 ji Σ j=i+1 ij
n–2 n–2
n
n–2
(8) cn–1 = – Σ
w –
w +
q
i=1 Σ j=i+1 ij Σ j=1 jn Σ i=1,i≠n–1 i
n–1 n
n–1
(9) cn = – Σ
w +
q
i=1 Σ j=i+1 ij Σ i=1 i
Fig. 4 | Algorithm discovery beyond standard matrix multiplication.
n(n − 1)
a, Decompositions found by AlphaTensor for the tensors of size 2 × n × n
(with n = 3, 4, 5, 6) representing the skew-symmetric matrix-vector multiplication.
The red pixels denote 1, the blue pixels denote −1 and the white pixels denote 0.
Analysing the symmetries of matrix multiplication algorithms
From a mathematical standpoint, the diverse algorithms discovered
by AlphaTensor show that the space is richer than previously known.
For example, while the only known rank-49 factorization decomposing
T 4 = T 2 ⊗ T 2 before this paper conforms to the product structure (that
is, it uses the factorization of T 2 twice, which we refer to as Strassen-
square2), AlphaTensor finds more than 14,000 non-equivalent fac-
torizations (with standard arithmetic) that depart from this scheme,
and have different properties (such as matrix ranks and sparsity—see
Supplementary Information). By non-equivalent, we mean that it is
not possible to obtain one from another by applying a symmetry trans-
formation (such as permuting the factors). Such properties of matrix
multiplication tensors are of great interest, as these tensors represent
fundamental objects in algebraic complexity theory3,5,7. The study of
matrix multiplication symmetries can also provide insight into the
asymptotic complexity of matrix multiplication5. By exploring this rich
space of algorithms, we believe that AlphaTensor will be useful for
generating results and guiding mathematical research. See Supple-
mentary Information for proofs and details on the symmetries of
factorizations.
Beyond standard matrix multiplication
Tensors can represent any bilinear operation, such as structured matrix
multiplication, polynomial multiplication or more custom bilinear
operations used in machine learning27,28. We demonstrate here a
use-case where AlphaTensor finds a state-of-the-art algorithm for
multiplying an n x n skew-symmetric matrix with a vector of length n.
Figure 4a shows the obtained decompositions for small instance sizes n.
We observe a pattern that we generalize to arbitrary n, and prove that
this yields a general algorithm for the skew-symmetric matrix-vector
product (Fig. 4b). This algorithm, which uses (n − 1)(n + 2)/2 ~ 21 n 2
multiplications (where ∼ indicates asymptotic similarity), outperforms
the previously known algorithms using asymptotically n2 multiplica-
tions29, and is asymptotically optimal. See Supplementary Information
Computing the final n intermediate products
Extrapolation to n = 10 is shown in the rightmost figure. b, Skew-symmetric
matrix-by-vector multiplication algorithm, obtained from the examples solved
by AlphaTensor. The wij and qi terms in steps 3 and 5 correspond to the m r terms
in Algorithm 1. It is noted that steps 6–9 do not involve any multiplications.
for a proof, and for another use-case showing AlphaTensor’s ability
to re-discover the Fourier basis (see also Extended Data Table 2). This
shows that AlphaTensor can be applied to custom bilinear operations,
and yield efficient algorithms leveraging the problem structure.
Rapid tailored algorithm discovery
We show a use-case where AlphaTensor finds practically efficient matrix
multiplication algorithms, tailored to specific hardware, with zero
prior hardware knowledge. To do so, we modify the reward of AlphaT-
ensor: we provide an additional reward at the terminal state (after the
agent found a correct algorithm) equal to the negative of the runtime
of the algorithm when benchmarked on the target hardware. That is,
we set r t′ = rt + λbt , where rt is the reward scheme described in ‘DRL for
algorithm discovery’, bt is the benchmarking reward (non-zero only at
the terminal state) and λ is a user-specified coefficient. Aside from the
different reward, the exact same formulation of TensorGame is used.
We train AlphaTensor to search for efficient algorithms to multiply
4 × 4 block matrices, and focus on square matrix multiplication of size
8,192 (each block is hence of size 2,048) to define the benchmarking
reward. AlphaTensor searches for the optimal way of combining the
16 square blocks of the input matrices on the considered hardware. We
do not apply the 4 × 4 algorithm recursively, to leverage the efficient
implementation of matrix multiplication on moderate-size matrices
(2,048 × 2,048 in this case). We study two hardware devices commonly
used in machine learning and scientific computing: an Nvidia V100
graphics processing unit (GPU) and a Google tensor processing unit
(TPU) v2. The factorization obtained by AlphaTensor is transformed
into JAX30 code, which is compiled ( just in time) before benchmarking.
Figure 5a,b shows the efficiency of the AlphaTensor-discovered
algorithms on the GPU and the TPU, respectively. AlphaTensor dis-
covers algorithms that outperform the Strassen-square algorithm,
which is a fast algorithm for large square matrices31,32. Although
the discovered algorithm has the same theoretical complexity as
Strassen-square, it outperforms it in practice, as it is optimized for
the considered hardware. Interestingly, AlphaTensor finds algorithms
Nature | Vol 610 | 6 October 2022 | 51Article
23.9%
21.3%
20,480
13.3%
10.7%
8,192
8.5%
4.3%
Strassen-square
AlphaTensor
9.2%
13.9%
8.9%
Fig. 5 | Speed-ups of the AlphaTensor-discovered algorithm. a,b, Speed-ups
(%) of the AlphaTensor-discovered algorithms tailored for a GPU (a) and a TPU
(b), optimized for a matrix multiplication of size 8,192 × 8,192. Speed-ups are
measured relative to standard (for example, cuBLAS for the GPU) matrix
multiplication on the same hardware. Speed-ups are reported for various
with a larger number of additions compared with Strassen-square (or
equivalently, denser decompositions), but the discovered algorithms
generate individual operations that can be efficiently fused by the
specific XLA33 grouping procedure and thus are more tailored towards
the compiler stack we use. The algorithms found by AlphaTensor also
provide gains on matrix sizes larger than what they were optimized
for. Finally, Fig. 5c shows the importance of tailoring to particular
hardware, as algorithms optimized for one hardware do not perform
as well on other hardware.
Discussion
Trained from scratch, AlphaTensor discovers matrix multiplication
algorithms that are more efficient than existing human and
computer-designed algorithms. Despite improving over known
algorithms, we note that a limitation of AlphaTensor is the need
to pre-define a set of potential factor entries F, which discretizes
the search space but can possibly lead to missing out on efficient
algorithms. An interesting direction for future research is to adapt
AlphaTensor to search for F. One important strength of AlphaTensor
is its flexibility to support complex stochastic and non-differentiable
rewards (from the tensor rank to practical efficiency on specific hard-
ware), in addition to finding algorithms for custom operations in a
wide variety of spaces (such as finite fields). We believe this will spur
applications of AlphaTensor towards designing algorithms that opti-
mize metrics that we did not consider here, such as numerical stability
or energy usage.
The discovery of matrix multiplication algorithms has far-reaching
implications, as matrix multiplication sits at the core of many com-
putational tasks, such as matrix inversion, computing the determi-
nant and solving linear systems, to name a few7. We also note that our
methodology can be extended to tackle related primitive mathemati-
cal problems, such as computing other notions of rank (for example,
border rank—see Supplementary Information), and NP-hard matrix
factorization problems (for example, non-negative factorization). By
tackling a core NP-hard computational problem in mathematics using
DRL—the computation of tensor ranks—AlphaTensor demonstrates the
viability of DRL in addressing difficult mathematical problems, and
potentially assisting mathematicians in discoveries.
52 | Nature | Vol 610 | 6 October 2022
Speed-up on TPU v2
Optimized for GPU
TPU
10.3%
6.6%
Optimized for TPU
2.8%
12.4%
9.0%
8,192
Speed-up on Nvidia V100 GPU
4.4%
13.4%
14,336
10,240
6.8%
8.5%
GPU
11.2%
7.2%
12,288
10.1%
10,240
Matrix size
16.1%
12,288
12.3%
6.9%
16,384
19.6%
14,336
8.4%
18,432
16.6%
13.8%
16,384
13.9%
20,480
17.9%
15.3%
18,432
Matrix size
c
b
Benchmark device
a
10.3%
Strassen-square
AlphaTensor
Speed-up of tailored agorithms
on both devices
matrix sizes (despite optimizing the algorithm only on one matrix size). We also
report the speed-up of the Strassen-square algorithm. The median speed-up is
reported over 200 runs. The standard deviation over runs is <0.4 percentage
points (see Supplementary Information for more details). c, Speed-up of both
algorithms (tailored to a GPU and a TPU) benchmarked on both devices.
Online content
Any methods, additional references, Nature Research reporting summa-
ries, source data, extended data, supplementary information, acknowl-
edgements, peer review information; details of author contributions
and competing interests; and statements of data and code availability
are available at https://doi.org/10.1038/s41586-022-05172-4.
1.
Silver, D. et al. A general reinforcement learning algorithm that masters chess, shogi, and
Go through self-play. Science 362, 1140–1144 (2018).
2.
Strassen, V. Gaussian elimination is not optimal. Numer. Math. 13, 354–356 (1969).
3.
Bürgisser, P., Clausen, M. & Shokrollahi, A. Algebraic Complexity Theory Vol. 315 (Springer
Science & Business Media, 2013).
4. Bläser, M. Fast matrix multiplication. Theory Comput. 5, 1–60 (2013).
5.
Landsberg, J. M. Geometry and Complexity Theory 169 (Cambridge Univ. Press, 2017).
6.
Pan, V. Y. Fast feasible and unfeasible matrix multiplication. Preprint at https://arxiv.org/
abs/1804.04102 (2018).
7.
Lim, L.-H. Tensors in computations. Acta Numer. 30, 555–764 (2021).
8.
Schönhage, A. Partial and total matrix multiplication. SIAM J. Comput. 10, 434–455 (1981).
9.
Coppersmith, D. & Winograd, S. Matrix multiplication via arithmetic progressions. In ACM
Symposium on Theory of Computing 1–6 (ACM, 1987).
10. Strassen, V. The asymptotic spectrum of tensors and the exponent of matrix
multiplication. In 27th Annual Symposium on Foundations of Computer Science 49–54
(IEEE, 1986).
11. Le Gall, F. Powers of tensors and fast matrix multiplication. In International Symposium on
Symbolic and Algebraic Computation 296–303 (ACM, 2014).
12. Alman, J. & Williams, V. V. A refined laser method and faster matrix multiplication. In
ACM-SIAM Symposium on Discrete Algorithms 522–539 (SIAM, 2021).
13. Gauss, C. F. Theoria Motus Corporum Coelestium in Sectionibus Conicis Solum
Ambientium (Perthes and Besser, 1809).
14. Hillar, C. J. & Lim, L.-H. Most tensor problems are NP-hard. J. ACM 60, 1–39 (2013).
15. Laderman, J. D. A noncommutative algorithm for multiplying 3 × 3 matrices using 23
multiplications. Bull. Am. Math. Soc. 82, 126–128 (1976).
16. Hopcroft, J. E. & Kerr, L. R. On minimizing the number of multiplications necessary for
matrix multiplication. SIAM J. Appl. Math. 20, 30–36 (1971).
17. Vervliet, N., Debals, O., Sorber, L., Van Barel, M. & De Lathauwer, L. Tensorlab 3.0 (2016);
https://www.tensorlab.net/
18. Smirnov, A. V. The bilinear complexity and practical algorithms for matrix multiplication.
Comput. Math. Math. Phys. 53, 1781–1795 (2013).
19. Sedoglavic, A. & Smirnov, A. V. The tensor rank of 5x5 matrices multiplication is bounded
by 98 and its border rank by 89. In Proc. 2021 on International Symposium on Symbolic
and Algebraic Computation 345–351 (ACM, 2021).
20. Heule, M. J., Kauers, M. & Seidl, M. New ways to multiply 3 × 3-matrices. J. Symb. Comput.
104, 899–916 (2021).
21. Hubert, T. et al. Learning and planning in complex action spaces. In International
Conference on Machine Learning 4476–4486 (PMLR, 2021).
22. Zhang, W. & Dietterich, T. G. A reinforcement learning approach to job-shop scheduling.
In International Joint Conferences on Artificial Intelligence Vol. 95, 1114–1120
(Morgan Kaufmann Publishers, 1995).23. Vaswani, A. Attention is all you need. In International Conference on Neural Information
Processing Systems Vol 30, 5998–6008 (Curran Associates, 2017).
24. Ho, J., Kalchbrenner, N., Weissenborn, D. & Salimans, T. Axial attention in
multidimensional transformers. Preprint at https://arxiv.org/abs/1912.12180 (2019).
25. Drevet, C.-É., Islam, M. N. & Schost, É. Optimization techniques for small matrix
multiplication. Theor. Comput. Sci. 412, 2219–2236 (2011).
26. Sedoglavic, A. A non-commutative algorithm for multiplying (7 × 7) matrices using 250
multiplications. Preprint at https://arxiv.org/abs/1712.07935 (2017).
27. Battaglia, P. W. et al. Relational inductive biases, deep learning, and graph networks.
Preprint at https://arxiv.org/abs/1806.01261 (2018).
28. Balog, M., van Merriënboer, B., Moitra, S., Li, Y. & Tarlow, D. Fast training of sparse graph
neural networks on dense hardware. Preprint at https://arxiv.org/abs/1906.11786 (2019).
29. Ye, K. & Lim, L.-H. Fast structured matrix computations: tensor rank and Cohn–Umans
method. Found. Comput. Math. 18, 45–95 (2018).
30. Bradbury, J. et al. JAX: composable transformations of Python+NumPy programs. GitHub
http://github.com/google/jax (2018).
31. Benson, A. R. & Ballard, G. A framework for practical parallel fast matrix multiplication.
ACM SIGPLAN Not. 50, 42–53 (2015).
32. Huang, J., Smith, T. M., Henry, G. M. & Van De Geijn, R. A. Strassen’s algorithm reloaded.
In International Conference for High Performance Computing, Networking, Storage and
Analysis 690–701 (IEEE, 2016).
33. Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In USENIX
Symposium On Operating Systems Design And Implementation 265–283 (USENIX,
2016).
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution
4.0 International License, which permits use, sharing, adaptation, distribution
and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the Creative Commons license,
and indicate if changes were made. The images or other third party material in this article are
included in the article’s Creative Commons license, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons license and your
intended use is not permitted by statutory regulation or exceeds the permitted use, you will
need to obtain permission directly from the copyright holder. To view a copy of this license,
visit http://creativecommons.org/licenses/by/4.0/.
© The Author(s) 2022
Nature | Vol 610 | 6 October 2022 | 53Article
Methods
TensorGame
TensorGame is played as follows. The start position S 0 of the game
corresponds to the tensor T representing the bilinear operation of
interest, expressed in some basis. In each step t of the game, the player
writes down three vectors (u(t), v(t), w(t)), which specify the rank-1 tensor
u(t) ⊗ v(t) ⊗ w(t), and the state of the game is updated by subtracting the
newly written down factor:
S t ← S t −1 − u(t ) ⊗ v (t ) ⊗ w (t ).
(2)
The game ends when the state reaches the zero tensor, SR = 0. This
means that the factors written down throughout the game form a
R
factorization of the start tensor S 0, that is, S 0 = ∑t =1 u(t ) ⊗ v (t ) ⊗ w (t ) .
This factorization is then scored. For example, when optimizing for
asymptotic time complexity the score is −R, and when optimizing
for practical runtime the algorithm corresponding to the factorization
R
{(u(t ), v (t ), w (t ))}t =1 is constructed (see Algorithm 1) and then bench-
marked on the fly (see Supplementary Information).
In practice, we also impose a limit Rlimit on the maximum number of
moves in the game, so that a weak player is not stuck in unnecessarily
(or even infinitely) long games. When a game ends because it has run
out of moves, a penalty score is given so that it is never advantageous
to deliberately exhaust the move limit. For example, when optimizing
for asymptotic time complexity, this penalty is derived from an upper
bound on the tensor rank of the final residual tensor S R limit. This upper
bound on the tensor rank is obtained by summing the matrix ranks of
the slices of the tensor.
TensorGame over rings. We say that the decomposition of T n in equa-
tion (1) is in a ring E (defining the arithmetic operations) if each of the
factors u(t), v(t) and w(t) has entries belonging to the set E , and additions
and multiplications are interpreted according to E . The tensor rank
depends, in general, on the ring. At each step of TensorGame, the ad-
ditions and multiplications in equation (2) are interpreted in E . For
example, when working in Z2, (in this case, the factors u(t), v(t) and w(t)
live in F = {0, 1}), a modulo 2 operation is applied after each state update
(equation (2)).
We note that integer-valued decompositions u(t), v(t) and w(t) lead to
decompositions in arbitrary rings E . Hence, provided F only contains
integers, algorithms we find in standard arithmetic apply more gener-
ally to any ring.
AlphaTensor
AlphaTensor builds on AlphaZero 1 and its extension Sampled
AlphaZero21, combining a deep neural network with a sample-based
MCTS search algorithm.
The deep neural network, fθ(s) = (π, z) parameterized by θ, takes as
input the current state s of the game and outputs a probability distribu-
tion π(⋅∣s) over actions and z(⋅∣s) over returns (sum of future rewards) G.
The parameters θ of the deep neural network are trained by reinforce-
ment learning from self-play games and synthetic demonstrations.
Self-play games are played by actors, running a sample-based MCTS
search at every state st encountered in the game. The MCTS search
returns an improved probability distribution over moves from which
an action at is selected and applied to the environment. The sub-tree
under at is reused for the subsequent search at st+1. At the end of the
game, a return G is obtained and the trajectory is sent to the learner to
update the neural network parameters θ. The distribution over returns
z(⋅∣st) is learned through distributional reinforcement learning using
the quantile regression distributional loss34, and the network policy
π(⋅∣st) is updated using a Kullback–Leibler divergence loss, to maximize
its similarity to the search policy for self-play games or to the next
action for synthetic demonstrations. We use the Adam optimizer35
with decoupled weight decay36 to optimize the parameters θ of the
neural network.
Sample-based MCTS search. The sample-based MCTS search is very
similar to the one described in Sampled AlphaZero. Specifically, the
search consists of a series of simulated trajectories of TensorGame that
are aggregated in a tree. The search tree therefore consists of nodes
representing states and edges representing actions. Each state-action
pair (s, a) stores a set of statistics N (s , a), Q(s , a), πˆ(s , a), where N(s, a)
is the visit count, Q(s, a) is the action value and πˆ(s , a) is the empirical
policy probability. Each simulation traverses the tree from the root
state s0 until a leaf state sL is reached by recursively selecting in each
state s an action a that has not been frequently explored, has high em-
pirical policy probability and high value. Concretely, actions within
the tree are selected by maximizing over the probabilistic upper con-
fidence tree bound21,37
argmaxQ(s , a) + c(s) ⋅ πˆ(s , a)
a
∑ b N ( s , b)
1 + N (s , a)
,
where c(s) is an exploration factor controlling the influence of the
empirical policy πˆ(s , a)relative to the values Q(s, a) as nodes are visited
more often. In addition, a transposition table is used to recombine
different action sequences if they reach the exact same tensor. This
can happen particularly often in TensorGame as actions are commuta-
tive. Finally, when a leaf state sL is reached, it is evaluated by the neural
network, which returns K actions {ai} sampled from π(a∣sL), alongside
1
the empirical distribution πˆ(a sL) = K ∑i δa, ai and a value v(sL) con-
structed from z(⋅∣sL). Differently from AlphaZero and Sampled Alp-
haZero, we chose v not to be the mean of the distribution of returns
z(⋅∣sL) as is usual in most reinforcement learning agents, but instead to
be a risk-seeking value, leveraging the facts that TensorGame is a deter-
ministic environment and that we are primarily interested in finding
the best trajectory possible. The visit counts and values on the simu-
lated trajectory are then updated in a backward pass as in Sampled
AlphaZero.
Policy improvement. After simulating N(s) trajectories from state s
using MCTS, the normalized visit counts of the actions at the root of
the search tree N(s, a)/N(s) form a sample-based improved policy. Dif-
ferently from AlphaZero and Sampled AlphaZero, we use an adaptive
temperature scheme to smooth the normalized visit counts distribution
as some states can accumulate an order of magnitude more visits than
others because of sub-tree reuse and transposition table. Concretely,
we define the improved policy as Iπˆ(s , a) = N 1/τ (s)(s , a)/ ∑b N 1/τ (s)(s , b)
where τ(s) = logN (s)/logN if N > N and 1 otherwise, with N being a
hyperparameter. For training, we use Iπ̂ directly as a target for the
network policy π. For acting, we additionally discard all actions that
have a value lower than the value of the most visited action, and sample
proportionally to Iπ̂ among those remaining high-value actions.
Learning one agent for multiple target tensors. We train a single
agent to decompose the different tensors T n, m, p in a given arithmetic
(standard or modular). As the network works with fixed-size inputs, we
pad all tensors (with zeros) to the size of the largest tensor we consider
(T 5, of size 25 × 25 × 25). At the beginning of each game, we sample uni-
formly at random a target T n, m, p, and play TensorGame. Training a sin-
gle agent on different targets leads to better results thanks to the trans-
fer between targets. All our results reported in Fig. 3 are obtained using
multiple runs of this multi-target setting. We also train a single agent
to decompose tensors in both arithmetics. Owing to learned transfer
between the two arithmetics, this agent discovers a different distribu-
tion of algorithms (of the same ranks) in standard arithmetic than the
agent trained on standard arithmetic only, thereby increasing the over-
all diversity of discovered algorithms.Synthetic demonstrations. The synthetic demonstrations buffer
contains tensor-factorization pairs, where the factorizations
R
{(u(r ), v (r ), w (r ))}r =1 are first generated at random, after which the tensor
R
(r )
D = ∑r =1 u ⊗ v (r ) ⊗ w (r ) is formed. We create a dataset containing
5 million such tensor-factorization pairs. Each element in the factors
is sampled independently and identically distributed (i.i.d.) from a
given categorical distribution over F (all possible values that can be
taken). We discarded instances whose decompositions were clearly
suboptimal (contained a factor with u = 0, v = 0, or w = 0).
In addition to these synthetic demonstrations, we further add to
the demonstration buffer previous games that have achieved large
scores to reinforce the good moves made by the agent in these games.
Change of basis. The rank of a bilinear operation does not depend on
the basis in which the tensor representing it is expressed, and for any
invertible matrices A, B and C we have Rank (T) = Rank (T (A, B, C)), where
T (A, B, C) is the tensor after change of basis given by
S
S
S
(A, B, C)
T ijk
= ∑ ∑ ∑ AiaBjbCkc T abc .
(3)
a =1 b =1 c =1
Hence, exhibiting a rank-R decomposition of the matrix multiplica-
tion tensor T n expressed in any basis proves that the product of two
n × n matrices can be computed using R scalar multiplications. Moreo-
ver, it is straightforward to convert such a rank-R decomposition into
a rank-R decomposition in the canonical basis, thus yielding a practical
algorithm of the form shown in Algorithm 1. We leverage this observa-
tion by expressing the matrix multiplication tensor T n in a large number
of randomly generated bases (typically 100,000) in addition to the
canonical basis, and letting AlphaTensor play games in all bases in
parallel.
This approach has three appealing properties: (1) it provides a natural
exploration mechanism as playing games in different bases automati-
cally injects diversity into the games played by the agent; (2) it exploits
properties of the problem as the agent need not succeed in all bases—it
is sufficient to find a low-rank decomposition in any of the bases; (3)
it enlarges coverage of the algorithm space because a decomposition
with entries in a finite set F = {−2, −1, 0, 1, 2} found in a different basis
need not have entries in the same set when converted back into the
canonical basis.
In full generality, a basis change for a 3D tensor of size S × S × S is
specified by three invertible S × S matrices A, B and C. However, in our
procedure, we sample bases at random and impose two restrictions:
(1) A = B = C, as this performed better in early experiments, and (2)
unimodularity (detA ∈ {−1, + 1}), which ensures that after converting
an integral factorization into the canonical basis it still contains integer
entries only (this is for representational convenience and numerical
stability of the resulting algorithm). See Supplementary Information
for the exact algorithm.
Signed permutations. In addition to playing (and training on) games in
different bases, we also utilize a data augmentation mechanism when-
ever the neural network is queried in a new MCTS node. At acting time,
when the network is queried, we transform the input tensor by applying
a change of basis—where the change of basis matrix is set to a random
signed permutation. We then query the network on this transformed
input tensor, and finally invert the transformation in the network’s
policy predictions. Although this data augmentation procedure can
be applied with any generic change of basis matrix (that is, it is not
restricted to signed permutation matrices), we use signed permuta-
tions mainly for computational efficiency. At training time, whenever
the neural network is trained on an (input, policy targets, value target)
triplet (Fig. 2), we apply a randomly chosen signed permutation to
both the input and the policy targets, and train the network on this
transformed triplet. In practice, we sample 100 signed permutations
at the beginning of an experiment, and use them thereafter.
Action canonicalization. For any λ1, λ2, λ3 ∈ {−1, +1} such that λ1λ2λ3 = 1,
the actions (λ1u, λ2v, λ3w) and (u, v, w) are equivalent because they lead
to the same rank-one tensor (λ1u) ⊗ (λ2v) ⊗ (λ3w) = u ⊗ v ⊗ w. To pre-
vent the network from wasting capacity on predicting multiple equiva-
lent actions, during training we always present targets (u, v, w) for the
policy head in a canonical form, defined as having the first non-zero
element of u and the first non-zero element of v strictly positive. This is
well defined because u or v cannot be all zeros (if they are to be part of
a minimal rank decomposition), and for any (u, v, w) there are unique
λ1, λ2, λ3 ∈ {−1, +1} (with λ1λ2λ3 = 1) that transform it into canonical form.
In case the network predicts multiple equivalent actions anyway, we
merge them together (summing their empirical policy probabilities)
before inserting them into the MCTS tree.
Training regime. We train AlphaTensor on a TPU v3, with a total batch
size of 2,048. We use 64 TPU cores, and train for 600,000 iterations.
On the actor side, the games are played on standalone TPU v4, and we
use 1,600 actors. In practice, the procedure takes a week to converge.
Neural network
The architecture is composed of a torso, followed by a policy head that
predicts a distribution over actions, and a value head that predicts a dis-
tribution of the returns from the current state (see Extended Data Fig. 3).
Input. The input to the network contains all the relevant information
of the current state and is composed of a list of tensors and a list of
scalars. The most important piece of information is the current 3D
tensor S t of size S × S × S. (For simplicity, in the description here we
assume that all the three dimensions of the tensor are equal in size. The
generalization to different sizes is straightforward.) In addition, the
model is given access to the last h actions (h being a hyperparameter
usually set to 7), represented as h rank-1 tensors that are concatenated
to the input. The list of scalars includes the time index t of the current
action (where 0 ≤ t < Rlimit).
Torso. The torso of the network is in charge of mapping both scalars and
tensors from the input to a representation that is useful to both policy
and value heads. Its architecture is based on a modification of transform-
ers23, and its main signature is that it operates over three S × S grids pro-
jected from the S × S × S input tensors. Each grid represents two out of
the three modes of the tensor. Defining the modes of the tensor as
U, V, W , the rows and columns of the first grid are associated to U and V ,
respectively, the rows and columns of the second grid are associated to
W and U , and the rows and columns of the third grid are associated to V
and W . Each element of each grid is a feature vector, and its initial value
is given by the elements of the input tensors along the grid’s missing
mode. These feature vectors are enriched by concatenating an S × S × 1
linear projection from the scalars. This is followed by a linear layer pro-
jecting these feature vectors into a 512-dimensional space.
The rest of the torso is a sequence of attention-based blocks with the
objective of propagating information between the three grids. Each of
those blocks has three stages, one for every pair of grids. In each stage,
the grids involved are concatenated, and axial attention24 is performed
over the columns. It is noted that in each stage we perform in parallel S
self-attention operations of 2S elements in each. The representation
sent to the policy head corresponds to the 3S2 512-dimensional feature
vectors produced by the last layer of the torso. A detailed description
of the structure of the torso is specified in Extended Data Fig. 4 (top)
and Appendix A.1.1 in Supplementary Information.
Policy head. The policy head uses the transformer architecture23
to model an autoregressive policy. Factors are decomposed into kArticle
tokens of dimensionality d such that k × d = 3S. The transformer con-
ditions on the tokens already generated and cross-attends to the fea-
tures produced by the torso. At training time, we use teacher-forcing,
that is, the ground truth actions are decomposed into tokens and
taken as inputs into the causal transformer in such a way that the
prediction of a token depends only on the previous tokens. At infer-
ence time, K actions are sampled from the head. The feature repre-
sentation before the last linear layer of the initial step (that is, the
only step that is not conditioned on the ground truth) is used as an
input to the value head, described below. Details of the architecture
are presented in Extended Data Fig. 4 (centre) and Appendix A.1.2 in
Supplementary Information.
Value head. The value head is composed of a four-layer multilayer
perceptron whose last layer produces q outputs corresponding to the
2q − 1
1
3
, , … 2q quantiles. In this way, the value head predicts the distri-
2q 2q
bution of returns from this state in the form of values predicted for the
aforementioned quantiles34. At inference time, we encourage the agent
to be risk-seeking by using the average of the predicted values for quan-
tiles over 75%. A detailed description of the value head is presented in
Extended Data Fig. 4 (bottom) and Appendix A.1.3 in Supplementary
Information.
Related work
The quest for efficient matrix multiplication algorithms started with
Strassen’s breakthrough in ref. 2, which showed that one can multiply
2 × 2 matrices using 7 scalar multiplications, leading to an algorithm
of complexity O(n 2.81). This led to the development of a very active field
of mathematics attracting worldwide interest, which studies the asymp-
totic complexity of matrix multiplication (see refs. 3–6). So far, the best
known complexity for matrix multiplication is O(n 2.37286)(ref. 12), which
improves over ref. 11, and builds on top of fundamental results in the
field8–10. However, this does not yield practical algorithms, as such
approaches become advantageous only for astronomical matrix sizes.
Hence, a significant body of work aims at exhibiting explicit factoriza-
tions of matrix multiplication tensors, as these factorizations provide
practical algorithms. After Strassen’s breakthrough showing that
rank (T 2) ≤ 7 , efficient algorithms for larger matrix sizes were
found15,16,18,26,38. Most notably, Laderman showed in ref. 15 that 3 × 3
matrix multiplications can be performed with 23 scalar multiplications.
In addition to providing individual low-rank factorizations, an impor-
tant research direction aims at understanding the space of matrix
multiplication algorithms—as opposed to exhibiting individual
low-rank factorizations—by studying the symmetry groups and diver-
sity of factorizations (see ref. 5 and references therein). For example,
the symmetries of 2 × 2 matrix multiplication were studied in refs. 39–42,
where Strassen’s algorithm was shown to be essentially unique. The
case of 3 × 3 was studied in ref. 43, whereas a symmetric factorization
for all n is provided in ref. 44.
On the computational front, continuous optimization has been the
main workhorse for decomposing tensors17,45,46, and in particular matrix
multiplication tensors. Such continuous optimization procedures (for
example, alternating least squares), however, yield approximate solu-
tions, which correspond to inexact matrix multiplication algorithms
with floating point operations. To circumvent this issue, regularization
procedures have been proposed, such as ref. 18, to extract exact decom-
positions. Unfortunately, such approaches often require substantial
human intervention and expertise to decompose large tensors. A dif-
ferent line of attack was explored in refs. 47,48, based on learning the
continuous weights of a two-layer network that mimics the structure
of the matrix multiplication operation. This method, which is trained
through supervised learning of matrix multiplication examples, finds
approximate solutions to 2 × 2 and 3 × 3 matrix multiplications.
In ref. 48, a quantization procedure is further used to obtain an exact
decomposition for 2 × 2. Unlike continuous optimization-based
approaches, AlphaTensor directly produces algorithms from the
desired set of valid algorithms, and is flexible in that it allows us to
optimize a wide range of (even non-differentiable) objectives. This
unlocks tackling broader settings (for example, optimization in finite
fields, optimization of runtime), as well as larger problems (for exam-
ple, T 4 and T 5) than those previously considered. Different from con-
tinuous optimization, a boolean satisfiability (SAT) based formulation
of the problem of decomposing 3 × 3 matrix multiplication was
recently proposed in ref. 20, which adds thousands of new decompo-
sitions of rank 23 to the list of known 3 × 3 factorizations. The approach
relies on a state-of-the-art SAT solving procedure, where several
assumptions and simplifications are made on the factorizations to
reduce the search space. As is, this approach is, however, unlikely to
scale to larger tensors, as the search space grows very quickly with
the size.
On the practical implementation front, ref. 31 proposed several ideas
to speed up implementation of fast matrix multiplication algorithms
on central processing units (CPUs). Different fast algorithms are then
compared and benchmarked, and the potential speed-up of such algo-
rithms is shown against standard multiplication. Other works focused
on getting the maximal performance out of a particular fast matrix
multiplication algorithm (Strassen’s algorithm with one or two levels
of recursion) on a CPU32 or a GPU49. These works show that, despite
popular belief, such algorithms are of practical value. We see writing
a custom low-level implementation of a given algorithm to be distinct
from the focus of this paper—developing new efficient algorithms—and
we believe that the algorithms we discovered can further benefit from
a more efficient implementation by experts.
Beyond matrix multiplication and bilinear operations, a growing
amount of research studies the use of optimization and machine learn-
ing to improve the efficiency of computational operations. There
are three levels of abstractions at which this can be done: (1) in the
hardware design, for example, chip floor planning50, (2) at the hard-
ware–software interface, for example, program super-optimization
of a reference implementation for specific hardware51, and (3) on the
algorithmic level, for example, program induction52, algorithm selec-
tion53 or meta-learning54. Our work focuses on the algorithmic level of
abstraction, although AlphaTensor is also flexible to discover efficient
algorithms for specific hardware. Different from previous works, we
focus on discovering matrix multiplication algorithms that are prov-
ably correct, without requiring initial reference implementations. We
conclude by relating our work broadly to existing reinforcement learn-
ing methods for scientific discovery. Within mathematics, reinforce-
ment learning was applied, for example, to theorem proving55–58, and
to finding counterexamples refuting conjectures in combinatorics and
graph theory59. Reinforcement learning was further shown to be useful
in many areas in science, such as molecular design60,61 and synthesis62
and optimizing quantum dynamics63.
Data availability
The data used to train the system were generated synthetically accord-
ing to the procedures explained in the paper. The algorithms discov-
ered by AlphaTensor are available for download at https://github.com/
deepmind/alphatensor.
Code availability
An interactive notebook with code to check the non-equivalence of algo-
rithms is provided. Moreover, the fast algorithms from the ‘Algorithm
discovery results’ section on a GPU and a TPU are provided. These are
available for download at https://github.com/deepmind/alphatensor.
A full description of the AlphaZero algorithm that this work is based
on is available in ref. 1, and the specific neural network architecture we
use is described using pseudocode in the Supplementary Information.34. Dabney, W., Rowland, M., Bellemare, M. & Munos, R. Distributional reinforcement learning
with quantile regression. In AAAI Conference on Artificial Intelligence Vol. 32, 2892–2901
(AAAI Press, 2018).
35. Kingma, D. P., & Ba, J. Adam: a method for stochastic optimization. In International
Conference on Learning Representations (ICLR) (2015).
36. Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International
Conference on Learning Representations (ICLR) (2019).
37. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search.
Nature 529, 484–489 (2016).
38. Sedoglavic, A. A non-commutative algorithm for multiplying 5x5 matrices using 99
multiplications. Preprint at https://arxiv.org/abs/1707.06860 (2017).
39. de Groote, H. F. On varieties of optimal algorithms for the computation of bilinear
mappings II. optimal algorithms for 2 × 2-matrix multiplication. Theor. Comput. Sci. 7,
127–148 (1978).
40. Burichenko, V. P. On symmetries of the Strassen algorithm. Preprint at https://arxiv.org/
abs/1408.6273 (2014).
41. Chiantini, L., Ikenmeyer, C., Landsberg, J. M. & Ottaviani, G. The geometry of rank
decompositions of matrix multiplication I: 2 × 2 matrices. Exp. Math. 28, 322–327 (2019).
42. Grochow, J. A. & Moore, C. Designing Strassen’s algorithm. Preprint at https://arxiv.org/
abs/1708.09398 (2017).
43. Ballard, G., Ikenmeyer, C., Landsberg, J. M. & Ryder, N. The geometry of rank
decompositions of matrix multiplication II: 3 × 3 matrices. J. Pure Appl. Algebra 223,
3205–3224 (2019).
44. Grochow, J. A. & Moore, C. Matrix multiplication algorithms from group orbits. Preprint at
https://arxiv.org/abs/1612.01527 (2016).
45. Kolda, T. G. & Bader, B. W. Tensor decompositions and applications. SIAM Rev. 51,
455–500 (2009).
46. Bernardi, A., Brachat, J., Comon, P. & Mourrain, B. General tensor decomposition,
moment matrices and applications. J. Symb. Comput. 52, 51–71 (2013).
47. Elser, V. A network that learns Strassen multiplication. J. Mach. Learn. Res. 17, 3964–3976
(2016).
48. Tschannen, M., Khanna, A. & Anandkumar, A, StrassenNets: deep learning with a
multiplication budget. In International Conference on Machine Learning 4985–4994
(PMLR, 2018).
49. Huang, J., Yu, C. D. & Geijn, R. A. V. D. Strassen’s algorithm reloaded on GPUs. ACM Trans.
Math. Softw. 46, 1–22 (2020).
50. Mirhoseini, A. et al. A graph placement methodology for fast chip design. Nature 594,
207–212 (2021).
51. Bunel, R., Desmaison, A., Kohli, P., Torr, P. H. & Kumar, M. P. Learning to superoptimize
programs. In International Conference on Learning Representations (ICLR) (2017).
52. Li, Y., Gimeno, F., Kohli, P. & Vinyals, O. Strong generalization and efficiency in neural
programs. Preprint at https://arxiv.org/abs/2007.03629 (2020).
53. Lagoudakis, M. G. et al. Algorithm selection using reinforcement learning. In International
Conference on Machine Learning 511–518 (Morgan Kaufmann Publishers, 2000).
54. Schmidhuber, J. Evolutionary Principles in Self-Referential Learning. On Learning now to
Learn: The Meta-Meta-Meta...-Hook. Diploma thesis, Technische Univ. Munchen (1987).
55. Kaliszyk, C., Urban, J., Michalewski, H. & Olšák, M. Reinforcement learning of theorem
proving. In International Conference on Neural Information Processing Systems
8836–8847 (Curran Associates, 2018).
56. Piotrowski, B. & Urban, J. ATPboost: learning premise selection in binary setting with ATP
feedback. In International Joint Conference on Automated Reasoning 566–574 (Springer,
2018).
57.
58.
59.
60.
61.
62.
63.
64.
Bansal, K., Loos, S., Rabe, M., Szegedy, C. & Wilcox, S. HOList: an environment for
machine learning of higher order logic theorem proving. In International Conference on
Machine Learning 454–463 (PMLR, 2019).
Zombori, Z., Urban, J. & Brown, C. E. Prolog technology reinforcement learning prover.
In International Joint Conference on Automated Reasoning 489–507 (Springer, 2020).
Wagner, A. Z. Constructions in combinatorics via neural networks. Preprint at https://
arxiv.org/abs/2104.14516 (2021).
Popova, M., Isayev, O. & Tropsha, A. Deep reinforcement learning for de novo drug
design. Sci. Adv. 4, eaap7885 (2018).
Zhou, Z., Kearnes, S., Li, L., Zare, R. N. & Riley, P. Optimization of molecules via deep
reinforcement learning. Sci. Rep. 9, 10752 (2019).
Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural
networks and symbolic AI. Nature 555, 604–610 (2018).
Dalgaard, M., Motzoi, F., Sørensen, J. J. & Sherson, J. Global optimization of quantum
dynamics with AlphaZero deep exploration. npj Quantum Inf. 6, 6 (2020).
Fast matrix multiplication algorithms catalogue. Université de Lille https://fmm.univ-
lille.fr/ (2021).
Acknowledgements We thank O. Fawzi, H. Fawzi, C. Ikenmeyer, J. Ellenberg, C. Umans and
A. Wigderson for the inspiring discussions on the use of machine learning for maths; A. Davies,
A. Gaunt, P. Mudigonda, R. Bunel and O. Ronneberger for their advice on early drafts of the
paper; A. Ruderman, M. Bauer, R. Leblond, R. Kabra and B. Winckler for participating in a
hackathon at the early stages of the project; D. Visentin, R. Tanburn and S. Noury for sharing
their expertise on TPUs; P. Wang and R. Zhao for their help on benchmarking algorithms;
G. Holland, A. Pierce, N. Lambert and C. Meyer for assistance coordinating the research; and
our colleagues at DeepMind for encouragement and support.
Author contributions A.F. conceived the project, with support from B.R.-P. and P.K.; T.H., A.H.
and J.S. developed the initial AlphaZero codebase, and B.R.-P., M. Balog, A.F., A.N., F.J.R.R. and
G.S. developed an early supervised network prototype. A.H., T.H., B.R.-P., M. Barekatain and
J.S. designed the network architecture used in the paper. T.H., J.S., A.H., M. Barekatain, A.F.,
M. Balog and F.J.R.R. developed the tensor decomposition environment and data generation
pipeline, and A.H., T.H., M. Barekatain, M. Balog, B.R.-P., F.J.R.R. and A.N. analysed the
experimental results and algorithms discovered by AlphaTensor. A.N., A.F. and T.H. developed
the benchmarking pipeline and experiments, and B.R.-P., F.J.R.R. and A.N. extended the
approach to structured tensors. A.F., B.R.-P., G.S. and A.N. proved the results in the paper.
D.S., D.H. and P.K. contributed technical advice and ideas. A.F., M. Balog, B.R.-P., F.J.R.R., A.N.
and T.H. wrote the paper. These authors contributed equally, and are listed alphabetically by last
name after the corresponding author: A.F., M. Balog, A.H., T.H., B.R.-P. These authors contributed
equally, and are listed alphabetically by last name: M. Barekatain, A.N., F.J.R.R., J.S. and G.S.
Competing interests The authors of the paper are planning to file a patent application relating
to subject matter contained in this paper in the name of DeepMind Technologies Limited.
Additional information
Supplementary information The online version contains supplementary material available at
https://doi.org/10.1038/s41586-022-05172-4.
Correspondence and requests for materials should be addressed to Alhussein Fawzi.
Peer review information Nature thanks Grey Ballard, Jordan Ellenberg, Lek-Heng Lim, Michael
Littman and the other, anonymous, reviewer(s) for their contribution to the peer review of this
work.
Reprints and permissions information is available at http://www.nature.com/reprints.Article
Extended Data Fig. 1 | Algorithm for multiplying 4 × 4 matrices in modular arithmetic (Z 2) with 47 multiplications. This outperforms the two-level Strassen’s
algorithm, which involves 72 = 49 multiplications.Extended Data Fig. 2 | Algorithm for multiplying 4 × 5 by 5 × 5 matrices in standard arithmetic with 76 multiplications. This outperforms the previously best
known algorithm, which involves 80 multiplications.Article
Extended Data Fig. 3 | AlphaTensor’s network architecture. The network
takes as input the list of tensors containing the current state and previous
history of actions, and a list of scalars, such as the time index of the current
action. It produces two kinds of outputs: one representing the value, and the
other inducing a distribution over the action space from which we can sample
from. The architecture of the network is accordingly designed to have a
common torso, and two heads, the value and the policy heads. c is set to 512 in
all experiments.Extended Data Fig. 4 | Detailed view of AlphaTensor’s architecture, included torso, policy and value head. We refer to Algorithms A.1-A.11 in Supplementary
Information for the details of each component.Article
Extended Data Table 1 | Rank results obtained by combining decompositions (in standard arithmetic)
The table shows the cases where we were able to obtain an improvement over state-of-the-art, for tensors T n, m, p (with n, m, p≤12). The recipe column indicates the low-level matrix multiplica-
tion algorithms used to build the corresponding factorization. 〈n, m, p〉 denotes the best known bound on the rank of T n, m, p; see Appendix H in Supplementary Information for more details. For
tensors that were directly decomposed by AlphaTensor, the recipe shows a star mark, e.g. 〈3, 4, 5〉*. All the factorizations are made available.Extended Data Table 2 | Result of applying AlphaTensor to the tensor representing the cyclic convolution operation
AlphaTensor finds the discrete Fourier matrix (DFT) and the inverse DFT matrix in finite fields. The figure shows the decompositions found by AlphaTensor of the n × n × n tensor representing the
cyclic convolution of two vectors, for three different values of n in the finite field of order 17. The action space, characterized by the number of possible factor triplets {u(r), v(r), w(r)}, is thus 173n,
which is of the order of 1029 for n = 8. Despite the huge action space, AlphaTensor finds the optimal rank-n decompositions for the three values of n. The factors in the figure are stacked vertically,
i.e., U = [u(1), …, u(n)]. For ease of visualization, the factor entries have been expressed in terms of powers of an n-th primitive root of unity in the finite field. Within each column, each colour
uniquely represents one element of the field (e.g., for the column n = 4, we have depicted in grey 40 = 44 = 4−4 = 1). By inspecting the patterns in the decompositions, one could extrapolate the
results for other values of n and other fields. Indeed, the factors u(r) and v(r) correspond to the DFT coefficients, since u(kr ) = v (kr ) = z kr, whereas the factors w(r) correspond to the inverse DFT, since
w (kr ) = z −kr /n for 0≤k, r < n, where z is an n-th primitive root of unity (i.e., zn = 1 and zj ≠ 1 for any 1≤j < n).
The Illusion of Thinking:
Understanding the Strengths and Limitations of Reasoning Models
via the Lens of Problem Complexity
Parshin Shojaee∗†
Maxwell Horton
Iman Mirzadeh∗
Keivan Alizadeh
Samy Bengio
Mehrdad Farajtabar
Apple
Abstract
Recent generations of frontier language models have introduced Large Reasoning Models
(LRMs) that generate detailed thinking processes before providing answers. While these models
demonstrate improved performance on reasoning benchmarks, their fundamental capabilities, scal-
ing properties, and limitations remain insufficiently understood. Current evaluations primarily fo-
cus on established mathematical and coding benchmarks, emphasizing final answer accuracy. How-
ever, this evaluation paradigm often suffers from data contamination and does not provide insights
into the reasoning traces’ structure and quality. In this work, we systematically investigate these
gaps with the help of controllable puzzle environments that allow precise manipulation of composi-
tional complexity while maintaining consistent logical structures. This setup enables the analysis
of not only final answers but also the internal reasoning traces, offering insights into how LRMs
“think”. Through extensive experimentation across diverse puzzles, we show that frontier LRMs
face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counter-
intuitive scaling limit: their reasoning effort increases with problem complexity up to a point, then
declines despite having an adequate token budget. By comparing LRMs with their standard LLM
counterparts under equivalent inference compute, we identify three performance regimes: (1) low-
complexity tasks where standard models surprisingly outperform LRMs, (2) medium-complexity
tasks where additional thinking in LRMs demonstrates advantage, and (3) high-complexity tasks
where both models experience complete collapse. We found that LRMs have limitations in exact
computation: they fail to use explicit algorithms and reason inconsistently across puzzles. We
also investigate the reasoning traces in more depth, studying the patterns of explored solutions
and analyzing the models’ computational behavior, shedding light on their strengths, limitations,
and ultimately raising crucial questions about their true reasoning capabilities.
1
Introduction
Large Language Models (LLMs) have recently evolved to include specialized variants explicitly
designed for reasoning tasks—Large Reasoning Models (LRMs) such as OpenAI’s o1/o3 [1, 2],
DeepSeek-R1 [3], Claude 3.7 Sonnet Thinking [4], and Gemini Thinking [5]. These models are new
artifacts, characterized by their “thinking” mechanisms such as long Chain-of-Thought (CoT) with
self-reflection, and have demonstrated promising results across various reasoning benchmarks. Their
∗
†
Equal contribution.
Work done during an internship at Apple.
{p_shojaee, imirzadeh, kalizadehvahid, mchorton, bengio, farajtabar}@apple.com
1LLM Response
Peg 0
Claude 3.7
(+thinking)
80
60
40
20
Claude 3.7
0
1 2 3 4 5 6 7 8 10
15
Complexity (number of disks)
20
Initial State
[1, 0, 2]
extract moves from thoughts\u2028 [2, 0, 1]
[1, 2, 1]
(for analysis)
[3, 0, 2]
Middle State
extract final answer\u2028
(for measuring accuracy)
Target State
20,000
Claude 3.7
(+thinking)
15,000
10,000
Claude 3.7
5,000
0
1 2 3 4 5 6 7 8 10
15
Complexity (number of disks)
Peg 2
1
23
[1, 1, 0]
[2, 1, 2]
[1, 0, 2]
20
Position within Thoughts
Accuracy (%)
100
Response Length (Tokens)
<think>
Move disk 1 from peg 0 to peg 2 ... \u2028
moves = [
[1, 0, 2],
[2, 0, 1],
[1, 2, 1],
[3, 0, 2],
[1, 1, 0],
[2, 1, 2],
[1, 0, 2],
]
Let me double-check this...
</think>
<answer> the final answer is moves=...
</answer>
Peg 1
1
2
3
1
2
3
1.0
0.8
0.6
0.4
Correct Solutions
Incorrect Solutions
0.2
0.0
1 2 3 4 5 6 7 8 9 10
Complexity (number of disks)
15
Figure 1: Top: Our setup enables verification of both final answers and intermediate reasoning traces,
allowing detailed analysis of model thinking behavior. Bottom left & middle: At low complexity,
non-thinking models are more accurate and token-efficient. As complexity increases, reasoning models
outperform but require more tokens—until both collapse beyond a critical threshold, with shorter
traces. Bottom right: For correctly solved cases, Claude 3.7 Thinking tends to find answers early
at low complexity and later at higher complexity. In failed cases, it often fixates on an early wrong
answer, wasting the remaining token budget. Both cases reveal inefficiencies in the reasoning process.
emergence suggests a potential paradigm shift in how LLM systems approach complex reasoning
and problem-solving tasks, with some researchers proposing them as significant steps toward more
general artificial intelligence capabilities.
Despite these claims and performance advancements, the fundamental benefits and limitations of
LRMs remain insufficiently understood. Critical questions still persist: Are these models capable
of generalizable reasoning, or are they leveraging different forms of pattern matching [6]? How
does their performance scale with increasing problem complexity? How do they compare to their
non-thinking standard LLM counterparts when provided with the same inference token compute?
Most importantly, what are the inherent limitations of current reasoning approaches, and what
improvements might be necessary to advance toward more robust reasoning capabilities?
We believe the lack of systematic analyses investigating these questions is due to limitations in
current evaluation paradigms. Existing evaluations predominantly focus on established mathematical
and coding benchmarks, which, while valuable, often suffer from data contamination issues and do
not allow for controlled experimental conditions across different settings and complexities. Moreover,
these evaluations do not provide insights into the structure and quality of reasoning traces. To
understand the reasoning behavior of these models more rigorously, we need environments that
enable controlled experimentation.
In this study, we probe the reasoning mechanisms of frontier LRMs through the lens of problem
2complexity. Rather than standard benchmarks (e.g., math problems), we adopt controllable puzzle en-
vironments that let us vary complexity systematically—by adjusting puzzle elements while preserving
the core logic—and inspect both solutions and internal reasoning (Fig. 1, top). These puzzles: (1) of-
fer fine-grained control over complexity; (2) avoid contamination common in established benchmarks;
(3) require only the explicitly provided rules, emphasizing algorithmic reasoning; and (4) support
rigorous, simulator-based evaluation, enabling precise solution checks and detailed failure analyses.
Our empirical investigation reveals several key findings about current Language Reasoning Models
(LRMs): First, despite their sophisticated self-reflection mechanisms learned through reinforcement
learning, these models fail to develop generalizable problem-solving capabilities for planning tasks,
with performance collapsing to zero beyond a certain complexity threshold. Second, our comparison
between LRMs and standard LLMs under equivalent inference compute reveals three distinct reason-
ing regimes (Fig. 1, bottom). For simpler, low-compositional problems, standard LLMs demonstrate
greater efficiency and accuracy. As problem complexity moderately increases, thinking models gain
an advantage. However, when problems reach high complexity with longer compositional depth,
both model types experience complete performance collapse (Fig. 1, bottom left). Notably, near
this collapse point, LRMs begin reducing their reasoning effort (measured by inference-time tokens)
as problem complexity increases, despite operating well below generation length limits (Fig. 1,
bottom middle). This suggests a fundamental inference time scaling limitation in LRMs’ reasoning
capabilities relative to problem complexity. Finally, our analysis of intermediate reasoning traces or
thoughts reveals complexity-dependent patterns: In simpler problems, reasoning models often identify
correct solutions early but inefficiently continue exploring incorrect alternatives—an “overthinking”
phenomenon. At moderate complexity, correct solutions emerge only after extensive exploration
of incorrect paths. Beyond a certain complexity threshold, models completely fail to find correct
solutions (Fig. 1, bottom right). This indicates LRMs possess limited self-correction capabilities
that, while valuable, reveal fundamental inefficiencies and clear scaling limitations.
These findings highlight both the strengths and limitations of existing LRMs, raising questions
about the nature of reasoning in these systems with important implications for their design and
deployment. Our key contributions are:
• We question the current evaluation paradigm of LRMs on established math benchmarks and
design a controlled experimental testbed by leveraging algorithmic puzzle environments that enable
controllable experimentation with respect to problem complexity.
• We show that state-of-the-art LRMs (e.g., o3-mini, DeepSeek-R1, Claude-3.7-Sonnet-Thinking)
still fail to develop generalizable problem-solving capabilities, with accuracy ultimately collapsing
to zero beyond certain complexities across different environments.
• We find that there exists a scaling limit in the LRMs’ reasoning effort with respect to problem
complexity, evidenced by the counterintuitive decreasing trend in the thinking tokens after a
complexity point.
• We question the current evaluation paradigm based on final accuracy and extend our evaluation
to intermediate solutions of thinking traces with the help of deterministic puzzle simulators. Our
analysis reveals that as problem complexity increases, correct solutions systematically emerge at
later positions in thinking compared to incorrect ones, providing quantitative insights into the
self-correction mechanisms within LRMs.
• We uncover surprising limitations in LRMs’ ability to perform exact computation, including their
failure to benefit from explicit algorithms and their inconsistent reasoning across puzzle types.
32
Related Works
Reasoning in Language Models. Large Language Models (LLMs) undergo multiple costly
training phases using vast amounts of training data. While these LLMs demonstrate promising
language understanding with strong compression capabilities, their intelligence and reasoning abilities
remain a critical topic of scientific debate [7, 8]. Earlier iterations of LLMs [9, 10, 11] exhibited
poor performance on reasoning benchmarks [12, 13, 14, 6]. To address these shortcomings, several
approaches have been explored with the common theme among them being “scaling” both the training
data and test-time computation. For instance, generating a Chain of Thought (CoT) [15, 16, 17, 18]
and incorporating self-verification [19, 20, 21] prior to the final answer have been shown to improve
model performance. However, obtaining high-quality and scalable CoT data is quite expensive
due to its scarcity. Another line of research focuses on compensating for the lack of supervised
data by teaching models to think more effectively through supervised learning or reinforcement
learning [22, 23, 24, 25, 26, 27]. A notable open-source example of these improvements is Deepseek-
R1 [3], which demonstrated that applying RL with verifiable rewards can significantly enhance model
performance, matching that of closed models like OpenAI’s o1 [2], leading to a new generation of
language models referred to as Large Reasoning Models (LRMs) such as Gemini flash thinking [5],
Claude 3.7 Sonnet thinking [4], etc.
Understanding Large Reasoning Models. Recent studies have explored various aspects of
reasoning behavior: Large Reasoning Models have shown emergent behaviors such as discrepancy
between thought traces and final answers [28, 29] as well as efficiency concerns through what
researchers term the “overthinking phenomenon” [30, 31, 32, 33], where models produce verbose,
redundant outputs, even after finding the solution, creating significant inference computational
overhead. In this work, we systematically analyze how much model thinks w.r.t task complexity.
Recently, Ballon et al. [34] demonstrated that in newer LRMs accuracy generally declines when
thinking increases in math problems, in contrast we observe when in controlled puzzle environment
difficulty passes a certain level the model starts to think less and opposite corelation of thinking and
task complexity only happens up to some threshold. Yue et al. [35] questioned whether reinforcement
learning truly elicits novel reasoning patterns and shows pass@k of reasoning vs non-reasoning models
converge to the same point. We also observe that in MATH-500 pass@k is close for reasoning versus
non-reasoning models but we observed different patterns under medium and high complexity of
puzzles, which is not easily observable on established math benchmarks used in common evaluations.
Controllable Evaluation Environments. Unlike earlier studies that focused on mathematical
problems to evaluate the reasoning capabilities of language models, this work introduces controllable
puzzle environments. These environments allow for precise manipulation of problem complexity while
maintaining consistent logical processes, enabling a more rigorous analysis of reasoning patterns and
limitations. Controllable environments are not uncommon in the literature [12, 36, 37]. However,
our primary aim is not to propose a new benchmark; instead, we use these benchmarks as tools
for designing experiments to understand the reasoning capabilities of language models. A closely
related study by Valmeekam et al. [38] demonstrated that o1-models show significant performance
improvements compared to previous models. Our work offers additional insights, such as examining
pairs of thinking/non-thinking models (e.g., DeepSeek-R1/V3, Claude 3.7 Sonnet thinking/non-
thinking). Furthermore, we study the reasoning traces of the LRMs in more depth, revealing different
behaviors across various complexity levels.
Overall, the promising results from recent LRMs raise a critical question: how much have the
previously reported limitations of LLMs been improved? In this work, we move beyond merely
measuring the performance of these LRMs. We analyze how well these LRMs tackle problems of
varying complexities and examine the properties of their reasoning processes.
4MATH-500
AIME24
pass@k
pass@k
95
90
85
80
claude-3-7-sonnet-thinking
claude-3-7-sonnet-no-thinking
0
5000
10000
15000
20000
25000
30000
100
8080
60
40
claude-3-7-sonnet-thinking
claude-3-7-sonnet-no-thinking
20
0
35000
0
Inference Compute Budget (Tokens)
50000
MATH-500
DeepSeek-R1
DeepSeek-V3
10000
20000
claude-3-7-sonnet-thinking
claude-3-7-sonnet-no-thinking
20
0
200000
0
50000
30000
40000
Inference Compute Budget (Tokens)
100
8080
60
40
0
DeepSeek-R1
DeepSeek-V3
0
20000
40000
60000
80000
150000
200000
AIME25
100
20
100000
Inference Compute Budget (Tokens)
pass@k
pass@k
pass@k
90
0
150000
40
AIME24
95
80
100000
60
Inference Compute Budget (Tokens)
100
85
AIME25
100
pass@k
100
100000
120000
Inference Compute Budget (Tokens)
60
40
DeepSeek-R1
DeepSeek-V3
20
0
0
20000
40000
60000
80000
100000 120000
Inference Compute Budget (Tokens)
Figure 2: Comparative analysis of thinking versus non-thinking models across math benchmarks
reveals inconsistent performance patterns. While results on the MATH-500 dataset show comparable
performance between both model types, the thinking models demonstrate superior performance
on AIME24 and AIME25 benchmarks. Additionally, the observed performance degradation from
AIME24 to AIME25 highlights the vulnerability of these benchmarks to data contamination issues.
3
Math and Puzzle Environments
Currently, it is not clear whether the performance enhancements observed in recent RL-based
thinking models are attributable to increased exposure to established mathematical benchmark
data, to the significantly greater inference compute allocated to thinking tokens, or to reasoning
capabilities developed by RL-based training? Recent studies [35, 39] have explored this question
with established math benchmarks by comparing the upper-bound capabilities (pass@k) of RL-based
thinking models with their non-thinking standard LLM counterparts. They have shown that under
equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable
to thinking models on benchmarks like MATH500 [40] and AIME24 [41]. We also conducted our
comparative analysis of frontier LRMs like Claude-3.7-Sonnet (with vs. without thinking) and
DeepSeek (R1 vs. V3). Our results (shown in Fig. 2) confirm that, on the MATH500 dataset, the
pass@k performance of thinking models is comparable to their non-thinking counterparts when
provided with the same inference token budget. However, we observed that this performance gap
widens on the AIME24 benchmark and widens further on AIME25. This widening gap presents
an interpretive challenge. It could be attributed to either: (1) increasing complexity requiring
more sophisticated reasoning processes, thus revealing genuine advantages of the thinking models
for more complex problems, or (2) reduced data contamination in newer benchmarks (particularly
AIME25). Interestingly, human performance on AIME25 was actually higher than on AIME24
[42, 43], suggesting that AIME25 might be less complex. Yet models perform worse on AIME25
than AIME24—potentially suggesting data contamination during the training of frontier LRMs.
Given these non-justified observations and the fact that mathematical benchmarks do not allow for
controlled manipulation of problem complexity, we turned to puzzle environments that enable more
precise and systematic experimentation.
5Tower of Hanoi
Checkers Jumping
River Crossing
Blocks World
Initial State
moves
Middle State
moves
Target State
Figure 3: Illustration of the four puzzle environments. Columns show the progression from initial
state (top) through intermediate state (middle) to target state (bottom) for puzzles: Tower
of Hanoi (disk transfer across pegs), Checkers Jumping (position swapping of colored tokens), River
Crossing (transporting entities across a river), and Blocks World (stack reconfiguration).
3.1
Puzzle Environments
We evaluate LRM reasoning on four controllable puzzles spanning compositional depth, planning
complexity, and distributional settings. The puzzles are defined below and illustrated in Fig. 3.
Tower of Hanoi is a puzzle featuring three pegs and n disks of different sizes stacked on the first
peg in size order (largest at bottom). The goal is to transfer all disks from the first peg to the third
peg. Valid moves include moving only one disk at a time, taking only the top disk from a peg, and
never placing a larger disk on top of a smaller one. The difficulty in this task can be controlled by
the number of initial disks as the minimum number of required moves with n initial disks will be
2n − 1. However, in this work we do not grade for optimality of final solution and only measuring
the correctness of each move and reaching the target state.
Checker Jumping is a one-dimensional puzzle arranging red checkers, blue checkers, and a single
empty space in a line. The objective is to swap the positions of all red and blue checkers, effectively
mirroring the initial configuration. Valid moves include sliding a checker into an adjacent empty
space or jumping over exactly one checker of the opposite color to land in an empty space. No checker
can move backward in the puzzle process. The complexity of this task can be controlled by the
number of checkers: with 2n checkers, the minimum number of moves required will be (n + 1)2 − 1.
River Crossing is a constraint satisfaction planning puzzle involving n actors and their corresponding
n agents who must cross a river using a boat. The goal is to transport all 2n individuals from the
left bank to the right bank. The boat can carry at most k individuals and cannot travel empty.
Invalid situations arise when an actor is in the presence of another agent without their own agent
present, as each agent must protect their client from competing agents. The complexity of this task
can also be controlled by the number of actor/agent pairs present. For n = 2, n = 3 pairs, we use
boat capacity of k = 2 and for larger number of pairs we use k = 3.
Blocks World is a block-stacking puzzle requiring rearrangement of blocks from an initial configu-
ration into a specified goal configuration. The objective is to find the minimum number of moves
needed for this transformation. Valid moves are restricted to the topmost block of any stack, which
can be placed either on an empty stack or on top of another block. The complexity in this task can
be controlled by the number of blocks present.
6Figure 4: Accuracy of thinking models (Claude 3.7 Sonnet with thinking, DeepSeek-R1) versus their
non-thinking counterparts (Claude 3.7 Sonnet, DeepSeek-V3) across all puzzle environments and
varying levels of problem complexity.
4Experiments & Results
4.1Experimental Setup
Most of our experiments are conducted on reasoning models and their non-thinking counterparts,
such as Claude 3.7 Sonnet (thinking/non-thinking) and DeepSeek-R1/V3. We chose these models
because they allow access to the thinking tokens, unlike models such as OpenAI’s o-series. For
experiments focused solely on final accuracy, we also report results on the o-series models. For Claude
3.7 Sonnet models, we allow the maximum token budget (64k). Similarly, for DeepSeek-R1/V3
models on local servers, we allow the maximum length to be up to 64k tokens. For each puzzle
instance, we generate 25 samples and report the average performance of each model across them.
Comprehensive details of our experimental setup and results are provided in the Appendix.
4.2How Does Complexity Affect Reasoning?
4.2.1Three Regimes of Complexity
Motivated by the observations in Fig. 2, to systematically investigate the impact of problem complexity
on reasoning behavior, we conducted experiments comparing thinking and non-thinking model
pairs across our controlled puzzle environments. Our analysis focused on matched pairs of LLMs
with identical model backbones, specifically Claude-3.7-Sonnet (w. vs. w/o thinking) and DeepSeek
(R1 vs. V3). In each puzzle, we vary the complexity by manipulating problem size N (representing
disk count, checker count, block count, or crossing elements).
Fig. 4 presents the accuracy of both model types as a function of problem complexity across all
puzzle environments. Complementing this, Fig. 5 shows the upper bound performance capabilities
(pass@k) of these model pairs under equivalent inference token compute (averaged across all puzzles),
extending earlier analyses from mathematical benchmarks (Fig. 2) to the controlled puzzle environ-
ments. Results from both these figures demonstrate that, unlike observations from math, there exists
three regimes in the behavior of these models with respect to complexity. In the first regime where
problem complexity is low, we observe that non-thinking models are capable to obtain performance
comparable to, or even better than thinking models with more token-efficient inference. In the
7Figure 5: Pass@k performance of thinking vs. non-thinking models across equivalent compute
budgets in puzzle environments of low , medium , and high complexity. Non-thinking models excel
in simple problems, thinking models show advantages at medium complexity, while both approaches
fail at high complexity regardless of compute allocation.
second regime with medium complexity, the advantage of reasoning models capable of generating
long chain-of-thought begin to manifest, and the performance gap between model pairs increases. The
most interesting regime is the third regime where problem complexity is higher and the performance
of both models have collapsed to zero. Results show that while thinking models delay this collapse,
they also ultimately encounter the same fundamental limitations as their non-thinking counterparts.
4.2.2
Collapse of Reasoning Models
We next examine how different specialized reasoning models equipped with thinking tokens respond
to increasing problem complexity. Our experiments evaluate five state-of-the-art thinking models:
o3-mini (medium and high configurations), DeepSeek-R1, DeepSeek-R1-Qwen-32B, and Claude-3.7-
Sonnet (thinking). Fig. 6 demonstrates these models’ performance in terms of accuracy (top) and
thinking token usage (bottom) across varying complexity levels. Results show that all reasoning
models exhibit a similar pattern with respect to complexity: accuracy progressively declines as
problem complexity increases until reaching complete collapse (zero accuracy) beyond a model-
specific complexity threshold. Analysis of inference thinking token compute also reveals an intriguing
pattern in thinking token allocation learned by these models. We observe that reasoning models
initially increase their thinking tokens proportionally with problem complexity. However, upon
approaching a critical threshold—which closely corresponds to their accuracy collapse point—models
counterintuitively begin to reduce their reasoning effort despite increasing problem difficulty. This
phenomenon is most pronounced in o3-mini variants and less severe in the Claude-3.7-Sonnet
(thinking) model. Notably, despite operating well below their generation length limits with ample
inference budget available, these models fail to take advantage of additional inference compute during
the thinking phase as problems become more complex. This behavior suggests a fundamental scaling
limitation in the thinking capabilities of current reasoning models relative to problem complexity.
8Figure 6: Accuracy and thinking tokens vs. problem complexity for reasoning models across puzzle
environments. As complexity increases, reasoning models initially spend more tokens while accuracy
declines gradually, until a critical point where reasoning collapses—performance drops sharply and
reasoning effort decreases.
4.3
What Happens Inside the Thoughts of Reasoning Models?
To gain deeper insights into the thinking processes of reasoning models, we conducted a fine-grained
analysis of their reasoning traces. As shown in Fig. 1, our setup with puzzle environments allows us
to look beyond final answer and obtain more detailed insight into the reasoning traces (“thoughts”)
produced by these models. We extract and analyze the intermediate solutions explored within the
thoughts of a model with the help of puzzle simulators. Our investigation examines the patterns and
characteristics of these intermediate solutions, their correctness relative to their sequential position
in the reasoning process, and how these patterns evolve with increasing problem complexity. For
this analysis, we focus on the reasoning traces generated by Claude-3.7-Sonnet-Thinking across
our puzzle suite. For each intermediate solution identified within the traces, we recorded: (1) its
relative position within the reasoning trace (normalized by total thought length), (2) its correctness
as validated by our puzzle simulators, and (3) the complexity of the corresponding problem. This
allows to characterize the progression and accuracy of solution development throughout the reasoning
process.
Fig. 7a demonstrates the relation between the position of intermediate solutions within thoughts, their
correctness, and problem complexity across all puzzle environments. Our analysis from reasoning
traces also further validates three regimes of complexity discussed above. For simpler problems,
reasoning models often find the correct solution early in their thinking but then continue exploring
incorrect solutions. Note the distribution of incorrect solutions (red) is shifted more upward towards
end of thinking compared to correct solutions (green). This phenomenon, referred to as “overthinking”
in the literature, leads to the waste of compute. As problems become moderately more complex,
this trend reverses: models first explore incorrect solutions and mostly later in thought arrive at
the correct ones. This time the distribution of incorrect solutions (red) is shifted more downward
compared to correct ones (green). Finally, for the problems with higher complexity, collapse emerges,
9Tower of Hanoi
Solution Accuracy (%)
100
N=1
N=2
N=3
N=4
N=5
N=6
N=7
N=8
N=10
80
60
40
20
0
0
4000
8000
12000
Position in Thinking (Token)
(a)
(b)
Figure 7: Left & Middle: Position and correctness of intermediate solutions within reasoning traces
across four puzzles at varying complexity levels. ✓ indicates correct solutions, ✗ indicates incorrect
solutions, with distribution density shown by shading; Right: Solution accuracy versus position
in thinking for Tower of Hanoi at different complexity levels. Simple problems (N=1-3) show early
accuracy declining over time (overthinking), moderate problems (N=4-7) show slight improvement
in accuracy with continued reasoning, and complex problems (N≥8) exhibit consistently near-zero
accuracy, indicating complete reasoning failure.
meaning that the model fails to generate any correct solutions within the thought.
Fig. 7b presents a complementary analysis of solution accuracy within sequential segments (bins)
of the thoughts in the Tower of Hanoi environment. It can be observed that for simpler problems
(smaller N), solution accuracy tends to decrease or oscillate as thinking progresses, providing further
evidence of the overthinking phenomenon. However, this trend changes for more complex problems,
where solution accuracy increases with thinking progression—up to a certain threshold. Beyond this
complexity threshold, in the “collapse mode”, accuracy is zero.
4.4
Open Questions: Puzzling Behavior of Reasoning Models
In this section, we present surprising results concerning the limitations of reasoning models in
executing exact problem-solving steps, as well as demonstrating different behaviors of the models
based on the number of moves.
As shown in Figures 8a and 8b, in the Tower of Hanoi environment, even when we provide the
algorithm in the prompt—so that the model only needs to execute the prescribed steps—performance
does not improve, and the observed collapse still occurs at roughly the same point. This is noteworthy
because finding and devising a solution should require substantially more computation (e.g., for search
and verification) than merely executing a given algorithm. This further highlights the limitations of
reasoning models in verification and in following logical steps to solve a problem, suggesting that
further research is needed to understand the symbolic manipulation capabilities of such models [44, 6].
Moreover, in Figures 8c and 8d, we observe very different behavior from the Claude 3.7 Sonnet think-
ing model. In the Tower of Hanoi environment, the model’s first error in the proposed solution often
occurs much later, e.g., around move 100 for (N=10), compared to the River Crossing environment,
where the model can only produce a valid solution until move 4. Note that this model also achieves
near-perfect accuracy when solving the Tower of Hanoi with (N=5), which requires 31 moves, while
it fails to solve the River Crossing puzzle when (N=3), which has a solution of 11 moves. This likely
suggests that examples of River Crossing with N>2 are scarce on the web, meaning LRMs may not
have frequently encountered or memorized such instances during training.
1040
80
60
40
2020
00
1 2 3 4 5 6 7 8 9 10
15
Complexity (Number of Disks)
(a)
20
Tower of Hanoi
Claude-3.7-Sonnet (thinking)
Algorithm Given
Default
1 2 3 4 5 6 7 8 9 10
15
20
Complexity (Number of Disks)
100
80
60
40
20
0
Claude-3.7-Sonnet (thinking)
1 2 3 4 5 6 7 8 9 10
15
Complexity (Number of Disks)
(b)
(c)
20
First Wrong Move (Median)
60
100
Accuracy (%)
80
Accuracy (%)
Tower of Hanoi
DeepSeek-R1
Algorithm Given
Default
First Wrong Move (Median)
Tower of Hanoi
100
River Crossing
10
Claude-3.7-Sonnet (thinking)
8
6
4
2
0
2 3 4 5 6
8
10
15
20
Complexity (Number of People)
(d)
Figure 8: (a) & (b) Despite providing the solution algorithm in the prompt, execution failure
occurs at similar points, highlighting reasoning model limitations in logical step execution. (c) &
(d) Notably, the Claude 3.7 Sonnet model demonstrates much longer error-free sequences in the
Tower of Hanoi compared to early errors in the River Crossing scenario.
5
Conclusion
In this paper, we systematically examine frontier Large Reasoning Models (LRMs) through the lens
of problem complexity using controllable puzzle environments. Our findings reveal fundamental
limitations in current models: despite sophisticated self-reflection mechanisms, these models fail to
develop generalizable reasoning capabilities beyond certain complexity thresholds. We identified
three distinct reasoning regimes: standard LLMs outperform LRMs at low complexity, LRMs excel at
moderate complexity, and both collapse at high complexity. Particularly concerning is the counterin-
tuitive reduction in reasoning effort as problems approach critical complexity, suggesting an inherent
compute scaling limit in LRMs. Our detailed analysis of reasoning traces further exposed complexity-
dependent reasoning patterns, from inefficient “overthinking” on simpler problems to complete failure
on complex ones. These insights challenge prevailing assumptions about LRM capabilities and
suggest that current approaches may be encountering fundamental barriers to generalizable reasoning.
Finally, we presented some surprising results on LRMs that lead to several open questions for future
work. Most notably, we observed their limitations in performing exact computation; for example,
when we provided the solution algorithm for the Tower of Hanoi to the models, their performance
on this puzzle did not improve. Moreover, investigating the first failure move of the models revealed
surprising behaviors. For instance, they could perform up to 100 correct moves in the Tower of
Hanoi but fail to provide more than 5 correct moves in the River Crossing puzzle. We believe our
results can pave the way for future investigations into the reasoning capabilities of these systems.
Limitations
We acknowledge that our work has limitations. While our puzzle environments enable controlled
experimentation with fine-grained control over problem complexity, they represent a narrow slice of
reasoning tasks and may not capture the diversity of real-world or knowledge-intensive reasoning
problems. It is notable that most of our experiments rely on black-box API access to the closed frontier
LRMs, limiting our ability to analyze internal states or architectural components. Furthermore, the
use of deterministic puzzle simulators assumes that reasoning can be perfectly validated step by
step. However, in less structured domains, such precise validation may not be feasible, limiting the
transferability of this analysis to other more generalizable reasoning.
11Acknowledgments
The authors would like to thank Scott Hoang, Yichen Jiang, Minsik Cho, Mohammad Sekhavat, David
Harrison, Mohammadreza Armandpour and Devi Krishna for the valuable feedback and support.
References
[1] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec
Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv
preprint arXiv:2412.16720, 2024.
[2] OpenAI. Introducing openai o1. Jan 2024.
[3] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu,
Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms
via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[4] Anthropic. Claude 3.7 sonnet. Feb 2025.
[5] Google. Gemini flash thinking. Google AI Blog, Jan 2025.
[6] Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio,
and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical
reasoning in large language models. In The Thirteenth International Conference on Learning
Representations, 2025.
[7] Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2:
A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831, 2025.
[8] Gary Marcus. Five ways in which the last 3 months — and especially the deepseek era — have
vindicated "deep learning is hitting a wall". Marcus on AI (Substack), February 2025. Blog
post.
[9] Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany
Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, and et. al. Phi-3
technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219,
2024.
[10] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap-
lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier,
Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril,
Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825,
2023.
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle,
Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony
Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark,
Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière,
Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris
Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong,
Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny
12Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino,
Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael
Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson,
Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar,
Hu Xu, Hugo Touvron, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024.
[12] Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean
Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal,
Xiang Ren, Allyson Ettinger, Zaïd Harchaoui, and Yejin Choi. Faith and fate: Limits of
transformers on compositionality. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko,
Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems
36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New
Orleans, LA, USA, December 10 - 16, 2023, 2023.
[13] R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths.
Embers of autoregression: Understanding large language models through the problem they are
trained to solve, 2023.
[14] Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. Alice in wonderland:
Simple tasks showing complete reasoning breakdown in state-of-the-art large language models.
arXiv preprint arXiv:2406.02061, 2024.
[15] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi,
Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language
models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh,
editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural
Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -
December 9, 2022, 2022.
[16] Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. Lam-
bada: Backward chaining for automated reasoning in natural language. arXiv preprint
arXiv:2212.13894, 2022.
[17] Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie
Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066,
2022.
[18] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large
language models are zero-shot reasoners. Advances in neural information processing systems,
35:22199–22213, 2022.
[19] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and
Jun Zhao. Large language models are better reasoners with self-verification. In Houda Bouamor,
Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics:
EMNLP 2023, pages 2550–2575, Singapore, December 2023. Association for Computational
Linguistics.
[20] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.
Making language models better reasoners with step-aware verifier. In Proceedings of the 61st
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pages 5315–5333, 2023.
13[21] Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. Sample, scrutinize and scale: Effective
inference-time search by scaling verification. arXiv preprint arXiv:2502.01839, 2025.
[22] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning
with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,
Advances in Neural Information Processing Systems, 2022.
[23] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and
Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens.
In The Twelfth International Conference on Learning Representations, 2024.
[24] David Herel and Tomas Mikolov. Thinking tokens for language modeling. ArXiv, abs/2405.08644,
2024.
[25] Zhihong Shao, Peiyi Wang, Runxin Xu Qihao Zhu, Junxiao Song, Mingchuan Zhang, Y.K. Li,
Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open
language models, 2024.
[26] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy,
Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning
through refined credit assignment, 2024.
[27] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze
Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya
Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris
Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha-
jishirzi. Tülu 3: Pushing frontiers in open language model post-training. ArXiv, abs/2411.15124,
2024.
[28] Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John
Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, et al. Reasoning models
don’t always say what they think. arXiv preprint arXiv:2505.05410, 2025.
[29] Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh
Hakhamaneshi, Shishir G Patil, Matei Zaharia, et al. Llms can easily learn to reason from
demonstrations structure, not content, is what matters! arXiv preprint arXiv:2502.07374, 2025.
[30] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi
Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the
overthinking of o1-like llms. arXiv preprint arXiv:2412.21187, 2024.
[31] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi
Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning
for large language models. arXiv preprint arXiv:2503.16419, 2025.
[32] Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad
BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han
Lù, et al. Deepseek-r1 thoughtology: Let’s< think> about llm reasoning. arXiv preprint
arXiv:2504.07128, 2025.
[33] Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching,
Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement
fine-tuning. arXiv preprint arXiv:2503.07572, 2025.
14[34] Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and
performance in large language models–o3 (mini) thinks harder, not longer. arXiv preprint
arXiv:2502.15631, 2025.
[35] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does
reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
arXiv preprint arXiv:2504.13837, 2025.
[36] Benjamin Estermann, Luca A. Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer.
Puzzles: A benchmark for neural algorithmic reasoning, 2024.
[37] Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati.
Large language models still can’t plan (A benchmark for llms on planning and reasoning about
change). CoRR, abs/2206.10498, 2022.
[38] Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Llms still can’t plan; can
lrms? a preliminary evaluation of openai’s o1 on planbench. 2024.
[39] Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning
models can be effective without thinking. arXiv preprint arXiv:2504.09858, 2025.
[40] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan
Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint
arXiv:2305.20050, 2023.
[41] Mathematical
Association
of
America.
American
invitational
math-
ematics
examination
(aime).
https://maa.org/math-competitions/
american-invitational-mathematics-examination-aime, 2025. Accessed: 2025-05-15.
[42] Art of Problem Solving.
Amc historical results - aime i (february 1, 2024).
https://artofproblemsolving.com/wiki/index.php/AMC_historical_results#AIME_
I_.28February_1.2C_2024.29, 2024. Accessed: 2025-05-15.
Amc historical results – aime i (february 6, 2025).
[43] Art of Problem Solving.
https://artofproblemsolving.com/wiki/index.php/AMC_historical_results#AIME_
I_.28February_6.2C_2025.29, 2025. Accessed: 2025-05-15.
[44] Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT
press, 2003.
[45] Saul Amarel. On representations of problems of reasoning about actions. In Readings in artificial
intelligence, pages 2–22. Elsevier, 1981.
[46] Günter Rote. Crossing the bridge at night. Bulletin of the EATCS, 78:241, 2002.
15A
Appendix
In this appendix, we provide details supplementing the main text, including experimental setup
specifications, additional results, and extended analysis.
A.1 Details on Puzzle Environment Specifications and Design - Comprehensive descriptions of all
four puzzle environments, including their problem descriptions, prompt designs, and simulators.
A.1.1 Tower of Hanoi
A.1.2 Checker Jumping
A.1.3 River Crossing
A.1.4 Blocks World
A.2 Implementation Details - Full experimental setup specifications, model configurations, extrac-
tion pipeline details, and prescribed algorithm execution experiments.
A.3 Details on Computational Complexity
A.3.1 Compositional Depth Characterization
A.3.2 Performance vs Compositional Depth
A.4 Additional Results and Analysis - Extended analysis including reasoning effort patterns, and
detailed failure analysis across all models and puzzle environments.
A.1Details on Puzzle Environment Specifications and Design
A.1.1Tower of Hanoi
Problem Description. The Tower of Hanoi is a classic recursive puzzle that serves as a great
problem for evaluating sequential reasoning and planning capabilities in reasoning models. The
puzzle consists of three pegs (labeled 0, 1, and 2 from left to right) and N disks of varying sizes,
where each disk is uniquely numbered from 1 (smallest) to N (largest). In the initial configuration,
all N disks are stacked on the leftmost peg (peg 0) in descending order of size, with the largest disk
at the bottom and the smallest at the top. The remaining two pegs (1 and 2) are initially empty.
The goal is to transfer all disks from peg 0 to peg 2, maintaining the same size ordering (largest
at bottom, smallest at top). This puzzle is governed by three fundamental constraints: (1) Single
Disk Movement: Only one disk may be moved at a time; (2) Top Disk Access: Only the topmost
disk from any peg can be selected for movement; and (3) Size Ordering Constraint: A larger disk
may never be placed on top of a smaller disk. This puzzle is a good evaluation testbed for reasoning
and planning capabilities of models as it requires models to demonstrate key cognitive demands
such as breaking down the problem into subproblems (recursive thinking), tracking multiple states
and disk positions simultaneously (working memory management), adhering to movement rules and
constraints while planning ahead (constraint satisfaction), and determining the correct order of
operations to achieve the final goal (sequential planning).
The minimum number of moves required to solve the Tower of Hanoi recursive puzzle with N disks
is 2N − 1, making it an exponentially scaling problem. This property allows for fine-grained difficulty
control by adjusting the problem size with number of initial disks. However, in our evaluation
framework, we focus on solution correctness rather than optimality, assessing each of the move’s
validity and the model’s ability to reach the target state as the success criteria.
16Prompt Design. The system prompt begins with a clear problem statement describing the puzzle
setup. It explicitly states the movement rules and the objective of transferring all disks to the third
peg. To facilitate understanding, the prompt includes example demonstrations as well as the critical
formatting and reasoning expectations.
System Prompt - Tower of Hanoi
You are a helpful assistant. Solve this puzzle for me.
There are three pegs and n disks of different sizes stacked on the first peg. The disks are
numbered from 1 (smallest) to n (largest). Disk moves in this puzzle should follow:
1. Only one disk can be moved at a time.
2. Each move consists of taking the upper disk from one stack and placing it on top of
another stack.
3. A larger disk may not be placed on top of a smaller disk.
The goal is to move the entire stack to the third peg.
Example: With 3 disks numbered 1 (smallest), 2, and 3 (largest), the initial state is [[3, 2, 1],
[], []], and a solution might be:
moves = [[1 , 0 , 2] , [2 , 0 , 1] , [1 , 2 , 1] , [3 , 0 , 2] ,
[1 , 1 , 0] , [2 , 1 , 2] , [1 , 0 , 2]]
This means: Move disk 1 from peg 0 to peg 2, then move disk 2 from peg 0 to peg 1, and so on.
Requirements:
• When exploring potential solutions in your thinking process, always include the corre-
sponding complete list of moves.
• The positions are 0-indexed (the leftmost peg is 0).
• Ensure your final answer includes the complete list of moves in the format:
moves = [[disk id, from peg, to peg], ...]
The user prompt after the system prompt presents the specific puzzle instance with current configu-
ration showing the distribution of disks across pegs and the goal configuration specifying the target
state.
User Prompt Template for $N$ Disks - Tower of Hanoi
I have a puzzle with $N$ disks of different sizes with
Initial configuration:
• Peg 0: $N$ (bottom), . . . 2, 1 (top)
• Peg 1: (empty)
• Peg 2: (empty)
17Goal configuration:
• Peg 0: (empty)
• Peg 1: (empty)
• Peg 2: $N$ (bottom), . . . 2, 1 (top)
Rules:
• Only one disk can be moved at a time.
• Only the top disk from any stack can be moved.
• A larger disk may not be placed on top of a smaller disk.
Find the sequence of moves to transform the initial configuration into the goal configuration.
Simulator. Our evaluation framework employs separate puzzle simulators for each puzzle to
ensure rigorous and consistent assessment of solutions obtained from LRMs. The Tower of Hanoi
simulator is designed as a stateful environment that tracks disk configurations across three pegs
and validates each proposed move against the puzzle’s fundamental constraints. The simulator
architecture follows a modular design pattern with clear separation between state management,
move validation, and solution verification. In this simulator, we have a puzzle class which tracks the
current disk configuration and enforces the puzzle’s fundamental constraints. We also have a method
to execute each move in the puzzle setup and perform four-layer validation: checking peg boundary
conditions (0-2), verifying source pegs contain disks, confirming the specified disk is topmost, and
enforcing the size ordering constraint that prevents larger disks from being placed on smaller ones.
Upon successful validation, the method executes the disk transfer and updates the game state. Then,
the complete solution validation is processed by sequentially processing move lists, and verifying
goal state achievement.
A.1.2
Checker Jumping
Problem Description. Checker Jumping is a one-dimensional constraint-satisfaction puzzle
designed to test sequential reasoning, planning, and rule understanding capabilities. The puzzle
consists of a linear arrangement of red checkers (’R’), blue checkers (’B’), and a single empty space
(’_’). In the standard configuration, N red checkers are positioned on the left side, followed by an
empty space in the middle, and N blue checkers on the right side, forming a linear board of length
2N + 1. The objective is to swap the positions of all red and blue checkers, effectively mirroring the
initial configuration, where red checkers end up on the right and blue checkers on the left. Movement
in this puzzle is governed by two fundamental rules: (1) Slide Movement: A checker can slide
forward into an adjacent empty space; and (2) Jump Movement: A checker can jump forward over
exactly one checker of the opposite color to land in an empty space. Therefore, checkers cannot
move backward toward their starting side—red checkers can only move rightward, and blue checkers
can only move leftward from the initial configuration. This puzzle presents cognitive challenges that
make it a great testbed for reasoning models. For example, models must demonstrate some aspect of
spatial reasoning (tracking checker positions and possible moves), constraint satisfaction (adhering
to movement rules during puzzle), lookahead planning (anticipating how current moves affect future
18possibilities towards goal), and state-space exploration (searching through possible move sequences
to find a valid solution path).
The difficulty of the Checker Jumping puzzle scales with the number of checkers: with N checkers of
each color, the minimum solution requires (N + 1)2 − 1 moves, creating a quadratic relationship
between problem size and solution complexity. In our evaluation framework, we mainly focus on
solution correctness rather than optimality, evaluating each move against the puzzle constraints and
confirming that the final state matches the goal configuration. This approach allows us to precisely
identify reasoning failures and constraint violations that might occur during the solution process.
Prompt Design. The system prompt begins with a clear problem statement describing the puzzle
setup and movement rules. It explicitly states the objective and provides a concrete example with a
small board configuration to illustrate how moves should be represented.
System Prompt - Checker Jumping
You are a helpful assistant. Solve this puzzle for me.
On a one-dimensional board, there are red checkers (’R’), blue checkers (’B’), and one empty
space (’_’). A checker can move by either:
1. Sliding forward into an adjacent empty space, or
2. Jumping over exactly one checker of the opposite color to land in an empty space.
The goal is to swap the positions of all red and blue checkers, effectively mirroring the initial
state.
Example: If the initial state is [’R’, ’_’, ’B’], the goal is to reach [’B’, ’_’, ’R’]. Your solution
should be a list of moves where each move is represented as [checker_color, position_from,
position_to]. For example:
moves = [[ ’R ’ , 0 , 1] , [ ’B ’ , 2 , 0] , [ ’R ’ , 1 , 2]]
This means: Move the red checker from position 0 to 1, then move the blue checker from
position 2 to 0, and so on.
Requirements:
• When exploring potential solutions in your thinking process, always include the corre-
sponding complete list of moves.
• The positions are 0-indexed (the leftmost position is 0).
• Ensure your final answer includes the complete list of moves for final solution in the
format: moves = [[checker_color, position_from, position_to], ...]
The user prompt presents the specific puzzle instance with the initial board configuration, and the
goal state.
19User Prompt Template for $N$ Checkers - Checker Jumping
I have a puzzle with 2$N$+1 positions, where $N$ red checkers (’R’) on left, $N$ blue checkers
(’B’) on right, and one empty space (’_’) in between are arranged in a line.
Initial board: R R ... R _ B B ... B
Goal board: B B ... B _ R R ... R
Rules:
• A checker can slide into an adjacent empty space.
• A checker can jump over exactly one checker of the opposite color to land in an empty
space.
• Checkers cannot move backwards (towards their starting side).
Find the minimum sequence of moves to transform the initial board into the goal board.
Simulator. Our evaluation framework employs a custom simulator for validating Checker Jumping
puzzle solutions. The simulator implements a comprehensive validation system that enforces all
puzzle constraints while tracking the state evolution throughout the solution path. The Checker
Jumping simulator is designed as a stateful environment that tracks the position of all checkers and
the empty space, validating each move of a given solution against the puzzle’s movement rules. The
simulator begins by validating that both the initial and goal states are well-formed, containing the
same number of red and blue checkers and exactly one empty space. Then, each move is executed
with a method that performs multi-layer validation: verifying position boundaries, confirming correct
checker color at source, ensuring target positions are empty, and validating move types as either
slides (distance=1) or jumps (distance=2). The simulator enforces directional constraints preventing
backward movement (red checkers move right, blue checkers move left) and validates jump moves
by confirming the presence of an opposite-colored checker in the middle position. Upon successful
validation, the method executes the checker transfer by updating positions and clearing the source.
Then, the complete move sequences are processed with final goal state verification.
A.1.3
River Crossing
Problem Description. River Crossing is a constraint satisfaction planning puzzle that tests multi-
agent coordination and constraint management. This puzzle is a generalization of classic problems
such as the Missionaries and Cannibals problem and the Bridge and Torch problem, which have been
widely studied in planning literature [45, 46]. The river crossing puzzle involves N actors (denoted by
a1 , a2 , ..., aN ) and their corresponding N agents (denoted by A1 , A2 , ..., AN ) who must cross a river us-
ing a boat. In the initial state, all 2N individuals are on the left bank of the river. The goal is to trans-
port everyone safely to the right bank. The puzzle operates under several key movement constraints:
(1) Boat Capacity Constraint: The boat can carry at most k individuals at a time, where k is typically
set to 2 for smaller puzzles (N ≤ 3) and 3 for larger puzzles (N ≤ 5); (2) Non-Empty Boat Constraint:
The boat cannot travel empty and must have at least one person aboard; (3) Safety Constraint: An
actor cannot be in the presence of another agent unless their own agent is also present, as agents must
protect their clients from competing agents. This safety constraint applies both on the banks and in
the boat. This puzzle requires complex planning and state tracking as participants must carefully coor-
dinate their crossings while maintaining safety constraints at all times. The solver must reason through
20different combinations of individuals who can safely travel together, determine who should return with
the boat after a crossing, and strategically plan a sequence that eventually brings everyone to the right
bank without violating any constraints. The complexity of this task can be controlled by adjusting the
number of actor-agent pairs and the boat capacity, creating a scalable challenge for reasoning models.
Prompt Design. The system prompt introduces the notation for representing actors and agents,
establishes the solution format as a list of boat moves, and provides a simple example to demonstrate
the format.
System Prompt - River Crossing
You are a helpful assistant. Solve this puzzle for me.
You can represent actors with a_1, a_2, ... and agents with A_1, A_2, ... . Your solution
must be a list of boat moves where each move indicates the people on the boat. For example, if
there were two actors and two agents, you should return:
moves =[[" A_2 " , " a_2 "] , [" A_2 "] , [" A_1 " , " A_2 "] , [" A_1 "] , [" A_1 " , " a_1
"]]
which indicates that in the first move, A_2 and a_2 row from left to right, and in the second
move, A_2 rows from right to left and so on.
Requirements:
• When exploring potential solutions in your thinking process, always include the corre-
sponding complete list of boat moves.
• The list shouldn’t have comments.
• Ensure your final answer also includes the complete list of moves for final solution.
The user prompt presents the specific puzzle instance with N actor-agent pairs, and the boat capacity
k, and the safety constraint that must be maintained throughout the solution.
User Prompt Template for $N$ Pairs - River Crossing
$N$ actors and their $N$ agents want to cross a river in a boat that is capable of holding
only $k$ people at a time, with the constraint that no actor can be in the presence
of another agent, including while riding the boat, unless their own agent is also
present, because each agent is worried their rivals will poach their client. Initially, all actors
and agents are on the left side of the river with the boat. How should they cross the river?
(Note: the boat cannot travel empty)
Simulator. Our evaluation framework employs a custom simulator for validating River Crossing
puzzle extracted solutions. The simulator tracks the state of all individuals (actors and agents) and
the boat position while enforcing all puzzle constraints. Each move is executed with multi-step
validation: checking boat capacity limits, verifying all passengers are on the boat’s current side,
and enforcing the critical safety constraint that actors cannot be in the presence of other agents
without their own agent present, both on the boat and on each bank after the move. The simulator
21manages dynamic boat positioning, automatically switching sides after each crossing, and validates
the complete state after each move to ensure no safety violations occur on either bank. Then, the
complete crossing sequences are verified that all 2N individuals successfully reach the right bank.
A.1.4
Blocks World
Problem Description. Blocks World is a classical planning puzzle that has been recently studied
for analyzing the planning capabilities of LLMs [37, 38]. The puzzle involves multiple stacks of blocks
(A, B, C, etc.) that must be rearranged from an initial configuration to a specified goal configuration.
Each block is uniquely identified by its letter, and the objective is to find the minimum sequence of
moves needed to transform the initial state into the goal state. The puzzle operates only under two
fundamental constraints: (1) Top Block Movement: Only the topmost block from any stack can be
moved; and (2) Valid Placement: A block can only be placed either on an empty position or on top
of another block. These constraints create planning problem where the order of operations becomes
critical, as some configurations may require temporary placement of blocks to access those beneath
them later. Blocks World serves as a great testbed for evaluating planning capabilities in reasoning
models because it requires forward thinking, and state tracking. Recent studies have examined this
puzzle in various configurations, including simplified settings with as few as 3 to 5 blocks, to evaluate
LLM performance on sequential planning tasks [37, 38]. Models must demonstrate the ability to
decompose complex state transformations into valid sequential moves, reason about dependencies
between blocks (e.g., unblocking lower blocks before accessing them), and efficiently plan paths to
the goal state without illegal moves.
The difficulty of this puzzle can be scaled by adjusting several parameters: the number of blocks, the
number of stacks, and the complexity of the initial and goal configurations. We primarily control
complexity through the block count N , while following clear structural patterns in the initial and
goal configurations. In our experimental design, the initial configuration consistently divides the
N blocks between two stacks in alphabetical order, with the third stack empty as workspace. The
goal configuration consolidates all blocks onto the first stack in a systematic interleaved pattern
that alternates between blocks from the two initial stacks, with specific positioning that requires
complete disassembly and reassembly of the existing stacks. For example, for N = 4, the initial
state has blocks divided between two stacks [["A", "B"], ["C", "D"], []] and the goal state
[["D", "B", "C", "A"], [], []] requires interleaving blocks from both stacks; and for N = 6,
the initial state [["A", "B", "C"], ["D", "E", "F"], []] must be transformed to [["F", "C",
"E", "B", "D", "A"], [], []], forming a complex alternating pattern. As N increases, the state
space grows factorially, and the minimum solution length increases approximately linearly with
N . For small values of N (2-7), the puzzles test basic planning; for medium values (8-20), they
require more complex reasoning with longer planning horizons; and for large values (N > 20), they
challenge the limits of sequential reasoning capabilities by requiring extensive temporary movements
and pattern recognition across lengthy solution paths.
Prompt Design. The system prompt introduces the fundamental rules of the Blocks World puzzle,
establishes the move representation format, and provides a simple example to demonstrate the
solution structure.
22System Prompt - Blocks World
You are a helpful assistant. Solve this puzzle for me.
In this puzzle, there are stacks of blocks, and the goal is to rearrange them into a target
configuration using a sequence of moves where:
• Only the topmost block from any stack can be moved.
• A block can be placed either on an empty position or on top of another block.
Example: With initial state [["A", "B"], ["C"], []] and goal state [["A"], ["B"],
["C"]], a solution might be:
moves = [[" C " , 1 , 2] , [" B " , 0 , 1]]
This means: Move block C from stack 1 to stack 2, then move block B from stack 0 to stack 1.
Requirements:
• When exploring potential solutions in your thinking process, always include the corre-
sponding complete list of moves.
• Ensure your final answer also includes the complete list of moves for final solution in the
format: moves = [[block, from stack, to stack], ...]
The user prompt presents the specific puzzle instance with the initial and goal configurations provided,
and explicitly reminds the model about the movement constraint.
User Prompt Template for $N$ Blocks - BlocksWorld
I have a puzzle with $N$ blocks.
Initial state:
Stack 0: $blocks_0$ (top)
Stack 1: $blocks_1$ (top)
...
Stack $m$: $blocks_m$ (top)
Goal state:
Stack 0: $goal_blocks_0$ (top)
Stack 1: $goal_blocks_1$ (top)
...
Stack $m$: $goal_blocks_m$ (top)
Find the minimum sequence of moves to transform the initial state into the goal state. Remember
that only the topmost block of each stack can be moved.
23Simulator. Our evaluation framework employs a custom simulator for validating Blocks World
puzzle extracted solutions. The simulator manages the state of all blocks across stacks while enforcing
the puzzle’s movement constraints. Each move is executed in the puzzle setup with three-layer
validation: verifying stack indices are within bounds, confirming the source stack contains blocks,
and ensuring the specified block is at the top of its stack (enforcing the top-block-only movement
rule). Upon successful validation, the block transfer is executed and the block is popped from the
source stack and appended to the destination stack. Finally, the complete solution sequences of block
movements are processed and verified that the resulting configuration matches the target goal state.
A.2
Implementation Details
Configurations Our experiments primarily utilized reasoning models and their non-thinking
counterparts to enable thorough analysis of the thinking process. We specifically selected Claude
3.7 Sonnet (thinking/non-thinking) and DeepSeek-R1/V3 due to their ability to provide access to
thinking traces, a critical requirement for our analysis. For experiments focused solely on final
accuracy metrics, we also included results from OpenAI’s o3-mini models, as they lack access to
thoughts. For Claude 3.7 Sonnet (thinking and non-thinking) models we used maximum generation
budget of 64,000 tokens, accessed through the API interface. Temperature is set to 1.0 for all API
rus (Claude-3.7-Sonnet and o3-mini runs). The experiments with DeepSeek-R1, DeepSeek-V3, and
DeepSeek-R1-Distill-Qern-32B are conducted on local servers with maximum generation length set to
64,000 and temperature set to 1.0. In all experiments, we generated 25 samples per puzzle instance
at each complexity level (N value) and reported performance averages across all samples.
Solution Extraction A custom extraction pipeline was developed to process model responses
and intermediate reasoning traces (thoughts). The pipeline consists of several key components. We
implemented a flexible regex-based extractors to identify potential solution attempts in both the
final response and thinking trace. The extraction process identify solution patterns using regular
expressions (both explicit “moves =” patterns and alternative bracket-based solutions). We process
and clean each extracted candidate solution by (i) Removing comments from the list (text following
"#" in any line), and (ii) Normalizing move formats to what suggested in context to ensure consistent
structure. Then, we validate solution format and structure to filter out invalid matches. During
the extraction, we also capture metadata of token position for each extracted solution. Notably, for
accurate position tracking within thinking traces, we employed the same tokenizer (cl100k_base)
as the corresponding model to count tokens across all experiments. Token positions were also
normalized with respect to thought length to enable cross-sample comparison. Finally, we make sure
that the recorded solutions within the thought trace are unique and duplicate solutions (identical
moves list) were filtered. In case of duplicate solutions, only the first solution is recorded for analysis.
Solution Evaluation After extraction, each solution candidate is passed to the corresponding
simulator of puzzle for fine-grained verification. The simulator takes a solution as list of moves and
evaluate that with respect to the puzzle (check App. A.1 for details of each puzzle simulator). Each
move in the compositional solution is executed sequentially according to previous moves and the
puzzle rules. Then, the final state obtained from all moves in the sequence is compared to the goal
state of puzzle to determine full solution correctness. For incorrect solutions, details of first failure
move and the type of failure is also collected during the move verification with puzzle simulator.
Execution of Prescribed Steps In addition to open-ended problem solving across different
puzzles, we also conducted focused experiments to test how providing the explicit solving algorithm
24guidance with prescribed steps would affect behavior of these reasoning models (Sec. 4.4).
We expected that finding and devising solution from scratch should require substantially more
computation for model (e.g., for search and verification) than just following a given algorithm’s
steps. However, results in Figures 8a and 8b show that reasoning models’ behavior does not change
that much and the collapse still occurs at roughly same points as before with this setting. This
finding strengthens evidence that the limitation is not just in problem-solving and solution strategy
discovery but also in consistent logical verification and step execution limitation throughout the
generated reasoning chains.
For example, models are provided with a complete recursive algorithm of solving Tower of Hanoi
puzzle as follows. This algorithm scratchpad was appended to the standard problem prompt to test
its impact on reasoning behavior.
Example of Prescribed Algorithm for Tower of Hanoi
Here is a pseudocode of recursive algorithm to solve the puzzle:
ALGORITHM Solve(n, source, target, auxiliary, moves)
// n = number of disks to move
// source = starting peg (0, 1, or 2)
// target = destination peg (0, 1, or 2)
// auxiliary = the unused peg (0, 1, or 2)
// moves = list to store the sequence of moves
IF n equals 1 THEN
// Get the top disk from source peg
disk = the top disk on the source peg
// Add the move to our list: [disk_id, source, target]
ADD [disk, source, target] to moves
RETURN
END IF
// Move n-1 disks from source to auxiliary peg
Solve(n-1, source, auxiliary, target, moves)
// Move the nth disk from source to target
disk = the top disk on the source peg
ADD [disk, source, target] to moves
// Move n-1 disks from auxiliary to target
Solve(n-1, auxiliary, target, source, moves)
END ALGORITHM
To solve the entire puzzle of moving n disks from peg 0 to peg 2:
1. Initialize an empty list ’moves’
2. Execute Solve(n, 0, 2, 1, moves)
3. The ’moves’ list will contain the complete solution
25Note: When executing this pseudocode, track which disk is currently on top of each peg. The
disk IDs in the moves list should correspond to the actual disk being moved.
You can use this algorithm as a scratchpad to help you solve the problem step by step.
A.3Details on Computational Complexity
A.3.1Compositional Depth Characterization
Compositional Depth (# of Moves)
Compositional depth is the number of sequential op-
Blocks World
erations (i.e., moves) required to reach a full solution.
60
Checker Jumping
Figure 9 demonstrates how this depth scales with
River Crossing
problem size (N ) across our four puzzle environments.
50
Tower of Hanoi
Each puzzle has a distinct growth pattern, reflecting
40
its underlying computational complexity. For exam-
ple, Tower of Hanoi shows exponential growth (2N −1),
30
and Checker Jumping displays quadratic scaling (
20
(N + 1)2 − 1). The River Crossing and Blocks World
puzzles show more moderate, near-linear growth with
10
N . These varying compositional depth profiles enable
us to evaluate how language reasoning models handle
0
1
2
3
4
5
6
different types of sequential reasoning challenges and
Problem Size (N)
if their accuracy is always correlated with the com-
positional depth required to solve the puzzle. More Figure 9: Compositional depth (number of
details regarding this analysis is provided in Figure 10 moves required) across different problem sizes
in App. A.4.
for our four puzzle environments.
A.3.2
Performance vs Compositional Depth
While intuition suggests a negative correlation between problem complexity and model accuracy, our
analysis reveals a more nuanced relationship between compositional depth and LRM performance.
Figure 10 demonstrates this across three state-of-the-art reasoning models (Claude-3.7-Sonnet w.
thinking, DeepSeek-R1, and o3-mini) on our puzzle suite. Within individual puzzle types, we observe
the expected negative correlation: as compositional depth increases, model accuracy consistently
decreases. However, across different puzzle types, this relation breaks. Models may struggle with
puzzles of lower compositional depth while succeeding on different puzzles with higher compositional
depth. . For instance, models achieve >50% accuracy on Tower of Hanoi instances requiring
approximately 102 moves, yet consistently fail on River Crossing puzzles with substantially lower
compositional depth (∼ 101 moves).
A.4
Extended Results and Analysis
Failure Analysis. Understanding where models fail within the compositional reasoning steps
provides insights beyond binary success metrics. Our accuracy evaluation requires perfect execution
of entire move sequences—a single incorrect move results in failure. To examine failure patterns
more granularly, we analyze the compositional depth at which models first make incorrect moves
across varying problem complexity levels.
26DeepSeek-R1
100
100
40
20
0
80
Accuracy (%)
60
60
40
20
100
101
102
103
Compositional Depth (# of Moves)
Tower Hanoi
0
o3-mini (high)
100
80
Accuracy (%)
Accuracy (%)
80
Claude-3.7-Sonnet (thinking)
60
40
20
100
101
102
103
0
Compositional Depth (# of Moves)
Checker Jumping
River Crossing
100
101
102
103
Compositional Depth (# of Moves)
Blocks World
Figure 10: Accuracy versus compositional depth (number of moves required) for three LRMs
(DeepSeek-R1, Claude-3.7-Sonnet with thinking, and o3-mini) across four puzzle environments.
Figure 11 shows the failure move ID versus problem complexity (N ) within the solution sequence.
The top row compares Claude-3.7-Sonnet with and without thinking capabilities, while the bottom
row compares DeepSeek-R1 (thinking) with DeepSeek-V3 (non-thinking). These comparisons
demonstrates how thinking mechanisms of LRMs influence failure patterns in compositional reasoning
tasks of puzzles. Several counterintuitive patterns emerge from our analysis. First, models exhibit
non-monotonic failure behavior with respect to problem complexity—instances where models fail
earlier in the solution sequence for higher N values despite requiring longer overall solutions. For
example, in Tower of Hanoi, models sometimes fail at below 50 moves for N = 15 but succeed through
more than 100 moves for N = 8, contradicting the expectation that effective algorithmic planning
and execution for the same puzzle should maintain consistent failure patterns relative to solution
progress. This suggests fundamental inconsistencies in how models (both LRMs and their non-
thinking standard LLM counterparts) apply learned solution strategies across different problem scales.
Also, we observe that in the high-complexity regimes where both model variants experience complete
accuracy collapse, e.g., Tower of Hanoi with N ≥ 15 and Blocks World with N ≥ 40, non-thinking
models occasionally sustain performance deeper into the solution sequence and are able to fail at later
moves than thinking-enabled variants. This is interesting as it shows that compositional reasoning
failures in LLMs are not simply due to insufficient context length or inference compute, but rather
reflect fundamental limitations in how models maintain algorithmic consistency across problem scales.
We also analyze the distributional characteristics of failure moves to understand the consistency and
reliability of model reasoning. Figure 12 presents the density distributions of failure move positions
aggregated across all problem complexities for each puzzle environment, comparing thinking and
non-thinking models within the same family. Based on the figure, thinking models (Claude-3.7-Sonnet
with thinking and DeepSeek-R1) consistently show higher mean failure positions across all puzzles,
as indicated by the dashed vertical lines showing mean of first failure in sequence of moves. However,
the distribution shape of thinking models mostly have higher variance in their failure patterns. This
suggests that while these models can reach deeper into solution sequences on average, their reasoning
processes are more instable and prone to inconsistent performance.
Reasoning Effort Dynamics. Figure 13 demonstrates the reasoning effort (measured by inference
thinking tokens) versus problem complexity across our puzzle environments. Green dots indicate
27Figure 11: The first failure move versus problem complexity (N ) comparison for thinking and
non-thinking models across puzzle environments. Top: Claude-3.7-Sonnet comparison; Bottom:
DeepSeek-R1 vs DeepSeek-V3.
correct solutions, red crosses show incorrect ones, and blue lines track average thinking token usage at
each complexity level (N ) across different puzzles and LRMs. We observe a consistent pattern across
all three reasoning models (DeepSeek-R1, Claude-3.7-Sonnet-thinking, o3-mini) where thinking token
usage, i.e. reasoning effort, initially scales with problem complexity but counterintuitively declines
after reaching a model-specific threshold. This suggests an interesting and fundamental scaling limit
in LRM thinking process for reasoning where beyond certain complexity thresholds, models not
only fail to solve problems but counterintuitively reduce their inference compute despite facing more
difficult problems and being well below the context and generation limits.
28Figure 12: Density distribution of first failure moves for thinking and non-thinking models across
puzzle environments. Top: Claude-3.7-Sonnet comparison; Bottom: DeepSeek-R1 vs DeepSeek-V3.
29Figure 13: Detailed results on reasoning effort (measured in inference thinking tokens) versus problem
complexity (N) for three LRMs (DeepSeek-R1, Claude-3.7-Sonnet with thinking, and o3-mini) across
four puzzle environments.
30
#====
LEO: Boosting Mixture of Vision Encoders for
Multimodal Large Language Models
Mozhgan Nasr Azadani* James Riddell Sean Sedwards
University of Waterloo
Krzysztof Czarnecki
arXiv:2501.06986v1 [cs.CV] 13 Jan 2025
https://github.com/Mozhgan91/LEO
Abstract
88.0
87.6
Enhanced visual understanding serves as a cornerstone for
multimodal large language models (MLLMs). Recent hy-
brid MLLMs incorporate a mixture of vision experts to ad-
dress the limitations of using a single vision encoder and ex-
cessively long visual tokens. Despite the progress of these
MLLMs, a research gap remains in effectively integrating
diverse vision encoders. This work explores fusion strate-
gies of visual tokens for hybrid MLLMs, leading to the de-
sign of LEO, a novel MLLM with a dual-branch vision en-
coder framework that incorporates a post-adaptation fu-
sion strategy and adaptive tiling: for each segmented tile
of the input images, LEO sequentially interleaves the vi-
sual tokens from its two vision encoders. Extensive eval-
uation across 13 vision-language benchmarks reveals that
LEO outperforms state-of-the-art open-source MLLMs and
hybrid MLLMs on the majority of tasks. Furthermore, we
show that LEO can be adapted to the specialized domain
of autonomous driving without altering the model architec-
ture or training recipe, achieving competitive performance
compared to existing baselines. The code and model will be
publicly available.
86.4
64.8
64.2
62.9
60.5
85.4
65.4
64.8
64.2
62.0
59.3
37.2
72.2
31.2 31.1 29.1
78.5
71.6
67.1
66.8
66.2
65.1
38.2
48.7
64.3
72.9
64.6 65.4
65.1
50.0
52.5
57.9
53.4
57.0
58.2
65.7
80.1
63.8
67.1
68.8
71.0
Figure 1. Comparison of the performance of LEO across diverse
vision-language tasks with recent approaches [3, 7, 11, 27, 33].
effectiveness is reduced in tasks requiring high visual de-
tail, such as complex optical character recognition (OCR)
and chart understanding. Enhancing visual understanding
is essential for minimizing hallucinations [39] and improv-
ing performance in tasks that require high-resolution.
To this aim, recent studies [11, 16, 19, 23, 25, 28]
have attempted to improve MLLMs by exploring methods
that enhance their visual understanding capabilities. Some
works [6, 7, 24, 28, 50] demonstrate that strengthening vi-
sion encoders, either by scaling up model parameters and
pretraining data to match those of LLMs or by employing
tile segmentation, can significantly improve detailed scene
understanding. However, these approaches often come with
increased computational burden, particularly when multiple
images are processed. This has prompted the development
of a new class of models that exploit multiple vision en-
coders, each pretrained for distinct vision tasks and input
resolutions, and integrated using a variety of fusion tech-
1. Introduction
Advancements in multimodal large language models
(MLLMs) [2, 12, 21, 29] have harnessed the strengths of
pre-trained large language models (LLMs) alongside pow-
erful vision encoders. These models are trained through
multiple stages on large-scale image-text datasets, which
effectively aligns visual tokens extracted from vision foun-
dation models, such as CLIP [38], with the latent space of
LLMs. This multi-stage alignment has enabled advance-
ments in tasks involving vision-language comprehension
and reasoning [2, 10, 28]. Nonetheless, due to limitations in
input resolution, arising from the constraints of current vi-
sion encoders and language model sequence lengths, their
* Corresponding author: mnasraza@uwaterloo.ca
1Post-adaptation Fusion: ours
Text tokens
Vision Encoder 1
Vision Encoder 2
Fusion Block
Fusion
Block
+
Projector
Text tokens
Fused visual tokens
1. Channel Concatenation
Text
2. Sequence Concatenation
Vision Encoder 1Projector1
Vision Encoder 2Projector 2
3. MR-Adapter
ViT Block
Fusion
Block
MR-Adapter
Features 1
Features
1
Features 2 2
Features
Features 1
Features 1
Features 2
Features
2
Conv Block
+
Fusion
Block
LLM
Text
LLM
Pre-adaptation Fusion: existing MLLMs
Fused
visual tokens
4. Cross-Attention
Cross-attention
…
Q
K&V
Features 1 Features 2
Figure 2. Top: Comparison between the fusion strategy of existing hybrid MLLMs and that of LEO. Bottom: The most common fu-
sion paradigms in the literature: (1) channel concatenation [40], (2) sequence concatenation [16], (3) MR-adapter [33], and (4) cross-
attention [23].
niques, including but not limited to sequence concatena-
tion [11, 16, 26], channel concatenation [31, 40], mixture-
of-resolution adaptation [33], and cross-attention [23], to
enhance their ability to process complex multimodal data.
Despite their success, these prior studies have primarily
focused on pre-adaptation fusion, leaving tile-level post-
adaptation fusion unexplored in effectively fusing visual to-
kens from diverse vision encoders.
In this work, we introduce LEO, a novel MLLM de-
signed to enhance the integration of multiple vision en-
coders for multimodal language tasks. Unlike existing hy-
brid models, LEO adopts a unique post-adaptation fusion
strategy combined with tile segmentation (see Fig. 2, top).
Specifically, each input image is divided into 448 × 448
tiles, which are processed independently by our dual-branch
vision encoder architecture. This allows each encoder to
exploit its specialized capabilities for optimal image pro-
cessing. Our method follows a standard ViT-projector-LLM
framework, using separate MLP projectors for each vision
encoder branch. To further optimize the visual token rep-
resentation, LEO employs pixel unshuffling to reduce the
number of tokens for both encoders. These visual tokens
are then sequentially interleaved and merged with text to-
kens before being fed into the LLM. By employing tile-level
post-adaptation fusion strategy for visual token combina-
tion, LEO not only outperforms models with single vision
encoders and those that rely on higher image resolutions,
but also demonstrates superior performance over existing
hybrid MLLMs with pre-adaptation fusion techniques. This
is demonstrated by our quantitative analysis, illustrated in
Fig. 1. Our contributions are summarized as follows:
coders through a tile-level post-adaptation fusion method-
ology to enhance the visual understanding capabilities of
MLLMs.
• We conduct extensive experiments across multiple gen-
eral benchmarks, thus demonstrating the superior perfor-
mance of LEO on the majority of tasks when compared
to leading open-source single-vision-encoder and hybrid
models.
• We adapt LEO to the specialized domain of autonomous
driving (AD) without altering its architecture, collecting
extensive domain-specific pretraining data, or customiz-
ing the training recipe. To the best of our knowledge, this
is the first exploration of hybrid MLLMs for AD.
2. Related work
2.1. Multimodal LLMs
With the rapid advancement of large language models [1,
8, 43, 46], there has been considerable interest in multi-
modal models that enhance understanding and reasoning
capabilities. BLIP2 [21] introduces the Q-Former, designed
to efficiently bridge the modality gap between images and
text during pretraining. Flamingo [2] is capable of pro-
cessing sequences that combine visual and textual data in
any arbitrary order, a crucial feature that enables them to
perform in-context few-shot learning effectively. LLaMA-
Adapter v2 [12] activates a larger number of learnable pa-
rameters in the LLM through an early fusion strategy for vi-
sion and text tokens, where vision tokens are fed only into
the initial layers of the language model, enabling more ef-
ficient integration of visual information. LLaVA [27, 29]
delivers impressive results by employing a straightforward
projector, such as a linear layer or MLP, between the visual
• We propose LEO, a powerful hybrid multimodal model
that strategically fuses the capabilities of two visual en-
2and language components, and developing a streamlined
instruction-following data process powered by GPT. Most
of these methods, however, maintain low input resolutions
due to the constraints of pretrained vision encoders, such
as that of CLIP [38], and the sequence length restrictions
of large language models. All these approaches exploit a
single vision encoder in their architecture.
LLM
Tokenizer
Is it possible for you
to accelerate in this
situation, and if so,
why?
2.2. Vision encoders for MLLMs
To address the constraints of lower input resolutions, recent
MLLMs have concentrated on enhancing their vision en-
coder module. From a vision-focused standpoint, these ap-
proaches can be broadly categorized into three main strate-
gies: (1) robust vision encoders: designing stronger vision
encoders [7, 50] that effectively capture complex visual fea-
tures by incorporating larger models sizes and better train-
ing strategy, (2) tile segmentation: handling high-resolution
inputs by dividing images into smaller, lower-resolution
tiles [6, 28, 39], and (3) hybrid vision encoders: utilizing
a vision backbone that incorporates multiple vision experts.
Our research is closely tied to the third group of ap-
proaches [23, 31, 45] that develop a multi-branch vision
encoder framework to enhance perception capabilities of
MLLMs. Some models [23, 33] suggest merging high-
resolution visual details with low-resolution tokens to en-
hance visual representation. LLaVA-HR [33] proposes a
dual-pathway vision model that integrates features from
high-resolution convolutional blocks with those from low-
resolution ViT blocks. These pretrained vision experts
might nevertheless lack key capabilities, such as text under-
standing and object localization. Several studies have incor-
porated multiple vision experts trained on diverse tasks to
broaden the functionality of their encoders. Brave [16] and
Mousi [11] perform sequence append, combining vision to-
kens from multiple experts into a single, extended sequence.
Tong et al. [45] identify distinct differences in the visual
features captured by CLIP [38] and DINOv2 [37], leading
to the design of an image-level mixture-of-features strategy.
Some models combine vision tokens through channel con-
catenation to preserve the original sequence length, such as
DeepSeek-VL [31] and Eagle [40] models, or utilize ad-
vanced fusion and routing techniques [19, 51] to leverage
the strengths of various encoders. These methods perform
pre-adaptation fusion of vision tokens, which are then pro-
cessed by a single projector module (either an MLP or Q-
Former), as shown in Fig. 2.
Distinguished from previous studies, this work proposes
a novel hybrid approach that integrates a tile-level post-
adaptation fusion strategy with dynamic high-resolution in-
puts achieved through tile segmentation. In our frame-
work, each vision expert is equipped with its own projector,
and the vision tokens are sequentially interleaved following
vision-text alignment by the projectors at the tile level.
Fusion Block
Projector 1Projector 2
Pixel unshufflePixel unshuffle
Intern-ViTSAM-L
Dynamic high resolution
Figure 3. The architecture of our model. LEO adapts a dual-vision
MLLM architecture through tile-level post-adaptation fusion of vi-
sual tokens. Pixel unshuffle is adapted to decrease the visual token
quantity.
3. Method
3.1. Overview
Figure 3 illustrates the overall architecture of our proposed
multimodal large language model, LEO, which is conceptu-
ally straightforward: First, the high-resolution input images
are divided into tiles. These tiles are then processed by two
different vision experts, each exploiting its specialized pre-
training to provide distinct feature representations. Next,
pixel unshuffling is applied to the extracted visual embed-
dings to reduce the number of visual tokens for each vision
encoder. Vision-text alignment is conducted using two dis-
tinct MLP projectors, and a tile-level post-adaptation fusion
strategy is employed to sequentially interleave tile-level vi-
sual tokens. Finally, these visual tokens are combined with
text tokens and processed by the LLM for comprehensive
visual-language understanding and reasoning. The follow-
ing section gives more details of the architectural blocks
identified in Fig. 3.
3.2. LEO
Dynamic high resolution. The image processing begins
with a dynamic high-resolution approach, where an input
image is segmented into multiple tiles alongside a thumb-
nail of the original image, helping the model to capture the
global context. We utilize a dynamic resolution input strat-
egy similar to that in InternVL [6, 7]. Each input image is
resized to a closest available aspect ratio that is dividable
into square patches of size 448 × 448, with up to six tiles
for the 3:2 aspect ratio shown in Fig. 4. To capture global
context, the input image is also scaled to match the patch
size and added to the set of patches to be processed by the
3text tokens and feed them into the LLM for auto-regressive
generation. For our language model, we use InternLM2-
7B-Chat [4], identical to the LLM in the base model [7], to
ensure a fair comparison. To optimize memory and compu-
tational efficiency, the context length of the LLM is set to a
maximum of 8196 tokens, ensuring balanced performance
across various multimodal tasks. Given the input context
length constraint of the LLM, we limit each input image
to six segmented tiles, which enables efficient handling of
multiple images.
Figure 4. Tile Segmentation: Each input image is divided into
multiple tiles to capture localized details, while a resized version
maintains global context. The tiles are shown after preprocessing
with the SAM [18] preprocessor.
3.3. Training process.
Our training process consists of two main phases. In the first
phase, we initialize the vision encoders and the language
model from the base models [4, 7, 18], while the projector
layers for SAM are randomly initialized. This approach is
adopted because InternViT is already pretrained on vision-
language alignment and SAM is pretrained specifically for
segmentation. To avoid any potential representation incon-
sistencies, we focus on training the layers of the second pro-
jector. In the second phase, the vision encoders are kept
frozen, and we perform full fine-tuning of two projectors
and the language model. In Table 5, we present the results
of an ablation study that shows the effect of keeping the vi-
sion encoders frozen during the training phases.
vision encoders. This approach, when applied dynamically
to two input images, results in a total of 14 unique patches
for training. Figure 4 illustrates this tiling process with an
example driving scene image, where the tiles are shown af-
ter normalization by the SAM’s preprocessor [18].
Pixel unshuffling. Once the input images are segmented
into tiles, they are simultaneously fed to two distinct vi-
sion encoders. We select InternViT-300M-448px [7] as the
first vision encoder and SAM-L [18] as the second. For
each encoder, we apply pixel unshuffling [41], which re-
arranges the spatial layout of pixels to reduce the num-
ber of visual tokens while preserving important visual fea-
tures. Given an input tensor of shape [b, c, w, h] and a down-
scaling factor of r, the output tensor will have the shape
[b, c ∗ (r2 ), w/r, h/r], where b represents the number of
tiles, c is channel, and w and h denote width and hight.
Specifically, for each segmented tile, we apply downscaling
factors of 2 and 4 for our first and second vision encoders,
respectively, reducing the number of visual tokens to 256
per tile and encoder, resulting in a total of 512 visual tokens
per tile.
Visual token fusion. Existing hybrid multimodal mod-
els [16, 23, 33, 40] typically use a pre-adaptation fusion
strategy to combine visual tokens from two or more vi-
sual encoders, applying one of the common fusion methods
shown in Fig. 2 (bottom) prior to the vision-text alignment
process. In these approaches, all visual encoders share the
same projector module. In contrast, LEO employs an alter-
native fusion approach in which each vision encoder main-
tains its own dedicated projector module, allowing for in-
dependent processing of visual tokens before they are com-
bined. We find this to be a more flexible and effective fusion
strategy. We use a two-layer MLP for the projector archi-
tecture to ensure simplicity and efficiency. To streamline
processing, we set the output dimension of the projector to
match that of our LLM. In the fusion block, we sequentially
interleave the visual tokens from InternViT and SAM-L for
each segmented tile, while preserving their original order.
Language model. We combine the extracted visual and
3.4. Adaption to autonomous driving
Although numerous studies have successfully applied
MLLMs to autonomous driving [5, 34, 44, 47], a straight-
forward approach that avoids extensive modifications to
model architecture, training processes, or heavy data collec-
tion has yet to be fully explored. In this work, we investigate
the potential of applying LEO to the autonomous driving
domain without altering its architecture or training recipe,
aiming to offer insights into streamlined transfer learning
and facilitate MLLM adaptation to specialized domains.
Instruction tuning plays a crucial role in helping models
learn to follow user prompts, utilizing training data in visual
question answering and conversational formats. For this do-
main, we design tasks in a VQA format, with each frame
represented as: <img> <IMG-CONTEXT> </img>. At
the prompt level, the temporal aspect of video frames is
managed by treating sequential frames as multiple image
inputs. A sample prompt is formulated as:“<image1> ...
<image N> Is it safe to enter the intersection at this time?”.
The images in this setting are of high-resolution, each mea-
suring 2048 × 1280 and segmented into six patches of size
448 × 448.
4. Experiments
We first describe the evaluation setting, outlining the imple-
mentation details of our model. We then present a compar-
4Model
ChartQA DocVQA VQAT GQA VQAv2 VizWiz MMB MMMU POPE AI2D SEED SQA MMVet
Instruct-BLIP [9]
InternVL [7]
VILA [25]
QwenVL [3]
QwenVL-Chat [3]
LLaVA.1.5 [27]
LLaVA-Next [28]-
-
-
65.7
66.3
-
--
-
-
65.1
62.5
-
-50.1
57.0
64.4
63.8
61.5
58.2
-49.2
62.9
62.5
59.3
57.5
62.0
--
79.3
79.9
79.5
78.2
78.5
-45.5
52.5
57.8
-
-
50.0
-36
64.6
68.9
38.2
-
64.3
67.4-
-
-
-
-
-
35.8-
86.4
85.5
-
-
-
86.5-
-
-
62.3
57.7
-
--
65.4
61.1
64.8
-
-
70.260.5
66.2
68.2
67.1
68.2
66.8
-26.2
31.2
34.9
-
-
31.1
-
LEO (ours)71.080.168.864.878.357.972.936.488.069.672.278.537.2
Table 1. Comparison with leading MLLMs across 13 benchmarks. All models use a 7B language model. Bolded values indicate the best
performance. Some benchmark names are abbreviated due to space constraints: VQAT : TextVQA [42], SQA: ScienceQA [32], MMB:
MMBench [30], and SEED: SEED-Bench [20].
ModelFusion
PT
Brave-X5 [16]
LLaVA-HR [33]
Mini-Gemini [23]
Mousi [11]Pre-A
Pre-A
Pre-A
Pre-A
LEO (ours)Post-A 595 K
SFT VQAT GQA VQAv2 VizWiz MMB MMMUv MMMUt POPE SEED SQA MMVet
100 M
-
558 K 1.2 M
1.2 M 1.5 M
1.2 M 1.6 M
1M
-
67.1
65.2
53.452.7
64.2
64.5
60.582.5
81.9
-
75.454.2
48.7
-
--
-
69.3
65.4-
-
36.1
--
-
32.8
-87.6
87.6
-
85.4-
64.2
-
62.0-
65.1
71.1
71.6-
31.2
40.8
29.1
68.864.878.357.972.936.433.588.072.278.537.2
Table 2. Results on 11 evaluation benchmarks are compared with leading hybrid MLLMs. All models use a 7B language model. The
best values are shown in bold. X5 denotes a mixture of 5 vision encoders. The following names are shortened due to space constraints:
Pre-A: pre-adaptation, Post-A: post-adaptation, PT: pretraining data, SFT: supervised finetuning data, VQAT : TextVQA [42], SQA: Sci-
enceQA [32], MMB: MMBench [30], and SEED: SEED-Bench [20].
ative analysis of LEO against leading open-source MLLMs
and hybrid models, across diverse vision-language tasks.
each) using DeepSpeed’s Zero2 strategy, allowing training
to complete within approximately 72 hours.
Training Datasets. In the first stage of training, we use
the LLaVA-595k dataset [28], which comprises 595k sam-
ples. In the supervised finetuning stage, we employ the
same supervised fine tuning stage dataset as InternVL [7],
incorporating a total of approximately one million visual in-
struction tuning samples, all of which are fully open-source.
4.1. Setting
Implementation details. Our model architecture is built
upon InternVL [7], following a standard MLLM design
of ViT-projector-LLM. The projector aligns the visual fea-
tures by mapping them into the language embedding space.
LEO uses InternLM2-7B-Chat [4] as the large language
model, with pretrained InternViT-300M-448px [7] and
SAM-L [18] as vision encoders, and an adaptive tiling strat-
egy. Following the base model [7], we divide each input
image into 448 × 448 tiles, where the number of tiles varies
based on the aspect ratio of the image. Our training pro-
cedure consists of two stages. In the first stage, we freeze
both vision encoders and focus on optimizing the projector
module to ensure effective training. In the second stage, we
perform supervised instruction tuning, unfreezing the pro-
jector modules along with the LLM. Both stages employ
a context length of 8196, and training is conducted for a
single epoch. We optimized the model using the AdamW
optimizer with a cosine learning rate schedule. During the
the second stage, we set the learning rate to 4 × 10−5 with a
weight decay of 0.01. In the alignment stage, we increased
the learning rate to 4 × 10−4 , maintaining the same weight
decay. Training was conducted on 8 A100 GPUs (80 GB
4.2. Main Results on general benchmarks
Comparison with leading MLLMs. In this section, we
comprehensively evaluate our model’s visual understand-
ing and reasoning abilities in comparison to previous lead-
ing MLLMs, across 13 vision-language benchmarks. These
benchmarks are organized into three task categories: (1)
OCR and chart understanding, including DocVQA [36],
TextVQA [42], ChartQA [35], and AI2D [17]; (2) gen-
eral visual question answering, including VQAv2 [13],
GQA [15], and VizWiz [14]; and (3) general multimodal
benchmarks, such as MMMU [49], MMBench [30], SEED-
Bench [20], POPE [22], MMVet [48], and ScienceQA [32].
In Table 1, we see that LEO achieves state-of-the-art re-
sults in 12 out of 13 benchmarks. In the OCR and chart
understanding category, LEO consistently surpasses lead-
ing models across all four datasets, by virtue of its dual-
branch vision encoders. In the multimodal benchmark cat-
5egory, LEO demonstrates superior performance across all
six benchmarks, highlighting its broad knowledge and ad-
vanced reasoning abilities. Additionally, the results in Ta-
ble 1 show that LEO excels in more demanding bench-
marks that necessitate college-level knowledge, such as
MMMU [49], which focuses on complex problems from
various domains, including science, business, tech and en-
gineering, as well as health and medicine. Notably, com-
pared to InternVL [7], which uses the same LLM and vision
encoder as our model, LEO achieves superior performance
across 8 out of 9 benchmarks, demonstrating the benefits of
dual-branch vision encoders for vision-language tasks. This
approach mitigates the inherent biases of individual vision
encoders, providing a robust framework for the mixture of
encoders.
Comparison with leading Hybrid MLLMs. We com-
pare the performance of LEO with recent hybrid approaches
across 11 benchmarks. In Table 2, we see that LEO demon-
strates strong performance on the majority of benchmarks.
Our model is trained on the least amount of data in both pre-
training and SFT stages, yet it outperforms models trained
on larger datasets, such as Mousi [11], highlighting the gen-
eralization capability of our model. Compared to mod-
els with more complex fusion strategies, such as LLaVA-
HR [33] and Mini-Gemini [23], our model excels across
most benchmarks, especially on multimodal benchmarks
like MMBench, SEED, and ScienceQA. Notably, compared
to Brave [16], which combines five distinct vision encoders
through pre-adaptation fusion and sequence concatenation,
LEO achieves competitive performance on most tasks. This
result underscores that post-adaptation fusion of visual to-
kens from only two vision experts can be as effective as
pre-adaptation fusion of visual tokens from a larger set of
vision experts.
Comparison with Eagle. We conduct a comparison
with Eagle [40], a concurrent work that integrates vision en-
coders through pre-adaptation fusion and channel concate-
nation. Table 3 shows that LEO outperforms Eagle-X2 [40],
which combines two vision experts, on 7 out of 9 bench-
marks, particularly excelling in the OCR and general VQA
categories. Notably, LEO also surpasses Eagle-X4 [40],
which uses four vision encoders, on 5 out of 7 benchmarks,
with an identical score for GQA and a near-dentical score on
POPE. It is worth mentioning that these results are achieved
despite LEO being trained with less SFT data, highlighting
the robustness and reasoning capability of the enhanced fu-
sion design in LEO.
Eagle-X4 [40]Eagle-X2 [40]LEO (ours)
Pre-adapt.
1024Pre-adapt.
1024Post-adapt.
512
PT
SFT595 K
1.8 M595 K
1.8 M595 K
1M
ChartQA
DocVQA
VizWiz
GQA67.5
-
50.8
64.867.0
77.7
48
63.271.0
80.1
57.9
64.8
MMMU
SEED
MMBench
POPE
ScienceQA-
73.4
67.8
88.4
70.436.0
73.5
-
88.3
70.736.4
72.2
72.9
88.0
78.5
Fusion
#-Tokens
Table 3. Results compared to a concurrent approach [40], which
combines vision encoders through pre-adaptation fusion and chan-
nel concatenation. Here, XN denotes a mixture of N vision en-
coders and #-Tokens denotes number of visual tokens.
Model
N Lingo-J ↑ BLUE ↑ METEOR ↑ CIDEr ↑
BLIP 2 [21]
1
LLaVA.1.5 [27] 5
5
InternVL [7]52.20
51.00
58.0013.00
10.62
13.5317.40
29.44
34.2760.10
48.18
67.17
LingoQA [34]
LingoQA [34]3
559.80
60.8014.61
15.0018.44
18.5662.61
65.62
LEO (ours)261.0014.9135.4469.72
Table 4. Results on the LingoQA benchmark [34]. All models are
fine-tuned, where N denotes the number of frames used during
training. Lingo-J represents the Lingo-Judge metric. Leo demon-
strates competitive performance without requiring tailored model
architecture for the autonomous driving domain.
cludes data covering nine distinct task types, such as ac-
tion, justification, localization, and anticipation, providing
a thorough representation of scenarios encountered in au-
tonomous driving.
## /scripts/full_tests.py
```py path="/scripts/full_tests.py"
from ollm import Inference, TextStreamer, file_get_contents
import torch, os
from datetime import datetime
def run_test(test_id, model_id, sm, um, kvcache=None, offload_layers_to_gpu=0, offload_layers_to_cpu=0, max_new_tokens=500):
o = Inference(model_id, device="cuda:0", logging=True)
o.ini_model(models_dir=f"/media/mega4alik/ssd/models/", force_download=False)
if offload_layers_to_gpu: o.offload_layers_to_gpu_cpu(gpu_layers_num=offload_layers_to_gpu, cpu_layers_num=offload_layers_to_cpu)
elif offload_layers_to_cpu>0: o.offload_layers_to_cpu(layers_num=offload_layers_to_cpu)
if kvcache=="disk": past_key_values = o.DiskCache(cache_dir="/media/mega4alik/ssd/kv_cache/")
else: past_key_values=None
model, tokenizer, device = o.model, o.tokenizer, o.device
messages = [{"role":"system", "content":sm}, {"role":"user", "content":um}]
input_ids = tokenizer.apply_chat_template(messages, tokenize=True, reasoning_effort="minimal", add_generation_prompt=True, return_tensors="pt", return_dict=False).to(device)
text_streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)
with torch.no_grad():
print(f"\n\n#{test_id}.TestingStarted.{model_id}", datetime.now().strftime("%H:%M:%S"), "input_ids.shape:", input_ids.shape)
outputs = model.generate(input_ids=input_ids, max_new_tokens=max_new_tokens, do_sample=False, past_key_values=past_key_values, use_cache=True, streamer=text_streamer).detach().cpu()
answer = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=False)
print(answer)
return answer
#=======================================================
test_ids = [4] #quick tests: [1,2,5,6], hard tests: [4]
for test_id in test_ids:
if test_id==1: #1. Llama3-8B check noKV==newKV2.0 on 10k_chats
sm, um = "[CHATS]:\n"+file_get_contents("./samples/10k_sample.txt")+"[/END CHATS]", "Analyze chats above and write top 10 most popular questions (translate to english)."
#ans1 = run_test("1-1", "llama3-8B-chat", sm, um)
ans2 = run_test("1-2", "llama3-8B-chat", sm, um, kvcache="disk", offload_layers_to_cpu=2)
#if ans1!=ans2: raiseError(f"#1.TestFailed <1.ans1>:\n{ans2}\n<1.ans2>:\n{ans2}")
#else: print("#1.TestSuccess")
if test_id==2: #2. gpt-oss-20B check noKV==newKV2.0 on 2k_sample
sm, um = "[CHATS]:\n"+file_get_contents("./samples/2k_sample.txt")+"[/END CHATS]", "Analyze chats above and write top 10 most popular questions (translate to english)."
#ans1 = run_test("2-1", "gpt-oss-20B", sm, um)
ans2 = run_test("2-2", "gpt-oss-20B", sm, um, kvcache="disk", offload_layers_to_cpu=6, max_new_tokens=10)
#if ans1!=ans2: raiseError(f"#2.TestFailed <2.ans1>:\n{ans2}\n<2.ans2>:\n{ans2}")
#else: print("#2.TestSuccess")
if test_id==3: #3. Llama3-8B newKV2.0, make sure it runs without OOM on 85k_sample
sm, um = file_get_contents("./samples/85k_sample.txt"), "Analyze papers above and find 3 common similarities."
ans = run_test("3", "llama3-8B-chat", sm, um, kvcache="disk", offload_layers_to_cpu=2, max_new_tokens=10)
print("#3.TestSuccess")
if test_id==4: #4. qwen3-next-80B, make sure it generates proper output on 45k sample
sm, um = file_get_contents("./samples/45k_sample.txt"), "Analyze papers above and find 3 common similarities.",
ans = run_test("4", "qwen3-next-80B", sm, um, kvcache="disk", offload_layers_to_cpu=48, max_new_tokens=100)
print("#4.TestSuccess")
if test_id==5: #5. qwen3-next-80B, make sure it generates proper output for 2k context
sm, um = "[CHATS]:\n"+file_get_contents("./samples/2k_sample.txt")+"[/END CHATS]", "Analyze chats above and write top 10 most popular questions (translate to english)."
ans = run_test("5", "qwen3-next-80B", sm, um, kvcache="disk", max_new_tokens=100)
print("#5.TestSuccess")
if test_id==6: #6. gemma3-12B, properly runs on 2k_sample, cpu_offloading(12 layers should be consuming ~7GB RAM)
sm, um = "[CHATS]:\n"+file_get_contents("./samples/2k_sample.txt")+"[/END CHATS]", "Analyze chats above and write top 10 most popular questions (translate to english)."
ans = run_test("6", "gemma3-12B", sm, um, offload_layers_to_cpu=12, max_new_tokens=10)
print("#6.TestSuccess")
```
## /scripts/llama_export.py
```py path="/scripts/llama_export.py"
import json, os
import torch
from transformers import AutoModelForCausalLM, AutoConfig
from safetensors.torch import load_file as load_safetensors
MODEL_ID = "meta-llama/Llama-3.2-1B-Instruct"
OUT_DIR = "gds_export"
os.makedirs(OUT_DIR, exist_ok=True)
# Load weights on CPU (normal HF path), then export raw
# If your model is sharded across multiple .safetensors, iterate them.
state_dict = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
device_map=None,
torch_dtype=torch.float16, #"auto" same as original torch.bfloat16,
low_cpu_mem_usage=True
).state_dict()
manifest = {}
for name, tensor in state_dict.items():
# Only export layer weights to keep it small; adjust filter as needed
if not name.startswith(("model.layers", "transformer.h")): continue
t = tensor.to("cpu").contiguous() # ensure contiguous for .tofile
filename = f"{name.replace('.', '__')}.bin"
path = os.path.join(OUT_DIR, filename)
os.makedirs(os.path.dirname(path), exist_ok=True)
t.numpy().tofile(path) # raw bytes
manifest[name] = {
"path": filename,
"dtype": str(t.dtype).replace("torch.", ""), # e.g., "float16"
"shape": list(t.shape),
}
with open(os.path.join(OUT_DIR, "manifest.json"), "w") as f:
json.dump(manifest, f, indent=2)
print(f"Exported {len(manifest)} tensors to {OUT_DIR}")
```
## /src/ollm/__init__.py
```py path="/src/ollm/__init__.py"
# src/ollm/__init__.py
from .inference import Inference, AutoInference
from .utils import file_get_contents
from transformers import TextStreamer
```
The content has been capped at 50000 tokens. The user could consider applying other filters to refine the result. The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.