How to Benchmark BitNet on CPU Without Misleading Yourself

A benchmark number is useful only when another developer can understand what was measured and repeat the workload. “My CPU runs BitNet at 20 tokens per second” leaves out the model file, runtime revision, prompt length, generated-token count, thread count, operating system, and warm-up behavior. Those omissions can change the conclusion more than a small kernel optimization.

Microsoft’s bitnet.cpp repository includes an end-to-end benchmark script and documents its model, prompt-token, generated-token, and thread arguments.[1] That makes it a better starting point than timing an interactive chat response with a stopwatch. The protocol below does not supply a result; it tells you how to produce and report one honestly.

Define the question before you run a command

CPU inference has at least two important phases:

• Prompt processing consumes the input context and can be sensitive to prompt length.
• Token generation produces new tokens one by one and is often what people mean by interactive speed.

A one-line “tokens per second” result can combine these phases in a way that obscures both. Choose a fixed prompt-token length and generated-token length, write them down, and compare like with like. If you change a thread count, model file, compiler, or runtime revision, label it as a new configuration rather than averaging it with the old one.

Capture a baseline environment

Before the first run, record:

CPU model and physical/logical core count:
RAM and free memory at test time:
Operating system and kernel:
BitNet repository commit:
Compiler and CMake versions:
Python version:
Model repository, file name, and file checksum:
Quantization type / kernel selection:

The model checksum is particularly valuable when a repository hosts several files with similar names. It tells a later reader exactly what was measured without asking them to infer it from a directory.

Run a fixed end-to-end workload

The official README documents this benchmark form:[1]

python utils/e2e_benchmark.py \
  -m /absolute/path/to/model.gguf \
  -p 512 \
  -n 200 \
  -t 4

Replace the values with the configuration you actually want to evaluate. Keep -p and -n fixed while comparing thread counts. Run a short warm-up first, then perform several measured runs. A single run can be distorted by cold file cache, background work, thermal behavior, or CPU frequency changes.

Treat thread count as an experiment, not a default

More threads do not guarantee more useful throughput. The best setting can depend on the kernel, CPU topology, memory bandwidth, and the workload. Test a small planned set—such as 1, 2, 4, and the number of physical cores—while keeping everything else constant. Record each raw result; report the median rather than the best accidental run.

Do not turn off safety controls or lock clock speeds merely to obtain a headline number. If you deliberately change a power profile, affinity setting, or governor, disclose it. A reader needs to know whether the figure represents a normal desktop, a tuned laboratory run, or a server configuration.

Publish a compact result table

Use a table like this in a project README or article:

Field	Value
Runtime revision	commit hash
Model file	repository, file, checksum
OS / CPU / RAM	exact system
Workload	prompt tokens, generated tokens
Threads	fixed value for this row
Runs	warm-up count and measured-run count
Result	median and range, with units
Notes	power mode, errors, or caveats

That table makes a slower result more useful than an unexplained faster one. It also gives future readers enough information to detect a regression after an upgrade.

Keep source results separate from local results

The BitNet project and its technical reports publish performance claims for specified conditions. Cite them as source results and link directly to the report; do not relabel them as an in-house benchmark.[1][2] Conversely, a local measurement should say “measured on” and include its environment. This simple wording prevents an article from implying a test that did not happen.

Editorial note

BitNet.XIN does not publish benchmark figures without the configuration and raw output needed to interpret them. This guide intentionally contains no invented hardware data. Use the official benchmark interface, keep the environment record, and publish caveats with any result.