Reproducible N/M runs: seeds, shots and versions

When are two runs the same experiment?

Suppose you run a Bell circuit and see 263 counts of 00, while a colleague runs the same circuit and sees 249. Whether that difference means anything depends on three questions: did the seed match, did the shot count match, and did the software and settings match?

This article separates what a seed controls from what it does not, gives a scale for ordinary sampling variation, and lists what to record.

Run the circuit

Paste the program from the Bell state article into the Playground:

module blog_bell;
@seed(23);
fn main() {
  let q = qreg[2];
  sample 512 {
    H(q[0]);
    CNOT(q[0], q[1]);
    let c0 = measure(q[0]);
    let c1 = measure(q[1]);
  }
  return q;
}

Note the 00 and 11 counts and the pre-measurement probability display.

Probabilities and counts are different quantities

H and CNOT prepare ( |00> + |11> ) / sqrt(2). In the ideal model, the statevector assigns probability 0.5 to 00 and to 11, and 0 to 01 and 10. Those values follow from the gates. They do not depend on the seed or on the shot count.

The histogram is a finite sample of 512 draws from that distribution. Because only 00 and 11 can occur here, the count of 00 fixes the count of 11. The 00 count can therefore be treated as a binomial variable with n = 512 and p = 0.5.

The @seed annotation fixes the pseudo-random sequence the local simulator uses to draw those samples. For a fixed program and a compatible runtime, the same seed reproduces the same sample. It does not describe the randomness of a physical device. As the Noise Lab states, a seed does not remove stochastic uncertainty: it repeats one sample, not a more accurate one.

How much variation is ordinary?

For a binomial count, the standard deviation is σ = √(n·p·(1−p)). With n = 512 and p = 0.5, σ = √128 ≈ 11.3 counts around a mean of 256. Roughly 95% of runs fall within 2σ, that is between 234 and 278 counts, or 45.70% to 54.30% of shots. The exact binomial probability of that interval is 95.3%.

With n = 4096, σ = √1024 = 32 counts around 2048. The absolute spread is larger, but relative to the number of shots it falls from 2.2% to 0.78%. The 2σ interval is 1984 to 2112 counts, or 48.4% to 51.6%. Eight times as many shots narrows the relative spread by √8 ≈ 2.8.

Comparing two runs

• Same seed and same inputs: expect identical counts. A difference means some input changed. Find it before interpreting the result.
• Different seeds: compare the gap with the sampling scale. The difference between two independent 00 counts has σ = √2 × 11.3 = 16 counts at 512 shots, about 3.1 percentage points, and about 45 counts (1.1 points) at 4096 shots. A gap of one or two such units is unremarkable.
• In this noiseless program, 01 and 10 have zero ideal probability. Any count there signals a changed program or an enabled noise model.
• When comparing command-line output, compare counts rather than whole files. The CLI reference classifies nm run output as runtime-timing, not byte-stable.

Three changes to predict

1. Run the unchanged program twice. Predict identical counts.
2. Change @seed(23) to @seed(24). Predict different counts, an unchanged probability display, and a 00 count most likely between 234 and 278.
3. Restore seed 23 and change sample 512 to sample 4096. Predict each outcome near 2048 ± 64, proportions roughly within 48.4% to 51.6%, and unchanged probabilities.

Write each prediction down before running the change.

What to record

• The complete source, including annotations.
• The N/M version. The language is 0.2.0; the VS Code extension has its own version. Record the installed extension version separately and check the release notes for published releases. A local unreleased package is not a public release.
• The simulator, backend and target profile that actually ran. Experiment Studio records the effective engine and any fallback for its preset circuits; for Playground runs, note it yourself.
• The seed and the shot count.
• The noise model and its probability, or "noiseless".
• The bit-order convention of the display you read. The Playground prints q[0] as the rightmost digit; for 00 and 11 the order does not matter, for 01 and 10 it does.
• Relevant feature settings, as the N/M 0.2 overview recommends.

nm workspace snapshot creates a deterministic read-only bundle. The documented example records a source hash, a run fingerprint, language and package versions, the engine and the seed. These FNV-1a fingerprints are reproducibility identifiers, not signatures. The shot count and bit order are not among the fields in that example, so keep them in your own notes.

Limits

• The documentation does not promise that a seed produces identical counts across N/M versions or engines. Treat a change of version or engine as a new run and compare statistically.
• A seed does not control hardware randomness. Hardware adds device noise and calibration drift; everything here is a local noiseless simulation.
• The 2σ interval is a rule of thumb. Trying seeds until a histogram looks balanced defeats the purpose of seeding.
• Matching histograms do not prove entanglement; see the Bell state article.

Next steps and sources

• Build a Bell state with N/M: the circuit used here.
• N/M 0.2.0: capabilities and limits: version and capability boundaries.
• Release notes and workspace snapshots: version records and reproducibility metadata.
• NIST/SEMATECH e-Handbook: Binomial distribution: mean np and standard deviation √(np(1−p)).

The counts and intervals in this article describe an ideal local simulation; they are not a hardware result or a performance claim.