Test the audio engine with numbers, not with ears

The same note goes into two engines. Each engine saves a WAV file. The computer compares the two files.

A = reference (already sounds good) B = my engine
A desk seen from above. From left to right: a small keyboard, two engine boxes A and B, two WAV file cards, a speaker with a red no-sign, two screens, a tablet with a checklist, and headphones. An orange arrow runs along the front of the desk from the tablet back to engine B.
1

One note

Send the same note to both engines.

note = A4        // 440 Hz
A small keyboard. One key is lit yellow. Its display reads A4 440 Hz. An arrow leaves the keyboard and splits in two.
2

Two engines

A is the reference. B is my engine. Both get the same note.

A = reference.render(note)
B = myEngine.render(note)
Two boxes. The back one has a blue badge A and the word REFERENCE. The front one has an orange badge B and the words MY ENGINE. One arrow splits and goes into both.
3

Save to a file. No speaker.

Each engine writes a WAV file. A WAV file is a list of numbers.

A.wav = [0.00, 0.31, 0.59, 0.81, ...]
B.wav = [ ... ]
// no speaker, no microphone
Two paper cards marked A WAV and B WAV, each with a wave drawn on it and rows of numbers. Behind them is a speaker covered by a red no-sign.
4

Subtract: A − B

A flat line means the two files are the same. Here the line is not flat, so B is different.

diff = A - B
max(|diff|) = 0.27   // 0 = same
A screen with three rows. Row A is a blue wave. Row B is an orange wave with a slightly different shape. Row A minus B is a small red wave, not a flat line.
5

FFT shows where

FFT shows every overtone. At 1320 Hz, B is 12 dB lower than A. That is the bug.

fft(B) - fft(A)
 440 Hz    0 dB   PASS
 880 Hz    0 dB   PASS
1320 Hz  -12 dB   FAIL
A screen titled FFT with eight pairs of bars, blue for A and orange for B, labelled 440 to 3520 Hz. The third pair, at 1320 Hz, has a red box around it and the label minus 12 dB: the orange bar is shorter.
6

PASS or FAIL

The computer checks every number. FAIL: fix the code and run again. It takes seconds and needs no person.

Hz     pitch       PASS
dB     level       PASS
ms     timing      PASS
NOISE              PASS
FFT    overtones   FAIL

while (FAIL) { fix(); test(); }
A tablet with a checklist: Hz, dB, ms and NOISE have green check marks, FFT has a red cross, and the bottom line reads 4 of 5 FAIL. An orange arrow marked FAIL and FIX leads back along the desk. A green arrow marked PASS leads to headphones.
7

Ears: one time, at the end

When every check is PASS, listen one time. Ears answer one question: does it sound beautiful?

5 / 5  PASS  →  listen 1×
Headphones on a stand at the end of the desk. A green arrow marked PASS points to them.

Programs to use

First: the engine must save a WAV file

My engine isHow to get the WAV
a plug-in (VST3, AU)REAPER or Pedalboard loads it and saves the WAV
a web appOfflineAudioContext (built into the browser)
my own programadd one command: engine --render note.mid out.wav

To build the test

JobProgramCost
Play the same note through A and B, save WAVREAPER reaper -renderproject test.rpppaid, 60-day full trial
The same job from Python, no windowPedalboardfree
Read the WAV, do A - B and FFTPython + NumPy + SciPyfree
Say PASS or FAILpytestfree
Run after every code changeGit + a CI runner (GitHub Actions)free to start
Look at a WAV when a test failsSonic Visualiser or Audacityfree
Check that a plug-in does not crashpluginvalfree
Blind listening at the end (step 7)foobar2000 + ABX Comparatorfree

To compare against (A = reference)

Kind of soundProgramCost
Instruments from recordingsKontakt Playerfree
Piano built by mathPianoteqpaid; trial has 8 silent notes and stops after 20 minutes
Wind instruments built by mathSWAMpaid
Guitar ampNeural DSPpaid, free trial
Clean effects: EQ, compressorFabFilterpaid, 30-day full trial
ReverbValhalla Supermassivefree
SynthesizerVital or SerumVital Basic is free; Serum is paid
The same recordings in both enginesSalamander Grand Pianofree

Save the WAV files of A one time, during the free trial. After that, compare with the saved files. Measure their sound. Do not copy their recordings into your product.

Free and paid were checked on 2026-11-17. Check each site for the terms today.

Ears and numbers

EarsNumbers
Time for one testminutesseconds
Same result every timenoyes
Shows where the bug isnoyes: 1320 Hz, -12 dB
Runs after every code changenoyes
Knows if it sounds beautifulyesno