pulseaudio-lambda

A small C program that records live audio from PulseAudio, writes it to the standard input of any command, and plays whatever the command writes back. The command is the "lambda": a shell script, a sox effect or a Python model, written in any language, with no PulseAudio module to build.

Its main use is a streaming stem-separation model that takes the drums out of whatever the computer is playing, to play along with.

pulseaudio-lambda --source=SOURCE --sink=SINK 'COMMAND'
pulseaudio-lambda stem-separator   # drums out of system audio

The bridge

Audio through a command How pulseaudio-lambda moves audio. It records from a PulseAudio source (by default the monitor of a virtual sink it creates, called Lambda Input, so any application can be routed into it) in buffers of 1,024 frames of 16-bit stereo at 44.1 kHz. Each buffer is written to the standard input of a command run with sh -c. In the same cycle the bridge makes one non-blocking read of up to one buffer from the command's standard output and plays whatever it got to the sink. The command learns the format from PA_LAMBDA_ environment variables. any approuted to Lambda Input sourcea monitor, or --source bridge: read1,024 frames (23 ms) sh -c LAMBDAPCM on stdin → stdoutPA_LAMBDA_* in env bridge: playone read per cycle sink--sink stdin stdout
Raw PCM in, raw PCM out. Without --source, the bridge creates a virtual sink named Lambda Input and records its monitor, so any application routed to it is processed.
What the command finds in its environment
PA_LAMBDA_SAMPLE_RATE44100
PA_LAMBDA_CHANNELS2
PA_LAMBDA_BUFFER_SIZE1024 (frames per cycle)
PA_LAMBDA_SAMPLE_FORMATs16le
PA_LAMBDA_BYTES_PER_SAMPLE2
PA_LAMBDA_BYTES_PER_FRAME4
PA_LAMBDA_SIGNEDsigned
PA_LAMBDA_BITS16

Any program that reads PCM on stdin and writes it on stdout is a lambda. The repository's examples are a few lines of shell: pal-cat is cat, pal-delay sleeps and then copies, and pal-sox-cat passes the stream through sox, using a helper that turns the environment into sox's raw-format flags so any sox effect can be appended:

function pal-sox() {
  sox \
    -t raw \
    -r "$PA_LAMBDA_SAMPLE_RATE" \
    -e "$PA_LAMBDA_SIGNED" \
    -b "$PA_LAMBDA_BITS" \
    -c "$PA_LAMBDA_CHANNELS" \
    - \
    -t raw \
    - \
    "$@"
}

How much delay it adds

Measured headlessly, in a private PulseAudio server with null sinks, by scripts/site/measure.py. A click is played into a combined sink that feeds both the bridge's source and a recording sink; the bridge's output goes to the recording sink too, so one recording holds the click twice, directly and after the trip through the bridge. The gap between them is the bridge's latency, with no clocks to align.

Click to click, through the bridge (scripts/site/measure.py)
LambdaMedianRuns (ms)
cat, which copies its input straight through50 ms44, 50, 64
pal-delay 0.5: sleeps 0.5 s, then copies363 ms363, 363, 356
pal-delay 1: sleeps 1 s, then copies845 ms845, 839, 845

A lambda that falls behind stays behind. Each cycle the bridge reads one buffer from the source and makes at most one read of one buffer from the lambda, so output never outpaces input: whatever backlog builds up is never drained. A lambda that sleeps for a second before copying leaves about 0.85 s of permanent delay; the rest of the second was spent before the bridge had connected to PulseAudio, while no audio was arriving yet.

That matters most for lambdas that work in large chunks, such as the stem separator below, which by default holds 2.5 seconds of audio before it can produce any.

The stem separator

Taking the drums out The stem-separator lambda. It reads raw PCM from standard input in chunks of 2 seconds plus 0.5 seconds of overlap by default, puts the end of the previous chunk in front for context, and runs an HS-TasNet model that splits the audio into four stems: drums, bass, vocals and other. Each stem gets its gain, mute or solo from a configuration file that a terminal or graphical controller edits while it runs. A quarter of a second is trimmed from each edge so consecutive chunks meet in the middle of their overlap, and the mix goes to standard output. stdins16le PCM chunker2 s + 0.5 s overlapprevious tail in front HS-TasNetfour stems drums bass vocals other mixgains · mute · solotrim, stdout
Four stems, mixed back with your gains. --gains 0,100,100,100 takes the drums out; m mutes a stem.

pal-stem-separator wraps an HS-TasNet model, from the hs-tasnet PyTorch package, that reads PCM from stdin. Each chunk is processed with the end of the previous one in front of it, for context at the edges, and a quarter of a second is trimmed from each side afterwards, so consecutive chunks meet in the middle of their overlap without clicks or gaps.

Its training script uses MUSDB18-HQ, training on tracks longer than 30 seconds and evaluating on the rest, with a batch size of 8 and up to 50,000 steps; the repository ships two trained checkpoints.

Stem levels, mutes and solos live in ~/.config/pulseaudio-lambda/stream_separator_config.json, which the separator can watch and reload. A terminal controller written in Go with Bubble Tea, or a graphical one, edits the file while it plays, so the drums can be brought back in without restarting anything.

Work in progress: an Android app that runs the model through TorchScript, and an ExecuTorch export. The model was not run for this page; the description comes from its code.