Martin Heinze | marts~

Tag: algorithmic composition

  • UnRAVEl: speculative composition in latent space

    RAVE models encode data from the audio domain into highly compressed latent representations. Based on statistical information retrieved from these encodings, a speculative compositional practice can be established inside the latent space of the models. It is derived from the improvisation tactics empirically proven in Latent Jamming.

    For a proof of concept (POC), I have written UnRAVEl, a set of Python scripts, that cover a three step process:

    • audio data encoding: 1-n audio files are encoded into arrays in the shape of a model’s latent space.
    • generation of synthetic data: 1-n encodings are evaluated for their data distribution. Based on the results, arrays of synthetic data are generated which are used to populate preset patterns and apply alterations to these patterns.
    • decoding of synthetic data into audio data: 1-n generated/ synthetic data arrays are being decoded and up sampled back to the audio domain using the same model as in the encoding process.

    Considerations and hypotheses

    Prior distribution vs. encoding audio

    While models contain statistics learned during training, encoding real world audio data through a model for evaluation can be more robust when it comes to deviations from the original data set. However, domain data similar to what the model has seen during training should yield most truthful statistics for informing synthetic data generation.

    Distribution

    VAEs like RAVE assume a Gaussian latent prior. Generating synthetic data using mean and standard deviation retrieved from the encodings should create truthful results.

    Data distribution in a set of encodings.

    In UnRAVEl, this is covered by normal distribution; other distribution types, e.g. uniform or correlation-based, are experimental but, depending on the model, can create more interesting output since they sample from a different value range and logic.

    Normal data distribution in a generated array.
    Uniform data distribution in a generated array.

    Tempo quantizing

    Latent embeddings created with RAVE are highly compressed representations of audio domain data. A sample rate of 44.1KHz corresponds to 21 data points times the number of latent dimensions in the model’s configuration. The audio domain resolution is high enough to be more or less irrelevant for tempo considerations, however, with the low resolution in latent space, limitations to achievable tempi are expressed by:

    60 * (model sample rate / model compression) / latent data points

    For example:

    60 * (44100 / 2048) / 11 = 117.45 BPM

    This leads to the following quantized tempi (in 4ths, 8ths for double time) achievable by looping k amount of latent data points.

    kTempo BPMDouble timeAudio
    …………
    7184.57–Example
    8161.49322.98Example
    9143.55–Example
    10129.19258.38Example
    11117.45–Example
    12107.66215.32Example
    …………

    Patterns

    The compositional approach in latent space exemplified in UnRAVEl is based on high level structural considerations, e.g. repeating (parts of) data arrays, replacing data points and/or slightly altering them while boundaries like value distribution or tempo quantizing need to be considered.

    Compositional ideas can be established defining patterns; in UnRAVEl four patterns have been implemented as a starting point.

    fibo

    A given array of shape (data points, latent dimensions) is repeated along the fibonacci series of integers. 1 corresponds to the first row in the array, 2 corresponds to the first two rows in the array, …, 8 corresponds to rows 0-7 and so on.

    orale

    An approximation to a standard sequence in electronic music building an array using a 3:1 scheme where the original array is repeated three times and a fourth time with subtle changes applied to its values. This sequence is then repeated and altered again in the same scheme of 3:1.

    blender

    Two arrays are blended into one another by replacing single data points sequentially after n repetitions, starting with the first value in the first dimension, followed by the first value in the second dimension and so on until the last value in the last dimension has been reached.

    swapper

    Values of randomly picked data points in two arrays of the same size are swapped. The altered array is repeated n times.

    Use in Pure Data

    UnRAVEl generates .npy arrays that can be decoded to the audio domain using the dedicated script. Alternatively, latent audio files are being written; these are basically multi channel (= latent dimensions), double precision (= for values outside -1/+1 boundary) .wav file at e.g. 21Hz resolution (if data source was 44.1KHz). This format works with an abstraction I’ve written in Pure Data: ch4ns0n/ch8ns0n (note that only models with 4 and 8 latent dimensions are supported, but the component is fairly easy to extent).

    Acknowledgements

  • Reykjavík Sunburn

    In Reykjavík Sunburn, four different neural audio models, trained on my own musical material — a corpus of electronic music conventionally written and produced — and a private voice dataset are employed in an improvisational setting inside Pure Data, a visual audio programming environment where I perform Latent Jamming, a real-time improvisation practice with neural audio models that embraces concepts of algorithmic and generative composition techniques. I act in real-time inside the models’ latent space, steering mood, density, and rhythmicality by exploring parameter settings in signal streams that resemble latent embeddings. By doing so, I aim to replace deterministic composition with guided exploration: tweak, listen, stabilize, vary. 

    Reykjavík Sunburn concludes that neural audio synthesis can extend creative practices in music performance and composition by leveraging the unpredictable behaviour of the models from a control interface that caters for creative intent.

    In Reykjavík Sunburn each two RAVE and vschaos2 models are being used:

    • Black Latents: a RAVE V2 model trained on the Black Plastics series – 28 tracks/ 3h of drum- and percussion-heavy electronic music. The resulting model generates mainly percussive output with rough textures and a generally high grittiness. In the framework, this model is used as a leading asset to generate the rhythmic baseline and general percussive structure. 
    • Nobsparse: a RAVE V2 model trained on a hybrid dataset of Tech House and sonically sparse Drum & Bass (about 4h of audio material). The model’s characteristics are relatively clear, sterile, and lightweight sounds, harmonic textures, and an isolated but dominant low end. Depending on the process development during the improvisation, this model serves as a secondary texture generator but can also replace Black Latent’s role in the composition. 
    • VSC2_Nobsparse: this vschaos2 model has been trained on the same dataset as the Nobsparse RAVE model. In the composition, this model is used to generate interchanging pads and drone-like noise textures for transitions or simply to enrich an ongoing section of the recording with a harmonic layer.
    • VSC2_Martha2023: being the only model trained on voice data, courtesy of my daughter, this model adds a layer of rhythmical, pseudo-vocal sound on top of the otherwise „instrumental“ generations of the three other models. 

    Output examples

    Reykjavík Sunburn (Take 1 Redux) received recognition at the AI Song Contest 2025 where it was selected to the finalist shortlist of 10 out of >150 submissions.

  • Latent Russando

    Latent Russando is a semi-generative compositional framework written in Pure Data dedicated to exploring musical qualities in working with generative neural nets for audio, conceived both as hybrid instruments and as autonomous actors.

    Practices from generative music and algorithmic composition are used as mediators between human performer and the generative abilities of the neural nets, displacing and circumventing concepts of authorship and genius by empowering multiple independent agents in an improvisation-driven, co-creative process.

    The work is based on Russando. Serenade for six German Sirens, op. 43 by Hallgrímur Vilhjálmsson, a heteronym of conceptual artist Georg Joachim Schmitt. The original piece was composed in 2008 and premiered in the context of the (also fictional) art exhibition cologne contemporary — international art biennale 08 at Asbach-Uralt Werke in Rüdesheim. It is a three-part composition of approx. 33 minutes in length, in which six German emergency and police sirens are alternately sounded together or alone. In consultation with the creator, I trained models based on two neural net architectures (RAVE, vschaos2, both courtesy of IRCAM, Paris) on the original piece.

    Public performances

    An exemplary instantiation, Nebuloso, a 7.0 output recording of the Latent Russando framework, was shortlisted for Soundcinema 2025, a recorded sound festival at FFT in Düsseldorf, where the framework was premiered in October 2025.

    In March 2026, Etereo, another 7.0 composition based on the framework, was publicly presented at Music for Cinemas, an event series dedicated to experimental music at Filmrauschpalast Moabit, Berlin.

    Fluidante, a quadrophonic recording from the framework, is presented during International Computer Music Conference (ICMC) 2026 in Hamburg.

  • Sinusoidal Run Rhythm: implementation in Pure Data

    In his research and book on Sinusoidal Run Rhythm, Steffen Krebber describes a way of generating non-discrete rhythmical patterns by adding up in-phase cosine functions in integer ratios.

    Source: https://steffenkrebber.de/research/sinusoidal-run-rhythm

    Sinusoidal Run Rhythm comes with shifts in timing relative to discrete rhythmic or polyrhythmic patterns and also adds volume weighting.

    This temporal shifting is significant because it illustrates how rhythms generated from wave interference behave differently from manually constructed or performed rhythms. These shifts and volume variations introduce a nuanced, fluid quality to the rhythm that is not easily replicable through traditional musical notation or performance. Therefore, Krebber considers Sinusoidal Run Rhythm to allow a subobjective perspective on rhythmic patterns.

    “‘sinusoidal run rhythm’ proposes a definition of rhythm as a wave. It does not conceive of time as discrete subdivisions, but makes it continuously quantifiable. Concurrently, through the aesthetics of wave additions, it does not present physicality as a merely subjective concept and thus liberates it from mystification.”

    Steffen Krebber (2024)

    In electronic music, similar kinds of rhythmic patterns often appear in low frequency oscillator based modulations e.g. in sound synthesis or filtering.

    Pure Data implementation

    PD-SRR is an implementation of Krebber’s concept of Sinusoidal Run Rhythm in Pure Data. It comes as both a standalone (pd-srr.pd) and modular (srrmod.pd) version, the latter for use in compositions. You can find it on GitHub.

    The standalone implementation works with Sinusoidal Run Rhythm applied to the amplitude modulation of a white noise generator similar to the web based application Steffen Krebber presents on his website.

    As suggested by Krebber, the implementation works with combinations of 2 partials (derived from the Farey sequence of order 8) and 3 partials (coprime triples up to 16) through dedicated selectors. For further experimentation, up to 4 partials can be set by using the manual selection option.

    The modular version of the implementation allows setting the partial combination from the calling patch and outputs the result through its outlet for use in signal modulation.

    In the following video, I’m showing three scenarios where srrmod.pd is used to modulate both sound synthesis parameters as well as a filter in a compact setup.

    It should be interesting to expand the concept of Sinusoidal Run Rhythm to larger compositions both incorporating it as rhythmical baseline for sound generators and modulators alike but also experiment with an application on a compositional level.

    Thanks to Steffen, whom I had the pleasure to first meet and talk to at the ArtSearch 2024 symposium at ligeti zentrum in Hamburg recently. His presentation on Sinusoidal Run Rhythm sparked quite a few new ideas on my end.

  • Risset rhythm: implementation in Pure Data

    Jean-Claude Risset described the auditory illusion of an “eternal accelerando”, where, similar to Shepard tones for pitch, a rhythm can be structured and played back in a way that creates the perception of constant acceleration.

    In his 2011 paper “Scheduling and composing with Risset eternal accelerando rhythms”, Dan Stowell provided a solution for implementing eternal accelerandos on (rhythmic) audio samples by employing variable play back rates and amplitudes distributed to a number of sample play back streams that run synchronized.

    Illustration of Risset rhythm streams in Stowell, 2011

    Pure Data implementation

    Risset Sampler is a Pure Data implementation of an eternal accelerando I programmed following Stowell’s paper.

    The sampler has 5 streams set up to play back any given sample in a loop to generate the eternal accelerando effect. The individual play back rates and depending amplitude envelopes for each stream are calculated based on Stowell’s formulas (2) and (3).

    In addition to the stand alone risset_sampler.pd abstraction, i’ve also provided a modular version in the repo that can be embedded into larger compositions.

    Risset Remixes

    With the modular version of the Risset Sampler (jaycee.pd), I’ve done experiments on composition level which resulted in the Risset Remixes.

    MARTSM<>N – Risset Remixes [datamarts/2KOMMA2]: Bandcamp, Nina

    For these, I’ve been using stems from my tracks Axe Why Dread, Ting and Double Dub. In each composition, sample loops separated from these stems are being played back using the jaycee.pd abstraction.

    Events within the compositions are triggered by the completion of sub stream cycles in each sampler, creating a generative, closed circuit system. On the remix of Ting, manual triggers were also applied during recording. 

  • Saatgut Proxy

    Saatgut Proxy is an experimental generative setup in Pure Data that creates both randomized and repeatable pathways through the latent space of two neural audio model architectures (RAVE, vschaos2) at the same time.

    The framework is based both on generalized abstractions that I have developed for the Latent Jamming use case and additional prototypes of techniques that I turned into dedicated abstractions later on.

    Saatgut Proxy was presented at ArtSearch symposium at ligeti zentrum in Hamburg during a lecture performance and along with a presentation on Latent Jamming and shared human/AI agency in electronic music creation at Storytellers+Machines 2024 conference at SODA (Manchester).

    Output examples

    The framework led to the following release artifacts:

    MARTSM=N – VARIA 3L [datamarts/2KOMMA1]: Nina

    MARTSM))N – Saatgut Proxy Reflux [datamarts/2KOMMA0]: Nina

    MARTSM))N – Saatgut Proxy [n/a]: Bandcamp

  • Fibonacci Jungle

    While singular generative composition techniques have already become an established part of the creative process in music writing, holistic approaches to generative music production in traditional electronic dance music genres yet seem under-represented both in theory and practice.

    Fibonacci Jungle is a POC for a simple to use generative framework for Jungle and Drum & Bass built on the Fibonacci number sequence as structural alternative to conventional meters and track build-up.

    The framework is implemented in Pure Data. It uses probability and randomization within a pre defined set of genre typical parameter settings (tempo, harmonics, sample selection). Fibonacci Jungle allows creating stand alone tracks in a Jungle and Drum & Bass aesthetics with only a few clicks and can be individually customized.

    For a detailed description of concept and implementation, see this paper and the below presentation video from Generative Music Prize 2024, hosted by IRCAM, where Fibonacci Jungle was awarded 2nd place.

    The source code for Fibonacci Jungle is publicly available on GitHub.

    Output examples

    Fibonacci Jungle Versions – an EP of recordings based on the Fibonacci Jungle framework. Each track/ version has been recorded multiple times and individually distributed through different channels (Bandcamp, Nina, Spotify).

  • Spoor

    Early prototypes and setups in latent embedding mimickry and establishing a control level baseline in latent space have led to Spoor, both name of a loosely coupled set of Latent Jamming techniques and two releases:

    MARTSM/\N – Spoor Widen [datamarts/1KOMMA9]: Nina

    MARTSM/\N – Spoor [n/a]: Bandcamp

    Below video shows the setup that lead to tracks Loom and Loom Rewood.

    Track Architects was based on the following patch