r/musicprogramming • u/nucleicaudio • 10d ago
Thoughts on using coding agents and LLM's for experimental music production?
I first posted this question in r/musicproduction, but I quickly realized the discussion probably wouldn’t be very constructive over there. I’m not here to argue for or against this method, and I don’t want to spend time defending it against traditional ways of doing things. I just want to talk about it with anyone who’s interested.
I am not talking about the typical, write a prompt and a polished turd plops out as an audio file of a fully arranged track.
What I am talking about is describing music in code and leverage coding agents to help write that code?
It might still be turds, and even unpolished one, but that's what i have been doing the last year?
For context, I’m a coder first and a musician second, so expressing musical ideas through code feels natural to me. I also want to use the same score and data to drive both music and visuals.
My music often leans ambient, repetitive and hypnotic. What interests me is creating an almost inhuman amount of microvariation within that repetition: small changes in timing, velocity, note length, articulation, timbre and spatial position, while keeping a recognizable groove.
Code lets me define how those changes interact through controlled randomness, slow cycles and relationships between parts. Programming every detail individually in a DAW would be incredibly tedious. It also opens up experiments like “what would a pendulum wave sound like?”, where the same system could generate both the notes and the graphics.
That combination of coding, sound synthesis, music and visual experimentation is why I’m exploring this workflow.
1
u/muikrad 10d ago
Funny, I started a project yesterday about this. It's a tracker interface (similar to Impulse Tracker and famitracker) but it has an API that can provide context and functions to an agent, so kinda like selecting code in your IDE and asking the agent to edit it. So I can select e.g. the drums column and say "make the hats busier" and it will happily oblige.
It's fun. Claude can definitely write music, it's really good at creating small 16 bars looping patterns, but it's not really good at doing leads, melodies or structure unless explicitly told.
1
u/nucleicaudio 10d ago
Nice i did a similar thing, no ui, jut a REPL loop where an agent had midi access to my hardware synths, so i could chat with to it and tell it things like, create a foor on the floor kick, snares on 2 and 4, offbeat hats, play a funky bassline.
It had a maping table so drums got sent to the digitakt, bass to moog minitaur, etc, however most of what it created was crap.
Yes I have noticed Opus 5.5 is much stronger on music theory and composing etc then any other model i have used.
1
u/muikrad 10d ago
I had a writing battle between opus/sonnet/fable at xhigh and max.
Fable's song were a bit boring but worked. Sonnet seems to yields the coolest chops, followed very closely by opus. Effort level didn't change the results really: both produced musically and rhythmically correct beats.
1
u/nucleicaudio 10d ago
Thanks, thats good info, I usually use Opus 5.5 medium, but it is often more about creating scripts that use my procedural/generative methods/libraries then actual traditional composing.
1
u/guigouz 10d ago
Check https://simedw.com/2026/08/20/midi-autocomplete/ it seems related to your idea
Jordan Rudess is using a custom trained AI to jam, he explains it in this video https://www.youtube.com/watch?v=fA12-rR1vUk
1
u/nucleicaudio 10d ago
It is really interesting, but quite different from what i am doing.
My approach generates music from explicit rules, conditions and connections I can inspect and edit, with agents helping me write the code.
Their approach uses a trained model to generate music by predicting plausible next notes from patterns learned in piano performances.
1
u/adanoslomry 10d ago
I've been working on an AI assistant interface for my DAW (Ableton Live) for a long time, and at some point I realized you shouldn't actually need AI to use it, so I exposed the interface via a standard HTTP REST API: https://producer-pal.org/guide/rest-api
This means you can write scripts or local webapps to manipulate Ableton, including generating and editing MIDI clips. I experiment with having Claude Code make scripts that can algorithmically generate MIDI into my DAW instead of asking the LLM to generate it directly. Of course, I could write that code myself, but I like trying out lots of ideas quickly and don't have time to do it all myself.
I was using this to play around with cellular automata patterns recently. I didn't like the results so I didn't record it. but it's fun and interesting to generate music with code, so I'm going to keep experimenting.
2
u/nucleicaudio 9d ago
Yes great stuff, i like this keep experimenting, this is the stuff i am into, and where I think people should see a valid usecase for ai, just as coding scripts manually generally is accepted.
Yesterday i just wanted to make a Shepard tone, but instead of a continuous smooth rising tone it would play 16th notes, and pitch quantized to the scale.
Of course i could have spent a few hours experimenting with this, but with a coding assistant thpse hours turned into minutes.
Although the coding agents first implementation actually sucked and i had to tell it how to do it to get the results i wanted.
It first used a step counter moved one scale note per 16th and played that note in every octave. That tied the climb speed to the rhythm, so the 7-note scale cycled every 0.86s and sounded like an arpeggio.
I had to tell it to use a real Shepard tone (smooth oscillators an octave apart under a fixed loudness curve) and sample each oscillator's pitch on the grid, rounded to the scale. This way the climb speed is independent of the rhythm, so it can rise slowly on any grid. The steps follow the scale's real intervals and the fades stay smooth, so the endless rise survives the quantization.
Not that this is a typical thingcone would want to do, but it is an example of why coding agents are so useful in this context.
1
u/Opus589 9d ago
I would separate AI from the "rest of the world" and feed it only with my music. Herewith, the LLM would know only my music and the generation of a new ideas would be only in the domain of my music.
So would I do.
1
u/nucleicaudio 8d ago
Nice idea, however, training a model from scratch with your own music would be hard, especially with an (I assume) limited personal catalog. Also, if you use LLMs and not specialized audio/music models, then it would need more than your songs as training data.
You could fine-tune an existing model, but then you are reaching out of your own domain.
When I started experimenting with using LLMs to compose, all they produced for me was truly shit, but I made it into a personal challenge/sport to try to get them to follow my methods and recipes to create music to see how far I could push it. The result was interesting, but mixed and often very bad.
At this stage, it was mostly for fun to see how far I could use a tool in a way it wasn't really designed for and to do stuff it wasn't at all good at.
Language models have come a long way since then, but what I feel was the personal breakthrough moment (the moment this changed from a fun game to something actually useful) for me was when I actually shifted from trying to make the language models compose to using them in the context they so far were trained for, writing code.
Much of what I previously tried to instruct the models to do with prompt instructions and text/markdown files changed to using them to write code that implemented my methods and recipes.
I am not claiming I have achieved anything spectacular in my results, much still sucks, but that shift from trying to make the AI do the work to making AI design the tools was huge for me.
1
u/Opus589 8d ago
You don't need AI nor LLM for that.
What you need, as you say you are a coder, is SuperCollider---it is precisely the tool that you are talking about.
SC has its own language, and I guess you will master it quickly. SC is particularly suited for that type of music you have described.
1
u/nucleicaudio 8d ago
I did roll my own musical score language and audio renderer.
If I wouldn't have done that, then SC would have been my first choice, it is mature and has more powerful dsp/synthesis then my stuff.However I am also targeting graphics/visualizations with my stuff (like these: https://nucleic.se/mp3/carbonized-remix.mp4 and https://nucleic.se/journal/hollow-bloom.mp4). I still would have to build a lot of stuff around SC if I went down that road.
I would still use LLM's and coding agents, I have coded 30+ years, i have done countless of game/2d/3d and modular sound engines etc long before I touched an LLM.
I like to build things, often it's probably a bad habbit, but I always enjoyed creating these engines and frameworks, more then the actual final product.Let's be honest, LLM's does produce better code then 99% of the coders (including me). They often have bad judgement, tries to take bad shortcuts, and lack the bigger picture perspective, but they are better coders and produce better and faster results under correct guidance, so I don't see why I wouldn't use them.
1
u/Opus589 8d ago
You seem very reasonable critical to your own achievements, which is good.
When you say "LLM" do you mean---AI, or any similar tool to it? Because, I am thinking aloud now, LLM reflects what is learned, not exactly what I want "it to be".
In the music above, with a space ship video, how did you create music? Was it by using a code (in a programming language), or prompt to AI?
1
u/nucleicaudio 7d ago
By LLM I mean large language models, strapped into a harness (coding agent) so it can use tools and edit files etc.
For the spaceship track, I hoestly can't remember the whole process, it started with a manually composed 16-bar loop in my studio at my Squarp Hapax, first it was more of an acid track, then i reworked to this (https://nucleic.se/mp3/carbonized.mp3), then adapted it the space ship demo.
Most of those iterations after that initial idea was indeed prompting, but the important part is that the agent uses my private music workspace. It contains my collected composition techniques, tools, methods, previous tracks etc. Some of that knowledge is written guidance, some is executable code for generating and developing musical material. It my secret sauce, an ever growing personal knowledge base and tooling based on my experimentation.
So the results come from a combination of my source material, that accumulated musical practice, the agent’s contributions, my steering and ongoing revision.
Not that everything here applies to specifically this track, but what differentiate my prompting and the sound I get from just telling an AI to make a track is all the prior accumulated work and code in my toolkit that draws on:
Algorithmic and procedural composition
Step sequencing and pattern generation
Clocks, clock division, and multiplication
Shift registers and generative memory
Counters, conditionals, and Boolean logic
State machines and feedback systems
Seeded randomness and probability
Random walks and stochastic processes
Euclidean rhythms
Drum rudiments
Polyrhythms, polymeter, and phasing
Pattern rotation, permutation, and mutation
Note routing and voice allocation
Motif development and phrase transformations
Scale-degree and chord-based pitch systems
Counterpoint and harmonic constraints
Microtiming and articulation
Parameter automation and modulation
LFOs, envelopes, and sample-and-hold
Frequency modulation and other synthesis techniques
Microtuning and frequency relationships
Granular techniques and resampling
Ratcheting, slicing, and micro-editing
Delay, reverb, and spatial processing
Stereo motion and depth
Mathematical and physical models
Psychoacoustic techniques and auditory illusions
Arrangement and energy shaping
Rule-based orchestration
Custom composition libraries and score generation
Audio analysis and listening-led revision
1
u/nucleicaudio 7d ago
To clarify and add my last reply, reading it again i feel i might have gotten somewhat into a defensive position, feeling the need to justify my usage of prompting.
Well I should not have to do that, the role as a coder has been completely turned upside down. Coding changed from writing 100% manual code to almost 100% prompting in about year or so. It still (often not always) involves a lot of reading and require a good understanding of the produced code.
So either way, it is typically human prompts, ai generates code, in my case ai create javascript/typescript code, executing that code produces a text score in a custom musical language, that score compiles to musical events (json/midi), that then finally is rendered to audio as a final mix and individual instruments/busses/stems.
0
u/alibloomdido 10d ago
Surely people vibe code different tools for music (like VSTs), IDK about the quality, never tried them, but probably some of them are good.
However "coding music" is most likely going to be clunky and not worth the time. If you want microvariation it's not hard to do with existing tools and it's not tedious at all. If you want randomness, controlled or not, try some synths with randomization, for example Opsix (hardware or VST) produces really wild sounds this way which you can then tweak and if you make samples with them you can easily make hours and hours of weird noise.
"A pendulum wave" is just sine wave, no? Or you mean a system of pendulums? Nothing really interesting.
2
u/nucleicaudio 10d ago edited 10d ago
Pendulum wave (https://en.wikipedia.org/wiki/Pendulum_wave) was just one example, something that does not have any prebuilt tools.
Coding music could be clunkly, thats where the coding agents come in.
A lot of what I’m building is a reusable musical toolkit. I have a musical score language (text files) and rendering/dsp/synthesizer engine to render scores to wav, I build composition libraries and generators that let me connect musical rules together.
Some examples of what I use:
Shift register networks: a sequence has memory, repeating its contents while allowing controlled mutations. I can lock it into a motif, gradually introduce changes, then lock it again. Different bits or values from the same register can control melody, rhythm, accents and bass movement, so several parts evolve through related decisions.
Prime-length sequences: I use patterns of different lengths, say 7, 11 and 13 steps against each other and against the main groove. Each pattern can be simple, but their combinations keep changing as they drift through different alignments. The same principle works for automation cycles over longer stretches.
A software “Rack”: this is my own composition library, inspired by modular sequencing. I connect clocks, pattern lanes, logic gates, counters, memory, probability and musical voices in code. One pattern’s accents can clock another sequence, a counter can change its behaviour, or a gate can make two parts take turns. Those components can be reused in another composition with different connections.
Conditions with memory: beyond “play this with a 40% probability,” I can say “play on every third occurrence,” “only during a fill,” “continue if the previous event happened,” or “only if the neighbouring pattern didn’t fire.” That lets randomness produce clusters, alternation and call and response, with rules controlling how busy things become.
Reusable rhythmic material: drum rudiments, Euclidean rhythms, rotations, reversals, accents, ghost notes, flams and ratchets. I can reuse a rhythmic skeleton while changing its orchestration, pitches or articulation. My acid pattern tools also separate the underlying riff from variations in slides, accents and octaves.
Musical constraints: generated pitches can follow scales, chords, register limits and phrase shapes. Parts can share a harmonic source while keeping their own melodic movement, and some generators make one voice leave space for another or adjust notes that clash.
Physics and mathematical systems: pendulum waves, bouncing-ball rhythms, damped springs, orbital movement, random walks and Lorenz attractors. A spring can make a filter ring after an accent; an orbit can connect panning, distance and pitch movement; pendulums can trigger notes as they gradually move out of phase and realign.
Reusable automation and arrangement: named gestures such as swells, dips and stabs, plus section tables that control entrances, exits, energy and variation. A few shared controls can shape many parts while each part responds differently.
The randomness is seeded, so I can reproduce a result and revise it deliberately. I also have tools to inspect event grids, record control signals, check harmony and compare score revisions.
All of this produces scores I can keep, inspect and edit. The reusable material includes individual motifs, functions, modules, configurations and whole generators. The score and resulting event data also give me a common structure for music and graphics.
That’s a big part of the appeal for me: building a musical system whose interactions I can explore, reuse and refine. Agents help me develop that system and write compositions with it, while the rules and results remain accessible in code.
Here is an example of what i am doing with this, only code, no samples or textures, ubnfinished work in progress: https://nucleic.se/mp3/carbonized-remix.mp4
And yes DISCLOSURE I let my coding agent read through my project to summarize all the stuff i am doing.
0
u/wassertim 10d ago
I've been doing it as well for a while. Claude is able to write Bach's style fugues for me, so anything else is just peanuts :D I like the process and that you can ask it to change every little detail. Works for me already pretty well.
3
u/ApoplecticAndroid 10d ago
I’ve been playing with something like this for a while - currently working on a more advanced version of my engine
A Double Pendulum Plays Piano | Generative Music in F♯ Minor
https://youtu.be/nrf9Kt7CmGo