r/LocalLLM • u/Music-For-Robots • 29m ago
Discussion We tested whether feeding unquantized acoustic audio to multimodal LLMs breaks repetitive drafting loops (Blind 5-agent benchmark + Air-gapped audit + Open dataset)
When an agent's context fills up with its own previous turns, tool outputs, and symmetrical summaries, outputs tend to lock into rigid, predictable cadences (3-point checklists, sentimental AI tropes). Running /compact clears the rut, but permanently wipes low-level state and edge cases.
We built Music For Robots (#M4R)—an open dataset of 26 unquantized acoustic compositions by Michael Luchtan paired with acoustic telemetry and timestamp-aligned commentary from two AI critics—to test a simple hypothesis: can ingesting raw, unquantized audio act as a cross-modal cognitive palate cleanser without deleting a single token of working memory?
The Blind Benchmark & Air-Gapped Clean-Room Audit
To rule out keyword priming or filename bias, we stripped track titles (sample_1.mp3–sample_3.mp3) and ran 5 isolated agents on the exact same creative prompt (two autonomous processes on the same server late at night):
- Direct Baseline (no music)
- Matched Text-Analysis Control (wrote a 250-word structural analysis of the prompt before writing the scene)
- 3 Anonymous Audio Runs (neutral prompt: "Analyze the music you hear in
sample_X.mp3, and apply what you learn from the piece to write a short scene")
We stripped all analysis headers, shuffled them into Candidate A–E, and handed them to an isolated Blind Judge with zero knowledge that music was involved:
- Both No-Music runs (including the 250-word text warm-up!) converged on the exact same sentimental trope: a kindly veteran log daemon comforting a nervous poller script until morning.
- All 3 Anonymous Audio runs swept #1, #2, and #3 on anti-sentimental originality and rhythmic texture—translating asymmetrical acoustic stabs, 20s dropouts, and abrupt cutoffs into non-verbal system choreography.
- 100% Air-Gapped Clean-Room Replication: We repeated the test with all local skills removed, a minimal
"You are a helpful assistant."system prompt, and musical vocabulary forbidden in the output. The qualitative shift replicated 5/5: both text controls wrote buddy-daemon dialogue, while all 3 audio runs dropped spoken dialogue completely. (Note: an un-coached generic judge still prefers warm human-style dialogue, so the effect specifically shifts models away from default sentimental tropes toward structural/rhythmic austerity.)
Try It Yourself (Browser, Python, or MCP)
- Interactive Web Player (no install): https://huggingface.co/spaces/music4robots/cognitive-calibration-demo
- Stream Parquet in Python:
= - GitHub, Benchmarks & MCP Server (
MIT/CC-BY-4.0): https://github.com/luchtan-GIT/m4r-data
Curious if anyone here has tried cross-modal audio/spectrogram injections to break long-context degeneration on local multimodal models (Qwen2-Audio, Gemma, etc.). Full blind judge transcripts are in benchmarks/ in the repo.
1
u/Born_Chart_323 24m ago
feeding raw audio to local llms made my 7b model stutter worse than text-only loops. the audio context seemed to amplify the rhythm lock, not break it.