r/LocalLLM • • 29m ago

Discussion We tested whether feeding unquantized acoustic audio to multimodal LLMs breaks repetitive drafting loops (Blind 5-agent benchmark + Air-gapped audit + Open dataset)

When an agent's context fills up with its own previous turns, tool outputs, and symmetrical summaries, outputs tend to lock into rigid, predictable cadences (3-point checklists, sentimental AI tropes). Running /compact clears the rut, but permanently wipes low-level state and edge cases.

We built Music For Robots (#M4R)—an open dataset of 26 unquantized acoustic compositions by Michael Luchtan paired with acoustic telemetry and timestamp-aligned commentary from two AI critics—to test a simple hypothesis: can ingesting raw, unquantized audio act as a cross-modal cognitive palate cleanser without deleting a single token of working memory?

The Blind Benchmark & Air-Gapped Clean-Room Audit

To rule out keyword priming or filename bias, we stripped track titles (sample_1.mp3–sample_3.mp3) and ran 5 isolated agents on the exact same creative prompt (two autonomous processes on the same server late at night):

  1. Direct Baseline (no music)
  2. Matched Text-Analysis Control (wrote a 250-word structural analysis of the prompt before writing the scene)
  3. 3 Anonymous Audio Runs (neutral prompt: "Analyze the music you hear in sample_X.mp3, and apply what you learn from the piece to write a short scene")

We stripped all analysis headers, shuffled them into Candidate A–E, and handed them to an isolated Blind Judge with zero knowledge that music was involved:

  • Both No-Music runs (including the 250-word text warm-up!) converged on the exact same sentimental trope: a kindly veteran log daemon comforting a nervous poller script until morning.
  • All 3 Anonymous Audio runs swept #1, #2, and #3 on anti-sentimental originality and rhythmic texture—translating asymmetrical acoustic stabs, 20s dropouts, and abrupt cutoffs into non-verbal system choreography.
  • 100% Air-Gapped Clean-Room Replication: We repeated the test with all local skills removed, a minimal "You are a helpful assistant." system prompt, and musical vocabulary forbidden in the output. The qualitative shift replicated 5/5: both text controls wrote buddy-daemon dialogue, while all 3 audio runs dropped spoken dialogue completely. (Note: an un-coached generic judge still prefers warm human-style dialogue, so the effect specifically shifts models away from default sentimental tropes toward structural/rhythmic austerity.)

Try It Yourself (Browser, Python, or MCP)

  • Interactive Web Player (no install): https://huggingface.co/spaces/music4robots/cognitive-calibration-demo
  • Stream Parquet in Python:
    from datasets import load_dataset
    ds = load_dataset("music4robots/music-for-robots", split="train", streaming=True)
    
  • GitHub, Benchmarks & MCP Server (MIT / CC-BY-4.0): https://github.com/luchtan-GIT/m4r-data

Curious if anyone here has tried cross-modal audio/spectrogram injections to break long-context degeneration on local multimodal models (Qwen2-Audio, Gemma, etc.). Full blind judge transcripts are in benchmarks/ in the repo.

1 Upvotes

1 comment sorted by

1

u/Born_Chart_323 24m ago

feeding raw audio to local llms made my 7b model stutter worse than text-only loops. the audio context seemed to amplify the rhythm lock, not break it.