It Hands You the Synth Patch, Not the Song

Rochester, N.Y. Most AI music tools hand you finished audio. When you're producing, you usually want a sound you can shape. WingmanPro gives you a real Surge XT synth preset instead. Drop in audio from your track, describe the sound you want ("dark slow riser") and you get a patch you can open and tweak like any other.

The first stage listens. librosa pulls out BPM, key, energy, brightness, noisiness and attack shape, and each one becomes a label a language model can read. Three Hugging Face models add vibe (CLAP, zero-shot), genre and AudioSet tags. That heavy ML lives in its own Python venv and runs as a subprocess that prints JSON, so the main app stays light.

The second stage writes. The audio context and your prompt go to an LLM (local Qwen through Ollama by default, Claude or Groq if you prefer) along with a reference of Surge XT's parameters and a template patch. The model writes patch XML, which I parse and validate before trusting it. Then I wrap it into the binary FXP preset format myself: chunk headers, Surge's plugin ID, all of it.

The desktop half uses DawDreamer to load Surge XT as a VST3, render MIDI notes to audio and play them back, inside a PyWebView window with an AI mode and a synth mode. DawDreamer exposes 2,855 plugin parameters; I mapped 68 of them to names a person would recognize. The patch XML is the single source of truth, and a round-trip check confirmed that changing the filter cutoff really changes the rendered audio.

The first working commit was the full path: text in, vibe-matched against the audio, preset out. Getting there took two days. The written Phase 2 plan swaps the LLM step for CLAP embeddings and a trained PyTorch decoder that maps straight to Surge parameters.