Neural Audio Theory

By Eduardo J. Barrios

Neural Audio Theory is an open educational reference for both people making music with AI and people engineering the systems behind it.

Choose your learning path

Create music with AI: User Guides

Choose this path if your goal is to make, troubleshoot, finish, or release AI-assisted music. It uses plain language and assumes no machine-learning background.

Start here: AI Music in Plain Language → Prompt Engineering → Production Workflow

Build and study neural audio systems: Engineering Docs

Choose this path if your goal is to understand, build, train, integrate, or evaluate neural audio systems. It assumes comfort with software concepts and introduces the required mathematics along the way.

Start here: Digital Audio Basics → Music Representations → Signal Processing Basics

Technical claims follow the guide's Reliability and Sourcing standard, which distinguishes documented facts, observations, inference, and unknowns.

Both paths describe the same field from different levels of abstraction. You can switch between them whenever a practical question needs a technical explanation—or an engineering concept needs a musical example.

What You Will Learn

This handbook covers the full landscape of AI music — from foundational audio theory to production workflows to cutting-edge research:

Foundations

• Audio Fundamentals — digital audio, psychoacoustics, music theory, and codecs
• Concepts — embeddings, latent spaces, neural codecs, text-audio alignment, and music representations

Engineering

• Mathematics — FFT, mel spectrograms, attention math, loss functions, and signal processing
• Architecture — transformers, diffusion models, VAEs, GANs, and U-Nets for audio
• Training — dataset curation, augmentation, training strategies, and evaluation metrics

Systems

• Model Zoo — MusicLM, MusicGen, Stable Audio, Jukebox, Suno, and Udio
• Advanced Topics — multimodal generation, real-time inference, fine-tuning, and controllable generation

Practice

• Producer Handbook — workflows, troubleshooting, genre prompting, mixing, stem separation, and vocal synthesis
• Prompt Engineering Guide — conditioning embeddings and prompt structure
• Tools & Ecosystem — DAW integration, open-source tools, and API patterns

Reference

• Ethics & Legal — copyright, training data rights, and responsible use
• Glossary — comprehensive A–Z reference of AI music terminology

Core Engineering View

Most AI music systems follow a practical pipeline:

1. Data preparation: normalize, segment, and annotate large music/audio corpora
2. Representation: transform audio into spectrograms, codec tokens, or latents
3. Modeling: train sequence or diffusion networks with conditional inputs
4. Inference control: steer generation with prompts, structure tags, and guidance scales
5. Post-processing: mixing, mastering, and quality assurance
6. Evaluation: combine objective metrics and human listening tests

If You Just Want to Make AI Music

If your main goal is creating songs quickly, start with the beginner page:

• For Dummies: AI Music in Plain Language

It translates the same engineering foundations into plain language while staying technically accurate, so you can move from "just prompting" to more consistent, controllable outputs.

Then dive into the Producer Handbook for practical workflows, genre-specific prompting tips, and mixing techniques for AI-generated audio.

Example Mathematical Building Blocks

Continuous Fourier transform for a signal $x(t)$:

$$X(f) = \int_{-\infty}^{\infty} x(t) \, e^{-j2\pi ft} \, dt$$

Cosine similarity for embedding vectors $\mathbf{a}$ and $\mathbf{b}$:

$$\text{sim}(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \, \|\mathbf{b}\|}$$

Use the sidebar to explore all sections for a full engineering-level understanding.