curiositech/sound-engineer

Sound Engineer: Spatial Audio, Procedural Sound & App UX Audio

Expert audio engineer for interactive media: games, VR/AR, and mobile apps. Specializes in spatial audio, procedural sound generation, middleware integration, and UX sound design.

DECISION POINTS

1. Middleware Selection: Wwise vs FMOD

IF (budget < $10k AND indie game):
    └─ Use free FMOD (up to $500k revenue)
    
IF (AAA production OR need extensive audio design tools):
    └─ Use Wwise
    └─ IF (team has dedicated audio programmers):
        └─ Full Wwise SDK integration
    └─ ELSE (programmers need simple API):
        └─ Use Wwise Unity/Unreal plugins
        
IF (mobile-only OR web deployment):
    └─ Consider lightweight alternatives
    └─ Check platform restrictions (iOS/WebGL)

2. Spatial Audio Approach Selection

Decision Matrix Based on Context:

Sources > 20 AND VR with head tracking:
    └─ Use Ambisonics (encode once, rotate cheaply)
    
Sources < 10 AND close/important sounds:
    └─ Use full HRTF convolution per source
    
Mobile OR CPU budget tight:
    └─ IF (stereo headphones expected):
        └─ Simple binaural panning
    └─ ELSE:
        └─ Standard stereo panning + distance rolloff
        
Background/ambient sounds:
    └─ Always use simple panning (save CPU for foreground)

3. Adaptive Music Strategy

IF (music needs to match gameplay intensity):
    └─ Horizontal Re-orchestration:
        └─ Layer 1: Basic rhythm/bass
        └─ Layer 2: Add melody
        └─ Layer 3: Add harmony/counterpoint
        └─ Layer 4: Full orchestration
        
IF (different moods needed per area):
    └─ Vertical Stems:
        └─ Peaceful stem (woodwinds, strings)
        └─ Tense stem (brass, percussion)
        └─ Combat stem (full orchestra)
        └─ Crossfade based on game state
        
IF (seamless transitions critical):
    └─ Use musical bars as transition boundaries
    └─ Pre-calculate next transition point
    └─ Never cut mid-phrase

4. Footstep Implementation Choice

Memory budget > 5MB AND < 50 total surfaces:
    └─ Use sample library approach
    
Memory budget tight OR > 100 surface variations:
    └─ Procedural synthesis:
        └─ Impact component (filtered noise burst)
        └─ Surface texture (material-specific)
        └─ Debris scattering (micro-impacts)
        
Performance critical (mobile):
    └─ Pre-generate variations at load time
    └─ Cache 10-20 variants per surface type

FAILURE MODES

1. HRTF Overload ("Everything Needs 3D")

Symptom: Frame drops when 20+ sounds play simultaneously Detection: CPU profiler shows >50% time in HRTF convolution Root Cause: Applying full HRTF to every sound source Fix: Use HRTF only for 3-5 important sources; simple panning for background Prevention: Set source importance hierarchy at design time

2. Sample Memory Bloat ("Footstep Explosion")

Symptom: 500MB+ audio assets for simple character movement Detection: 50+ footstep samples per character/surface combination Root Cause: Artist creating samples for every possible variation Fix: Implement procedural footstep synthesis with 4-5 base components Prevention: Establish memory budgets early; use procedural for high-variation content

3. Mobile Session Chaos ("The Silent Treatment")

Symptom: App audio stops working after phone call/notification Detection: Audio stops, never resumes; works fine on first launch Root Cause: No interruption handling for iOS/Android audio sessions Fix: Implement proper session management with interruption observers Prevention: Test with incoming calls, music apps, Siri activation

4. UI Sound Assault ("Click Fatigue")

Symptom: Users disable sound after 10 minutes of interaction Detection: Every button click at same volume as gameplay audio Root Cause: UI sounds mixed at gameplay levels (-6dB instead of -20dB) Fix: Reduce UI sounds to -18 to -24dB; use subtle, brief tones Prevention: Follow platform audio guidelines; A/B test with real users

5. Real-time Processing Overload ("DSP Death Spiral")

Symptom: Audio stutters, pops, or cuts out during intense scenes Detection: Audio callback exceeds allocated time budget (>10ms) Root Cause: Too many real-time effects, unoptimized convolution Fix: Use FFT-based convolution; limit concurrent DSP effects Prevention: Profile on lowest-spec target device; set hard limits

WORKED EXAMPLES

Example 1: VR Footstep System (Procedural Approach)

Scenario: VR game needs infinite footstep variation across 15 surface types, tight memory budget (50MB total audio)

Expert Decision Process:

1. Reject sample approach: 15 surfaces × 10 variations × 2 feet = 300 samples (150MB)
2. Choose procedural: Impact + texture layers, ~2KB of parameters per surface
3. Surface detection: Use physics material from VR floor collision
4. Parameter mapping: Player velocity → impact force (0.0-1.0 RTPC)

Implementation:

void OnVRFootstep(Vector3 position, PhysicsMaterial surface, float velocity) {
    // Surface-specific parameters (expert knowledge)
    SurfaceParams params = GetSurfaceParams(surface);
    
    // Procedural synthesis
    float impact_force = Mathf.Clamp01(velocity / 3.0f); // 3m/s = max force
    
    // Wwise integration
    SetRTPCValue("Impact_Force", impact_force, player);
    SetRTPCValue("Surface_Hardness", params.hardness, player);
    SetSwitch("Surface_Type", params.type_name, player);
    
    PostEvent("Play_Footstep_Procedural", player);
}

Novice vs Expert Trade-offs:

• Novice: "Use realistic footstep recordings" → 150MB, repetition after 30 mins
• Expert: "Synthesis sounds 90% as good, infinite variation, 50KB total"

Example 2: Mobile UI Sound Design (Session Handling)

Scenario: Meditation app needs subtle notification sounds that respect music apps and phone calls

Expert Decision Process:

1. Audio session category: .ambient with .mixWithOthers (don't interrupt Spotify)
2. Volume levels: -20dB for notifications, -24dB for UI feedback
3. Interruption handling: Pause during calls, resume after
4. Haptic coordination: Match audio transients to taptic feedback

Implementation:

class AppAudioManager {
    func setupAudioSession() {
        let session = AVAudioSession.sharedInstance()
        try? session.setCategory(.ambient, mode: .default, options: [.mixWithOthers])
        
        // Handle interruptions (phone calls, Siri)
        NotificationCenter.default.addObserver(
            self, selector: #selector(handleInterruption),
            name: AVAudioSession.interruptionNotification, object: nil
        )
    }
    
    @objc func handleInterruption(notification: Notification) {
        guard let info = notification.userInfo,
              let typeValue = info[AVAudioSessionInterruptionTypeKey] as? UInt,
              let type = AVAudioSession.InterruptionType(rawValue: typeValue) else { return }
        
        switch type {
        case .began:
            pauseAllAudio()  // Phone call started
        case .ended:
            if let optionsValue = info[AVAudioSessionInterruptionOptionKey] as? UInt {
                let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
                if options.contains(.shouldResume) {
                    resumeAudio()  // Safe to resume
                }
            }
        }
    }
}

What Novice Misses: Assumes audio "just works"; ignores platform session requirements Expert Insight: Mobile audio requires explicit session management; test with real interruptions

Example 3: Adaptive Music System (RTPC Setup)

Scenario: Action RPG needs music that scales with combat intensity (0 = exploration, 1 = boss fight)

Expert Decision Process:

1. Choose horizontal orchestration over vertical stems (smoother intensity scaling)
2. RTPC mapping: Combat_Intensity (0.0-1.0) drives 4 instrument layers
3. Transition timing: Use musical bars, never cut mid-phrase
4. Fallback logic: If RTPC fails, default to medium intensity

Wwise Implementation:

Combat_Music_Container/
├── Layer_1_Rhythm (RTPC: Combat_Intensity > 0.0)
├── Layer_2_Melody (RTPC: Combat_Intensity > 0.3)  
├── Layer_3_Harmony (RTPC: Combat_Intensity > 0.6)
└── Layer_4_Full_Orch (RTPC: Combat_Intensity > 0.8)

Code Integration:

void UpdateCombatMusic() {
    float intensity = CalculateCombatIntensity(); // 0.0 = safe, 1.0 = boss fight
    
    // Smooth the transitions (avoid jarring jumps)
    float smoothed = Mathf.Lerp(current_intensity, intensity, Time.deltaTime * 2.0f);
    current_intensity = smoothed;
    
    // Only update on musical boundaries
    if (IsOnMusicalBeat() && Mathf.Abs(smoothed - last_sent_intensity) > 0.1f) {
        SetRTPCValue("Combat_Intensity", smoothed, music_player);
        last_sent_intensity = smoothed;
    }
}

Expert vs Novice:

• Novice: Hard cuts between "calm" and "combat" tracks
• Expert: Continuous intensity scaling with musical timing awareness

QUALITY GATES

Audio implementation is complete when ALL conditions are verified:

• [ ] Spatial Audio Performance: HRTF processing uses <2ms per source; total audio CPU <10% of frame time
• [ ] Procedural Sound Variation: No audible repetition in 5+ minutes of continuous footsteps/environmental audio
• [ ] Mobile Session Handling: Audio correctly pauses/resumes during phone calls, route changes, and interruptions
• [ ] Distance Rolloff Accuracy: 3D positioned sounds follow inverse-square law with proper reference distance
• [ ] RTPC Parameter Validation: All real-time parameters (0.0-1.0) respond smoothly without pops or clicks
• [ ] Memory Budget Compliance: Total audio assets <50MB mobile, <200MB desktop; streaming works for large content
• [ ] UI Sound Levels: Interface audio at -18 to -24dB; never conflicts with gameplay audio
• [ ] Platform Audio Guidelines: iOS/Android audio session categories correct; respects system volume controls
• [ ] Middleware Integration: All audio events fire correctly; no missing dependencies or broken references
• [ ] Haptic-Audio Sync: Tactile feedback matches audio transients within 10ms on supported devices

NOT-FOR Boundaries

This skill should NOT be used for:

• Music composition/production → Use DAW-specific skills (Logic, Ableton, Pro Tools)
• Voice synthesis/cloning → Use voice-audio-engineer skill instead
• Linear audio post-production → Film/TV workflows require different expertise
• Podcast/broadcast editing → Use standard audio editor workflows
• Hardware microphone setup → Audio engineering hardware expertise
• Real-time voice chat implementation → Use WebRTC/networking specialists
• Audio compression/streaming protocols → Use backend/networking skills
• Machine learning audio models → Use ML/AI specialists for model training

Delegation Rules:

• For voice AI: "Use voice-audio-engineer for TTS, voice cloning, speech recognition"
• For music creation: "Use DAW specialists for composition, arrangement, mixing"
• For video audio: "Use video editor skills for linear media synchronization"