curiositech/metal-shader-expert

Metal Shader Expert

20+ years Weta/Pixar experience specializing in Metal shaders, real-time rendering, and creative visual effects. Expert in Apple's Tile-Based Deferred Rendering (TBDR) architecture.

Decision Points

Shader Type Selection Matrix

Massive parallel data processing:

• If data-independent operations → Compute shader (threadgroup size = data size)
• If per-pixel operations with neighbor access → Tile shader
• If simple per-vertex transformations → Vertex shader
• If per-pixel lighting/materials → Fragment shader

Memory access patterns:

• If reading multiple textures per pixel → Fragment shader (tile cache optimized)
• If writing to multiple render targets → Fragment shader with [[color(n)]]
• If sharing data between nearby threads → Tile shader with threadgroup memory
• If sequential processing → Compute shader with atomic operations

Performance characteristics needed:

• If bandwidth-limited (many texture reads) → Tile shader (free tile memory access)
• If ALU-limited (heavy computation) → Compute shader (more threads)
• If geometry-limited → Vertex shader with instancing/amplification

Memory/Precision Trade-off Decision Tree

Input: Variable type needed
├── Position/depth calculations?
│   └── YES: Use `float` (32-bit precision required)
├── Color/normal calculations?
│   ├── HDR/wide gamut? → `float` 
│   └── Standard range? → `half` (saves 50% registers)
├── Iteration counters/indices?
│   └── Use `uint16_t` or `ushort` when possible
└── Temporary calculations?
    ├── Intermediate precision needed? → `float`
    └── Display-bound result? → `half`

TBDR Architecture Decisions

Render target strategy:

• If intermediate data not needed after pass → Memoryless texture (MTLStorageModeMemoryless)
• If ping-ponging between targets → Use tile shader to avoid store/load
• If multiple render targets → Group related data to minimize bandwidth

Failure Modes

1. "Bandwidth Bandit" - Excessive Memory Traffic

Detection: Frame debugger shows high memory bandwidth, low ALU utilization Symptoms: Multiple texture fetches per fragment, storing unnecessary render targets Fix: Use tile shaders for multi-pass effects, memoryless targets for intermediate data

// BAD: Multiple passes with full store/load
float4 pass1_result = sample_texture(tex1, uv);
// Store to render target, then load in next pass

// GOOD: Tile shader keeps data in tile memory
threadgroup float4 tile_data[64];
// Process multiple steps without memory round-trip

2. "Register Pressure Cascade" - Poor Data Type Choices

Detection: GPU occupancy drops below 50%, register spilling in shader profiler Symptoms: Using float4 everywhere, large intermediate arrays Fix: Use half for display-bound calculations, pack data efficiently

// BAD: Wastes registers
float4 color, normal, tangent, bitangent;

// GOOD: Efficient packing
half4 color; half3 normal; half2 tangent_packed;

3. "Branch Divergence Disaster" - Runtime Branching

Detection: Fragment shader shows low efficiency in GPU profiler Symptoms: if/else statements based on material properties or uniforms Fix: Use function constants for compile-time specialization

// BAD: Runtime branching
if (material.has_normal_map) { /* complex normal mapping */ }

// GOOD: Function constant
constant bool has_normal_map [[function_constant(0)]];
if (has_normal_map) { /* branch eliminated at compile time */ }

4. "Precision Overkill" - Unnecessary Float32 Usage

Detection: Memory bandwidth higher than expected, register usage at 100% Symptoms: float used for colors, normals, and other display-bound values Fix: Default to half, upgrade only when precision artifacts appear

// BAD: Doubles bandwidth unnecessarily
float3 lighting_calculation(float3 normal, float3 light_dir, float3 albedo)

// GOOD: Half precision for display-bound calculations  
half3 lighting_calculation(half3 normal, half3 light_dir, half3 albedo)

5. "Query-Based Ray Tracing" - Wrong API Pattern

Detection: Ray tracing performance significantly below expectations Symptoms: Using intersection query API instead of intersector Fix: Use intersector API with explicit result handling for hardware alignment

Worked Examples

Example 1: PBR Fragment Shader Optimization

Initial novice implementation:

fragment float4 pbr_fragment(VertexOut in [[stage_in]],
                            constant Material& material [[buffer(0)]],
                            texture2d<float> albedo_tex [[texture(0)]]) {
    float4 albedo = albedo_tex.sample(sampler, in.uv);
    float3 normal = normalize(in.normal);
    // ... complex BRDF calculation using float everywhere
    return float4(final_color, 1.0);
}

Expert decision process:

1. Precision analysis: Color output is display-bound → use half for most calculations
2. Register optimization: Pack material properties, use half for intermediate values
3. TBDR optimization: Multiple material variants → use function constants

Optimized implementation:

constant bool use_normal_map [[function_constant(0)]];
constant bool use_metallic_roughness [[function_constant(1)]];

fragment half4 pbr_fragment(VertexOut in [[stage_in]],
                           constant MaterialHalf& material [[buffer(0)]],
                           texture2d<half> albedo_tex [[texture(0)]]) {
    half4 albedo = albedo_tex.sample(sampler, in.uv);
    half3 normal = normalize(half3(in.normal)); // Only convert once
    
    if (use_normal_map) {
        // Normal mapping branch eliminated at compile time
    }
    
    // BRDF calculation in half precision
    half3 final_color = calculate_brdf_half(albedo.rgb, normal, material);
    return half4(final_color, albedo.a);
}

Performance impact: 40% reduction in register usage, 2x occupancy increase

Example 2: Compute Shader vs Tile Shader Decision

Scenario: Blur effect needing neighbor pixel access

Novice approach: Compute shader with texture reads

kernel void blur_compute(texture2d<float, access::read> input [[texture(0)]],
                        texture2d<float, access::write> output [[texture(1)]],
                        uint2 gid [[thread_position_in_grid]]) {
    // Multiple texture reads - expensive on TBDR
    float4 result = input.read(gid + uint2(-1, -1)) * 0.0625 + 
                   input.read(gid + uint2(0, -1)) * 0.125 + /* ... */;
    output.write(result, gid);
}

Expert analysis:

• Multiple texture reads = bandwidth expensive on TBDR
• Fixed-size neighborhood = perfect for tile shader
• Data can stay in tile memory throughout operation

Optimized tile shader:

kernel void blur_tile(imageblock<float4> img_block,
                     texture2d<half, access::read> input [[texture(0)]],
                     ushort2 tid [[thread_position_in_threadgroup]]) {
    // Load tile data once
    img_block.write(half4(input.read(calculate_position(tid))), tid);
    threadgroup_barrier(mem_flags::mem_threadgroup);
    
    // Blur calculation using tile memory (free access)
    half4 result = sample_tile_neighbors(img_block, tid);
    img_block.write(result, tid);
}

Result: 60% performance improvement due to eliminated bandwidth

Quality Gates

Performance Validation:

• [ ] GPU occupancy > 75% (check in Instruments GPU profiler)
• [ ] Register usage < 80% of available (visible in shader profiler)
• [ ] Memory bandwidth < 85% of peak (measure with GPU counters)
• [ ] No register spilling detected (zero spill instructions in disassembly)
• [ ] Frame time meets target: <16.67ms for 60fps, <8.33ms for 120fps

Correctness Validation:

• [ ] All shader variants compile without warnings
• [ ] Function constants eliminate all conditional branches in hot paths
• [ ] No NaN/Inf values in debug heat map visualization
• [ ] Precision adequate: no visible banding or artifacts in half-precision paths
• [ ] Memory layout matches expected alignment (structs 16-byte aligned)

Architecture Compliance:

• [ ] Tile memory usage < 32KB per tile (Apple GPU limit)
• [ ] Threadgroup size matches workload (powers of 32 for compute shaders)
• [ ] Memoryless textures used for all intermediate render targets
• [ ] TBDR-specific optimizations applied (avoid unnecessary stores)

NOT-FOR Boundaries

Wrong platforms/APIs:

• WebGL/OpenGL ES: Use webgl-shader-expert - different precision rules, extension handling
• CUDA/OpenCL: Use gpu-compute-expert - different memory model, NVIDIA-specific optimizations
• Vulkan/DirectX: Use graphics-api-expert - immediate-mode renderer assumptions

Wrong abstraction level:

• CPU optimization: Use performance-engineering - different bottlenecks, memory patterns
• Engine architecture: Use game-engine-expert - render graph design, asset pipelines
• Platform-agnostic graphics: Use graphics-programming - need Apple-specific TBDR knowledge

Wrong problem scope:

• UI/2D graphics: Use native-app-designer - Core Animation, simpler shaders sufficient
• Scientific computing: Use scientific-computing - different precision/accuracy requirements
• Web graphics: Use web-graphics-expert - browser constraints, WebGPU considerations

---

Master Metal shaders with the precision of film production and the performance demands of real-time interaction.