igbuend/output-filter-pattern

Output Filter Security Pattern

Filter data before sending it to an external entity, ensuring that only necessary and authorized data elements are transmitted. This prevents excessive data exposure and enforces data minimization.

Problem Addressed

Entity receives excessive data: System sends more data than the receiver needs or is authorized to see, leading to:

• Exposure of sensitive data (PII, credentials, internal identifiers)
• Privacy violations (GDPR, CCPA)
• Increased attack surface
• Data leakage through traffic interception

Core Principle

Never rely on the client to filter sensitive data.

Data filtering must occur at the server/API level before sending, not at the client level after receiving.

Why Client-Side Filtering Fails

| Attack Vector | Description | |---------------|-------------| | Traffic sniffing | Attackers intercept API responses before client filtering | | Direct API calls | Attackers bypass client applications entirely | | Client manipulation | Attackers modify client code to reveal hidden data | | Mobile app reverse engineering | Attackers extract API endpoints and call directly |

Core Components

| Role | Type | Responsibility | |------|------|----------------| | Sender | Entity | System that sends data | | Output Filter | Enforcement Point | Filters outgoing data | | Filter Specification | Information Point | Defines what data to include/exclude | | Receiver | Entity | External entity receiving data |

Data Elements

• raw_data: Complete data before filtering
• filtered_data: Data after filtering (only necessary elements)
• filter_spec: Rules defining what to include/exclude
• context: Information about receiver, request type, authorization level

Pattern Flow

Sender → [raw_data] → Output Filter
Output Filter → [get_filter_spec(context)] → Filter Specification
Filter Specification → [filter_spec] → Output Filter
Output Filter → [apply filter to raw_data] → Output Filter
Output Filter → [filtered_data] → Receiver

1. Sender prepares complete data
2. Output Filter retrieves filter specification based on context
3. Output Filter applies specification, removing unauthorized/unnecessary data
4. Only filtered data is sent to Receiver

Filtering Strategies

Allowlist (Preferred)

Explicitly define which fields to include:

Include: [id, name, email, created_at]
// All other fields excluded by default

Blocklist (Use with Caution)

Define which fields to exclude:

Exclude: [ssn, password_hash, internal_id, credit_card]
// All other fields included by default

Warning: Blocklists fail when new sensitive fields are added but not blocked.

Context-Based Filtering

Different filters based on:

• User role/authorization level
• Request type (list vs. detail)
• API version
• Consumer type (internal vs. external)

Implementation Approaches

1. Response Schema Validation

Define explicit response schemas for each endpoint:

• Only fields in schema are returned
• Validates responses match contract
• Catches accidental data exposure

2. Data Transfer Objects (DTOs)

Create specific response objects:

// Instead of returning database entity directly
User (DB) → UserResponseDTO (filtered)

UserResponseDTO {
    id, name, email, avatar_url
    // Excludes: password_hash, ssn, internal_flags
}

3. Projection/Selection

Query only needed fields from data source:

SELECT id, name, email FROM users
// Instead of SELECT * FROM users

4. View-Specific Serialization

Custom serializers per context:

• Public API serializer (minimal fields)
• Admin API serializer (more fields)
• Internal API serializer (full access)

Security Considerations

Avoid Generic Serialization

Never use generic methods that expose all properties:

// AVOID
return user.to_json()
return user.to_dict()
return serialize(user)

// PREFER
return {
    'id': user.id,
    'name': user.name,
    'email': user.email
}

Protect Nested Objects

Data exposure often occurs in nested/related objects:

// User response includes related data
{
    "id": 1,
    "name": "Alice",
    "manager": {
        "id": 2,
        "name": "Bob",
        "salary": 150000  // LEAK! Manager salary exposed
    }
}

Filter nested objects recursively.

Error Responses

Don't leak data in error messages:

• Avoid stack traces with internal data
• Don't reveal database structure
• Use generic error codes externally

Pagination Metadata

Even metadata can leak information:

• Total counts may reveal data existence
• Filter appropriately

Different Endpoints, Same Data

Ensure consistent filtering across:

• List vs. detail endpoints
• Search endpoints
• Export/download features
• Related resource endpoints

Common Data to Filter

| Category | Sensitive Fields | |----------|------------------| | Authentication | password_hash, auth_tokens, API keys, session_ids | | Personal (PII) | SSN, tax_id, credit_card, bank_account | | Internal | internal_id, debug_flags, feature_flags | | System | created_by_system, internal_notes, audit_data | | Relationships | Detailed data of related entities | | Metadata | IP addresses, user agents, timestamps (if sensitive) |

Relationship to Other Patterns

Output Filter vs. Authorization

• Authorization: Can this user access this resource?
• Output Filter: What properties of this resource should user see?

Both are required. User may be authorized to see a resource but not all its properties.

Output Filter vs. Input Validation

• Input Validation: Filter/validate incoming data
• Output Filter: Filter outgoing data

Complementary patterns for complete data flow security.

Implementation Checklist

• [ ] All API endpoints have defined response schemas
• [ ] Response schemas use allowlist approach
• [ ] No generic serialization (to_json, to_dict) for external responses
• [ ] Nested/related objects filtered appropriately
• [ ] Error responses don't leak sensitive data
• [ ] Context-based filtering for different user roles
• [ ] List and detail endpoints have appropriate schemas
• [ ] Export/download features apply same filters
• [ ] Response schema validation in place
• [ ] DTOs or projections used instead of raw entities
• [ ] Regular review of response schemas for new sensitive fields

Testing for Excessive Data Exposure

1. Compare responses to schemas: Actual vs. documented
2. Test with different roles: Verify role-based filtering
3. Inspect raw API responses: Use proxy tools (Burp, ZAP)
4. Review database queries: Check for SELECT *
5. Audit new fields: Ensure added to filter specs

Related Patterns

• Data Validation (filter incoming data)
• Authorisation (determine resource access)
• Selective Encrypted Transmission (protect data in transit)
• Log Entity Actions (log data access)

References

• Source: https://securitypatterns.distrinet-research.be/patterns/05_01_001__output_filter/
• OWASP API Security Top 10 - API3:2019 Excessive Data Exposure
• OWASP API Security Top 10 2023 - Broken Object Property Level Authorization
• GDPR Article 5(1)(c) - Data Minimization Principle