oborchers/api-design

Define Your Public API Surface and Defend It

The public API is the contract between your library and every downstream user. Once a symbol is public, removing or changing it is a breaking change. Getting the boundary right -- what is exported, what is private, how errors are communicated, how complexity is layered -- is the most consequential design decision in a Python library. Get it wrong and you end up like Pydantic v1: an unintentionally wide API surface that required a full rewrite to fix.

httpx is the gold standard. Every public symbol is re-exported from __init__.py. Every implementation module is underscore-prefixed (_client.py, _config.py). The exception hierarchy is a carefully designed tree. Sync and async clients share all non-I/O logic in a _BaseClient. Study httpx before designing your own API.

Public API Surface: __all__ and Underscore Modules

Every module that exports symbols must define __all__. This documents intent, controls from module import *, and tells tools (mypy, pyright, mkdocstrings) what is public.

| Rule | Rationale | |------|-----------| | Define __all__ in every public module | Documents the contract; tools rely on it | | Re-export everything from __init__.py | Users write from mylib import Client, never from mylib._client import Client | | Prefix all implementation modules with _ | _client.py, _config.py, _exceptions.py make the private boundary unambiguous | | Sort __all__ alphabetically | Easy to review in diffs, easy to check for completeness | | Include type aliases and protocols in __all__ | Users need them for type-checking; easy to forget |

| Bad | Good | |-----|------| | from mylib.client import Client | from mylib import Client | | No __all__, everything implicitly public | Explicit __all__ listing every public name | | utils.py with mixed public/private helpers | _utils.py for internal, re-export public helpers from __init__.py |

Expose less than you think you need. You can always make something public later; you cannot easily make it private. If it is in __all__, it is a contract. If it starts with an underscore, you can change it freely.

Progressive Disclosure

Design APIs that serve beginners and experts simultaneously. Simple things should be simple; complex things should be possible.

# Layer 1: Module-level convenience (simplest)
response = httpx.get("https://api.example.com/users")

# Layer 2: Configured client (intermediate)
client = httpx.Client(base_url="https://api.example.com", timeout=30.0)
response = client.get("/users")

# Layer 3: Full customization (advanced)
transport = httpx.HTTPTransport(retries=3)
client = httpx.Client(transport=transport, timeout=httpx.Timeout(5.0, connect=10.0))

Rules for progressive disclosure:

1. The simplest call requires the fewest arguments. Use sensible defaults for everything.
2. Complexity is opt-in, not opt-out. A user who needs no auth should never see auth parameters.
3. Use keyword-only arguments (* in the signature) to prevent positional footguns as signatures grow.
4. Module-level functions for the common case, classes for the configured case. This is the httpx/requests pattern.
5. Config objects instead of dozens of kwargs. httpx.Timeout(5.0, connect=10.0) is more discoverable than timeout=5.0, connect_timeout=10.0.

Exception Hierarchy

A well-designed exception hierarchy lets users catch errors at exactly the right granularity. It is part of __all__ and must be as carefully designed as your classes.

class MyLibError(Exception):
    """Base exception. `except MyLibError` catches everything."""

class ConfigurationError(MyLibError):
    """Raised when configuration is invalid."""

class ConnectionError(MyLibError):
    """Raised when a connection fails."""

class TimeoutError(ConnectionError):
    """Subclasses ConnectionError -- timeout is a type of connection failure."""

class AuthenticationError(MyLibError):
    """Raised when authentication fails."""

class ValidationError(MyLibError):
    """Carries structured error data, not just a string."""
    def __init__(self, errors: list[dict[str, Any]]) -> None:
        self.errors = errors
        super().__init__(f"{len(errors)} validation error(s)")

| Rule | Example | |------|---------| | Always provide a base exception class | except MyLibError as catch-all | | Carry structured data, not just strings | httpx's HTTPStatusError.response, Pydantic's .errors() | | Use inheritance to group related errors | except TransportError catches all network issues | | Never raise bare Exception or ValueError | Users cannot distinguish your errors from others | | Name exceptions as nouns | TimeoutError, not TimedOut | | Document which methods raise which exceptions | Part of the API contract |

Async/Sync Dual API

Follow httpx's _BaseClient pattern: share all non-I/O logic in a base class, implement genuinely separate sync and async I/O paths.

class _BaseClient:
    """Shared logic: URL merging, headers, cookies, auth -- no I/O."""
    def _build_request(self, method: str, url: str, **kwargs) -> Request:
        ...

class Client(_BaseClient):
    """Synchronous client with blocking transport."""
    def send(self, request: Request) -> Response:
        return self._transport.handle_request(request)

class AsyncClient(_BaseClient):
    """Asynchronous client with async transport."""
    async def send(self, request: Request) -> Response:
        return await self._transport.handle_async_request(request)

| Bad | Good | |-----|------| | Wrap async with asyncio.run() in sync methods | Separate sync/async transport implementations | | Duplicate all non-I/O logic in both clients | Share logic in _BaseClient | | Only provide async API | Always provide sync; add async if doing I/O |

Never use asyncio.run() as a sync wrapper. It fails if an event loop is already running (Jupyter, async frameworks) and prevents connection pooling.

Plugin Architecture

Choose the right extensibility pattern based on your scale.

| Pattern | Complexity | When to Use | Exemplar | |---------|-----------|-------------|----------| | pluggy | High | Full plugin ecosystem with hooks | pytest, tox | | Entry points | Medium | Installed packages register themselves | pytest plugin discovery | | Protocols | Low | Third-party opt-in without importing your lib | Rich (__rich_repr__) | | Decorator registry | Low | Internal extensibility within your package | Click, Flask |

For protocols, use dunder names (__mylib_serialize__), make them @runtime_checkable, keep them to one method, and always provide a fallback for objects that do not implement the protocol. The key insight from Rich: objects do not need to import or subclass anything from your library to participate.

For entry points, define them in pyproject.toml:

[project.entry-points."mylib.plugins"]
my_plugin = "my_plugin_package:MyPlugin"

Discover them at runtime with importlib.metadata.entry_points(group="mylib.plugins").

Method Naming and Signatures

Use consistent verb-noun naming across the entire API. Pydantic v2 learned from v1's inconsistency by adopting a uniform model_ prefix.

| Verb | Meaning | Example | |------|---------|---------| | get | Retrieve (may raise if missing) | client.get() | | create | Make a new resource | Session.create() | | build | Construct from parts | Request.build() | | validate | Check and convert | model_validate() | | dump | Serialize to format | model_dump(), model_dump_json() | | load | Deserialize from format | json.load() |

Force keyword-only arguments after the first positional with * in the signature. Use sensible, secure defaults (verify=True, follow_redirects=False). Return rich objects for complex operations (httpx's Response carries status, headers, content, and raise_for_status()), primitives for simple queries, and self for builder/configuration methods.

Configuration Patterns

Use frozen dataclasses for configuration objects. Validate in __post_init__. Provide a from_env() classmethod for environment variable integration without requiring it.

@dataclass(frozen=True)
class ClientConfig:
    base_url: str
    timeout: float = 30.0
    max_retries: int = 3

    def __post_init__(self) -> None:
        if self.timeout <= 0:
            raise ConfigurationError("timeout must be positive")

    @classmethod
    def from_env(cls, prefix: str = "MYLIB_") -> "ClientConfig":
        return cls(
            base_url=os.environ.get(f"{prefix}BASE_URL", ""),
            timeout=float(os.environ.get(f"{prefix}TIMEOUT", cls.timeout)),
        )

Review Checklist

When reviewing code for API design:

• [ ] __all__ is defined in every public module, sorted alphabetically, and includes all public symbols (classes, functions, exceptions, type aliases)
• [ ] All implementation modules are underscore-prefixed (_client.py, _config.py) and users never import from them directly
• [ ] Public symbols are re-exported from the top-level __init__.py
• [ ] The API supports progressive disclosure: module-level functions for simple use, configurable classes for advanced use
• [ ] All parameters beyond the first positional are keyword-only (use * separator)
• [ ] A base exception class exists and all library exceptions inherit from it
• [ ] Exceptions carry structured data (not just string messages) and are included in __all__
• [ ] No bare Exception, ValueError, or TypeError is raised from library code
• [ ] Async and sync clients share non-I/O logic in a base class with separate transport implementations
• [ ] No asyncio.run() wrappers are used to bridge async to sync
• [ ] Plugin extension points use the appropriate pattern (pluggy for full ecosystems, Protocols for third-party opt-in, entry points for auto-discovery)
• [ ] Method naming follows a consistent verb-noun pattern across the entire API surface
• [ ] Mutable default arguments are avoided (use None sentinel + factory)
• [ ] Importing the library produces no side effects (no network calls, no logging config, no global state mutation)
• [ ] Internal state is not exposed through public attributes; use read-only properties that return values, not mutable objects