elbwalker/walkeros-understanding-destinations

Understanding walkerOS Destinations

Overview

Destinations receive processed events from the collector and deliver them to third-party tools (analytics, marketing, data warehouses).

Core principle: Destinations transform and deliver. They don't capture or process—that's sources and collector.

Destination Interface

See packages/core/src/types/destination.ts for canonical interface.

| Method | Purpose | Required | | --------------------------- | -------------------------- | ------------ | | init(context) | Load scripts, authenticate | Optional | | push(event, context) | Transform and send event | Required | | pushBatch(batch, context) | Batch processing | Optional | | destroy(context) | Cleanup on shutdown | Optional | | config | Settings, mapping, consent | Required |

destroy?: DestroyFn — Optional cleanup method. Called during command('shutdown'). Use to close DB connections, flush buffers, or release SDK clients. Receives { id, config, env, logger }.

The env Pattern

Destinations use dependency injection via env for external APIs. This enables testing without mocking.

// Destination defines its env type
export interface Env extends DestinationWeb.Env {
  window: {
    gtag: Gtag.Gtag;
    dataLayer: unknown[];
  };
}

// Destination uses env, not globals
async function push(event, context) {
  const { env } = context;
  env.window.gtag('event', mappedName, mappedData);
}

Testing with env

REQUIRED SKILL: See testing-strategy for full testing patterns.

import { mockEnv } from '@walkeros/core';
import { examples } from '../dev';

const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
  calls.push({ path, args });
});

await destination.push(event, { ...context, env: testEnv });

expect(calls).toContainEqual({
  path: ['window', 'gtag'],
  args: ['event', 'purchase', expect.any(Object)],
});

Destination Config

config: {
  settings: { /* destination-specific */ },
  mapping: { /* event transformation rules */ },
  data: { /* global data mapping */ },
  consent: { /* required consent states */ },
  policy: { /* processing rules */ },
  queue: boolean,  // queue events
  dryRun: boolean, // test mode
  queueMax: number,  // consent-queued events cap (default 1000)
  dlqMax: number,    // dead-letter queue cap (default 100)
}

Buffer bounds

Each destination keeps two internal buffers: queuePush (consent-denied events) and dlq (failed pushes). Both are size-bounded with FIFO drop-oldest eviction. Defaults: queueMax: 1000, dlqMax: 100. Set either on a destination's config to override per destination. Operators read drop counts from collector.status.dropped[stepId('destination', id)]?.queue (consent-denied evictions) and ?.dlq (DLQ evictions). Build the key with stepId() from @walkeros/core. Point-in-time sizes stay on collector.status.destinations[id].queuePushSize / dlqSize.

Batch scheduling

Set config.batch on a destination (with pushBatch implemented) to batch all of its events into one shared buffer. No '* *' wildcard mapping rule is needed. The configuration shape is batch?: number | { wait?, size?, age? } at both the destination-config layer and the mapping-rule layer.

• wait (ms): debounce window. The timer resets on every push. Legacy form batch: 1000 is shorthand for { wait: 1000 }.
• size: hard count cap. Default 1000. Flushes immediately when reached.
• age (ms): hard age cap since the first entry of the current window. Default 30000. Prevents debounce starvation under sustained load.

A mapping rule's own batch splits that entity-action into its own buffer and overrides config.batch per field (rule ?? config ?? default). To batch only specific events, omit config.batch and set batch on those rules. Pending batches flush on shutdown.

Per-event metadata

Batch.entries[] carries per-event { event, ingest?, respond?, rule?, data? }. Destinations that need per-event request IDs or HTTP responses (BigQuery, mParticle, HubSpot) should read entries instead of assuming all events in a batch share one ingest. batch.events and batch.data are derived views kept for backward compatibility.

Failure handling

If pushBatch throws (or returns a rejected Promise), the entire batch is routed to the destination's dlq and status.destinations[id].failed is incremented by the batch size. Per-item retry is the destination SDK's responsibility (BigQuery, Kafka, HubSpot each have their own backoff semantics). Counters (count, out) are bumped only after a successful flush.

Operators also see status.destinations[id].inFlightBatch: the number of events buffered but not yet delivered.

Require vs Consent

Two separate mechanisms control when destinations receive events:

| Mechanism | Purpose | Scope | Effect | | --------- | -------------------- | ----------------- | -------------------------------------------------------------------------------------- | | require | Delay initialization | Whole destination | Destination stays in pending until all required events fire (e.g., walker consent) | | consent | Filter events | Per-event | Events without matching consent are silently skipped or queued |

Require gates the destination lifecycle. A destination with require: ["consent"] does not exist in the collector until a "walker consent" event fires. Until then, events are queued internally.

Consent gates individual event delivery. A destination with consent: { marketing: true } only receives events where the collector's consent state (or the event's own consent field) includes { marketing: true }.

State refresh on flush: When queued events are flushed to a destination, they receive the current collector state (consent, user, globals) — not the stale state from when they were originally captured. Any state-mutation command (walker consent, walker user, walker globals, etc.) triggers a flush attempt. The consent gate still applies: events without required consent simply return to the queue.

Both can be combined:

{
  "config": {
    "require": ["consent"],
    "consent": { "marketing": true }
  }
}

This means: don't initialize until consent fires, then only accept events with marketing consent.

Simulation impact: require causes "destination not found" errors in flow_simulate because the destination stays pending. Remove require temporarily for simulation testing.

Policy

Policy modifies the event BEFORE mapping rules run. Defined at config level (applies to all events) or rule level (applies to specific events):

{
  "config": {
    "policy": {
      "user_data.email": {
        "key": "user.email",
        "consent": { "marketing": true }
      }
    }
  }
}

Policy supports consent-gated field injection — fields are only added when the required consent is present in the event.

Destination Paths

| Type | Path | Examples | | ------ | ------------------------------- | ------------------------------------ | | Web | packages/web/destinations/ | gtag, meta, api, piwikpro, plausible | | Server | packages/server/destinations/ | aws, gcp, meta |

Template Destination

Use as starting point: packages/web/destinations/plausible/

Transformer Wiring

Destinations can wire to post-collector transformer chains via the before property:

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'  // First transformer to run before this destination
  }
}

The transformer chain runs after collector enrichment, before the destination receives events. Each destination can have its own chain. See understanding-transformers for chain details.

Consent: Two-Layer Pattern

Destinations that integrate vendor SDKs typically need two consent layers:

Layer 1: config.consent — gates walkerOS event delivery. If consent is not granted, events don't reach the destination. This is the primary barrier.

Layer 2: on('consent') — controls vendor SDK internals. Even when walkerOS stops sending events, the vendor SDK may still run its own behaviors (DOM capture, polling, fetching configs). Use on('consent') to pause/resume these.

on(type, context) {
  if (type !== 'consent') return;
  const consent = context.data;
  // Derive from config.consent keys — don't hardcode consent names
  const granted = Object.keys(config.consent || {}).every(k => consent[k]);
  vendorSdk.setOptOut(!granted);
}

Both layers are needed for complete consent compliance. config.consent prevents data flow. on('consent') prevents vendor SDK side effects.

Response Delegation (env.respond)

Destinations can customize HTTP responses by calling context.env.respond?.({ body, status?, headers? }). This is useful for destinations that need to signal success/failure back to the HTTP caller. First call wins (idempotent). The respond function is optional — only present when the source provides one.

Setup (optional)

Destinations can implement an optional setup() lifecycle to provision external resources, for example a BigQuery dataset and table, a Pub/Sub topic, or a warehouse schema. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup destination.<name>.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-destination, walkeros-understanding-sources, walkeros-understanding-stores, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related Skills

• walkeros-understanding-flow - How destinations fit in architecture
• walkeros-understanding-mapping - Configure transformations
• walkeros-create-destination - Create new destination

Source Files:

• packages/core/src/types/destination.ts - Interface

Package READMEs:

• packages/web/destinations/gtag/README.md - gtag example
• packages/web/destinations/plausible/README.md - Plausible (template)

Documentation:

• Website: Destinations - Overview
• Website: Create Your Own - Guide