skills/walkeros-create-source/SKILL.md
Use when creating a new walkerOS source. Example-driven workflow starting with research and examples before implementation.
npx skillsauth add elbwalker/walkeros walkeros-create-sourceInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
Before starting, read these skills:
Flow.StepExample pattern, createTrigger, Three Type ZonesA source's identity is split into two fields:
| Field | Meaning | Examples |
| ----------------- | --------------------------------------------------------------- | ---------------------------------------------- |
| source.type | The kind of source (its role / mechanism) | browser, dataLayer, cookiefirst, fetch |
| source.platform | The runtime that hosts the source (web, server, app, ...) | web, server |
| Platform | Input | Example types |
| -------- | ----------------------- | ----------------------------------- |
| web | DOM events, dataLayer | browser, dataLayer |
| server | HTTP requests, webhooks | gcp, express, lambda, fetch |
SourceMapEvery source registers its type literal and any source-specific source.*
fields by augmenting SourceMap from @walkeros/core. This is how the
collector and downstream consumers know about the new source kind without
loosening the union to string.
Add this to the source's src/types.ts (or src/types/index.ts):
import type { Source, Elb } from '@walkeros/core';
declare module '@walkeros/core' {
interface SourceMap {
// Replace `mySource` with the source's package-level identifier.
mySource: {
type: 'mySource'; // matches the literal you return from Source.Init
platform?: 'web'; // 'web' | 'server' | 'app' | ...
// Add any extra fields the source surfaces in `event.source.*` here.
// e.g. `version?: string;` is already on the base Source - only add
// truly source-specific keys.
};
}
}
Reference implementations: packages/web/sources/browser/src/types/index.ts,
packages/web/sources/demo/src/types.ts. Conflicting declarations cause compile
errors on purpose, this surfaces naming collisions early.
| Category | Purpose | Examples | Key Concern |
| ------------------ | ----------------------------------------- | ----------------------- | -------------------- |
| Transformation | Convert external format → walkerOS events | dataLayer, fetch | Mapping accuracy |
| Transport | Receive events from specific platform | gcp, aws, express | Platform integration |
| Complexity | Template | When to Use |
| --------------------- | -------------- | ------------------------------------- |
| Simple transformation | fetch/ | Generic HTTP handler, data conversion |
| Platform transport | gcp/, aws/ | Cloud platform integration |
| Browser interception | dataLayer/ | DOM events, array interception |
1. Research → Deeply understand external system, SDK, and data format
2. Classify → Determine source type and integration approach
3. Examples → Define in/out pairs FIRST (start with the end result)
4. Mapping → Define input → walkerOS event transformation
5. Scaffold → Copy template and configure
6. Convention → Add walkerOS.json metadata and buildDev
7. Implement → Build using examples as test fixtures
8. Test → Verify against example variations
9. Document → Write README
Goal: Deeply understand the external system before writing any code. Research quality determines implementation quality.
Always prefer the vendor's official SDK package over raw HTTP API calls. The SDK handles transport, data formatting, and platform specifics - don't reinvent these.
npm install @vendor/sdk and read the actual source# Search npm for official packages
npm search [vendor-name]
npm search @[vendor]
# Install and inspect actual types
npm install @vendor/sdk
ls node_modules/@vendor/sdk/lib/esm/
Go beyond just the primary event payload. Most external systems provide multiple data channels:
| Data Channel | Examples | walkerOS Handling |
| ---------------- | ----------------------------------- | -------------------- |
| Event payload | Request body, DOM event data | Default push() |
| Headers/metadata | Auth tokens, content-type, origin | context or user |
| Query params | UTM parameters, tracking IDs | data or context |
| Platform context | Cloud function metadata, Lambda ctx | source or custom |
| Identity | User ID, session ID, device ID | user |
| Consent signals | Opt-in/out flags, consent string | consent |
Review similar sources in the codebase:
# List existing sources
ls packages/web/sources/
ls packages/server/sources/
# Reference implementations
# - dataLayer: DOM-based, array interception
# - express: HTTP middleware
# - fetch: Generic HTTP handler (simplest server pattern)
# - gcp: Cloud Functions specific
If working with human oversight, pause here to confirm:
Goal: Understand what the source captures and how it delivers data, which determines implementation complexity.
| Category | Description | Mapping Needed | Example Sources |
| ------------------ | ------------------------------------------ | ------------------------------- | ------------------------ |
| Transformation | Converts external event format to walkerOS | Essential - must map fields | dataLayer, fetch |
| Transport | Receives events from a specific platform | Structural - platform unwrap | gcp, aws, express |
| Interception | Intercepts existing data flows | Varies - depends on data format | dataLayer, CMP sources |
| Approach | When to use | Pattern | | --------------------------- | ----------------------------------- | --------------------------------------------------------- | | Platform SDK as host | SDK provides typed request/response | Use SDK types, wrap handler in walkerOS source | | DOM interception | Capture browser-side events | Listen to DOM events, intercept arrays/globals | | HTTP handler | Generic webhook/API receiver | Parse request, extract events, forward to collector | | Callback/event listener | Platform provides event emitter | Register listener, transform events, forward to collector |
Prefer the vendor SDK - it provides typed request/response objects and handles platform specifics. Raw HTTP parsing is a fallback when no SDK exists.
When using the vendor SDK:
Mandatory. Examples are the test fixtures for Phase 8. Define expected
trigger / in / out triples FIRST - start with the end result in mind.
Without examples, you cannot test. Even for simple sources, step examples are
the single source of truth for tests, simulations, and documentation.
Authoritative pattern: See using-step-examples for the Three Type Zones,
createTriggercontract, and CI integration. This skill reuses that contract - do not diverge.
mkdir -p packages/<platform>/sources/[name]/src/examples
mkdir -p packages/<platform>/sources/[name]/src/{schemas,types}
All reference sources in the monorepo use this exact layout in src/examples/.
Match it - no inputs.ts, outputs.ts, requests.ts, or standalone
mapping.ts.
| File | Required? | Purpose |
| --------------------- | --------- | ------------------------------------------------------------------- |
| examples/step.ts | yes | Flow.StepExample entries with trigger / in / out triples |
| examples/trigger.ts | yes | createTrigger implementation following Trigger.CreateFn |
| examples/index.ts | yes | Barrel exports: env (if present), step, createTrigger |
| examples/env.ts | if needed | Mock env for platform deps (browser window/document, express, etc.) |
env.ts is included whenever the source touches platform globals or injected
deps - all web sources and every server source that wraps a platform SDK ship
one. Sources whose tests drive the collector entirely through trigger.ts (e.g.
web/sources/session) may omit it. When in doubt, include it.
The old inputs.ts / outputs.ts / requests.ts / mapping.ts files are gone
Flow.StepExample entry in step.ts.Sources are the inverse of destinations in the Three Type Zones model:
| Zone | Source semantics |
| --------- | ------------------------------------------------------------------------------------- |
| trigger | How to simulate the invocation (HTTP method, DOM event type, cloud event) |
| in | External trigger content - HTTP request, DOM HTML, SDK payload (NOT a walkerOS event) |
| out | The walkerOS event(s) the source should emit (WalkerOS.Event) |
Where a destination does WalkerOS.Event → vendor output, a source does
external content → WalkerOS.Event. Read
using-step-examples before authoring
entries.
No any. Every example value must be explicitly typed.
trigger uses the local source trigger type or a platform-native type
(e.g. 'load' | 'click' for DOM, HTTP method strings for server sources).in uses the vendor / platform SDK types imported from the official
package whenever available (Express Request, Fetch Request, API Gateway
APIGatewayProxyEvent, Lambda Context, GCP CloudEvent, etc.). Do not
invent local request types when the platform publishes them.out uses WalkerOS.Event (or DeepPartialEvent for fragments).Flow.StepExample from @walkeros/core.Env type from ../types.createTrigger is typed as Trigger.CreateFn<Content, Result> - the
Content and Result generics come from the source's own types module.examples/step.tsimport type { Flow } from '@walkeros/core';
// One step example per captured trigger / input shape.
// `trigger` tells createTrigger how to simulate the invocation.
// `in` is the platform-specific content (HTTP request, DOM HTML, SDK payload) -
// typed against the platform SDK's published types where available.
// `out` is the walkerOS event the source is expected to emit.
// Set `title` + `description` for public examples; mark test-only fixtures
// with `public: false`. See
// [walkeros-using-step-examples](../walkeros-using-step-examples/SKILL.md).
export const pageView: Flow.StepExample = {
trigger: {
type: 'load',
options: {
url: 'https://example.com/docs',
title: 'Documentation',
},
},
in: '', // no external content - DOM-driven trigger
out: {
name: 'page view',
data: { domain: 'example.com', title: 'Documentation', id: '/docs' },
entity: 'page',
action: 'view',
trigger: 'load',
source: {
type: 'browser',
platform: 'web',
url: 'https://example.com/docs',
},
},
};
// Server example: HTTP POST carrying a walker event payload.
export const orderComplete: Flow.StepExample = {
trigger: { type: 'POST' },
in: {
method: 'POST',
path: '/collect',
body: { name: 'order complete', data: { id: 'ORD-123', total: 149.97 } },
},
out: {
name: 'order complete',
data: { id: 'ORD-123', total: 149.97 },
entity: 'order',
action: 'complete',
},
};
examples/index.ts (barrel)export * as env from './env'; // omit if the source has no env.ts
export * as step from './step';
export { createTrigger, trigger } from './trigger';
examples/trigger.ts - createTriggerEvery source exports a createTrigger following the unified
Trigger.CreateFn<Content, Result> interface. It simulates real-world
invocations from the outside - no source instance access, full blackbox.
import type { Trigger } from '@walkeros/core';
import { startFlow } from '@walkeros/collector';
export const createTrigger: Trigger.CreateFn<Content, Result> = async (
config,
) => {
let flow: Trigger.FlowHandle | undefined;
const trigger: Trigger.Fn<Content, Result> =
(type?: string) => async (content) => {
if (!flow) {
const result = await startFlow(config);
flow = { collector: result.collector, elb: result.elb };
}
// Package-specific: make real HTTP request, inject DOM, dispatch SDK call.
// Return the Result type declared by this source.
return /* ... */;
};
return {
get flow() {
return flow;
},
trigger,
};
};
Reference implementations:
packages/web/sources/browser/src/examples/trigger.ts - DOM
injection + native event dispatchpackages/web/sources/session/src/examples/trigger.ts - no
env.ts, trigger drives collector directlypackages/server/sources/express/src/examples/trigger.ts - real
HTTP fetch() to running serverpackages/web/sources/cmps/usercentrics/src/examples/trigger.ts - dispatches
CMP events, asserts on collector consent statepackages/server/sources/fetch/src/examples/trigger.ts - accesses source
instance via collector.sources, calls source.push() with platform-native
Requestpackages/server/sources/aws/src/lambda/examples/trigger.ts -
constructs API Gateway event + Lambda contextpackages/server/sources/gcp/src/cloudfunction/examples/trigger.ts -
synthesizes mock req/res (matching GCP Functions Framework)The examples authored here are the Phase 8 test fixtures. No parallel fixtures allowed.
src/index.test.ts MUST iterate examples via
it.each(Object.entries(examples.step)).examples.step, add it to step.ts
first, then consume it from the test.examples.createTrigger(config) and dispatch each example's
trigger.type + in content, asserting the collector receives out.See the canonical source tests under
packages/web/sources/browser/src/index.test.ts and
packages/server/sources/express/src/index.test.ts.
dev.tsexport * as schemas from './schemas';
export * as examples from './examples';
src/examples/step.ts - one Flow.StepExample per captured trigger /
input shape, typed trigger / in / outsrc/examples/trigger.ts - exports createTrigger typed as
Trigger.CreateFn<Content, Result>src/examples/index.ts - barrel exports step, createTrigger, and
env (when present)src/examples/env.ts - included whenever the source touches platform
globals or injected deps; typed against local Env; no real networkinputs.ts, outputs.ts, requests.ts, or mapping.ts
filesany, no
reinvented request / response shapessrc/index.test.ts iterates examples.step via
it.each(Object.entries(...))examples.stepnpm run build passes - examples compile against published typestrigger + in → source push → matches outGoal: Document transformation from input format to walkerOS events.
Mapping lives inside each Flow.StepExample entry in step.ts - no
separate mapping.ts file. Sources typically carry the mapping either in the
source's own settings (see dataLayer for an example) or inline via the
trigger → in → out relationship: the in content is the raw platform
payload; the out is the walkerOS event after the source's transformation.
For each entry in step.ts, trace:
Input: examples.step.pageView.trigger + examples.step.pageView.in
↓ createTrigger dispatches the trigger
↓ Source receives platform content, runs its transformation
↓ Source calls env.push / collector.push
Output: Should match examples.step.pageView.out (a WalkerOS.Event)
(trigger, in) to outTemplate sources:
packages/web/sources/dataLayer/packages/server/sources/fetch/ (simplest pattern)cp -r packages/server/sources/fetch packages/server/sources/[name]
cd packages/server/sources/[name]
# Update package.json: name, description, repository.directory
Directory structure:
packages/server/sources/[name]/
├── src/
│ ├── index.ts # Main export
│ ├── index.test.ts # Tests against examples
│ ├── dev.ts # Exports schemas and examples
│ ├── examples/
│ ├── schemas/
│ └── types/
├── package.json
├── tsconfig.json
├── tsup.config.ts
├── jest.config.mjs
└── README.md
Sources can wire to transformer chains via next in the init config:
sources: {
mySource: {
code: sourceMySource,
config: { settings: { /* ... */ } },
next: 'validate' // Events go through validator before collector
}
}
Every walkerOS package ships a walkerOS.json file for CDN-based schema
discovery.
walkerOS field to package.json{
"walkerOS": { "type": "source", "platform": "web" },
"keywords": ["walkerOS", "walkerOS-source", ...]
}
buildDev() in tsup.config.tsReplace buildModules({ entry: ['src/dev.ts'] }) with buildDev():
import { buildDev } from '@walkeros/config/tsup';
// In defineConfig array:
buildDev(),
This auto-generates dist/walkerOS.json from your Zod schemas at build time.
If your source has capabilities, behaviors, or troubleshooting patterns not
obvious from schemas alone, add hints. See walkeros-writing-documentation
skill for full guidelines.
Create src/hints.ts:
import type { Hint } from '@walkeros/core';
export const hints: Hint.Hints = {
'capture-timing': {
text: 'Describes when events are captured. See settings schema for options.',
code: [{ lang: 'json', code: '{ "settings": { ... } }' }],
},
};
Export from src/dev.ts:
export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';
Guidelines:
walkerOS field in package.json with type and platformbuildDev() in tsup.config.tsdist/walkerOS.jsonwalkerOS and walkerOS-sourceIf your package wraps a third-party npm dep that cannot be ESM-bundled (uses
__dirname, ships a .node binary, etc.), declare it under
walkerOS.bundle.external in your package.json. See
walkeros-using-cli → Bundle externals
for the complete contract.
Now write code to produce the outputs defined in Phase 3.
| File | Purpose | Template |
| ------------------ | ---------------------- | ------------------------------------------- |
| types/index.ts | Type definitions | types.ts |
| schemas/index.ts | Zod validation schemas | schemas.ts |
| index.ts | Main source | index.ts |
config, env, logger, id from
contextenv.push() to send events to the collectorenv with fallback to globals/imports:
env.express ?? express. This enables testing without mocking globals.logger?.error() for errors only, not routine
operations{ type, config, push } objectdestroy method: Implement if the source holds resources (HTTP
servers, timers, connections) that need cleanup on shutdownnpm run build passesnpm run verify:touched -- <source-name> passes (L1: typecheck + lint +
test)A source package can implement an optional setup() function to provision
external resources idempotently: Pub/Sub subscriptions, webhook registrations on
upstream platforms, queue declarations, SQS queues, polling cursors, inbound API
keys. Setup runs only when an operator explicitly types
walkeros setup source.<name>. The runtime never auto-invokes it from init(),
push, or destroy().
The framework provides the slot, the CLI command, and a resolveSetup helper.
The package owns: what setup means, idempotency, error handling, return value.
For background on how setup fits the source lifecycle, see understanding-sources.
// types/index.ts
import type { CoreSource } from '@walkeros/core';
export interface Settings {
/* runtime push settings */
}
export interface InitSettings {
/* one-time init settings */
}
export interface Mapping {
/* event extraction mapping */
}
export interface Env {
/* injected platform deps (SDK clients, request handlers, etc.) */
}
// The package's own setup options interface.
// Becomes the U slot of Types; surfaces as `config.setup: boolean | Setup` for users.
export interface Setup {
// package-specific provisioning options
// e.g. for Pub/Sub source: subscriptionName, ackDeadlineSeconds, filter
subscriptionName?: string;
ackDeadlineSeconds?: number;
}
export type Types = CoreSource.Types<
Settings,
Mapping,
Env,
InitSettings,
Setup
>;
// setup.ts
import type { CoreSource, SetupFn } from '@walkeros/core';
import { resolveSetup } from '@walkeros/core';
import type { Setup, Types } from './types';
const DEFAULT_SETUP: Setup = {
ackDeadlineSeconds: 60,
};
export const setup: SetupFn<
CoreSource.Config<Types>,
CoreSource.Env<Types>
> = async ({ config, env, logger }) => {
const options = resolveSetup(config.setup, DEFAULT_SETUP);
if (!options) return; // config.setup is false or unset
// Package-specific provisioning, idempotent.
// Returning a structured object (e.g. { subscriptionCreated: true })
// makes that data available to operators via `walkeros setup ... | jq`.
};
Wire it in your default export:
// index.ts
import { setup } from './setup';
export default {
type: 'my-source',
init: /* ... */,
setup,
};
Implement setup() when your source needs first-time provisioning of upstream
resources before events can be received: Pub/Sub subscriptions bound to a topic,
webhook registrations on upstream SaaS platforms (Stripe, GitHub, Shopify), SQS
queue declarations, message broker bindings, polling cursors. Skip it when your
source only listens on an HTTP port the runtime already owns or intercepts data
already present (DOM events, dataLayer pushes).
walkeros setup <kind>.<name>. Never by runtime push, init,
or destroy.IF NOT EXISTS on SQL, native idempotent operations where available.
The framework does not retry, track state, or detect drift.setup() when useful for operator scripting. The
CLI emits non-undefined return values as JSON to stdout.setup: true (boolean form) is meaningless because
mandatory fields have no safe defaults (e.g., GitHub webhook webhookUrl,
Pub/Sub source topicName), reject the boolean form with a clear runtime
error listing required fields:if (config.setup === true) {
throw new Error(
'github-webhook source setup requires explicit options: ' +
'{ webhookUrl, repo, events }. There is no safe default.',
);
}
Tests verify implementation against the examples from Phase 3. If examples are incomplete, tests will be incomplete.
See testing-strategy for the shared env / dev-examples conventions this phase depends on.
Verify implementation produces expected outputs.
Use the test template: index.test.ts. Canonical references:
packages/web/sources/browser/src/index.test.tspackages/server/sources/express/src/index.test.tsit.each(Object.entries(examples.step)) is mandatory - one iteration per
step example. Do not write per-feature tests with hand-rolled payloads.createTrigger - construct the trigger with startFlow
config, then dispatch each example's trigger.type + in content.createSourceContext() helper for any direct context construction.examples.step or examples.env. If you need
something new, add it to examples first.examples.step if needed).npm run verify:touched -- <source-name> passes (L1)it.each(Object.entries(examples.step))examples.step[...].outFollow the writing-documentation skill for:
apps/quickstart/Key requirements for source documentation:
Beyond
understanding-development
requirements (build, test, lint, no any):
dev.ts exports schemas and exampleswalkerOS.json generated at build timewalkerOS field in package.json| What | Where |
| --------------- | ----------------------------------- |
| Web template | packages/web/sources/dataLayer/ |
| Server template | packages/server/sources/fetch/ |
| Source types | packages/core/src/types/source.ts |
| Event creation | packages/core/src/lib/event.ts |
Flow.StepExample + createTrigger pattern, Three Type Zonestesting
Use when wiring `@walkeros/transformer-ga4` into a server flow, overriding default GA4 event mappings, dropping events, adding custom event keys, or troubleshooting GA4 Measurement Protocol decoding. Covers the `before`-chain wiring contract, configuration recipes, and per-field patching with extend/remove.
development
Use when adding read-through caching to a walkerOS store, memoizing a slow API/Sheets backing, composing multi-tier cache chains, or deduplicating concurrent store reads. Covers recipes, TTL choice, error policy, and observability counters.
development
Use when writing or updating walkerOS documentation - README, website docs, or skills. Covers quality standards, example validation, and DRY patterns.
testing
Use when writing, simulating, validating, or testing with walkerOS step examples. Covers the complete lifecycle from authoring examples to CI integration.