elbwalker/walkeros-understanding-stores

Understanding walkerOS Stores

Overview

Stores provide key-value storage that other components (sources, transformers, destinations) consume via environment injection. They are the 4th component type in Flow.Json alongside sources, transformers, and destinations.

Core principle: Stores are passive infrastructure. They don't process events or participate in chains — they provide state that other components read and write.

Store interface

See packages/core/src/types/store.ts for the canonical interface.

Instance

| Property | Type | Purpose | Required | | --------- | ---------------------------------- | ---------------------------- | ------------ | | type | string | Store type identifier | Required | | config | Store.Config | Settings and env | Required | | get | (key) => StoreValue \| undefined | Read a value | Required | | set | (key, value, ttl?) => void | Write a value (optional TTL) | Required | | delete | (key) => void | Remove a value | Required | | destroy | DestroyFn | Cleanup on shutdown | Optional |

All methods can be sync or async (return Promise).

Value type and file mode

Stores hold one canonical value type: structured data (StoreValue), with binary (Uint8Array) as a first-class leaf. StoreValue is string | number | boolean | null | Uint8Array | StoreValue[] | { [key: string]: StoreValue } (undefined is reserved as the "miss" sentinel and is never a stored value). A shared core codec (serializeStoreValue / deserializeStoreValue) round-trips that value to and from each backing.

Store.Config.file?: boolean (default false) picks the mode, decided once at init:

• Structured (default): values are StoreValue data, serialized by the shared codec.
• File (file: true): a byte-native backend (fs, S3, GCS) persists raw bytes byte-exact. set() accepts a Uint8Array or string and stores it untouched; get() hands the exact bytes back. Use for serving assets such as walker.js. The Sheets store is structured-only and rejects file: true at init.

TTL is owned by the cache layer, not the store. The store persists values; a cache wrapper manages expiry. flow_validate warns when a store sets both file: true and cache, and when a @walkeros/server-transformer-file is wired to a byte-native store that does not set file: true.

Init function (context pattern)

Stores use the same context-based init pattern as other components:

import type { Store } from '@walkeros/core';

export const storeMyStore: Store.Init = (context) => {
  const { config, env, logger, id } = context;
  const settings = config.settings || {};

  return {
    type: 'my-store',
    config: context.config as Store.Config,
    get(key) {
      /* ... */
    },
    set(key, value, ttl) {
      /* ... */
    },
    delete(key) {
      /* ... */
    },
    destroy() {
      /* cleanup */
    },
  };
};

Context contains:

| Property | Type | Purpose | | ----------- | -------------------- | ------------------------- | | config | Store.Config | Settings from flow config | | env | Store.Env | Environment dependencies | | logger | Logger.Instance | Scoped logger | | id | string | Store identifier | | collector | Collector.Instance | Reference to collector |

Lifecycle

Stores have the simplest lifecycle of all component types:

Startup:  Stores → Destinations → Transformers → Sources
Shutdown: Sources → Destinations → Transformers → Stores

• Init first: Stores init before all other components so they're available when sources/transformers/destinations start
• Destroy last: Stores destroy after all other components so transformers can flush during their own destroy
• No lazy init: Unlike destinations, stores don't support require or deferred activation — they are always eager

Wiring stores via $store.

Bundled mode (flow.json)

Use $store.storeId in a component's env to inject a store instance:

{
  "stores": {
    "data": {
      "package": "@walkeros/server-store-fs",
      "config": { "settings": { "basePath": "./data" } },
      "cache": { "rules": [{ "ttl": 60 }] }
    }
  },
  "transformers": {
    "fingerprint": {
      "package": "@walkeros/server-transformer-fingerprint",
      "env": { "store": "$store.data" }
    }
  }
}

The bundler resolves $store.data to a runtime reference. Invalid references are caught at build time. walkeros validate also catches typos at validation time, including unknown store names and the colon-instead-of-dot mistake (e.g. $store:data is flagged with the suggested form $store.data).

The cache field enables the built-in in-memory cache tier (__cache) on top of any backing store. No separate memory store declaration is needed.

Integrated mode (TypeScript)

Pass store instances directly — no $store. prefix needed:

import { startFlow } from '@walkeros/collector';
import { storeFsInit } from '@walkeros/server-store-fs';
import { transformerFingerprint } from '@walkeros/server-transformer-fingerprint';

const { collector } = await startFlow({
  stores: {
    data: {
      code: storeFsInit,
      config: { settings: { basePath: './data' } },
      cache: { rules: [{ ttl: 60 }] },
    },
  },
  transformers: {
    fingerprint: {
      code: transformerFingerprint,
      env: { store: collector.stores.data }, // Direct reference
    },
  },
});

Note: In integrated mode, you wire the store instance directly in env rather than using the $store. string prefix (that's a bundler feature).

Available stores

Built-in cache tier (__cache)

The collector ships a built-in in-memory cache with LRU eviction, TTL, and entry/byte caps. Enable it on any store by setting Flow.Store.cache. No separate package import needed:

{
  "stores": {
    "files": {
      "package": "@walkeros/server-store-fs",
      "cache": { "rules": [{ "ttl": 60 }] }
    }
  }
}

Use this for the common "cache in front of a slow backing store" pattern (API, GCS, Sheets, etc.).

@walkeros/server-store-fs (filesystem)

File-based store for serving static assets. Server-only.

import { storeFsInit } from '@walkeros/server-store-fs';

@walkeros/server-store-s3 (S3-compatible object storage)

S3-compatible store using s3mini (~20 KB, zero dependencies). Works with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO, and any S3-compatible provider. Structured by default (stored as application/json); set file: true to serve raw bytes byte-exact with the real mime. Server-only.

import { storeS3Init } from '@walkeros/server-store-s3';

Settings:

| Setting | Type | Required | Default | Purpose | | ----------------- | -------- | -------- | -------- | -------------------------- | | bucket | string | Yes | — | S3 bucket name | | endpoint | string | Yes | — | S3-compatible endpoint URL | | accessKeyId | string | Yes | — | S3 access key ID | | secretAccessKey | string | Yes | — | S3 secret access key | | region | string | No | "auto" | AWS region (SigV4 signing) | | prefix | string | No | — | Key prefix for scoping |

Primary use case: Serving static files in managed deployments (Mode D) where files live in a bucket rather than being baked into a Docker image.

@walkeros/server-store-gcs (Google Cloud Storage)

Zero-dependency GCS store using raw fetch + GCS JSON API. Built-in auth: ADC on Cloud Run / GKE, or explicit service account JWT. Server-only.

import { storeGcsInit } from '@walkeros/server-store-gcs';

Settings:

| Setting | Type | Required | Default | Purpose | | -------- | -------- | -------- | ------- | ---------------------- | | bucket | string | Yes | — | GCS bucket name | | prefix | string | No | — | Key prefix for scoping |

Credentials live at config.credentials (sibling of settings): string | object SA JSON for non-GCP envs, $env-resolvable; omit for ADC. The deprecated settings.credentials still works.

Primary use case: Serving static files on GCP infrastructure (Cloud Run, GKE) where ADC provides seamless authentication.

@walkeros/server-store-sheets (Google Sheets)

Zero-dependency Google Sheets store using raw fetch + Sheets v4 REST API. One row per key, one cell per value (JSON-serialized). Built-in auth shared with the GCS store. Server-only. Structured-only: cells hold structured JSON, so it rejects file: true at init and rejects values carrying a binary (Uint8Array) leaf. Use fs, S3, or GCS for byte-exact serving.

import { storeSheetsInit } from '@walkeros/server-store-sheets';

Settings:

| Setting | Type | Required | Default | Purpose | | ------------ | -------- | -------- | ---------- | -------------------------------- | | id | string | Yes | — | Spreadsheet ID (segment in URL) | | sheet | string | No | 'Sheet1' | Sheet (tab) name | | key | string | No | 'A' | Column letter for keys | | value | string | No | 'B' | Column letter for JSON values | | headerRows | number | No | 1 | Header rows to skip when reading |

Credentials live at config.credentials (sibling of settings): string | object SA JSON for non-GCP envs, $env-resolvable; omit for ADC. The deprecated settings.credentials still works.

Primary use case: Demos and small prototypes where the spreadsheet is the operator-facing UI for tweaking lookup data. Quota: 60 reads/min and 60 writes/min per project. Enable the built-in cache via Flow.Store.cache on the store declaration to absorb the quota, otherwise quota burns in seconds. Not a production CRM substitute. See Website: Sheets Store for the cache-wiring example.

Stores in flow config

Declare any store consumed by one or more components in the stores section of the flow config and wire it via $store.<id> in component env. The built-in cache tier (Flow.Store.cache) covers the "fast in-memory cache in front of a slow backing store" pattern without a separate memory store.

Accessing stores at runtime

After startFlow(), stores are available on the collector instance:

const { collector } = await startFlow({
  stores: {
    files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
  },
});

// Read/write
await collector.stores.files.set('key', 'value', 60000); // 60s TTL
const value = await collector.stores.files.get('key');
await collector.stores.files.delete('key');

Hooks

Store operations (get, set, delete) are wrapped with useHooks during initialization, enabling pre/post interception via the collector's hooks system.

Available hook names:

| Hook name | Pre hook | Post hook | | ------------- | ---------------- | ----------------- | | StoreGet | preStoreGet | postStoreGet | | StoreSet | preStoreSet | postStoreSet | | StoreDelete | preStoreDelete | postStoreDelete |

Hooks fire on every store operation regardless of which component triggered it (cache system, transformer via env, destination via env, direct access on collector.stores).

const { collector } = await startFlow({
  stores: {
    files: { code: storeFsInit, config: { settings: { basePath: './data' } } },
  },
});

// Intercept all store reads
collector.hooks.preStoreGet = ({ fn }, key) => {
  console.log('Reading key:', key);
  return fn(key);
};

Key differences from other components

| Aspect | Sources/Transformers/Destinations | Stores | | ------------- | --------------------------------- | ------------------------ | | Event flow | Participate in push chain | No push, no chain | | next/before | Chain connection fields | None (passive) | | Lifecycle | Init after stores | Init first, destroy last | | require | Deferred activation supported | Always eager | | Interface | push(event, context) | get/set/delete(key) |

Dev exports (dev.ts)

Every store package must export schemas and examples via dev.ts, matching the convention used by sources and destinations:

Required files

src/
├── schemas/
│   ├── settings.ts   # Zod schema for store settings
│   └── index.ts      # Re-export + zodToSchema() conversion
├── examples/
│   └── index.ts      # Example Store.Config objects
└── dev.ts            # export * as schemas; export * as examples

Schema pattern

// src/schemas/settings.ts
import { z } from '@walkeros/core/dev';

export const SettingsSchema = z.object({
  mySetting: z.string().describe('Setting description for docs and MCP'),
});

export type Settings = z.infer<typeof SettingsSchema>;

// src/schemas/index.ts
import { zodToSchema } from '@walkeros/core/dev';
import { SettingsSchema } from './settings';
export { SettingsSchema, type Settings } from './settings';
export const settings = zodToSchema(SettingsSchema);

tsup + package.json

// tsup.config.ts
import { defineConfig, buildModules, buildDev } from '@walkeros/config/tsup';
export default defineConfig([buildModules(), buildDev()]);

Add ./dev export to package.json:

"./dev": {
  "types": "./dist/dev.d.ts",
  "import": "./dist/dev.mjs",
  "require": "./dist/dev.js"
}

Why this matters

• buildDev() generates dist/walkerOS.json at build time
• MCP tools fetch walkerOS.json from CDN for package discovery and schema validation
• Website docs use <PropertyTable schema={schemas.settings} /> instead of hardcoded markdown tables
• Without dev.ts, a store is invisible to MCP and docs tables rot

Hints (Optional)

Stores can optionally export hints — lightweight, actionable context for AI agents beyond what schemas and examples already provide. Create src/hints.ts:

import type { Hint } from '@walkeros/core';

export const hints: Hint.Hints = {
  'persistence-behavior': {
    text: 'Describes persistence guarantees. See settings schema for options.',
  },
};

Export from src/dev.ts alongside schemas and examples:

export * as schemas from './schemas';
export * as examples from './examples';
export { hints } from './hints';

Most stores don't need hints — only add them for non-obvious behaviors, prerequisites, or troubleshooting patterns.

Setup (optional)

Stores can implement an optional setup() lifecycle to provision external resources, for example creating a SQLite table, initializing an S3 bucket, or running a one-off schema migration. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup store.<name>.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-destination, walkeros-create-source, walkeros-understanding-destinations, walkeros-understanding-sources, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related skills

• walkeros-understanding-flow - Architecture and Flow structure
• walkeros-understanding-transformers - Transformer interface and env pattern
• walkeros-using-store-cache - Store-level cache and the declarative state block for fetch/stash without $code:
• walkeros-using-cli - Bundling flows with stores

Source files:

• packages/core/src/types/store.ts - Store types
• packages/collector/src/store.ts - initStores implementation
• packages/server/stores/fs/src/ - Filesystem store package
• packages/server/stores/s3/src/ - S3 store package
• packages/server/stores/gcs/src/ - GCS store package
• packages/server/stores/sheets/src/ - Google Sheets store package

Documentation:

• Website: Flow - Flow configuration with stores section
• Website: Stores - Stores overview
• Website: S3 Store - S3 store documentation
• Website: FS Store - Filesystem store documentation
• Website: GCS Store - GCS store documentation
• Website: Sheets Store - Google Sheets store documentation