platform-engineering-labs/formae-plugin-new

Build a New Resource Plugin

Create a complete formae resource plugin for a cloud provider or service, following the plugin SDK tutorial and using TDD for the implementation.

MANDATORY RULE: Install Before Testing

After ANY code changes to the plugin, you MUST run make install before running conformance tests. The conformance tests run against the INSTALLED plugin binary, not the source code. Skipping this step means you're testing stale code.

Workflow

1. Gather requirements

Ask the user an open-ended question: what provider or service do they want to build a plugin for, and which resource types should it support? Do NOT present a multiple-choice list of providers — there are thousands of possible targets and we cannot guess. Just ask.

Collect the following from the user:

• Provider/service: e.g., Cloudflare, Datadog, GitHub
• Resource types: Once the user names the provider, research the provider's API thoroughly before suggesting resource types. Present a well-researched list organized in implementation waves (e.g., "Wave 1: core resources, Wave 2: networking, Wave 3: IAM/security"). The user should be able to pick individual resources or entire waves. Do NOT present a hastily assembled shortlist — take the time to understand the provider's full resource catalog first.
• Credentials: how should authentication be configured? (env vars, config files, API keys)
• Namespace convention: The provider namespace becomes the uppercase prefix in formae resource type IDs (e.g., AWS, GCP, ATLAS). The scaffolder will uppercase mixed-case input automatically and emit a notice when it does so.
• Polymorphic structures: are there resources with different field shapes per subtype? (e.g., DNS records with different fields per record type)
• Development mode: autonomous or guided?
  • Autonomous: TDD through the entire implementation without pausing for review — for one-shotting a plugin quickly
  • Guided: pause at checkpoints for user review before continuing — for production-grade plugins where the user wants to curate schema design and review each step

2. Research the provider API

Research the provider's API to understand:

• Resource models: required vs optional fields, immutable fields (createOnly), read-only fields
• CRUD operations: endpoints, request/response shapes, error codes
• Authentication: how credentials are passed
• Async operations: does the API return immediately or require polling?
• Value normalization: does the API normalize values? (e.g., FQDN vs short name, case changes, default values injected server-side)

Look at how Terraform or Pulumi model equivalent resources for schema design reference.

If resources have polymorphic structures (like DNS record types with different field shapes), design the schema to accommodate all variants.

Guided mode checkpoint: present API research findings and proposed resource model to the user for approval before continuing.

2a. External binary dependencies

If your plugin shells out to an external binary (e.g., atlas, helm, kubectl), the binary must be available in the formae agent's $PATH at runtime. The plugin SDK does not yet ship a first-class mechanism for bundling extra binaries with plugin distributions; the conventions below are the current stop-gap.

Prior art. Formae core ships the pkl-reader-helm external binary using the pattern at formae/scripts/install-helm-reader.sh. This is the canonical reference for any plugin that depends on a runtime binary — read it before designing your own.

Plugin-side convention. On plugin initialization, call exec.LookPath("<binary>") and fail fast with a clear error message naming both the binary and the install path. Example error:

plugin atlas: required binary `atlas` not found in $PATH.
Install with: /path/to/install-atlas.sh
See: https://atlasgo.io/getting-started

Optionally expose a health-check endpoint that reports binary availability so operators can diagnose deployment issues without invoking real CRUD operations.

Multi-arch handling. Mirror the uname -s / uname -m mapping pattern from formae/scripts/install-helm-reader.sh to select the correct release asset per platform.

Dev-mode bootstrap. Until the SDK supports bundled binaries, mirror the install script in your plugin's own repository (e.g., scripts/install-atlas.sh). Have make install invoke it so fresh dev machines and CI runners can bootstrap automatically.

Known SDK limitation. Track the follow-up "plugin ships extra binaries" SDK enhancement — when that lands, plugins will declare their binary dependencies in formae-plugin.pkl and the agent host will install them at plugin registration time.

3. Scaffold the plugin

Run the scaffolding command:

formae plugin init --no-input --name <name> --namespace <NS> --description "<desc>" --author "<author>" --module-path "<path>" --license Apache-2.0

Verify the scaffolded project:

• Check the directory structure: ls -R
• Check the Makefile and formae-plugin.pkl
• Confirm it builds: make build

4. Follow the tutorial

Fetch the plugin SDK tutorial from https://docs.formae.io/en/latest/plugin-sdk/tutorial/ and follow it start to finish, adapting each lesson to the target provider.

Reference prior art plugins for general examples (single resource, simple CRUD): AWS, Azure, GCP, OCI, OVH — each at https://github.com/platform-engineering-labs/formae-plugin-<provider>.

For specific advanced patterns, the canonical references are:

| Pattern | Canonical reference | |---|---| | Resolvable computed outputs (custom Read, output-port story) | formae-plugin-aws/schema/pkl/ses/emailidentity.pkl | | Polymorphic Target auth (abstract + discriminated subclasses) | formae-plugin-k8s/schema/pkl/main/k8s.pkl | | Cross-plugin Target Resolvables (Target Config field accepts upstream resolvable) | formae-plugin-grafana/schema/pkl/grafana.pkl | | Collection Resolvables (Mapping<String, formae.Resolvable>) | formae-plugin-compose — see the Target Config file with the Mapping field | | Synthetic identifiers (identifier = "Label" for logical resources without natural server-side IDs) | formae-plugin-atlas — Migration resource schema |

Tutorial lessons to follow (adapt to target provider):

1. Schema — define PKL resource types with annotations (@ResourceType, @CreateOnly, @ReadOnly)
2. Target configuration — set up provider authentication and connection settings
3. Plugin configuration — RateLimit, LabelConfig, DiscoveryFilters
4. Create — implement resource creation
5. Read — implement resource reading (handle NotFound correctly)
6. Update — implement resource updates
7. Delete — implement resource deletion (handle NotFound as success)
8. List — implement resource discovery
9. Error handling — use OperationErrorCode patterns from the SDK
10. Conformance tests — run make conformance-test

Target Config field serialization (G-6). Plugin Go structs SHOULD declare camelCase json tags on Config fields (e.g., `json:"host"`). PKL fields should be single-name camelCase. Do NOT use the hidden lowercase / fixed PascalCase = lowercase pattern preserved in older plugins (k8s, Grafana) — that's a wart from Go structs lacking json tags, not a convention. Canonical reference for the clean pattern: the AWS plugin's Target Config classes.

Target Config field immutability (G-8). @formae.ConfigFieldHint.createOnly defaults to true (immutable), the opposite of @formae.FieldHint on resource fields (defaults to false, mutable). This is by-design: a Target's identity is typically defined by immutable environment fields (AWS account ID, region, server URL), so changing one means replacing the Target — and everything in it. Always declare createOnly explicitly on ConfigFieldHint annotations so intent is visible in PKL:

@formae.ConfigFieldHint { createOnly = false }  // mutable
hidden host: String

@formae.ConfigFieldHint { createOnly = true }   // immutable (also the default)
hidden region: String

MANDATORY: TDD for steps 4–8. Each tutorial lesson includes integration tests (e.g., TestCreate, TestRead, TestReadNotFound, TestUpdate, TestDelete, TestDeleteNotFound, TestList). For EVERY CRUD operation you MUST follow this exact loop:

1. Fetch the tutorial page for that operation (e.g., https://docs.formae.io/en/latest/plugin-sdk/tutorial/05-create/)
2. Write the integration test FIRST — adapt the tutorial's test to the target provider. The test must compile but fail because the operation is not yet implemented.
3. Run the test — confirm it fails for the right reason (not implemented, not a compile error).
4. Implement the operation — write the minimum code to make the test pass.
5. Run make install && go test -tags=integration ./... — confirm the test passes.
6. Move to the next operation — do NOT skip ahead.

NEVER write implementation code before its corresponding test. NEVER skip writing tests. If you implement Create without first writing TestCreate, you are doing it wrong.

Guided mode checkpoints (skip all in autonomous mode):

• After step 1 (schema): present PKL resource definitions for review
• After step 2 (target): present target config approach for review
• After each CRUD operation (steps 4–8): present the failing test for review, then present the passing implementation for review
• After step 8 (list/discovery): present discovery implementation for review

In guided mode, do not proceed past a checkpoint until the user approves. If the user requests changes, make them and re-present.

5. Conformance tests

Run make install && make conformance-test.

Fix any failures and re-run. All CRUD and discovery tests must pass before continuing.

6. Local end-to-end test

Create a test project and verify the full lifecycle:

1. formae project init in a temporary directory
2. Write a simple forma file using the new plugin's resource types
3. formae apply --mode reconcile --simulate to verify the plan
4. formae apply --mode reconcile to create real resources
5. formae inventory to verify resources are managed
6. formae destroy to clean up

7. Examples and documentation

• Add a working example in the examples/ directory
• Update README.md with:
  • Target configuration instructions
  • Credentials setup
  • Supported resource types with descriptions
  • Example usage

8. Definition of Done

Before reporting completion, verify each item:

• [ ] All target resource types are implemented with full CRUD
• [ ] Conformance tests pass (make install && make conformance-test)
• [ ] Local end-to-end test passes (create, read, update, destroy)
• [ ] Working example in examples/
• [ ] README has: target config, credentials, supported resources, examples

Important

• After ANY code changes, ALWAYS run make install before make conformance-test — tests run against the installed binary
• "Plugin not found" errors mean the plugin isn't installed or is at the wrong path — run make install and check the output
• Cloud APIs often normalize values (FQDN vs short name, case changes, default values) — preserve user input where possible and normalize for idempotency where the API requires it
• NEVER skip conformance tests — they verify the full CRUD lifecycle through the formae agent
• NEVER skip the local end-to-end test — conformance tests alone don't verify the full integration
• NEVER ask the user to run tests — run them yourself and report results