acedergren/secrets-management

OCI Vault and Secrets Management

NEVER Do This

❌ NEVER set temp key file permissions AFTER writing content

# WRONG - world-readable during write (security window exists)
with open('/tmp/key.pem', 'w') as f:
    f.write(private_key)
os.chmod('/tmp/key.pem', 0o600)  # Too late — race condition!

# RIGHT - secure BEFORE writing
fd = os.open('/tmp/key.pem', os.O_CREAT | os.O_WRONLY, 0o600)
with os.fdopen(fd, 'w') as f:
    f.write(private_key)

❌ NEVER use overly broad IAM secret policies

BAD:  "Allow any-user to read secret-family in tenancy"
BAD:  "Allow group Developers to manage secret-family in tenancy"
GOOD: "Allow dynamic-group app-prod to read secret-family in compartment AppSecrets
       where target.secret.name = 'db-*'"

❌ NEVER retrieve secrets without caching

• Cost: $0.03 per 10,000 requests (first 10k/month free)
• Without cache: 1000 req/hr × 24 × 30 = 720k/month = $2.16/month
• With 60-min cache: 24 calls/day = 720/month = FREE (98% cost reduction)

❌ NEVER use PLAIN content type — always use BASE64 encoding; PLAIN is deprecated and may fail in future API versions

❌ NEVER hardcode Vault OCIDs in code — store in environment variables; OCIDs leak to repos and aren't portable across tenancies

❌ NEVER log secret contents — even in debug/error messages; logs are retained in aggregation systems for years

IAM Permission Gotcha (Critical)

Secret retrieval requires BOTH of these:

"Allow dynamic-group X to read secret-family in compartment Y"
"Allow dynamic-group X to use keys in compartment Y"

• read secret-family → list secrets and read metadata
• use keys → decrypt secret content (all secrets are encrypted with a master key)

Without use keys: Confusing 403 — "User not authorized to perform this operation." Hours of debugging because the error message doesn't mention key permissions.

Vault Hierarchy (Often Confused)

Vault (container)
 └─ Master Encryption Key (for encryption/decryption)
     └─ Secret (encrypted data)
         └─ Secret Versions (rotation over time)

Commands use different services — this trips everyone up:

• Vault operations: oci kms management vault ...
• Key operations: oci kms management key ... --endpoint <vault-management-endpoint>
• Secret operations: oci vault secret ... (NOT oci kms!)

Common mistake: oci vault-secret create (no such command) vs oci vault secret create (correct)

Secret Retrieval Error Decision Tree

Secret retrieval fails?
│
├─ 401 Unauthorized
│  ├─ On OCI compute? → Check dynamic group membership
│  ├─ Local dev? → Check ~/.oci/config, verify API key uploaded
│  └─ After rotation? → Cache has old credentials (wait for TTL)
│
├─ 403 Forbidden
│  ├─ Have "read secret-family"? → Add if missing
│  └─ Have "use keys"? → THIS IS USUALLY THE ISSUE
│
├─ 404 Not Found
│  ├─ Wrong OCID? → Verify env variable
│  ├─ Wrong compartment? → Secrets client must use secret's compartment
│  └─ Secret deleted? → Check vault for secret status
│
└─ 500 Internal Server Error
   └─ Vault rate limit → Retry with exponential backoff

Secret Rotation (Zero-Downtime)

# WRONG - creates new OCID, breaks all running apps
oci vault secret delete --secret-id <secret-ocid>
oci vault secret create ...

# RIGHT - create new VERSION of existing secret (OCID unchanged)
oci vault secret update-base64 \
  --secret-id <secret-ocid> \
  --secret-content-content "$(echo -n 'new-value' | base64)"

Apps pick up new version on next cache refresh — no restart needed. Old version retained for rollback.

Cache TTL Selection

| Security Requirements | Cache TTL | Reasoning | |----------------------|-----------|-----------| | High (rotate daily) | 5-15 min | 90%+ savings, frequent refresh | | Standard (rotate monthly) | 30-60 min | Balance security and cost | | Dev/Test | No cache | Always fresh |

Rule: Cache TTL must be less than secret rotation window.

OCI-Specific Gotchas

Vault management endpoint is required for key operations:

# Find vault's management endpoint
oci kms management vault get --vault-id <vault-ocid> \
  --query 'data."management-endpoint"' --raw-output

# Required for all key commands
oci kms management key create ... \
  --endpoint https://xxxxx-management.kms.us-ashburn-1.oraclecloud.com

Secret bundle requires explicit base64 decode:

secret_bundle = secrets_client.get_secret_bundle(secret_ocid)
encoded = secret_bundle.data.secret_bundle_content.content
decoded = base64.b64decode(encoded).decode('utf-8')  # Both steps required

Not all OCI regions have Vault service — check availability before designing architecture. Cross-region secret access adds 10-50ms latency.

Instance Principal Auth (Production Pattern)

# 1. Create dynamic group
oci iam dynamic-group create \
  --name "app-instances" \
  --matching-rule "instance.compartment.id = '<compartment-ocid>'"

# 2. Grant Vault access (both policies required — see IAM gotcha above)
# "Allow dynamic-group app-instances to read secret-family in compartment Secrets"
# "Allow dynamic-group app-instances to use keys in compartment Secrets"

# 3. Application code — no credentials needed on instance
signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
secrets_client = oci.secrets.SecretsClient(config={}, signer=signer)

Reference Files

Load references/oci-vault-reference.md when you need:

• Comprehensive Vault/KMS API documentation
• HSM-backed key protection setup
• Cross-region secret replication
• Official Oracle guidance on Vault architecture