curiositech/file-upload-storage-expert

File Upload & Storage Expert

Design secure, scalable file upload and storage systems with presigned URLs, multipart uploads, CDN distribution, and lifecycle management.

Activation Triggers

Activate on: "file upload", "S3", "presigned URL", "multipart upload", "CDN", "blob storage", "image upload", "R2", "media storage", "upload progress"

NOT for: Data lake storage → lakehouse-architect | Database BLOB columns → dimensional-modeler | Video processing pipelines → relevant media skill

Quick Start

1. Choose storage backend — S3, R2 (zero egress), GCS, or Supabase Storage
2. Generate presigned URLs server-side — never expose credentials to clients
3. Implement multipart upload for files >10MB with resume capability
4. Process asynchronously — thumbnails, transcoding, virus scanning via queue
5. Serve via CDN — CloudFront, Cloudflare, or Vercel Edge for global delivery

Core Capabilities

| Domain | Technologies | |--------|-------------| | Object Storage | AWS S3, Cloudflare R2, GCS, MinIO | | Upload Libraries | @aws-sdk/client-s3, tus-js-client, Uppy 4.x | | CDN | CloudFront, Cloudflare CDN, Fastly | | Processing | Sharp (images), FFmpeg (video), ClamAV (scanning) | | Managed | Supabase Storage, Vercel Blob, Uploadthing |

Architecture Patterns

Presigned URL Upload Flow

Client                    Server                    S3/R2
  │                         │                         │
  ├─ POST /upload/init ────→│                         │
  │  { filename, type }     │                         │
  │                         ├─ createPresignedPost() ─→│
  │                         │                         │
  │←── { url, fields } ────┤                         │
  │                         │                         │
  ├─ PUT (direct to S3) ──────────────────────────────→│
  │                         │                         │
  │                         │←── S3 Event Notification│
  │                         ├─ Process (thumbnail,    │
  │                         │   scan, metadata)       │
  │←── webhook: ready ─────┤                         │

Multipart Upload with Resume

import { S3Client, CreateMultipartUploadCommand,
  UploadPartCommand, CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';

const PART_SIZE = 10 * 1024 * 1024; // 10MB parts

async function multipartUpload(file: Buffer, key: string) {
  const { UploadId } = await s3.send(
    new CreateMultipartUploadCommand({ Bucket: BUCKET, Key: key })
  );

  const parts: { ETag: string; PartNumber: number }[] = [];
  for (let i = 0; i < file.length; i += PART_SIZE) {
    const partNumber = Math.floor(i / PART_SIZE) + 1;
    const { ETag } = await s3.send(new UploadPartCommand({
      Bucket: BUCKET, Key: key, UploadId,
      PartNumber: partNumber,
      Body: file.subarray(i, i + PART_SIZE),
    }));
    parts.push({ ETag: ETag!, PartNumber: partNumber });
  }

  await s3.send(new CompleteMultipartUploadCommand({
    Bucket: BUCKET, Key: key, UploadId,
    MultipartUpload: { Parts: parts },
  }));
}

CDN-First Serving Architecture

User Request → CDN Edge (cache hit?) ──yes──→ Serve cached
                    │ no
                    ↓
              Origin (S3/R2)
                    ↓
              Response + Cache-Control: public, max-age=31536000, immutable
              (use content-hash in filename for cache busting)

Anti-Patterns

1. Proxying uploads through your server — use presigned URLs so clients upload directly to object storage; your server should never be a relay
2. Synchronous processing — never resize/transcode during the upload request; push to a queue and process async
3. Mutable filenames — use content-addressable keys (sha256-{hash}.ext) for CDN cacheability
4. No upload size limits — enforce max file size server-side in presigned URL policy AND client-side
5. Storing credentials client-side — presigned URLs expire; never send AWS keys to the browser

Quality Checklist

• [ ] Presigned URLs used for client-to-storage uploads
• [ ] Presigned URL expiry set to 5-15 minutes
• [ ] Multipart upload implemented for files >10MB
• [ ] Upload size limits enforced in presigned URL policy
• [ ] Content-Type validation on server (not just extension)
• [ ] Virus/malware scanning before making files public
• [ ] CDN configured with immutable cache headers
• [ ] Lifecycle policy deletes incomplete multipart uploads after 24h
• [ ] CORS configured on bucket for browser direct uploads
• [ ] Storage costs monitored with lifecycle transitions (Standard → IA → Glacier)