d-subrahmanyam/cloud-storage

Cloud Object Storage

AWS S3 (Recommended Default)

Setup (SDK v3)

import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

const s3 = new S3Client({ region: process.env.AWS_REGION });

Upload

// Direct upload
await s3.send(new PutObjectCommand({
  Bucket: process.env.S3_BUCKET,
  Key: `uploads/${userId}/${filename}`,
  Body: buffer,
  ContentType: mimeType,
  Metadata: { 'uploaded-by': userId },
}));

// Presigned URL (client uploads directly to S3)
const url = await getSignedUrl(s3, new PutObjectCommand({
  Bucket: process.env.S3_BUCKET,
  Key: `uploads/${key}`,
  ContentType: mimeType,
}), { expiresIn: 3600 });

Presigned Download

const url = await getSignedUrl(s3, new GetObjectCommand({
  Bucket: process.env.S3_BUCKET,
  Key: fileKey,
}), { expiresIn: 3600 });

Multipart Upload (large files)

import { CreateMultipartUploadCommand, UploadPartCommand, CompleteMultipartUploadCommand } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';

// Simplified with lib-storage
const upload = new Upload({
  client: s3,
  params: { Bucket: bucket, Key: key, Body: readableStream },
  partSize: 10 * 1024 * 1024, // 10MB parts
  leavePartsOnError: false,
});
upload.on('httpUploadProgress', (progress) => console.log(progress));
await upload.done();

Python (boto3)

import boto3
from botocore.config import Config

s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

# Upload
s3.upload_fileobj(file_obj, bucket, key, ExtraArgs={'ContentType': mime_type})

# Presigned URL
url = s3.generate_presigned_url('put_object',
    Params={'Bucket': bucket, 'Key': key, 'ContentType': mime_type},
    ExpiresIn=3600)

Java (Spring)

@Service
public class S3Service {
    private final S3Client s3;

    public String generatePresignedUploadUrl(String key, String contentType) {
        var presigner = S3Presigner.builder().region(Region.of(region)).build();
        var request = PutObjectPresignRequest.builder()
            .signatureDuration(Duration.ofHours(1))
            .putObjectRequest(b -> b.bucket(bucket).key(key).contentType(contentType))
            .build();
        return presigner.presignPutObject(request).url().toString();
    }
}

Bucket Policy Patterns

Public read for assets

{
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::my-bucket/public/*"
  }]
}

CORS for direct uploads

{
  "CORSRules": [{
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT", "POST"],
    "AllowedOrigins": ["https://myapp.com"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3600
  }]
}

Anti-Patterns

| Anti-Pattern | Fix | |--------------|-----| | Proxying large files through server | Use presigned URLs for direct client upload/download | | No Content-Type on upload | Always set ContentType to enable correct browser behavior | | Predictable file keys | Use UUIDs or hashed paths to prevent enumeration | | No lifecycle rules | Set expiration for temp uploads, transition old files to Glacier | | Long-lived presigned URLs | Keep expiry short (1h for uploads, 15m for downloads) | | No multipart for large files | Use multipart upload for files > 100MB |

Quick Troubleshooting

| Issue | Cause | Fix | |-------|-------|-----| | 403 on presigned URL | Expired, wrong region, or CORS | Check expiry, region, CORS config | | SignatureDoesNotMatch | ContentType mismatch | Ensure client sends same ContentType used in signing | | Slow uploads | Single-part for large files | Use multipart upload with parallel parts | | CORS errors | Missing CORS configuration | Add CORSRules to bucket | | AccessDenied on public files | Missing bucket policy | Add public read policy for the prefix |

Production Checklist

• [ ] Presigned URLs for all client uploads (never proxy large files)
• [ ] ContentType set on every upload
• [ ] UUID-based keys (prevent enumeration)
• [ ] CORS configured for direct uploads
• [ ] Lifecycle rules for temp/old files
• [ ] Encryption at rest enabled (SSE-S3 or SSE-KMS)
• [ ] Access logging enabled
• [ ] Versioning enabled for critical buckets
• [ ] CloudFront CDN for public assets