DevOps & Cloud Infrastructure

Produksi 24/7 · multi-AZ · autoscaling tanpa downtime

Infrastruktur produksi Shorts Studio AI

Platform cloud native berbasis microservices di Kubernetes: highly available, secure by default, observable penuh, dan hemat biaya untuk ribuan pengguna aktif serta jutaan video yang diproses setiap bulan.

Uptime target

99,95%

Zona ketersediaan

3 AZ

Autoscale maks

150 pod worker

RTO region

< 4 jam

Arsitektur Cloud

Alur permintaan dari frontend hingga observability

Container & Service Spec

Resource limit, probe, dan standar container stateless

Media Processing Service

Worker

Scratch di emptyDir, hasil langsung ke object storage, spot friendly.

Runtime
FFmpeg + NVENC container
Replika
2 – 120
CPU
2 / 8
Memory
4Gi / 16Gi
GPU
1 × NVIDIA L4 (nvidia.com/gpu)
Port
8084
Liveness
/health
Readiness
/ready
Stateless
Ya
Scale on
Render jobs, GPU utilization, Queue wait time
# syntax=docker/dockerfile:1.7
# ---- build ----
FROM node:20-alpine AS build
WORKDIR /app
COPY package.json bun.lockb ./
RUN npm ci --omit=dev=false
COPY . .
RUN npm run build:media

# ---- runtime ----
FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg nodejs npm && rm -rf /var/lib/apt/lists/*
WORKDIR /app
ENV NODE_ENV=production PORT=8084
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules

USER 10001:10001
EXPOSE 8084
HEALTHCHECK --interval=15s --timeout=3s --retries=3 \
  CMD wget -qO- http://127.0.0.1:8084/health || exit 1
CMD ["node", "dist/media/server.js"]
  • Multi-stage build, base image distroless / alpine yang di-pin dengan digest.
  • Jalan sebagai non-root user (uid 10001), filesystem root read-only.
  • HEALTHCHECK, graceful shutdown SIGTERM, dan timeout drain 30 detik.
  • Tidak ada secret di layer image — semua lewat env dari Secret Manager.
  • Image di-scan (Trivy) dan ditandatangani (cosign) sebelum push registry.
  • Semua container stateless: data sementara di emptyDir, hasil ke object storage.

Struktur Kubernetes

Objek inti yang digunakan di setiap cluster

Deployment

Semua service stateless (frontend, API, orchestrator, worker).

StatefulSet

PostgreSQL, Redis Cluster, Loki, Prometheus.

Service

ClusterIP internal + headless untuk StatefulSet.

Ingress

NGINX/Envoy Ingress + cert-manager TLS 1.3, WAF di depan.

ConfigMap

Konfigurasi non-sensitif per environment.

Secret

Disinkronkan dari Secret Manager via External Secrets Operator.

PersistentVolume

PVC gp3/SSD untuk database, Prometheus TSDB, Loki chunk.

HorizontalPodAutoscaler

CPU, memory, queue depth, render job, latency.

NetworkPolicy

Default deny, izin eksplisit antar tier.

PodDisruptionBudget

minAvailable 2 agar upgrade node tanpa downtime.

Strategi Autoscaling

Skala otomatis tanpa downtime (maxUnavailable 0 + PDB)

SinyalTargetBerlaku untukSumber metrik
CPU utilization70%Semua servicemetrics-server
Memory utilization75%API, Orchestratormetrics-server
Queue length (BullMQ)30 job/podWorker, OrchestratorPrometheus adapter
Render jobs aktif2 job/GPU podMedia ProcessingCustom metric
Pengguna aktif (SSE session)500/podGateway, WorkflowCustom metric
Latency p95< 400 msGateway, APIOpenTelemetry

CI/CD Pipeline

GitHub Actions — gagal di satu tahap = pipeline berhenti + notifikasi

  1. 1

    Lint

    ESLint + Prettier + tsgo typecheck

    Gagal → pipeline berhenti

  2. 2

    Unit Test

    Vitest, coverage minimal 80%

    Coverage turun → blok

  3. 3

    Integration Test

    Testcontainers Postgres + Redis

    Gagal → blok

  4. 4

    Security Scan

    npm audit, Trivy FS, Semgrep, gitleaks

    High/Critical → blok

  5. 5

    Build Docker Image

    Buildx multi-arch + SBOM (syft)

    Build error → blok

  6. 6

    Push Registry

    Scan image + cosign sign

    Vuln critical → blok

  7. 7

    Deploy Staging

    Helm upgrade --atomic namespace staging

    Rollback otomatis

  8. 8

    Smoke Test

    Playwright happy-path + API contract

    Gagal → rollback

  9. 9

    Deploy Production

    Canary 10% → 50% → 100% (Argo Rollouts)

    Manual approval

  10. 10

    Health Check

    Probe semua /health + query sintetis

    Gagal → rollback otomatis

  11. 11

    Monitoring

    Bake time 30 menit, SLO burn-rate watch

    Alert → rollback

name: production-pipeline
on:
  push: { branches: [main] }

concurrency:
  group: prod-${{ github.ref }}
  cancel-in-progress: false

jobs:
  quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      - run: bun run lint
      - run: bun run typecheck
      - run: bun run test:unit -- --coverage
      - run: bun run test:integration

  security:
    needs: quality
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: gitleaks/gitleaks-action@v2
      - uses: aquasecurity/trivy-action@master
        with: { scan-type: fs, severity: HIGH,CRITICAL, exit-code: "1" }

  build:
    needs: security
    runs-on: ubuntu-latest
    permissions: { id-token: write, packages: write }
    strategy:
      matrix:
        service: [frontend, api-gateway, auth, workflow, ai-orchestrator, media, notification, analytics]
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: registry.shortsstudio.ai/${{ matrix.service }}:${{ github.sha }}
          provenance: true
          sbom: true
      - run: cosign sign --yes registry.shortsstudio.ai/${{ matrix.service }}:${{ github.sha }}

  staging:
    needs: build
    environment: staging
    runs-on: ubuntu-latest
    steps:
      - run: helm upgrade --install shorts ./charts --namespace shorts-staging --atomic --timeout 10m --set image.tag=${{ github.sha }}
      - run: bun run test:smoke -- --base-url https://staging.shortsstudio.ai

  production:
    needs: staging
    environment: production   # manual approval
    runs-on: ubuntu-latest
    steps:
      - run: helm upgrade --install shorts ./charts --namespace shorts-prod --atomic --timeout 15m --set image.tag=${{ github.sha }}
      - run: bun run scripts/health-gate.ts
      - if: failure()
        run: helm rollback shorts --namespace shorts-prod

  notify:
    if: always()
    needs: [quality, security, build, staging, production]
    runs-on: ubuntu-latest
    steps:
      - run: bun run scripts/notify.ts --status ${{ job.status }}

Environment Terpisah

Config, secret, storage, database, monitoring, dan logging sendiri-sendiri

Domain

app.shortsstudio.ai

Cluster

k8s-prod multi-AZ (3 zona)

Database

pg-prod primary + 2 replica + PITR

Storage

bucket prod-* + replikasi lintas region

Replika

3–150 (HPA)

Deploy

Manual approval + canary 10% → 50% → 100%

Secret Management

Nol secret di dalam kode — semua dari Secret Manager / KMS

SecretPenyimpananRotasiKonsumen
AI_PROVIDER_KEYSSecret Manager90 hariAI Orchestrator
JWT_SIGNING_KEYSecret Manager (KMS)30 hariAuth
OAUTH_CLIENT_SECRETSecret Manager180 hariAuth
DATABASE_URLSecret Manager + IRSA90 hariAPI, Workflow
STORAGE_CREDENTIALSWorkload IdentityOtomatisMedia, API
ENCRYPTION_MASTER_KEYKMS (HSM-backed)365 hariData at rest
REDIS_AUTHSecret Manager90 hariQueue, Cache
WEBHOOK_SIGNING_SECRETSecret Manager180 hariNotification

Object Storage & Lifecycle

Bucket terpisah per jenis aset

prod-images

CDN

IA 30 hari → Glacier 180 hari

Retensi: 365 hari

prod-videos

CDN

IA 60 hari → Glacier 365 hari

Retensi: 3 tahun

prod-audio

CDN

IA 30 hari

Retensi: 365 hari

prod-subtitles

CDN

Standard

Retensi: 365 hari

prod-thumbnails

CDN

Standard + cache 1 tahun

Retensi: 365 hari

prod-temp

Private

Hapus otomatis 24 jam

Retensi: 1 hari

prod-backup

Private

Immutable object lock

Retensi: 365 hari

prod-logs

Private

Compact 30 hari → arsip

Retensi: 90 hari

CDN & Cache Policy

Kompresi Brotli, HTTP/3, dan cache immutable

Static JS/CSS (hashed)

public, max-age=31536000, immutable

Brotli + HTTP/3

Fonts

public, max-age=31536000, immutable

Preload + subset

Thumbnail & Image

public, max-age=604800, stale-while-revalidate=86400

AVIF/WebP auto

Video preview (HLS)

public, max-age=86400

Range request + edge segment cache

API JSON

no-store

Hanya lewat gateway, tidak di-cache CDN

Database High Availability

Primary → replica → backup → disaster recovery

Primary

Postgres 16, multi-AZ, PgBouncer connection pooling (transaction mode).

Read Replica ×2

Query analitik & dashboard diarahkan ke replica, lag dipantau < 2s.

Automatic Failover

Patroni + leader election, promosi replica < 30 detik.

Backup & PITR

WAL archiving kontinu ke object storage, restore ke titik waktu mana pun.

Disaster Recovery

Snapshot lintas region harian + Terraform untuk rebuild infra.

Queue Infrastructure

BullMQ di Redis Cluster dengan prioritas bertingkat

AntreanPrioritasConcurrencyRetry
workflowHigh405× exponential 2s→60s
renderingHigh123× + fallback CPU
thumbnailMedium303× exponential
seoMedium253× exponential
exportMedium153× + resume upload
notificationLow505× linear
retry-dlqBackground5Manual replay

Backup Strategy

Retensi 7 / 30 / 90 / 365 hari dengan uji pemulihan berkala

PostgreSQL (full + WAL)

Full harian 02:00 UTC, WAL kontinu

Retensi: 7 / 30 / 90 / 365 hari

Uji restore: Mingguan (otomatis ke sandbox)

Object Storage metadata

Harian

Retensi: 30 / 90 hari

Uji restore: Bulanan

Workflow state & Project Memory

Tiap 6 jam

Retensi: 30 hari

Uji restore: Bulanan

Configuration (Helm, Terraform)

Setiap commit (GitOps)

Retensi: 365 hari

Uji restore: Tiap rilis

Secrets (KMS envelope)

Harian terenkripsi

Retensi: 90 hari

Uji restore: Kuartalan

Monitoring

Prometheus + Grafana, dashboard real-time

Compute

CPU per serviceMemory pressureGPU utilizationPod restart

Traffic

API latency p50/p95/p99RPSError rate 4xx/5xxSuccess rate

Pipeline

Queue depthJob wait timeWorkflow completionRender time per video

Data

DB connectionsReplica lagSlow queryRedis hit ratioStorage usage

AI

Provider latencyToken usageFallback rateBiaya per video

Logging & Tracing

Loki + Winston, korelasi penuh via OpenTelemetry

Field wajib setiap log

  • timestamp (ISO 8601, UTC)
  • requestId (ULID per request)
  • traceId / spanId (OpenTelemetry)
  • projectId
  • userId + tenantId
  • service + version
  • severity (debug|info|warn|error|fatal)
  • event (api.request, workflow.stage, render.job, security.audit)
  • durationMs + statusCode

Span OpenTelemetry

  • http.request

    Gateway → service, atribut tenant & route.

  • auth.verify

    Validasi JWT, RBAC, MFA challenge.

  • workflow.stage

    Satu span per tahap dari 20 tahap pipeline.

  • ai.request

    Provider, model, token in/out, biaya, retry.

  • render.job

    Timeline build, encode, GPU backend, durasi.

  • db.query

    Statement hash, rows, waktu, replica/primary.

  • queue.publish / queue.consume

    Antrean, prioritas, waktu tunggu.

  • storage.upload

    Bucket, ukuran objek, waktu unggah.

Alerting

Aturan alert dan kanal notifikasi tim

API Down

Critical

probe_success == 0 selama 2 menit

Kanal: Paging + email

Queue Macet

Critical

queue_depth > 500 dan proses < 1/menit

Kanal: Paging

Database Lambat

High

p95 query > 1s selama 5 menit

Kanal: Chat tim

GPU Penuh

High

GPU util > 95% selama 10 menit

Kanal: Chat tim

Storage Hampir Penuh

Medium

Kapasitas terpakai > 85%

Kanal: Email

Render Gagal

High

Failure rate > 5% dalam 15 menit

Kanal: Paging

Lonjakan Error

Critical

5xx rate > 2% (burn-rate SLO 14×)

Kanal: Paging

Biaya AI Melebihi Ambang

High

Biaya harian > 120% anggaran

Kanal: Email + chat

Security Produksi

Secure by default di setiap lapisan

Transport

  • HTTPS only + HSTS preload
  • TLS 1.3
  • mTLS antar service (mesh)

Perimeter

  • WAF (OWASP CRS)
  • DDoS protection
  • Firewall / private subnet
  • Rate limiting per tenant & IP

Identity

  • RBAC granular
  • MFA wajib untuk admin
  • Least privilege IAM
  • Session rotation + revocation

Data

  • Encryption at rest (KMS)
  • Encryption in transit
  • Field-level encryption PII
  • Signed URL berumur pendek

Aplikasi

  • Input validation (Zod)
  • Output sanitization
  • CSP + security headers
  • Audit trail immutable

Supply chain

  • Dependency scanning
  • Container scanning
  • Image signing (cosign)
  • SBOM tiap rilis

Disaster Recovery

Skenario, RTO/RPO, dan langkah pemulihan

SkenarioRTORPOTindakan
Kegagalan pod / node< 1 menit0Reschedule otomatis, PDB menjaga kapasitas.
Kegagalan zona (AZ)< 5 menit< 1 menitMulti-AZ spread + failover replica.
Kerusakan database< 30 menit< 5 menitPITR dari WAL + promosi replica.
Kegagalan region< 4 jam< 15 menitBackup lintas region + restore infra via Terraform.
Data terhapus / korup< 2 jam< 1 jamRestore snapshot bertanggal + object lock.
Insiden keamananIsolasi < 15 menit0Rotasi kredensial, cabut token, forensik log.

Cost Optimization

Aktifkan tuas penghematan untuk estimasi biaya operasional bulanan

Estimasi biaya bulanan

$27,720

Baseline $42.000 · hemat $14,280 (34%)

Termasuk compute GPU render, database HA, storage, CDN, dan panggilan AI provider.

Multi-Tenant

Isolasi data terjamin di setiap tier

Single User

Isolasi: Row-level security per userId

Kuota: 50 video/bulan

Ekstra: Shared worker pool

Team Workspace

Isolasi: RLS per workspaceId + audit log

Kuota: 1.000 video/bulan

Ekstra: Prioritas antrean medium

Enterprise Workspace

Isolasi: Skema/DB terpisah + bucket khusus + KMS key sendiri

Kuota: Kustom

Ekstra: Node pool dedicated, SSO, SLA 99.95%

Compliance

Siap audit GDPR, SOC 2, dan ISO 27001

GDPR

Consent managementRight to delete (kaskade + purge storage)Data export terstrukturDPA & data residency

SOC 2

Audit log immutableChange management via CI/CDAccess review kuartalanIncident response terdokumentasi

ISO 27001

Risk registerKebijakan retensi dataVendor assessmentBusiness continuity + uji DR

Production Readiness Checklist

Wajib hijau sebelum rilis ke produksi

  • Semua layanan memiliki liveness & readiness probe.
  • Semua endpoint terdokumentasi (OpenAPI + contract test).
  • Semua secret terenkripsi di Secret Manager, nol secret di kode.
  • Monitoring aktif dengan dashboard & SLO burn-rate.
  • Logging terstruktur aktif dengan trace correlation.
  • Backup otomatis berjalan dan uji restore lulus.
  • CI/CD otomatis dengan gate lint, test, dan security scan.
  • Rollback otomatis (helm rollback / Argo Rollouts) teruji.
  • Autoscaling aktif untuk CPU, memory, queue, dan render job.
  • Disaster recovery telah diuji sesuai RTO/RPO.

Panduan Deployment

Staging → approval → canary → bake time

  1. 1. Persiapan

    Pastikan CI hijau, changelog siap, dan feature flag dinonaktifkan untuk fitur baru.

  2. 2. Staging

    helm upgrade --install ke namespace staging dengan --atomic, jalankan smoke test.

  3. 3. Validasi

    Cek metrik staging 15 menit: latency, error rate, render success.

  4. 4. Approval

    Manual approval di GitHub Environment production oleh reviewer on-call.

  5. 5. Canary

    Argo Rollouts 10% selama 10 menit → 50% → 100% dengan analisis metrik otomatis.

  6. 6. Health gate

    Probe semua service, sintetis end-to-end satu video pendek.

  7. 7. Bake & monitor

    Pantau 30 menit; jika burn-rate melewati ambang, rollback otomatis.

  8. 8. Penutup

    Aktifkan feature flag bertahap, tulis catatan rilis, tutup change ticket.

Checklist Harian DevOps

Rutinitas operasional tim on-call

  1. 09:00Review dashboard SLO, error budget, dan alert semalam.
  2. 09:30Cek queue depth, DLQ, dan job gagal — replay bila perlu.
  3. 10:00Periksa replica lag, slow query, dan kapasitas storage.
  4. 11:00Tinjau biaya AI & infra harian vs anggaran.
  5. 14:00Verifikasi backup terakhir sukses + sampling restore.
  6. 15:00Review hasil dependency & container scan terbaru.
  7. 17:00Catat perubahan produksi ke changelog dan audit trail.
  8. MingguanGame day / uji failover, review kapasitas dan on-call handover.