Infrastruktur produksi Shorts Studio AI
Platform cloud native berbasis microservices di Kubernetes: highly available, secure by default, observable penuh, dan hemat biaya untuk ribuan pengguna aktif serta jutaan video yang diproses setiap bulan.
Uptime target
99,95%
Zona ketersediaan
3 AZ
Autoscale maks
150 pod worker
RTO region
< 4 jam
Arsitektur Cloud
Alur permintaan dari frontend hingga observability
Container & Service Spec
Resource limit, probe, dan standar container stateless
Media Processing Service
WorkerScratch di emptyDir, hasil langsung ke object storage, spot friendly.
- Runtime
- FFmpeg + NVENC container
- Replika
- 2 – 120
- CPU
- 2 / 8
- Memory
- 4Gi / 16Gi
- GPU
- 1 × NVIDIA L4 (nvidia.com/gpu)
- Port
- 8084
- Liveness
- /health
- Readiness
- /ready
- Stateless
- Ya
- Scale on
- Render jobs, GPU utilization, Queue wait time
# syntax=docker/dockerfile:1.7 # ---- build ---- FROM node:20-alpine AS build WORKDIR /app COPY package.json bun.lockb ./ RUN npm ci --omit=dev=false COPY . . RUN npm run build:media # ---- runtime ---- FROM nvidia/cuda:12.4.1-runtime-ubuntu22.04 AS runtime RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg nodejs npm && rm -rf /var/lib/apt/lists/* WORKDIR /app ENV NODE_ENV=production PORT=8084 COPY --from=build /app/dist ./dist COPY --from=build /app/node_modules ./node_modules USER 10001:10001 EXPOSE 8084 HEALTHCHECK --interval=15s --timeout=3s --retries=3 \ CMD wget -qO- http://127.0.0.1:8084/health || exit 1 CMD ["node", "dist/media/server.js"]
- Multi-stage build, base image distroless / alpine yang di-pin dengan digest.
- Jalan sebagai non-root user (uid 10001), filesystem root read-only.
- HEALTHCHECK, graceful shutdown SIGTERM, dan timeout drain 30 detik.
- Tidak ada secret di layer image — semua lewat env dari Secret Manager.
- Image di-scan (Trivy) dan ditandatangani (cosign) sebelum push registry.
- Semua container stateless: data sementara di emptyDir, hasil ke object storage.
Struktur Kubernetes
Objek inti yang digunakan di setiap cluster
Deployment
Semua service stateless (frontend, API, orchestrator, worker).
StatefulSet
PostgreSQL, Redis Cluster, Loki, Prometheus.
Service
ClusterIP internal + headless untuk StatefulSet.
Ingress
NGINX/Envoy Ingress + cert-manager TLS 1.3, WAF di depan.
ConfigMap
Konfigurasi non-sensitif per environment.
Secret
Disinkronkan dari Secret Manager via External Secrets Operator.
PersistentVolume
PVC gp3/SSD untuk database, Prometheus TSDB, Loki chunk.
HorizontalPodAutoscaler
CPU, memory, queue depth, render job, latency.
NetworkPolicy
Default deny, izin eksplisit antar tier.
PodDisruptionBudget
minAvailable 2 agar upgrade node tanpa downtime.
Strategi Autoscaling
Skala otomatis tanpa downtime (maxUnavailable 0 + PDB)
| Sinyal | Target | Berlaku untuk | Sumber metrik |
|---|---|---|---|
| CPU utilization | 70% | Semua service | metrics-server |
| Memory utilization | 75% | API, Orchestrator | metrics-server |
| Queue length (BullMQ) | 30 job/pod | Worker, Orchestrator | Prometheus adapter |
| Render jobs aktif | 2 job/GPU pod | Media Processing | Custom metric |
| Pengguna aktif (SSE session) | 500/pod | Gateway, Workflow | Custom metric |
| Latency p95 | < 400 ms | Gateway, API | OpenTelemetry |
CI/CD Pipeline
GitHub Actions — gagal di satu tahap = pipeline berhenti + notifikasi
- 1
Lint
ESLint + Prettier + tsgo typecheck
Gagal → pipeline berhenti
- 2
Unit Test
Vitest, coverage minimal 80%
Coverage turun → blok
- 3
Integration Test
Testcontainers Postgres + Redis
Gagal → blok
- 4
Security Scan
npm audit, Trivy FS, Semgrep, gitleaks
High/Critical → blok
- 5
Build Docker Image
Buildx multi-arch + SBOM (syft)
Build error → blok
- 6
Push Registry
Scan image + cosign sign
Vuln critical → blok
- 7
Deploy Staging
Helm upgrade --atomic namespace staging
Rollback otomatis
- 8
Smoke Test
Playwright happy-path + API contract
Gagal → rollback
- 9
Deploy Production
Canary 10% → 50% → 100% (Argo Rollouts)
Manual approval
- 10
Health Check
Probe semua /health + query sintetis
Gagal → rollback otomatis
- 11
Monitoring
Bake time 30 menit, SLO burn-rate watch
Alert → rollback
name: production-pipeline
on:
push: { branches: [main] }
concurrency:
group: prod-${{ github.ref }}
cancel-in-progress: false
jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: oven-sh/setup-bun@v2
- run: bun install --frozen-lockfile
- run: bun run lint
- run: bun run typecheck
- run: bun run test:unit -- --coverage
- run: bun run test:integration
security:
needs: quality
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: gitleaks/gitleaks-action@v2
- uses: aquasecurity/trivy-action@master
with: { scan-type: fs, severity: HIGH,CRITICAL, exit-code: "1" }
build:
needs: security
runs-on: ubuntu-latest
permissions: { id-token: write, packages: write }
strategy:
matrix:
service: [frontend, api-gateway, auth, workflow, ai-orchestrator, media, notification, analytics]
steps:
- uses: actions/checkout@v4
- uses: docker/build-push-action@v6
with:
push: true
tags: registry.shortsstudio.ai/${{ matrix.service }}:${{ github.sha }}
provenance: true
sbom: true
- run: cosign sign --yes registry.shortsstudio.ai/${{ matrix.service }}:${{ github.sha }}
staging:
needs: build
environment: staging
runs-on: ubuntu-latest
steps:
- run: helm upgrade --install shorts ./charts --namespace shorts-staging --atomic --timeout 10m --set image.tag=${{ github.sha }}
- run: bun run test:smoke -- --base-url https://staging.shortsstudio.ai
production:
needs: staging
environment: production # manual approval
runs-on: ubuntu-latest
steps:
- run: helm upgrade --install shorts ./charts --namespace shorts-prod --atomic --timeout 15m --set image.tag=${{ github.sha }}
- run: bun run scripts/health-gate.ts
- if: failure()
run: helm rollback shorts --namespace shorts-prod
notify:
if: always()
needs: [quality, security, build, staging, production]
runs-on: ubuntu-latest
steps:
- run: bun run scripts/notify.ts --status ${{ job.status }}Environment Terpisah
Config, secret, storage, database, monitoring, dan logging sendiri-sendiri
Domain
app.shortsstudio.ai
Cluster
k8s-prod multi-AZ (3 zona)
Database
pg-prod primary + 2 replica + PITR
Storage
bucket prod-* + replikasi lintas region
Replika
3–150 (HPA)
Deploy
Manual approval + canary 10% → 50% → 100%
Secret Management
Nol secret di dalam kode — semua dari Secret Manager / KMS
| Secret | Penyimpanan | Rotasi | Konsumen |
|---|---|---|---|
| AI_PROVIDER_KEYS | Secret Manager | 90 hari | AI Orchestrator |
| JWT_SIGNING_KEY | Secret Manager (KMS) | 30 hari | Auth |
| OAUTH_CLIENT_SECRET | Secret Manager | 180 hari | Auth |
| DATABASE_URL | Secret Manager + IRSA | 90 hari | API, Workflow |
| STORAGE_CREDENTIALS | Workload Identity | Otomatis | Media, API |
| ENCRYPTION_MASTER_KEY | KMS (HSM-backed) | 365 hari | Data at rest |
| REDIS_AUTH | Secret Manager | 90 hari | Queue, Cache |
| WEBHOOK_SIGNING_SECRET | Secret Manager | 180 hari | Notification |
Object Storage & Lifecycle
Bucket terpisah per jenis aset
prod-images
CDNIA 30 hari → Glacier 180 hari
Retensi: 365 hari
prod-videos
CDNIA 60 hari → Glacier 365 hari
Retensi: 3 tahun
prod-audio
CDNIA 30 hari
Retensi: 365 hari
prod-subtitles
CDNStandard
Retensi: 365 hari
prod-thumbnails
CDNStandard + cache 1 tahun
Retensi: 365 hari
prod-temp
PrivateHapus otomatis 24 jam
Retensi: 1 hari
prod-backup
PrivateImmutable object lock
Retensi: 365 hari
prod-logs
PrivateCompact 30 hari → arsip
Retensi: 90 hari
CDN & Cache Policy
Kompresi Brotli, HTTP/3, dan cache immutable
Static JS/CSS (hashed)
public, max-age=31536000, immutable
Brotli + HTTP/3
Fonts
public, max-age=31536000, immutable
Preload + subset
Thumbnail & Image
public, max-age=604800, stale-while-revalidate=86400
AVIF/WebP auto
Video preview (HLS)
public, max-age=86400
Range request + edge segment cache
API JSON
no-store
Hanya lewat gateway, tidak di-cache CDN
Database High Availability
Primary → replica → backup → disaster recovery
Primary
Postgres 16, multi-AZ, PgBouncer connection pooling (transaction mode).
Read Replica ×2
Query analitik & dashboard diarahkan ke replica, lag dipantau < 2s.
Automatic Failover
Patroni + leader election, promosi replica < 30 detik.
Backup & PITR
WAL archiving kontinu ke object storage, restore ke titik waktu mana pun.
Disaster Recovery
Snapshot lintas region harian + Terraform untuk rebuild infra.
Queue Infrastructure
BullMQ di Redis Cluster dengan prioritas bertingkat
| Antrean | Prioritas | Concurrency | Retry |
|---|---|---|---|
| workflow | High | 40 | 5× exponential 2s→60s |
| rendering | High | 12 | 3× + fallback CPU |
| thumbnail | Medium | 30 | 3× exponential |
| seo | Medium | 25 | 3× exponential |
| export | Medium | 15 | 3× + resume upload |
| notification | Low | 50 | 5× linear |
| retry-dlq | Background | 5 | Manual replay |
Backup Strategy
Retensi 7 / 30 / 90 / 365 hari dengan uji pemulihan berkala
PostgreSQL (full + WAL)
Full harian 02:00 UTC, WAL kontinu
Retensi: 7 / 30 / 90 / 365 hari
Uji restore: Mingguan (otomatis ke sandbox)
Object Storage metadata
Harian
Retensi: 30 / 90 hari
Uji restore: Bulanan
Workflow state & Project Memory
Tiap 6 jam
Retensi: 30 hari
Uji restore: Bulanan
Configuration (Helm, Terraform)
Setiap commit (GitOps)
Retensi: 365 hari
Uji restore: Tiap rilis
Secrets (KMS envelope)
Harian terenkripsi
Retensi: 90 hari
Uji restore: Kuartalan
Monitoring
Prometheus + Grafana, dashboard real-time
Compute
Traffic
Pipeline
Data
AI
Logging & Tracing
Loki + Winston, korelasi penuh via OpenTelemetry
Field wajib setiap log
- timestamp (ISO 8601, UTC)
- requestId (ULID per request)
- traceId / spanId (OpenTelemetry)
- projectId
- userId + tenantId
- service + version
- severity (debug|info|warn|error|fatal)
- event (api.request, workflow.stage, render.job, security.audit)
- durationMs + statusCode
Span OpenTelemetry
http.request
Gateway → service, atribut tenant & route.
auth.verify
Validasi JWT, RBAC, MFA challenge.
workflow.stage
Satu span per tahap dari 20 tahap pipeline.
ai.request
Provider, model, token in/out, biaya, retry.
render.job
Timeline build, encode, GPU backend, durasi.
db.query
Statement hash, rows, waktu, replica/primary.
queue.publish / queue.consume
Antrean, prioritas, waktu tunggu.
storage.upload
Bucket, ukuran objek, waktu unggah.
Alerting
Aturan alert dan kanal notifikasi tim
API Down
Criticalprobe_success == 0 selama 2 menit
Kanal: Paging + email
Queue Macet
Criticalqueue_depth > 500 dan proses < 1/menit
Kanal: Paging
Database Lambat
Highp95 query > 1s selama 5 menit
Kanal: Chat tim
GPU Penuh
HighGPU util > 95% selama 10 menit
Kanal: Chat tim
Storage Hampir Penuh
MediumKapasitas terpakai > 85%
Kanal: Email
Render Gagal
HighFailure rate > 5% dalam 15 menit
Kanal: Paging
Lonjakan Error
Critical5xx rate > 2% (burn-rate SLO 14×)
Kanal: Paging
Biaya AI Melebihi Ambang
HighBiaya harian > 120% anggaran
Kanal: Email + chat
Security Produksi
Secure by default di setiap lapisan
Transport
- • HTTPS only + HSTS preload
- • TLS 1.3
- • mTLS antar service (mesh)
Perimeter
- • WAF (OWASP CRS)
- • DDoS protection
- • Firewall / private subnet
- • Rate limiting per tenant & IP
Identity
- • RBAC granular
- • MFA wajib untuk admin
- • Least privilege IAM
- • Session rotation + revocation
Data
- • Encryption at rest (KMS)
- • Encryption in transit
- • Field-level encryption PII
- • Signed URL berumur pendek
Aplikasi
- • Input validation (Zod)
- • Output sanitization
- • CSP + security headers
- • Audit trail immutable
Supply chain
- • Dependency scanning
- • Container scanning
- • Image signing (cosign)
- • SBOM tiap rilis
Disaster Recovery
Skenario, RTO/RPO, dan langkah pemulihan
| Skenario | RTO | RPO | Tindakan |
|---|---|---|---|
| Kegagalan pod / node | < 1 menit | 0 | Reschedule otomatis, PDB menjaga kapasitas. |
| Kegagalan zona (AZ) | < 5 menit | < 1 menit | Multi-AZ spread + failover replica. |
| Kerusakan database | < 30 menit | < 5 menit | PITR dari WAL + promosi replica. |
| Kegagalan region | < 4 jam | < 15 menit | Backup lintas region + restore infra via Terraform. |
| Data terhapus / korup | < 2 jam | < 1 jam | Restore snapshot bertanggal + object lock. |
| Insiden keamanan | Isolasi < 15 menit | 0 | Rotasi kredensial, cabut token, forensik log. |
Cost Optimization
Aktifkan tuas penghematan untuk estimasi biaya operasional bulanan
Estimasi biaya bulanan
$27,720
Baseline $42.000 · hemat $14,280 (34%)
Termasuk compute GPU render, database HA, storage, CDN, dan panggilan AI provider.
Multi-Tenant
Isolasi data terjamin di setiap tier
Single User
Isolasi: Row-level security per userId
Kuota: 50 video/bulan
Ekstra: Shared worker pool
Team Workspace
Isolasi: RLS per workspaceId + audit log
Kuota: 1.000 video/bulan
Ekstra: Prioritas antrean medium
Enterprise Workspace
Isolasi: Skema/DB terpisah + bucket khusus + KMS key sendiri
Kuota: Kustom
Ekstra: Node pool dedicated, SSO, SLA 99.95%
Compliance
Siap audit GDPR, SOC 2, dan ISO 27001
GDPR
SOC 2
ISO 27001
Production Readiness Checklist
Wajib hijau sebelum rilis ke produksi
- ✓Semua layanan memiliki liveness & readiness probe.
- ✓Semua endpoint terdokumentasi (OpenAPI + contract test).
- ✓Semua secret terenkripsi di Secret Manager, nol secret di kode.
- ✓Monitoring aktif dengan dashboard & SLO burn-rate.
- ✓Logging terstruktur aktif dengan trace correlation.
- ✓Backup otomatis berjalan dan uji restore lulus.
- ✓CI/CD otomatis dengan gate lint, test, dan security scan.
- ✓Rollback otomatis (helm rollback / Argo Rollouts) teruji.
- ✓Autoscaling aktif untuk CPU, memory, queue, dan render job.
- ✓Disaster recovery telah diuji sesuai RTO/RPO.
Panduan Deployment
Staging → approval → canary → bake time
1. Persiapan
Pastikan CI hijau, changelog siap, dan feature flag dinonaktifkan untuk fitur baru.
2. Staging
helm upgrade --install ke namespace staging dengan --atomic, jalankan smoke test.
3. Validasi
Cek metrik staging 15 menit: latency, error rate, render success.
4. Approval
Manual approval di GitHub Environment production oleh reviewer on-call.
5. Canary
Argo Rollouts 10% selama 10 menit → 50% → 100% dengan analisis metrik otomatis.
6. Health gate
Probe semua service, sintetis end-to-end satu video pendek.
7. Bake & monitor
Pantau 30 menit; jika burn-rate melewati ambang, rollback otomatis.
8. Penutup
Aktifkan feature flag bertahap, tulis catatan rilis, tutup change ticket.
Checklist Harian DevOps
Rutinitas operasional tim on-call
- 09:00Review dashboard SLO, error budget, dan alert semalam.
- 09:30Cek queue depth, DLQ, dan job gagal — replay bila perlu.
- 10:00Periksa replica lag, slow query, dan kapasitas storage.
- 11:00Tinjau biaya AI & infra harian vs anggaran.
- 14:00Verifikasi backup terakhir sukses + sampling restore.
- 15:00Review hasil dependency & container scan terbaru.
- 17:00Catat perubahan produksi ke changelog dan audit trail.
- MingguanGame day / uji failover, review kapasitas dan on-call handover.