morpheus-engine-pool

Shared heavy-engine worker fleet for the Morpheus forge substrate. Forges submit jobs via HTTP + BullMQ; this pool runs the engines.

• ARCHITECTURE.md — protocol reference, per-engine details, mode matrix, ops runbook
• MIGRATING.md — walkthrough for moving an existing service's local ffmpeg/IM/libvips/whisper paths to the pool
• clients/typescript/ — typed TS SDK

Phase status:

• Phase 1 ✓ — ffmpeg, image (libvips + ImageMagick), whisper-proxy
• Phase 2 ✓ — file_url inputs (≤500 MB streamed)
• Phase 2.1 ✓ — output_url + output_urls[] outputs
• Phase 2.2 ✓ — aux_files for filter-graph siblings; streaming PUT
• Phase 2.3 ✓ — mode: "probe" for stderr-emitting filter graphs

Why

Every forge was bundling ffmpeg, ImageMagick, Whisper into its own Docker image — ~3 GB of duplicated layers. Engine pool = one fleet, pre-warmed, one place to tune GPU/codec/memory.

Architecture doc: ~/Github/rspace-online-dev/docs/ontology/MORPHEUS-FORGES-SDK.md §"Engine pool sharing".

Local development

cd ~/Github/morpheus-engine-pool

# Install deps
bun install

# Start Redis (Docker)
docker run -d -p 6379:6379 --name ep-redis redis:7-alpine

# Start server (terminal 1)
REDIS_URL=redis://localhost:6379 bun run src/server.ts

# Start worker (terminal 2)
REDIS_URL=redis://localhost:6379 bun run src/worker.ts

Run unit tests

bun test tests/test_smoke.ts

Integration tests are skipped unless services are running AND INTEGRATION=1:

INTEGRATION=1 bun test tests/test_smoke.ts

Manual smoke test (full round-trip)

Requires: server + Redis + ffmpeg + worker running.

# 1. Health check
curl -s http://localhost:8000/health | jq

# 2. List engines
curl -s http://localhost:8000/engines | jq .engines.ffmpeg

# 3. Submit ffmpeg job (MP4 → GIF, first 3 seconds)
curl -s -X POST http://localhost:8000/jobs/ffmpeg \
  -F "file=@/path/to/input.mp4" \
  -F 'args={"to":"gif","start":0,"duration":3}' | jq

# 4. Poll (replace JOB_ID)
curl -s http://localhost:8000/jobs/JOB_ID | jq .status

# 5. Retrieve result bytes when status=success
curl -s http://localhost:8000/jobs/JOB_ID | jq -r .result_bytes_base64 | base64 -d > out.gif

Docker Compose (local, no Netcup)

# Build image
docker build -t morpheus-engine-pool:latest .

# Bring up stack (2 ffmpeg workers)
docker compose up --scale engine-pool-worker-ffmpeg=2

# Verify
curl http://localhost:8000/health

Note: docker-compose.yml references external network traefik-public. For local testing without Traefik, add a temporary override:

docker compose -f docker-compose.yml -f docker-compose.local.yml up

docker-compose.local.yml:

services:
  engine-pool-server:
    ports:
      - "8000:8000"
    networks:
      - engine-pool-internal
networks:
  traefik-public:
    driver: bridge
    name: traefik-public-local

API reference

POST /jobs/:engine

Submit a job. Multipart form — provide EITHER inline file OR remote file_url (one of the two is required):

Field	Type	Required	Description
file	file	one of these	Input bytes (≤10 MB total). Repeats allowed for ffmpeg concat mode.
file_url	string	one of these	Public-or-internal URL the worker fetches. Bypasses the 10 MB inline limit; capped at MAX_FETCH_BYTES (default 500 MB). Repeats allowed for concat.
file_url_headers	string	no	JSON object of headers applied to all file_url fetches in this request (e.g. {"CF-Access-Client-Id":"…","CF-Access-Client-Secret":"…"}).
args	string	yes	JSON string with engine-specific options.
output_url	string	no	Phase 2.1 — caller-supplied pre-signed PUT URL. Worker uploads result there and the job response carries output_url instead of inline base64. Use for outputs >10 MB or when the caller already has presigned R2/S3 URLs.
output_urls	string (repeated)	no	Decompose mode only — one PUT URL per cut, length must equal args.cuts.length.
output_url_headers	string	no	JSON object headers applied to every PUT (signed-URL conventions).

POST /jobs/* is gated by Authorization: Bearer <ENGINE_POOL_AUTH_TOKEN> when the env var is set on the server. /health, /engines, /stats stay open so monitors and the calibration script work without credentials.

ffmpeg args:

{
  "to": "gif",
  "start": 0,
  "duration": 3,
  "extra": []
}

Returns 202 Accepted:

{
  "jobId": "...",
  "status": "queued",
  "queuedAt": "2026-04-30T...",
  "pollUrl": "/jobs/..."
}

GET /jobs/:jobId

Poll status. States: queued | running | success | failed.

On success:

{
  "jobId": "...",
  "status": "success",
  "engine": "ffmpeg",
  "elapsed_ms": 1234,
  "result_bytes_base64": "<base64>",
  "completedAt": "..."
}

On failed:

{
  "jobId": "...",
  "status": "failed",
  "error": "ffmpeg exited 1\n...",
  "failedAt": "..."
}

Results expire after 1 hour (success) or 24 hours (failed).

GET /engines

Lists registered engines, their accepted/output formats, and args shapes. Phase 1 engines (imagemagick, whisper, libvips) are listed but return 503 when submitted to — no workers yet.

GET /health

Redis reachability + per-engine active status. Returns 200 if Redis is reachable, 503 otherwise.

How forges integrate (Phase 0)

Instead of bundling ffmpeg in clip-forge or a future media-forge:

// Before (forge bundles ffmpeg):
const result = await ffmpegConvert(inputBytes, { to: 'gif', duration: 3 });

// After (forge submits job to engine pool):
const form = new FormData();
form.append('file', new Blob([inputBytes]), 'input.mp4');
form.append('args', JSON.stringify({ to: 'gif', duration: 3 }));

const { jobId } = await fetch('http://engine-pool-server:8000/jobs/ffmpeg', {
  method: 'POST', body: form,
}).then(r => r.json());

// Poll (or wire up a webhook/SSE — Phase 1)
const result = await pollUntilDone(jobId);
const gifBytes = Buffer.from(result.result_bytes_base64, 'base64');

The engine pool URL would be an env var (ENGINE_POOL_URL) in each forge's .env, fetched at startup via Infisical (or plain .env for v1 since there are no API keys).

Phase 1 checklist

• Wire ImageMagick worker (src/worker-imagemagick.ts, same pattern)
• Wire Whisper worker (src/worker-whisper.ts)
• Wire libvips worker (src/worker-libvips.ts)
• Phase 2: file_url inputs bypass 10 MB inline ceiling (capped at MAX_FETCH_BYTES, default 500 MB)
• Phase 2.1: output_url (single) + output_urls[] (decompose) — worker PUTs result to caller-supplied pre-signed URL
• Phase 2.2: large-output streaming (chunked PUT for results that don't fit in the worker's RAM)
• Webhook / SSE callbacks so forges don't need to poll
• Uptime Kuma push monitor for queue depth
• Add enginepool.jeffemmett.com to Cloudflare tunnel for cross-service access
• Update clip-forge to submit ffmpeg jobs here instead of local subprocess

Deploy to Netcup (when ready)

See pre-deploy-checklist.md (generated by infra-manager) or follow standard forge pattern:

ssh netcup
cd /opt/apps/morpheus-engine-pool
git pull
docker compose build
docker compose up -d
docker compose ps

Traefik will auto-discover enginepool.jeffemmett.com once the container is on the traefik-public network. Add the hostname to /root/cloudflared/config.yml if external access is needed.