gemstone-nexus · Phase 6
GemStone-Nexus — AWS Deploy Runbook
Metadata
| id | doc_id | title | category | source_path | summary | content_hash | indexed_at |
|---|---|---|---|---|---|---|---|
| 44 | AWS-RUNBOOK | GemStone-Nexus — AWS Deploy Runbook | reference | DevResources/docs/aws-runbook.md | > **Status (Stage 2):** Ohio T4 host is live (`i-04333dae160ab0daa`, us-east-2). > Day-to-day Deploy is documented in `DOC-Deploy-Pipeline.md` and driven by > `scripts/run-remote-docker-deploy.ps1` / `scripts/aws/deploy.sh`. > The us-east-1 “Phase 1 baseline” section below is historical plan context; > do not re-run those skeletons against Ohio without review. | 87f63904a35989babd95d47ed0ada56b3f2e58c9 | 2026-07-22 18:33:18 |
Content
# GemStone-Nexus — AWS Deploy Runbook
> **Status (Stage 2):** Ohio T4 host is live (`i-04333dae160ab0daa`, us-east-2).
> Day-to-day Deploy is documented in `DOC-Deploy-Pipeline.md` and driven by
> `scripts/run-remote-docker-deploy.ps1` / `scripts/aws/deploy.sh`.
> The us-east-1 “Phase 1 baseline” section below is historical plan context;
> do not re-run those skeletons against Ohio without review.
Source of truth for the Ohio stack: `DevResources/plans/ohio-terraform-plan.md`
and `infra/terraform/ohio/`.
## Target architecture
```
GoDaddy DNS (gemstone.dataright.net)
│ A record -> Elastic IP (Ohio live: 18.116.188.14)
▼
[ g4dn.xlarge / T4 | us-east-2 live host ]
nginx (TLS, Let's Encrypt HTTP-01) -> api (FastAPI)
├─> ollama (GPU, T4 16GB)
├─> whisper (STT)
├─> piper (TTS)
└─> ledger.db (standalone SQLite)
nvme0n1 30G = OS only
nvme1n1 128G = /mnt/docker-data (docker data-root + Ollama models + ledger + staging)
│
▼ S3 gateway endpoint (vpce-0661ccc0d47e88054) — ledger backups + staged files
```
The stack is self-contained in `DataRight-vpc`. There is **no runtime dependency**
on an unrelated on-prem host (`10.0.0.228`).
## Fixed references (from the plan)
| Item | Value |
|------|-------|
| Region | `us-east-1` |
| VPC | `DataRight-vpc` = `vpc-053baa45dd75c045a` (`10.0.0.0/16`) |
| Public subnet (primary) | `us-east-1a` = `subnet-019ffac204b77426b` (`10.0.0.0/20`) |
| Public subnet (fallback) | `us-east-1b` = `subnet-0a6e88b619d84a175` |
| IGW | `igw-0a828410ab4af913c` |
| S3 gateway endpoint | `vpce-0661ccc0d47e88054` |
| Golden launch template | `DataRightGoldenAMITemplate` = `lt-0e1491fcfe8f99a13` |
| Golden AMI base to bake from | `ami-0f8a61b66d1accaee` |
| New GPU launch template | `DataRightGemStoneGPUSpot` (derived; golden stays pristine) |
| Instance type | `g4dn.xlarge` (NVIDIA T4 16GB) — spot |
| Persistent data EBS | 128G `/dev/sdb`, `DeleteOnTermination=false` |
| Hostname | `gemstone.dataright.net` (Let's Encrypt, GoDaddy DNS) |
| Deploy path | `/home/dr/AppDev/gemstone-nexus` (`DEVELOPMENT_DIRECTORY`) |
| Primary region (live) | `us-east-2` (Ohio) — instance `i-04333dae160ab0daa` |
## Phase 1 — AWS baseline (do this first; not done yet)
1. **Bake `golden-v4-gpu`** from `ami-0f8a61b66d1accaee`: run the v4 Ansible
playbook, then add the NVIDIA driver + `nvidia-container-toolkit`. Preserve the
two-disk layout and baked images. Register the new AMI.
2. **New security group** in `DataRight-vpc`: inbound `443` (world), `80` (world —
Let's Encrypt HTTP-01 + redirect), `22` (your IP only); outbound all. The
golden template's `DataRightDev` SG lives in the default VPC and **cannot** be
reused (SGs are VPC-scoped).
3. **Create launch template `DataRightGemStoneGPUSpot`** derived from the golden
template, setting: GPU AMI, `InstanceType=g4dn.xlarge`, spot market options,
subnet + `AssociatePublicIpAddress=true`, key pair, IAM instance profile
(S3 + Secrets Manager), and the 128G `/dev/sdb` with `DeleteOnTermination=false`.
4. **Launch** the spot instance (`scripts/aws/bring-up.sh`).
5. **Mount** the persistent 128G volume at `/mnt/docker-data`
(`scripts/aws/mount-ebs.sh`). Format only on first provision.
6. **Associate the Elastic IP** and set the GoDaddy A record
(`scripts/aws/associate-eip.sh`).
7. **Verify** GPU is visible to Docker (`nvidia-smi` inside a container) and that
a bare `/health` responds over HTTPS.
## Phase 2/3 — Deploy the stack (this repo)
Run on the host after Phase 1 (`scripts/aws/deploy.sh`):
1. Clone/sync this repo into `/home/dr/AppDev/gemstone-nexus`.
2. Create `.env` from `.env.example`. On AWS set `APP_SECRETS_ARN` +
`SKIP_AWS_SECRETS=0` so the API loads Gemini keys from Secrets Manager via its
IAM instance profile. Set `TLS_SAN_DNS=gemstone.dataright.net`.
3. Provision Let's Encrypt: `sudo TLS_EMAIL=... ./scripts/aws/provision-letsencrypt.sh --recreate-nginx`
(HTTP-01 webroot; recreates **only** the nginx container). Without LE files,
nginx falls back to a self-signed cert in `/etc/gemstone/certs`.
4. `docker compose up -d --build`. The one-shot `ollama-init` service pulls
`OLLAMA_CHAT_MODEL` onto the GPU on first boot.
5. Verify: `curl -I https://gemstone.dataright.net/healthz` (no `-k`) and
`python scripts/verify_gemstone_dev.py https://gemstone.dataright.net`.
## Spot + persistence
- The 128G EBS is **persistent** and re-attached on relaunch, so Docker
data-root + Ollama models + `ledger.db` survive spot interruption (no multi-GB
re-pull).
- **Ledger durability:** snapshot `ledger.db` into `/mnt/docker-data/staging`
nightly and sync to S3 via the gateway endpoint. `scripts/aws/snapshot.sh`
captures a full EBS snapshot for point-in-time backups.
- **Elastic IP** re-associated on relaunch; stable hostname means the
Let's Encrypt cert survives relaunches.
- A 2-minute spot interruption notice handler should flush/upload the ledger
before reclaim (to be wired in Phase 4).
## Cost (us-east-1 estimate)
- Dev-hours (~176/mo, auto-stop): ~$35 compute + ~$13 EBS + ~$3.65 EIP ≈ **$52/mo**
- Always-on (730/mo): ≈ **$162/mo**
## Open items (not blocking scaffolding)
1. AWS credentials/profile for create actions.
2. GoDaddy DNS: manual A record vs. API automation.
## Temporary on-demand fallback (2026-07-22)
Ohio Spot `i-04333dae160ab0daa` / `sir-1s4zg8in` hit `capacity-not-available`. Per ops directive, traffic was moved to a **temporary on-demand** host while Spot is watched for cutover.
| Role | ID | Notes |
|------|----|-------|
| **Serving now (on-demand)** | `i-0f5f3b32dfa9c6a12` | `g4dn.xlarge` us-east-2; EIP `18.116.188.14` / `eipalloc-00556d8ca50728abd`; tag `DataRight-GemStone-gpu-TEMP-OD` |
| Original Spot (terminated) | `i-04333dae160ab0daa` | Cancelled `sir-1s4zg8in`; **do not** expect StartInstances on this ID |
| Persistent data EBS | `vol-0a1fff82ee46e9095` | `/dev/sdb` ? `/mnt/docker-data` (Docker root, models, ledger) |
| Current root EBS | `vol-0591c165d1467bd01` | Recreated from `ami-011a95758542dcaac` (prior root was lost on Spot terminate before DeleteOnTermination=false applied) |
Both volumes on the OD instance have `DeleteOnTermination=false`.
### Cutover back to Spot (manual / when capacity returns)
1. Confirm Spot capacity (`get-spot-placement-scores` score > 1, or a probe `request-spot-instances` fulfills).
2. `docker compose ps` healthy on OD; optionally snapshot root + data.
3. Stop OD `i-0f5f3b32dfa9c6a12`; detach root + data volumes (`DeleteOnTermination=false`).
4. Launch Spot (`g4dn.xlarge`, same subnet/SG/AMI or LT `lt-05dd1ba1dec324866`) in **us-east-2a**, stop it, replace root with `vol-0591c165d1467bd01`, attach data volume, start.
5. Associate EIP `eipalloc-00556d8ca50728abd` to the Spot instance; verify `curl -I https://gemstone.dataright.net/healthz` and `/` without `-k` (GET may be required if HEAD returns 405).
6. Terminate the temporary OD only after Spot is healthy and EIP moved.
SSH on golden/OD recovery image: `ubuntu` (+ key). App user `dr` home exists; deploy path `/home/dr/AppDev/gemstone-nexus`.