gemstone-nexus · Phase 6
GemStone-Nexus — Ohio (us-east-2) Terraform Bring-up Plan
Metadata
| id | doc_id | title | category | source_path | summary | content_hash | indexed_at |
|---|---|---|---|---|---|---|---|
| 46 | OHIO-TERRAFORM-PLAN | GemStone-Nexus — Ohio (us-east-2) Terraform Bring-up Plan | plan | DevResources/plans/ohio-terraform-plan.md | Recreate the GemStone dev stack in **us-east-2 (Ohio)** — the home datacenter — as | dd78f67315683c5d1234a370591c5b643a32fecb | 2026-07-22 18:33:18 |
Content
# GemStone-Nexus — Ohio (us-east-2) Terraform Bring-up Plan
Recreate the GemStone dev stack in **us-east-2 (Ohio)** — the home datacenter — as
**Terraform-managed** infrastructure: a mirrored DataRight network, a duplicated GPU
GoldenAMI, fresh Elastic IP(s), and a spot-first launch path, so we can resume work with
spot cost savings.
> Guardrail: per the cloud-infra approval rule, Terraform is authored here but **nothing is
> applied** without explicit per-step approval. The design deliberately separates "stage
> infra" (cheap, no GPU) from "launch GPU instance" (billable) so `apply` never starts a
> GPU box on its own.
## Why Ohio
- Home datacenter; g4dn.xlarge spot ~**$0.163/hr** (vs us-east-1 ~$0.25, on-demand ~$0.526) — cheapest of the regions checked.
- us-east-1 g4dn spot capacity is currently depleted (placement score 1/10 across all AZs).
## What exists today (discovery)
Source — **us-east-1** (to mirror):
- VPC `vpc-053baa45dd75c045a` `DataRight-vpc` (`10.0.0.0/16`)
- Public subnets: `10.0.0.0/20` (us-east-1a), `10.0.16.0/20` (us-east-1b); Private: `10.0.128.0/20`, `10.0.144.0/20`
- IGW `igw-0a828410ab4af913c`; S3 gateway endpoint `vpce-0661ccc0d47e88054`
- SG `gemstone-nexus-sg`: ingress 443 + 80 (0.0.0.0/0), 22 (`71.208.0.159/32`), egress all
- GPU AMI `ami-03639d222ae9be0a2` `golden-v4-gpu` (30G `/dev/sda1` + 128G `/dev/sdb` gp3 + ephemeral), snapshots `snap-0b39665b0aa5f2e47`, `snap-0681ec8c786de0622`
- EIP `50.19.136.7` (region-bound; stays in us-east-1)
Target — **us-east-2** (Ohio):
- Only the **default VPC** `vpc-5e9b5335` (`172.31.0.0/16`) — no DataRight network yet
- **No** self-owned AMIs, **no** EIPs, no GemStone key pair
- Spot g4dn.xlarge: us-east-2a $0.1663, us-east-2b $0.1629, us-east-2c $0.1749
## Target architecture (Ohio)
```mermaid
flowchart TB
dns["GoDaddy DNS: GemStone.DataRight.net"]
subgraph oh [us-east-2 DataRight-vpc 10.2.0.0/16]
igw["Internet Gateway"]
subgraph pub [public subnet us-east-2a]
eip["New Elastic IP"]
inst["g4dn.xlarge (spot-first) from golden-v4-gpu-oh"]
data["128G /mnt/docker-data (persist)"]
end
s3ep["S3 gateway endpoint (free)"]
igw --- pub
eip --> inst
inst --- data
inst -. backups .-> s3ep
end
amiSrc["us-east-1 AMI ami-03639d222ae9be0a2"] -->|"aws_ami_copy (cross-region)"| amiOh["golden-v4-gpu-oh (Ohio)"]
amiOh --> inst
dns -->|"A -> new EIP"| eip
```
## Terraform layout
New directory in this repo (segregated, IaC-managed):
```
infra/terraform/ohio/
versions.tf # terraform >=1.6, aws provider ~>5.x
providers.tf # provider aws (us-east-2) + aliased provider aws.use1 (source for AMI copy)
variables.tf # ssh_cidr(default 0.0.0.0/0), vpc_cidr, az_primary, launch_instance(bool,false), use_spot(bool,true), instance_type, public_key_path, source_ami_id
network.tf # VPC, 2 public + 2 private subnets, IGW, public RT, S3 gateway endpoint + assoc
security.tf # aws_security_group gemstone-nexus-sg (443/80/22 all 0.0.0.0/0), aws_key_pair (imported public key)
ami.tf # aws_ami_copy golden-v4-gpu-dr-oh from the corrected us-east-1 GoldenAMI
compute.tf # aws_launch_template DataRightGemStoneGPUSpot + optional aws_instance + aws_eip + assoc (count = var.launch_instance ? 1 : 0)
outputs.tf # vpc_id, subnet_ids, sg_id, ami_id, launch_template_id, eip_public_ip
terraform.tfvars # concrete values (admin_cidr=71.208.0.159/32, etc.)
```
State: **local** for Phase A (`terraform.tfstate`, gitignored). Before Phase B, migrate to an
**S3 backend with native lockfile** (`use_lockfile = true`, Terraform >= 1.10; no DynamoDB) via
`terraform init -migrate-state`. State bucket: versioned, SSE-S3 encrypted, Block Public Access on.
### Key resource details (mirrors us-east-1 exactly unless noted)
- **VPC** `DataRight-vpc` `10.2.0.0/16` in us-east-2 (distinct range chosen to keep future peering/VPN with us-east-1 and on-prem `10.0.0.x` clean). Subnets: public `10.2.0.0/20` (2a), `10.2.16.0/20` (2b); private `10.2.128.0/20` (2a), `10.2.144.0/20` (2b). IGW + public route table `0.0.0.0/0 -> igw`. S3 **gateway** endpoint (free, no NAT).
- **Security group** `gemstone-nexus-sg`: 443/80/22 all from `0.0.0.0/0`, egress all. SSH is intentionally open (not IP-restricted) to allow mobile/remote management, matching the existing DataRight pattern; access is controlled by the `dr` credentials.
- **Key pair** `gemstone-nexus-key`: import the **public** half of your existing PEM (`ssh-keygen -y -f C:\Users\Mark\.ssh\gemstone-nexus-key.pem`) so the same key works in both regions. This is a secondary/break-glass path (`ubuntu` + key); the primary access follows the DataRight pattern (`dr` + password via plink) — see below.
- **AMI** via `aws_ami_copy` from the **corrected** `golden-v4-gpu-dr` (source region us-east-1) → `golden-v4-gpu-dr-oh`. This copies both snapshots into Ohio. GPU drivers + two-disk layout + Docker data-root=/mnt/docker-data + the DataRight host contract (below) are baked in, so a launched instance is immediately GPU-ready, plink-accessible as `dr`, with the data volume mounted.
- **Launch template** `DataRightGemStoneGPUSpot`: instance_type `g4dn.xlarge`, AMI = copied Ohio AMI, SG + key + public subnet, block devices 30G `/dev/sda1` + 128G `/dev/sdb` with **DeleteOnTermination=false** (persist models/ledger). Spot options gated by `var.use_spot` (persistent, interruption=stop). On-demand is the documented fallback (flip `use_spot=false`).
- **Instance + EIP** (optional, `count = var.launch_instance ? 1 : 0`): allocate a new EIP and associate to the launched instance. Default `launch_instance=false` so staging apply does not create a billable GPU box.
### DataRight host contract (dr user + plink access) — must be baked in
Discovery on the current GPU AMI showed it is **stock Ubuntu + NVIDIA + two-disk only** — it is missing the DataRight golden host contract (no `dr` user, `PasswordAuthentication no`, no `/home/dr/AppDev`, no version marker). To follow our pattern we **re-bake a corrected GoldenAMI** (Phase 0) that includes:
- **`dr` sudo user** (in `sudo` + `docker` groups), password set from **`DATARIGHT_DEV_SECRETS`** at bake time (not stored in user-data/IMDS).
- **`PasswordAuthentication yes`** so plink (`DR_SSH_USER=dr`, `DR_SSH_PASSWORD`) works, matching StackOne/OmniFlow. SSH (22) is open to `0.0.0.0/0` per the standard pattern to permit mobile/remote management; access is gated by the `dr` credentials.
- **`/home/dr/AppDev`** (+ `/home/dr/admin`) and the `/etc/dataright-golden-version` marker.
- Prefer establishing this via the DataRight Bootstrap `provision-host.yml` roles (base/user/secrets/docker/firewall) during the bake for fidelity with the standard.
`DATARIGHT_DEV_SECRETS` is **duplicated to us-east-2 now (not deferred)** and used to retrieve the `dr` password (and other credentials) during the build, exactly matching the pattern in the other repos/initiatives. Because the password is applied during the bake, the running instance does not need Secrets Manager access at boot (IAM instance profile stays deferred for the app runtime).
The GemStone-Nexus repo also gains the standard deploy pattern (from the OmniFlow/StackOne convention): `DR_SSH_HOST/USER/PASSWORD/HOSTKEY`, `DEVELOPMENT_DIRECTORY=/home/dr/AppDev`, `DR_REPO_NAME` in `.env.example`, and a plink-based `scripts/run-remote-docker-deploy.ps1` targeting `/home/dr/AppDev/gemstone-nexus`.
### Representative HCL (the two non-obvious bits)
```hcl
# ami.tf — cross-region duplicate of the GPU GoldenAMI
resource "aws_ami_copy" "golden_gpu" {
name = "golden-v4-gpu-dr-oh"
source_ami_id = var.source_ami_id # corrected golden-v4-gpu-dr (Phase 0 output)
source_ami_region = "us-east-1"
tags = { Project = "GemStone-Nexus", Name = "golden-v4-gpu-dr-oh" }
}
# compute.tf — spot-first, on-demand via var.use_spot=false
resource "aws_launch_template" "gpu" {
name = "DataRightGemStoneGPUSpot"
image_id = aws_ami_copy.golden_gpu.id
instance_type = var.instance_type
key_name = aws_key_pair.gemstone.key_name
dynamic "instance_market_options" {
for_each = var.use_spot ? [1] : []
content {
market_type = "spot"
spot_options { spot_instance_type = "persistent" instance_interruption_behavior = "stop" }
}
}
# network interface (public subnet + SG + public IP), 30G/128G(persist) BDMs, tags...
}
```
## Phased execution (each phase = explicit approval)
- **Phase 0 — Re-bake corrected GoldenAMI (us-east-1, separate approval):** provision the DataRight host contract onto a build instance (reuse the running us-east-1 instance `i-0276f2c824906c3e9`, ideally via Bootstrap `provision-host.yml`): create `dr` (sudo + docker, password from `DATARIGHT_DEV_SECRETS`), enable `PasswordAuthentication yes`, create `/home/dr/AppDev` + `/home/dr/admin`, write the golden marker. Create AMI **`golden-v4-gpu-dr`**. Also **replicate `DATARIGHT_DEV_SECRETS` to us-east-2**. Output: `source_ami_id` for Terraform.
- **Phase A — Stage infra (cheap, no GPU):** `terraform apply` with `launch_instance=false`. Creates VPC/subnets/IGW/routes/S3 endpoint (free), SG + key (free), and the AMI copy of `golden-v4-gpu-dr` (small snapshot storage). Result: ready to launch, ~$0/hr compute.
- **Phase B — Launch GPU (billable, separate approval):** set `launch_instance=true`, confirm `use_spot=true`, `apply`. If spot returns `InsufficientInstanceCapacity`, I STOP and report (no auto on-demand). You then decide: retry, or set `use_spot=false` for on-demand.
- **Phase C — DNS + resume work:** you point `GemStone.DataRight.net` at the new Ohio EIP; deploy the GemStone app (later phase), Let's Encrypt, verify chat/voice/GPU.
- **Phase D — decommission us-east-1 (confirmed):** once Ohio is live and verified, terminate the us-east-1 on-demand instance and delete the us-east-1 GemStone resources tagged `Project=GemStone-Nexus` (instance, launch template, `golden-v4-gpu` AMI + 2 snapshots, SG, key pair, and the `50.19.136.7` EIP if you don't want to keep paying for it). Executed only after Ohio passes verification.
## Costs (us-east-2)
One-time:
- Cross-region AMI/snapshot copy data transfer: ~$0.02/GB of actual snapshot data (near-empty volumes) ≈ **<$0.50 one-time**.
Recurring — always present once staged:
| Item | Cost |
|---|---|
| VPC, subnets, IGW, route tables, SG, key pair | Free |
| S3 gateway endpoint | Free |
| Ohio AMI snapshots (OS + near-empty 128G) | ~$0.30-0.50/mo |
| New Elastic IP (charged whether or not associated) | ~$3.65/mo |
| **Staged subtotal (no GPU running)** | **~$4-4.5/mo** |
Recurring — when the GPU instance exists/runs:
| Item | Rate | Notes |
|---|---|---|
| g4dn.xlarge **spot** (Ohio) | ~$0.163/hr | primary |
| g4dn.xlarge **on-demand** | ~$0.526/hr | fallback |
| EBS 128G gp3 (persistent data) | ~$10.24/mo | persists across relaunch |
| EBS 30G gp3 root | ~$2.40/mo | ephemeral (DeleteOnTermination) |
Scenario totals (incl. EIP + snapshots + EBS):
- **Dev, spot, ~88 hrs/mo (4h×22d):** ~$14 compute + ~$12.6 EBS + ~$4.15 = **~$31/mo**
- **Always-on spot (730 hrs):** ~$119 + ~$12.6 + ~$4.15 = **~$136/mo**
- **Always-on on-demand (730 hrs):** ~$384 + ~$12.6 + ~$4.15 = **~$401/mo**
Cost control: keep `launch_instance=false` between sessions (or `stop` the instance); spot
+ auto-stop keeps dev around ~$30/mo. The 128G data volume persists so restarts need no re-pull.
## Decisions (locked)
1. **VPC CIDR:** distinct **`10.2.0.0/16`** (avoids overlap with us-east-1 `10.0.0.0/16` and on-prem `10.0.0.x`).
2. **us-east-1 teardown:** **terminate + clean up** us-east-1 GemStone resources once Ohio is live (Phase D).
3. **Terraform state:** **local for Phase A**, then migrate to an **S3 backend with native lockfile** (`use_lockfile = true`, Terraform >= 1.10, no DynamoDB) before Phase B via `terraform init -migrate-state`. Creating the SSE-S3-encrypted, versioned, private state bucket is a separate approved step before the migration.
4. **Host contract:** **re-bake** a corrected `golden-v4-gpu-dr` AMI including the `dr` sudo user (password from secrets), `PasswordAuthentication yes` for plink, `/home/dr/AppDev`, docker group, and marker — then copy that to Ohio (Phase 0).
5. **Secret:** use **`DATARIGHT_DEV_SECRETS`**, **duplicated to us-east-2 now (not deferred)**; used to retrieve credentials during the build; `dr` password applied at image time so the instance needs no Secrets Manager access at boot.
6. **SSH:** open (`22` from `0.0.0.0/0`) to permit mobile/remote management, matching the existing pattern — not IP-restricted.
## Explicitly NOT in this plan (later, separate approval)
GemStone app deployment, Let's Encrypt/TLS, GoDaddy DNS record change, IAM instance profile + S3 backup bucket. This plan gets Ohio "functionally ready to resume work" at the infrastructure layer.