gemstone-nexus · Phase 6
DOC08 — Playbook
Metadata
| id | doc_id | title | category | source_path | summary | content_hash | indexed_at |
|---|---|---|---|---|---|---|---|
| 26 | DOC08 | DOC08 — Playbook | playbook | DevResources/docs/DOC08-Playbook.md | 1. Query the sessions table for the session_id; confirm state and check interruption_snapshot is non-null. 2. If interruption_snapshot is null and state is PAUSED, this is a defect: the snapshot write failed silently. Escalate immediately; do not attempt manual state repair without engineering. 3. If state is PAUSED but the session-ready chime should already have fired, confirm cue scheduling; stuck-before-chime is distinct from stuck-after-chime (RESUMABLE). 4. If interrupt… | 941ae1407da6ade03137695c45d484ab813f54c5 | 2026-07-22 18:33:18 |
Content
# DOC08 — Playbook
*Operational runbooks for common production scenarios*
**Initiative:** GemStone · **Version:** v2.0 (Rewrite) · **Status:** Approved for Build · **Date:** July 17, 2026
## 1. Runbook: Session Stuck in PAUSED or RESUMABLE
1. Query the sessions table for the session_id; confirm state and check interruption_snapshot is non-null.
2. If interruption_snapshot is null and state is PAUSED, this is a defect: the snapshot write failed silently. Escalate immediately; do not attempt manual state repair without engineering.
3. If state is PAUSED but the session-ready chime should already have fired, confirm cue scheduling; stuck-before-chime is distinct from stuck-after-chime (RESUMABLE).
4. If interruption_snapshot is present and state is RESUMABLE (or PAUSED ready to resume), verify the practitioner's app can reach POST /api/sessions/{id}/resume; a 4xx/5xx here indicates an API-layer issue, not a data issue.
5. Confirm the resume restores last_turn correctly by comparing against the transcript stored in Atlas Vault for that session_id.
## 2. Runbook: Suspected Tier Violation
1. Pull the session record; confirm the tier field value has never changed by checking the audit log (tier is immutable; any change event is itself the defect).
2. Identify which action triggered the suspected violation and check whether it appears in RESTRICTED_ACTIONS for that tier in the TierGuard configuration.
3. If the action was permitted but should not have been, this is a TierGuard configuration defect, not a data defect; do not patch the individual session, fix the enforcement rule.
4. Confirm no secondary tier-check code path exists elsewhere in the codebase; a second enforcement point is itself a defect regardless of whether it agrees with TierGuard.
## 3. Runbook: Didymus Task Reached RUNNING Without Approval
1. This should be structurally impossible; treat as a Severity 1 defect immediately upon discovery.
2. Freeze the worker that executed the task; do not allow it to lease further tasks until the schema constraint is confirmed intact.
3. Inspect the database constraint directly (not application code) to confirm the approved_by/approved_at NOT NULL and CHECK constraints are present and unmodified.
4. If the constraint was bypassed via direct database write rather than the API, this is an access-control issue, not a Didymus logic issue; escalate accordingly.
## 4. Runbook: Vectorization Failure
1. Confirm the chunked-hold state: raw audio should be present and intact, not deleted, for any session in a failed vectorization state.
2. Confirm both flags fired: in-app notification delivered to the practitioner and a system-log entry created, within 5 minutes of the failure per DOC03 exit criteria.
3. If only one flag fired, this is a defect in the dual-flagging mechanism itself, not a one-off failure; escalate to engineering rather than manually notifying the practitioner as a workaround.
4. Once the underlying failure (transcription, indexing, or vectorization stage) is identified, re-run the pipeline against the held audio chunks; do not discard audio until vectorization succeeds.
## 5. Escalation Matrix
| Scenario | Severity | Escalate To |
| --- | --- | --- |
| Task reaches RUNNING without approval | Sev 1, immediate | NEXTECH principal + engineering lead |
| Tier violation confirmed | Sev 1, immediate | NEXTECH principal + engineering lead |
| Repeated/systemic vectorization failures | Sev 2, same-day | Engineering lead |
| Wake-word false-trigger rate above threshold in field data | Sev 3, next sprint | Engineering lead, informs Gate 2 tuning |
| HUME AI status inquiry | Informational | NEXTECH principal, no timeline commitment given |
## 6. Co-dev Deploy (Stage 2 operating model)
Primary build host is the Ohio T4 EC2. Windows authors code; Docker runs only on the host.
1. Confirm instance up: `i-04333dae160ab0daa` (public IP may change — verify via AWS CLI).
2. Pin SSH host key if relaunched: `.\scripts\remote-exec.ps1 -RefreshHostKey`; update `DR_SSH_HOSTKEY` in `DevResources/project.env`.
3. Ensure worktree: `.\scripts\ensure-ec2-dev-dir.ps1` → `DEVELOPMENT_DIRECTORY=/home/dr/AppDev/gemstone-nexus`.
4. **Deploy** after a deliberate commit+push: `.\scripts\run-remote-docker-deploy.ps1` (pull → `docker compose up` → health). On-host equivalent: `scripts/aws/deploy.sh`.
5. PAT and `DR_SSH_PASSWORD` come from Secrets Manager `DATARIGHT_DEV_SECRETS` — never commit them. Keep the EC2 `origin` URL free of embedded tokens.
6. Full procedure: `DevResources/docs/DOC-Deploy-Pipeline.md`. Atomic concurrency: every Deploy that changes schema/API/UI must update ledger + DOC12/DOC07 in the same change set (Stage 3+).
---
NEXTECH Business Technology Management LLC · nex-tech.biz · (941) 499-5505