> For the complete documentation index, see [llms.txt](https://decentralabs-2.gitbook.io/decentralabs-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://decentralabs-2.gitbook.io/decentralabs-docs/lab-gateway/operations-and-verification/operations-and-health.md). # Operations and health This runbook describes the safe operational checks for a Compose-managed Lab Gateway. It applies to Full and Lite access planes; the expected dependencies differ by mode. ## Health endpoints Public health intentionally contains only an aggregate result. It is safe for load balancers and basic monitoring and does not disclose keys, queues, hosts, or dependency diagnostics. | Endpoint | Audience | Meaning | | ----------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /health` | Public | Liveness/readiness of the public gateway edge. | | `GET /gateway/health` | Public | Aggregate readiness of the local access-plane dependencies. In Full mode it also requires the backend service; in Lite mode the remote issuer-trust check is available through the protected details endpoint. | | `GET /ops/health` | Public | Aggregate Ops Worker readiness. | | `GET /health/details` | Lab Manager operator | Detailed backend, network, certificate, and configuration diagnostics. | | `GET /gateway/health/details` | Lab Manager operator | Detailed local dependency and Lite issuer-trust diagnostics. | | `GET /ops/health/details` | Lab Manager operator | Detailed Ops Worker diagnostics. | | `GET /gateway/mode` | Public | Reports the selected edge mode without dependency details. Use it to confirm Full versus Lite after a configuration change. | Marketplace also uses `GET /public/labs/status?labIds=1,2` as a bounded per-lab projection. It returns only the state, signal age and stable reason for each requested lab. A fresh Lab Station heartbeat is authoritative; when no fresh heartbeat exists, the Ops Worker may probe the configured Guacamole RDP, VNC or SSH port. That fallback is reported as `reachable` or `not_ready`, and never as Lab Station `ready`; an absent fallback target remains `unknown`. Host names, addresses, ports and raw Station telemetry remain operator-only. The projection also separates `access` (`ready`, `busy`, `inaccessible` or `unknown`), `wake` (`verified`, `configured`, `failed` or `unknown`) and `availability` (`now`, `on_demand`, `recoverable`, `unavailable` or `unknown`). `wake.verified` requires a recent persisted wake operation that sent WoL and then observed the host answering; a configured MAC and ping target alone only produce `wake.configured`. Successful WoL evidence remains verified for seven days; a weekly Wake Ops verification can refresh that evidence automatically. Physical-lab entries use the persisted Station heartbeat, with the bounded Guacamole target probe and WoL evidence as independent fallback/enrichment signals. FMU entries use the private `fmu-runner` health signal in both local and Station mode. In Station mode the runner verifies the internal Lab Station FMU executor and its protected capacity endpoint, so FMU readiness does not depend on Guacamole, WoL or the physical-lab heartbeat. FMU responses expose bounded `executor` and `capacity` fields and do not expose a `wake` dimension. When available, `capabilities.physicalLab` and `capabilities.fmu` remain in the response so Marketplace can select the resource-specific projection. The public response has the stable shape below. `status` is `UP`, `PARTIAL`, or `DOWN`. `PARTIAL` means that the gateway is reachable but not all local dependencies are ready, such as incomplete provider or consumer registration. `DOWN` is reserved for the case where none of the checked dependencies is available. The endpoint returns `503` for a partial or unavailable gateway. ```json { "status": "UP", "service": "lab-gateway", "mode": "full", "public": true } ``` The browser view at `/gateway-health/` presents the same aggregate result and, after Lab Manager authentication, the service and infrastructure diagnostics. It is an operator aid, not a replacement for the public health contract. Power-controller reachability is intentionally not folded into the aggregate `/ops/health` result. A PDU can be offline or temporarily unreachable while the Ops Worker and the rest of the access plane remain healthy. Inspect configured controllers in Lab Manager `Energy`: the local catalog loads independently, while live discovery and outlet state are checked in the background. Use the controller `Refresh` action to force a fresh status request when diagnosing a private-network or SNMP issue. `GET /gateway/health/details` also exposes `services.demo` for the configured anonymous demo. Its status is one of: * `disabled`: both demo identifiers are empty; * `misconfigured`: the binding, Guacamole connection, managed principal, or exact `READ` grant is invalid. The MySQL reconciler also rejects a name collision, a different `DEMO_LAB_ID`, or any residual system/user/group, connection-group, sharing-profile permission or group membership; * `unready`: the gateway, Marketplace authority, or Station heartbeat cannot currently prove readiness; * `busy`: the Station has an active Windows session or local-only mode enabled, or the Marketplace authority denies the requested demo window; * `ready`: all local checks, the physical host and the Marketplace authority agree for a short probe window. The endpoint is protected by the Lab Manager operator policy and never exposes Guacamole credentials. The public `/gateway/health` remains an aggregate edge signal; a demo hand-off is permitted only when it is `UP` and the protected demo status is `ready`. The demo hand-off also has a synchronous physical lifecycle. It calls the internal Ops Worker with `demo:`, which resolves the bound Station, performs Wake-on-LAN when needed, runs `prepare-session` before the Guacamole token is exposed, and runs `release-session` without rebooting on expiry, failure or tunnel closure. The worker records `demo_start`, `demo_connection`, `demo_expiry`/`demo_failure`/`demo_disconnect`, and `demo_cleanup` alongside the low-level Station steps. OpenResty keeps the gateway slot in `pending` for the bounded `DEMO_PENDING_LEASE_SECONDS` window (30–60 seconds, default 45) and promotes it to `active` only after token issuance; failed or abandoned handoffs release the logical slot while physical cleanup remains retryable. This is an operational timeline, not a blockchain reservation. ![Gateway Health dashboard](https://2324164250-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhBavBrPgRBmGJAMRBdoV%2Fuploads%2Fgit-blob-4a5abdfe9de1f55c184b535e86d7c68c9ff98ba6%2Flab-gateway-health.png?alt=media) Use the normal Lab Manager authentication flow before opening a `details` endpoint. Do not add a token to the query string. ```bash # Public checks curl -k https://gateway.example.edu/health curl -k https://gateway.example.edu/gateway/health curl -k https://gateway.example.edu/ops/health # Start an operator session and save its path-scoped cookie for operator routes curl -k -c lab-manager-cookie.txt -X POST -d "token=" \ https://gateway.example.edu/lab-manager/login ``` ## First-line triage ```mermaid flowchart TD Start[Public health fails] --> Compose[docker compose ps] Compose --> Edge{OpenResty running?} Edge -->|No| EdgeLog[Inspect OpenResty logs and configuration] Edge -->|Yes| Mode[Check /gateway/mode and /gateway/health] Mode --> Full{Expected mode?} Full -->|No| Issuer[Correct ISSUER and restart the edge] Full -->|Yes, Full| Backend[Check blockchain-services and MySQL] Full -->|Yes, Lite| Remote[Check remote issuer DNS, TLS, JWKS/public key, and trust bundle] Backend --> Access[Check Guacamole, guacd, and Ops Worker] Remote --> Access Access --> Station[Check the private Station network only if the affected lab needs it] ``` | Symptom | Check | Typical next action | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `/health` is down | `docker compose ps`, `docker compose logs --tail=100 openresty` | Resolve edge startup, TLS, port binding, or configuration errors. | | `/gateway/health` is down in Full | `blockchain-services`, MySQL, Guacamole, `guacd`, and Ops Worker | Use protected details to identify the failed local dependency; do not treat a public response as detailed telemetry. | | `/gateway/health` is down in Lite | Remote issuer DNS/TLS, issuer public-key/JWKS sync, and per-Lite trust values | Confirm `ISSUER` is external and exact; refresh the Full-issued trust bundle or equivalent remote issuer configuration if keys rotated. | | End user sees Guacamole login | Access-code redemption and reservation state | Use the access-code flow, not a browser JWT or a manual Guacamole account. | | Guacamole connection fails | `guacd` and the private RDP/VNC/SSH route | Test only from the controlled lab network and verify the selected local connection ID. | | Ops action fails | `/ops/health/details`, host catalog, WinRM TLS and CIDR policy | Confirm host address is allowed by `WINRM_MANAGEMENT_CIDRS`; verify the encrypted credential reference. | | Power controller is `checking`, `unknown`, or `unreachable` | Lab Manager `Energy` status, controller host/port, private route, credentials and Ops Worker logs | Wait for the background status check, then use `Refresh`; for APC verify UDP `161`, the source-IP authorization and the credential's SNMP version. | | Heartbeat reports `WINRM_UNREACHABLE` | Station power state, WinRM listener/firewall, host address/port and the private management route | The Station may be powered off or WinRM may be unavailable; restore reachability and retry. | | Heartbeat reports `WINRM_AUTH_FAILED` | Host-card WinRM credentials, account permissions and the Ops Worker request ID | Recheck or replace the stored Station credential, then retry the heartbeat. | | Heartbeat reports `WINRM_HEARTBEAT_NOT_FOUND` | Station heartbeat path, `heartbeat_path` and the remote file-read log | Restore the configured heartbeat file or correct the host path, then retry. | | Heartbeat reports `WINRM_HEARTBEAT_INVALID` | Station heartbeat artifact and `status-json` output | Correct the malformed JSON produced by the Station, then retry; do not treat it as a TLS or authentication failure. | | Heartbeat stream fails for an incomplete Station | Host-card `WinRM credentials` and `WinRM TLS trust` statuses, then `/ops/health/details` and Ops Worker logs | Save the WinRM credential and save the Station certificate through the host card's `WinRM TLS trust` control (or copy it into the matching per-host `ops-data/winrm-certificates//` directory) before retrying heartbeat; retain any `requestId` for generic internal failures. | | FMU route returns `503` | `FMU_RUNNER_ENABLED` and active profile | Start exactly one FMU profile and verify the public audience and, for production, Station connectivity. | ## Useful Compose commands Run commands from the repository root: ```bash # Inspect the evaluated deployment surface docker compose config --services docker compose config --profiles # Base stack status and focused logs docker compose ps docker compose logs --tail=100 openresty docker compose logs --tail=100 blockchain-services docker compose logs --tail=100 ops-worker # Restart only the service that changed docker compose restart openresty docker compose restart blockchain-services # Rebuild a changed local service docker compose up -d --build openresty ``` `blockchain-services` is deliberately dormant in Lite mode. Its sleeping container is not a failure and must not be used as the local issuer-health signal. ## Operator access surfaces | Surface | Authentication | Intended purpose | | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `/lab-manager` and `/ops/**` | `LAB_MANAGER_TOKEN`, exchanged for a short-lived path-scoped cookie | Lab inventory, Station operations, operational timelines, and diagnostics. | | `/wallet-dashboard`, `/wallet/**`, `/billing/**`, `/institution-config` | `ADMIN_ACCESS_TOKEN`, exchanged for an administrative cookie | Institutional wallet, billing, and administrative configuration. | | `/guacamole/` manual login | Guacamole credentials with rate limits and IP bans | Operator break-glass/manual administration only. | | `/auth/access` | Short-lived opaque access code; gateway-local validation followed by commit/release | End-user reservation hand-off. | Keep administrative access behind the configured network policy, VPN, or allow-list. `POST /admin/logout` removes administrative cookies; never rely on cookies beyond their intended path or lifetime. ## State and backups Back up durable state before host replacement, schema maintenance, or key rotation. Use the institution's approved encrypted backup process. | Data | Why it matters | | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `blockchain-data/` | Encrypted institutional wallets and backend key material. | | `.env` and `blockchain-services/.env` | Deployment configuration and secret references. Keep outside source control. | | `OPS_SECRETS_KEY` | Required to decrypt stored WinRM credentials. It must be stable and recoverable. | | `OPS_POWER_CONFIG`, `OPS_POWER_CREDENTIALS_PATH`, and `OPS_SECRETS_KEY` | Provider-local power-controller catalog and encrypted APC/SNMP/NETIO credentials. Back them up together through the approved encrypted process; never expose the credential store or copy it into an unprotected backup. | | MySQL data volume | Guacamole, backend operational, and Ops Worker state. | | `lab-content/` | Lab Manager-generated metadata and uploaded provider content. | | AAS/FMU volumes when enabled | BaSyx shell data and FMU history, according to provider retention policy. | Do not back up by copying live files blindly from a running database container. Use a consistent database backup procedure and verify a restore in a non- production environment. ## Change and incident boundaries * Change one mode-sensitive value at a time and validate the Compose model before restart. See the [configuration reference](/decentralabs-docs/lab-gateway/getting-started-and-deployment/configuration.md). * A public `UP` does not prove that a reservation is authorized, a Guacamole session is usable, or a Station is ready. Correlate the relevant workflow, operational timeline, and on-chain state. * A local Ops state such as `ACTIVE` is operational telemetry, not the on-chain `ACCESS_AUTHORIZED` state. * Do not expose MySQL, Ops Worker internals, station WinRM, or FMU internal endpoints to simplify troubleshooting. ## Related documents * [Configuration reference](/decentralabs-docs/lab-gateway/getting-started-and-deployment/configuration.md) * [Deployment architectures](/decentralabs-docs/lab-gateway/getting-started-and-deployment/deployment-architectures.md) * [Lab Gateway and Lab Station operations](/decentralabs-docs/lab-gateway/architecture-and-connectivity/gateway-lab-station-operations.md) * [Guacamole session policy](/decentralabs-docs/lab-gateway/architecture-and-connectivity/guacamole-session-policy.md) * [Integration tests](/decentralabs-docs/lab-gateway/operations-and-verification/integration.md)