System Integrator Guide

This guide separates repository testing from the work of creating AHRI TRE testing, staging, and production environments. This repository provides application artifacts, protocol and configuration contracts, a disposable Integration fixture, and a durable non-production Local TRE. This repository does not provide production infrastructure or claim that Local TRE is a deployment template.

Choose the environment

Do not promote Integration volumes or Local TRE volumes into another environment. Promote reviewed, immutable release artifacts and versioned configuration instead.

Inputs supplied by this project

• the Rust binaries, libraries, public protocol, and C ABI described in the installation and API guides;
• versioned Application and Client configuration contracts;
• PostgreSQL metadata and DuckLake adapter behavior;
• configured container-visible Lake location as the canonical container-visible filesystem Lake location;
• JSON-over-HTTP control-plane behavior and OAuth/OIDC boundaries;
• release-versioned TRE Browser compatibility metadata; and
• validation commands and synthetic fixtures that contain no production data.

The Configuration And Secrets, Authentication, Sensitive Material Handling, and System Architecture pages define the application-facing boundaries. They are not infrastructure-as-code.

Create a testing, staging, or production environment

AHRI TRE does not select Kubernetes, virtual machines, a cloud, or an infrastructure-as-code tool for the integrator. The procedure below is the provider-neutral deployment plan that the integrator must express in the organization’s chosen platform. A deployment is not made by copying Local TRE: it is made by provisioning the following boundaries and satisfying the same Application configuration contract.

Create testing, staging, and production as separate logical Deployments. Each must have its own Deployment UUID, DNS names, PKI, OIDC registration, Application configuration, Injected secrets, Managed-secret root identity and store, PostgreSQL databases and roles, Lake storage, audit history, and backup policy. There is no testing, staging, or production switch in the application.

1. Record the deployment specification

Before creating resources, put a reviewed, non-secret specification under change control. At minimum, record:

• the environment name and immutable Deployment UUID;
• release artifact versions, checksums or image digests, and supported CPU architecture;
• the Trusted-runtime and Web HTTPS origins and certificate authorities;
• the OIDC issuer, public client identifiers, exact Web callback URI, and allowed audiences;
• PostgreSQL server, database, TLS, role, and backup design;
• the container-visible Lake base path, storage provider, encryption, capacity, retention, and recovery design;
• execution profiles, compute and image policy, scratch capacity, ingress, egress, and network-policy decisions; and
• service ownership, monitoring, recovery objectives, maintenance windows, and approval evidence.

Use synthetic data and test identities in testing. Staging should reproduce the production topology, trust path, policy, and upgrade procedure without sharing production data or credentials. Production uses only organization-approved services and controls. Capacity may differ, but security boundaries must not be silently removed in a lower environment.

2. Provision the platform boundaries

Translate the specification into reviewed infrastructure code and create:

1. Private data services. Provision TLS-enabled PostgreSQL, the metadata and DuckLake catalog databases, narrowly scoped administration/runtime/Web roles, and durable Lake storage. Mount the Lake at the exact absolute path that will be declared in Application configuration; inside trusted workloads this is the canonical configured container-visible Lake location boundary. Do not expose PostgreSQL or the Lake publicly.
2. Trusted service capacity. Provide persistent encrypted storage for /var/lib/ahri-tre/secrets, separately replaceable storage for /var/lib/ahri-tre/secrets-rotation, and encrypted local scratch. Only trusted runtime and operator workloads may receive these mounts or datastore administration capability.
3. Secret projections. Configure the platform secret store to project each injected://namespace/name beneath /run/secrets/namespace/name/ as a protected value file and a non-secret version file. The version must exactly match expected_version in configuration. Separately generate and back up the Deployment’s X25519 root identity, then project its value at /run/secrets/ahri-tre/root-identity/value. Never store the identity beside a backup of the encrypted Managed-secret store.
4. Network and Web edge. Put the static TRE Browser and ahri-tre-web behind one public HTTPS origin. Route the Web service through a private TLS listener, validate its certificate at the proxy, preserve Origin, X-Request-Id, and X-Protocol-Version, and allow only explicitly reviewed service-to-service and egress paths.
5. Operator and user separation. Give deployment automation a distinct trusted operator identity. User-controlled development, Jupyter, and pipeline workloads must not receive the Application configuration, Managed-secret store, root identity, raw Lake authority, or datastore-internal credentials.

Runtime mount contract

Project only the capability each workload needs. Platform-specific source names are deployment details; the container-visible targets and trust split are the application contract.

Run the Trusted runtime and trusted Secret-administration workloads under one deployment-dedicated service UID and GID. The effective service UID must own the Managed store; use the dedicated group only where read sharing is required. Managed-store directories may grant at most 0750, and regular files at most 0640. Neither may grant world access or group write. Unexpected ownership, excessive modes, inaccessible required paths, or unsupported locking is fatal; AHRI TRE reports safe metadata and never changes ownership or permissions.

Both Secret tiers limit each decrypted value to 64 KiB. Oversized Injected values are rejected before they are returned, and oversized Managed values are rejected before persistence. Treat this as an interface bound, not a prompt to split a larger credential across multiple Secret references.

Root rotation is the only workflow that also receives the next identity at /run/secrets/ahri-tre/root-identity-next/value and an empty writable staging store at /var/lib/ahri-tre/secrets-rotation. Long-running services must never receive either capability.

PKI and public trust

Keep the CA signing key outside AHRI TRE workloads. Inject only each service’s leaf certificate and private key into the service that terminates that identity. Place the public Client CA chain in Application configuration so config render-client can project it into the Client bootstrap. The Web edge has its own reviewed public/private TLS termination path; private service hops still validate certificate chains and hostnames. Development-generated Local certificates are not production trust material.

The Web Control Plane Datastore Deployment guide gives the current Web-to-Runtime topology, identity and authority split, same-origin proxy rules, and single-instance session-store limitation. If the intended service topology requires a capability that the selected AHRI TRE release does not implement, treat that as a release blocker rather than inventing a privileged side path.

3. Install one reviewed release

Build or obtain every executable and frontend artifact from one reviewed release. Verify checksums, signatures where supplied by the release process, architecture, compatibility metadata, dependency inventory, and vulnerability policy before installation. Pin immutable versions or image digests; never deploy latest, a developer worktree, a Local TRE volume, or an Integration fixture.

The Developer Installation Package Guide documents the current runtime package artifacts and their verification surface. The TRE Browser is a separately versioned companion artifact and must match the compatibility metadata of the selected backend release.

4. Author and validate Application configuration

Create one authoritative /etc/ahri-tre/config.toml for the Deployment. Start with the generated schema and minimal document; use the maintained complete example only as a structural reference because its endpoints, certificates, identities, and Secret references are placeholders:

ahri-tre config schema --format json > application-v1.schema.json
ahri-tre config init --output config.toml
# Edit config.toml from the reviewed deployment specification.
ahri-tre --config config.toml config validate
ahri-tre --config config.toml config show-effective \
  --for trusted-runtime
ahri-tre --config config.toml config show-effective --for web
ahri-tre --config config.toml config show-effective \
  --for execution-profile --profile research

The complete reference is examples/config/application-v1.toml. Replace every placeholder and remove unused declarations. Review the safe Effective output and configuration fingerprint, but keep the authored document and its provenance as the authority. Do not replace configuration with a set of environment variables.

After the required Secret projections and dependency endpoints have been staged, preflight each path that will be started or administered:

ahri-tre --config config.toml config preflight --for trusted-runtime
ahri-tre --config config.toml config preflight --for web
ahri-tre --config config.toml config preflight \
  --for execution-profile --profile research
ahri-tre --config config.toml config preflight \
  --for datastore-create --datastore research
ahri-tre --config config.toml config preflight --for secret-administration

This command uses the same selected Effective configuration and completeness rules as startup. It reads only the target’s required Secret material and performs bounded, safe dependency checks; it never creates, repairs, migrates, or otherwise mutates infrastructure. A successful result means those deployment-owned prerequisites were observable at that moment. It does not authenticate a TRE user, authorize a workflow, or prove service readiness. Trusted-runtime preflight reports authenticated client transport separately as not applicable, so it cannot be interpreted as client ready. Use --format json for exactly one machine-readable result document.

Render the public Client bootstrap from that same document and distribute it read-only to managed clients. It contains the Deployment identity, canonical Trusted-runtime origin, and public CA chain, not service topology or secrets:

ahri-tre --config config.toml config render-client --output client.toml

5. Bootstrap secrets and datastores

Run the following as a trusted, auditable operator workload with the final configuration, Injected-secret projection, root identity, and persistent Managed-secret volume mounted at their production paths:

ahri-tre --config /etc/ahri-tre/config.toml secrets init
ahri-tre --config /etc/ahri-tre/config.toml \
  secrets generate managed://postgresql/runtime-password --generator password
# Add every other managed:// reference declared by the selected service graphs.
ahri-tre --config /etc/ahri-tre/config.toml secrets verify --all

Use secrets put <reference> with its protected terminal prompt, or secrets put <reference> --stdin from a bounded secret-delivery mechanism, when a value is supplied externally. Do not pass a value as an argument or capture it in logs. secrets init is create-only and must not be rerun against an existing store.

Credential replacement is owner-routed. Rotate a generated Datastore database and DuckLake-catalog credential with only its logical identifier:

ahri-tre datastore rotate-lake-credential research

The Trusted runtime stages the new version, applies and verifies it at PostgreSQL, then activates it and removes the prior ciphertext. To remove an obsolete Managed Secret, select the intended Application configuration explicitly:

ahri-tre --config /etc/ahri-tre/config.toml secrets remove <managed://reference>

The request is bound to the selected Deployment UUID. Removal fails closed unless the immutable Application, durable and currently inspected Datastore bindings, durable Session records, and authentication-artifact state all prove the reference is absent; there is no force bypass.

Start the Trusted runtime with the final mounts and configuration, then use its protected local administration interface to reconcile each predeclared Datastore. For the research declaration in the reference configuration:

ahri-tre-runtime --config /etc/ahri-tre/config.toml
# From the trusted operator workload that can reach the runtime's local socket:
ahri-tre-runtime datastore reconcile research

Reconciliation creates an absent Datastore or validates its retained ready binding. It does not authorize copying a database or Lake from another logical Deployment. Apply release-supported migrations and datastore role provisioning before making a datastore visible to Web.

6. Start services and qualify the deployment

Start trusted services only after configuration validation, complete secret verification, PostgreSQL TLS, writable trusted scratch, Lake mounts, and datastore reconciliation succeed. Start ahri-tre-web on a private address, then enable the same-origin proxy and Browser only after /health, /ready, and safe /diagnostics checks pass. Follow the Web deployment guide for exact readiness semantics: /health alone does not prove datastore access.

Startup and readiness gates

The diagnostics operator guide documents request joins, evidence fields, exit/status meanings, bounded partial results and the tested Local TRE investigations for these gates.

These gates answer different questions and are not interchangeable:

A rendered Client bootstrap is only a validated artifact. Distribution, reload, and a successful authenticated remote operation are required before recording Client readiness. Likewise, ./dev-env doctor checks the contributor host and tooling; it is not product preflight, recovery verification, or service readiness.

Before accepting the environment, exercise with non-production records:

• TLS chain and hostname failures as well as the successful paths;
• OIDC login, callback, logout/expiry, and audience enforcement;
• expected datastore discovery and rejection of undeclared or unauthorized datastores;
• creation/opening of a Session and the configured Lake and catalog boundary;
• Study access request, custodial decision, audit, and provenance behavior;
• backup and exact same-Deployment restore of PostgreSQL, Lake data, configuration, encrypted Managed secrets and audit journal, with the root identity restored through its separate channel; and
• monitoring, alerting, restart, rollback, credential rotation, capacity exhaustion, and disaster-recovery runbooks.

Record the release identities, Deployment UUID, configuration fingerprint, test results, approvers, and exceptions without recording credentials, tokens, personal identities, Restricted local references, or restricted data.

7. Promote releases, not environments

Promote the same reviewed artifact digests from testing to staging and then to production. For each target, independently provision infrastructure, author and validate its configuration, inject its secrets, initialize its Deployment-bound Managed store, reconcile its datastores, and run its qualification gates. Never promote a Deployment UUID, root identity, secret-store ciphertext, database volume, Lake volume, OIDC credential, or private key between environments.

An update follows the same path: qualify the candidate in testing, rehearse the upgrade and rollback in staging from a representative non-production backup, approve the change, apply it to production, and verify readiness plus critical user journeys. Rollback must use the release’s documented compatibility and data-migration rules; replacing binaries alone is not a recovery plan.

Credential and trust rotation

Managed and Injected credentials

Use only the owning workflow for a Managed credential. secrets put and secrets generate are create-only; there is no generic overwrite or version selector. For a generated Datastore database and DuckLake catalog credential, invoke the protected owner route:

ahri-tre datastore rotate-lake-credential research

The Trusted runtime stages the successor, updates and verifies PostgreSQL, activates the new ciphertext, and removes the prior active ciphertext. A failure or uncertain commit fails closed and preserves recovery evidence. Already-open Sessions retain the exact versions with which they opened; new authorized operations resolve the new active version. Authentication artifacts are replaced through their authentication owner transaction, not by editing store files.

For an Injected secret, deployment tooling creates a new external version, updates the corresponding expected_version in Application configuration, projects the matching value and version files, validates/preflights the selected paths, and restarts every consumer. Processes snapshot Injected material at startup and never live-reload it.

Client CA rotation

Rotate the Client trust CA with an overlap:

1. Add the new public CA chain beside the old chain in Application configuration, validate it, render the replacement Client bootstrap, and distribute it atomically.
2. Restart every Managed runtime so its immutable bootstrap trusts both roots. Confirm distribution; rendering alone is not reachability evidence.
3. Issue and project a new Trusted-runtime leaf certificate and key chaining to the new CA, update its declared Injected version, and restart the Trusted runtime.
4. Prove authenticated Client operations through the new service certificate.
5. Remove the old CA from Application configuration, render and distribute the reduced bootstrap, restart clients again, and retire the old CA under the organization’s PKI policy.

A routine leaf renewal beneath an unchanged trusted CA does not require a bootstrap change, but the service still needs the newly versioned Injected projection and restart. A client that missed an advance trust update fails closed; the server cannot replace trust anchors over the connection they authenticate.

Managed-store root rotation and rollback

Root rotation is offline and non-destructive:

1. Quiesce every Secret consumer and writer. Revoke temporary diagnostic access, take and verify the current backup, and keep the old store and old identity as a matching recovery pair.

2. Mount the active store read-only at /var/lib/ahri-tre/secrets, an empty distinct writable staging volume at /var/lib/ahri-tre/secrets-rotation, and the current/next identities at their fixed paths.

3. Run:

   ahri-tre --config /etc/ahri-tre/config.toml secrets rotate-root
   

   Success means only that the staged store was completely re-encrypted and verified. The command neither activates mounts nor authorizes startup.

4. While services remain stopped, have deployment tooling switch the staged store and next identity together. Never form a mixed old/new pair.

5. In the actual new active topology, run:

   ahri-tre --config /etc/ahri-tre/config.toml secrets verify --all
   

   Start services only after this post-cutover gate passes.

If the new active pair fails verification, keep services stopped. Restore the retained old store and old identity together, repeat the same verify --all gate against that restored active topology, and start only if it passes. A double failure remains an operator recovery incident. AHRI TRE does not perform automatic rollback, remove a failed staging tree, resume a partial rotation, or export plaintext.

Backup and exact same-Deployment recovery

The initial supported backup is an offline, application-consistent procedure:

1. Quiesce every service and process capable of reading or writing the Managed store, PostgreSQL, or Lake state.
2. Snapshot the complete Application configuration, encrypted Managed-store directory and audit journal, PostgreSQL state, and Lake state at one reviewed recovery point. Record release identities and safe fingerprints.
3. Back up the matching X25519 identity through a separate external secret-custody channel. Never place it in or beside the ciphertext backup.
4. Mount the copied Application document, store, and separately retrieved identity into an isolated offline verification workload at the same fixed container paths and run secrets verify --all. Keep writers quiesced until this succeeds. A copied but unverified snapshot is not a completed backup.
5. Test PostgreSQL/Lake restoration and application qualification in a controlled non-production recovery exercise.

Recovery restores only the same logical Deployment: the same Deployment UUID, Application configuration, complete encrypted store/audit history, matching identity, and consistent PostgreSQL/Lake state. Deployment tooling restores files and mounts; AHRI TRE has no secrets restore command. Before any Secret-consuming service starts, run offline secrets verify --all in the restored active topology, then repeat dependency, service-readiness, and authenticated Client gates.

Changing the Deployment UUID, substituting another identity, copying selected ciphertexts into a new store, or using this procedure to promote environments fails closed and is unsupported. A new Deployment receives new trust roots and credentials.

Audit integrity and failure handling

Managed mutations append integrity-protected, safe audit records. The journal contains operation, logical reference, version, time, Deployment identity, and safe actor metadata—never Secret material. secrets verify --all checks the entire chain and every active envelope; it repairs nothing. List, inspect, human, JSON, log, and failure output remain safe to retain.

Treat missing, unreadable, permission-invalid, corrupt, identity-mismatched, Deployment-mismatched, lock-unsupported, or inspection-unavailable state as a hard failure. Do not delete staged material after an uncertain owner update, edit manifests, rename ciphertexts, skip an authority during removal, or use a force bypass. Preserve failed root-rotation staging for incident evidence and provide a newly empty destination only after reviewed disposal. Infrastructure break-glass identities and native recovery tools remain institution-owned and outside AHRI TRE configuration and Secret references.

Troubleshooting without weakening boundaries

• A retired-variable error names the variable only. Remove it from the process and deployment manifest; do not translate its value into another implicit input.
• An explicit configuration failure is final for that path. Correct the selected document, ownership, permissions, or schema; do not fall back to the canonical file.
• A preflight failure is safe selected-path evidence, not permission to create or repair a dependency. Inspect the named category through the owning platform and adapter.
• A TLS or Client-origin failure requires checking the distributed bootstrap, Deployment identity, CA overlap, DNS, and server leaf identity. There is no insecure or endpoint-override mode.
• A Managed-store integrity or identity failure keeps consumers stopped. Use offline secrets verify --all and an exact verified restore; never edit the store or audit journal.
• A root-rotation interruption keeps services stopped at the deployment tooling’s recovery marker. Resume the reviewed switch/verification procedure or restore and verify the old matching pair. Local dev-env recovery commands apply only to Local TRE.

Controls owned by the integrator

Before staging or production, design and review:

1. Identity and trust: approved OAuth/OIDC registration, redirect origins, certificate issuance and rotation, hostname verification, operator identities, and break-glass access.
2. Secret delivery: separate bootstrap, runtime, Web, Datastore, and operator capabilities; external secret custody; audit and rotation. Never bake credentials into an image or configuration document.
3. Network isolation: no public PostgreSQL or Lake endpoint; explicit workload-to-workload policy; controlled ingress and egress; separate administration and client paths.
4. Persistent data: PostgreSQL and Lake placement, encryption, retention, capacity, consistent backup, verified restore, disaster recovery, and data deletion policy.
5. Operations: health, metrics, logs, traces, alerts, on-call ownership, change control, rollback, dependency updates, and vulnerability response.
6. Governance: Study authorization, custodianship, audit/provenance, disclosure review, and any governed-compute isolation required by local policy.

Local TRE deliberately supplies no production backup, observability platform, RustFS/S3 service, JupyterHub, governed-compute worker, or container orchestrator. Its local CA and deterministic identities are test material.

Cut over retained Local TRE state

The replacement development-container migration is complete. Its legacy Compose/editor-attach and Git-volume instructions remain only in the historical one-time retirement section of README-dev-env.md; they are not a current Local, Integration, staging, or production operating path.

For the configuration-and-Secrets cutover, inspect this checkout’s retained Local TRE state through the supported interface:

./dev-env doctor
./dev-env local status

doctor checks host prerequisites only. local status reports exactly one safe action:

• no state change required: no Local deployment exists, or retained state already matches the current configuration/Secret and service fingerprint;
• service rebuild required: run ./dev-env local up to rebuild immutable services while retaining current configuration, roots, and durable volumes;
• guarded ./dev-env local reset: the retained Local deployment predates the cutover. Run the ownership-checked reset, type the printed Local identifier, then run ./dev-env local up to create the new authority layout.

Neither status nor up converts old state. Do not edit the root selector, rename volumes, copy old roots, or invoke raw Compose to bypass classification. Local up reports client ready only after an isolated workload uses the rendered Client bootstrap and an ephemeral Runtime credential for an authenticated stable-protocol HTTPS probe.

Local TRE has no supported backup, snapshot, restore, export, import, production isolation, or disaster-recovery guarantee. The Integration fixture is even narrower: ./dev-env integration run creates a command-scoped disposable PostgreSQL/Lake test environment, permits its fixture-only inputs only inside that runner, and removes successful state. Neither scope is a deployment template or production validation result.

Qualification gates

For every candidate release:

• run formatting, Clippy with warnings denied, and the full workspace tests;
• run the same Integration contract natively on linux/amd64 and linux/arm64, without QEMU or another architecture emulator;
• verify PostgreSQL TLS and the writable Lake mount at the configured container-visible Lake location through the Integration probes;
• verify declared image versions and compatibility labels, never latest;
• test invalid CA, hostname, and expiry failures;
• inspect images and runtime projections for source, private keys, credentials, and Restricted local references; and
• record platform, Docker/Compose, artifact revisions, profiles, and results without recording credentials or user identities.

Use README-dev-env.md for the repository-owned test and Local lifecycle. Production incidents and recovery must use the integrator’s reviewed runbooks, not Local TRE commands or raw Docker volume copies.

Run repository integration reproducibly from the checkout:

./dev-env doctor
./dev-env integration run

Arguments after -- may narrow Cargo tests for development, for example ./dev-env integration run -- -p ahri_tre_pgmeta; release qualification uses the complete runner. This fixture is the only documented command-scoped direct dependency exception and is not a production smoke test.

For an installed deployment, use release-packaged binaries and the final mounts—not dev-env—to record:

ahri-tre --config /etc/ahri-tre/config.toml config validate
ahri-tre --config /etc/ahri-tre/config.toml config show-effective \
  --for trusted-runtime --format json
ahri-tre --config /etc/ahri-tre/config.toml config preflight \
  --for trusted-runtime --format json
ahri-tre --config /etc/ahri-tre/config.toml secrets verify --all --format json

Repeat show-effective and preflight for Web, every admitted Execution profile, Datastore creation, and Secret administration path. Then record each running service’s documented readiness and an authenticated Client stable-protocol operation. Keep the exactly one JSON document from each reporting command as safe evidence, but do not record credentials, personal identity, private paths, or dependency-native errors.