Diagnostics And Observability
Use the reported request UUID, the owning service’s operational events, and the check’s evidence basis together. A successful transport, readable Lake directory, or previously opened Session does not establish current Datastore readiness. These diagnostics use existing process streams and Local TRE commands; no collector, dashboard, telemetry database, or external account is required.
P2.7 is complete. The coverage ledger records the delivered initial paths and validation. The fresh all-five acceptance passed Lake cases L1–L3, absent OAuth artifact case A1, expired artifact case A2, and verified cleanup. Under ADR-0019, A1 proves actual absence causes generic credential rejection. The rejected caller receives no owner-specific absence finding.
Find a reported request
From the checkout owning the Local TRE, inspect the named services:
./dev-env local status
./dev-env local logs web
./dev-env local logs trusted-runtime
Use the UUID from the Browser response’s x-request-id header or the protocol
response’s request_id. The following UUID is a recorded metadata-failure
example; replace it with the reported UUID for your deployment:
request_id=2581c020-a54b-48d3-8ecd-5dd10fa8363c
./dev-env local logs | rg --fixed-strings "$request_id"
local logs returns the last 200 redacted lines per service. local logs --follow trusted-runtime streams subsequent lines. Output includes an environment
identifier and may include Compose service prefixes; it is not a raw JSON file.
No match can mean the event is older than that window, was dropped, or belongs to
another process/deployment. It does not prove that the request never ran. Neither
command requires container IDs. Local status/log selection can initialize the
non-secret checkout identifier; it does not start services. Exit 69 means logs or
prerequisites are unavailable; 78 can mean incompatible retained checkout state.
Follow the Local lifecycle guide
for that state; diagnostic inspection does not authorize a reset.
Group matching events by service_role and process_instance_id, then compare
operation, stage, outcome, failure_category and duration_ms. Within one
process, parent_span_id points to a local span_id. Across processes, join by
request_id; there is no distributed span transport or global event ordering.
Callers can reuse UUIDs, so also use the operation, process and time window.
HTTP input UUIDs are validated/replaced and the resolved value is returned.
Correlation grants neither identity, replay rights, nor mutation idempotency.
Tracked Dataset admission adds operation_id. Join detached worker events to
admission with that ID and the initiating request ID; later polls have their own
request IDs. accepted means an Operation was admitted or read back, not that a
Dataset committed. Recovery joins by Operation ID without inventing the lost
request ID. A killed process cannot emit a trustworthy completion or orderly stop.
For a CLI workflow already authorized in an existing Session, retain stdout and
stderr separately. Here client.toml and analysis stand for your existing
Client document and Session name, not server configuration or a new Session:
umask 077
ahri-tre --config ./client.toml --session analysis domain list --format json > /tmp/tre-domain-result.json 2> /tmp/tre-domain-events.log
The result contains the protocol correlation; stderr contains CLI events. The Managed runtime has its own process stream. Do not expect it in Web’s logs. Keep authorized result payloads local; they can contain information excluded from general operational telemetry. Library and C ABI hosts own their logging sink.
Choose the diagnostic boundary
| Interface | Authority and evidence | Status and exit semantics |
|---|---|---|
ahri-tre doctor --format json (or text) | No configuration, daemon or login required. Local build and filesystem prerequisites only; remote checks are explicitly skipped. It does not create a write-test file. | Check status ok, warning, error, skipped. Any error exits 2; a required warning with --strict exits 1; otherwise 0. |
ahri-tre daemon doctor --format json | Inspects the running Managed runtime’s local process/protocol/registry state; never starts an absent daemon. Auxiliary adapter findings explain missing owner evidence. | Aggregate ready, degraded, unready, unknown; seven local checks determine it, not auxiliary adapter findings. A successful protocol response exits 0, so inspect its status too. Unreachable daemon exits 2 with safe daemon_connection guidance. |
ahri-tre --config ./config.toml config preflight --for execution-profile --profile research --format json | Operator Application document and selected Effective path. Local prerequisites, selected Secret availability, reachable endpoints, and authenticated service metadata checks where existing authority supports them. | Aggregate ready/not_ready; checks passed, failed, not_applicable. Failed required checks yield ok=false, code Operation, exit 3. Unsupported optional checks stay visible and cannot become authenticated success. |
ahri-tre --config ./client.toml session status analysis --format json | Existing Runtime-login owner and live Session; retained authentication, metadata and Lake findings only. No adapter reopen or fresh Secret resolution for evidence. | Successful lookup exits 0 even with a retained failed finding. Failed-open, closed or another owner’s Session returns not_found, exit 3. Rejected credentials deny protected findings. |
ahri-tre --config ./client.toml auth status --format json | Existing Runtime credential authorization, followed by owner-produced retained OAuth evidence when available. No implicit login or maintenance. | Valid Runtime status is HTTP 200/CLI 0 even if the OAuth finding is failed/expired. Rejected Runtime credentials return HTTP 401/CLI 6; existing rejected-credential projection cleanup still applies. |
Web GET /health | Process liveness only; independent of Runtime requests and their admission. | A running, responsive process can remain live while readiness fails. |
Web GET /ready and GET /diagnostics | Existing service-authenticated Runtime capability exchange. No PostgreSQL route, Lake mount, user Session or datastore credential. | Readiness HTTP 200/503; diagnostics HTTP 200 with check status in its body. Admission can reject either route. HTTP success alone is not readiness. |
The configuration path, profile and Session names above are placeholders for
already configured selections. Use config preflight --help to choose other
supported targets; do not supply an Application document to a user-side client.
Web paths refer to the Web service: the Local Browser image does not proxy
/ready. Use the deployed Web route and configured TLS trust, or the existing
./dev-env local test smoke wrapper for its defined Browser/Web checks.
Preflight’s filesystem_lake is directory access; lake_storage_endpoint is
remote TCP reachability; trusted_scratch is independent local posture. None
proves an authenticated object read, write success or usable catalog.
lake_catalog_authentication and lake_catalog_read remain optional
not_applicable/not_checked in operator preflight because that path has no
resolved catalog capability and opening the adapter can create scratch/TLS files.
Metadata administrator success cannot upgrade them.
authentication_provider checks TCP reachability only, with no discovery, token
exchange or validation. authentication_material, authentication_expiry and
authentication_validation stay not_checked without owner authority. Selected
client Secrets are opaque; token-shaped bytes do not become user OAuth evidence.
Preflight neither acquires a user’s credentials nor repairs permissions, migrates
metadata, writes Lake data, rotates Secrets, renews tokens or retries mutations.
Read observation fields and time
Existing check envelopes retain their own statuses and stable id, summary,
severity and safe evidence. Optional observation adds:
| Field | Meaning |
|---|---|
component | cli, filesystem, configuration, managed_runtime, trusted_runtime, authentication, session, metadata, lake |
scope | local, selected_path, owner_session, owner_login, web_runtime; these describe existing authority, not permission to probe |
observed_at_utc | UTC acquisition time, or time the decision not to check was made; rendering never refreshes it |
basis | Evidence depth from the table below; always interpret alongside check status |
origin | Optional runtime_login, session_open, workflow |
failure_category | Optional expired, unavailable, timeout, admission, authority_unavailable, connection, tls, authentication, compatibility, query |
next_action | A fixed suggested command, never executed by presentation |
Preflight also retains its outer check scope and optional category; use
observation.scope for the common authority vocabulary above. Diagnostic categories
and operational-event categories are separate closed vocabularies.
| Basis | What was observed |
|---|---|
local_build, local_state, local_filesystem | Build identity, local registry/process state, or filesystem posture |
configuration_only | A declaration; no dependency contact |
endpoint_reachability | A reachable endpoint; no authentication or adapter claim |
connection_attempt | Connection/TLS/authentication attempted, without established usable authority |
authenticated_adapter_observation | Evidence from the authorized adapter operation named by the finding |
authenticated_service_compatibility | Service exchange and required protocol capabilities checked; inspect status for compatibility |
historical_observation | Retained prior evidence; no new probe |
not_checked | No observation at that depth, including missing authority, failed prerequisites or admission |
Calculate age from the original observed_at_utc and your display time; don’t
substitute the time of the status response. Session-open catalog evidence is
historical even in the opening response. Current workflow failures can report
their own authenticated observations; subsequent Session inspection preserves
their acquisition time and presents them as historical. Missing evidence stays
skipped, not passed. Session findings use passed, warning, failed, skipped.
OAuth evidence at data.oauth_artifact_observation has a finding and separate
evaluated_at_utc: status may evaluate expiry now while retaining the original
artifact capture time (origin=runtime_login, scope=owner_login). This does not
validate the token with its provider. Runtime credential validity and OAuth
artifact expiry are distinct. Only the owning login can receive the observation.
Retrieve the shipped JSON contracts without login:
ahri-tre schema get cli.doctor.output.v1 --format json
ahri-tre schema get protocol.daemon.readiness.v2 --format json
ahri-tre schema get cli.config.preflight.v1 --format json
ahri-tre schema get protocol.session.lifecycle.v2 --format json
ahri-tre schema get cli.auth.status.v1 --format json
ahri-tre schema get protocol.runtime.credential.status.v2 --format json
cli.daemon.lifecycle.v1 describes private local lifecycle output and
cli.diagnostics.output.v1 generic local diagnostics. Consumers must accept the
documented optional observation fields; older reports may omit them. Fixed actions
are ahri-tre doctor, ahri-tre daemon doctor, ahri-tre config preflight --help,
ahri-tre auth status, ahri-tre session status, and
./dev-env local logs trusted-runtime. Supply your existing configuration and
Session selection where required. Advice to reauthenticate is an explicit next
step, never an automatic action taken by inspection.
Deadlines and partial results
Preflight allows five seconds per complete probe and fifteen seconds per active report, including DNS. Four process-wide worker permits bound outstanding work; timed-out native calls retain permits until they really finish. Independent checks and completed metadata stages survive another stage’s timeout. Unperformed checks retain timeout, admission or not-checked explanations. These budgets do not promise preemption of arbitrary native filesystem/configuration loading.
daemon doctor bounds the complete local exchange to five seconds, including
silent, trickled and oversized responses. Its transport also enforces the
one-MiB response-frame limit before waiting for a newline. Reused synchronous
local filesystem operations still depend on OS responsiveness.
Web uses the existing ten-second Runtime deadline for DNS, TLS, exchange and
validation. Readiness and diagnostics share the configured anonymous Browser
read budget regardless of cookies: defaults are six concurrent checks, burst 30
and 300 checks/minute. Six resolver slots remain occupied by uninterruptible
lookups after timeout. /health remains independent. A timeout or cancelled
forwarding event does not assert that remote/native work stopped or rolled back.
The Managed Runtime HTTP attempt also retains its ten-second deadline; no new
general workflow deadline is introduced by logging.
Representative investigations
These examples are verified against the recorded public acceptance runs linked
below, not faults to inject into a retained Local TRE. Begin with local status,
search the reported request, then inspect the owning service:
| Symptom | Read the evidence | Safe next step and acceptance |
|---|---|---|
| Metadata workflow fails, or Discovery returns an empty projection | ./dev-env local logs trusted-runtime: inspect metadata, metadata_connection or metadata_query. connection, tls, authentication, compatibility and query distinguish boundaries. HTTP 200/transport success can coexist with an adapter failure or omitted candidate. | Compare the selected operator preflight; ./dev-env local logs postgres locates the service boundary. Discovery ticket 02 and Managed metadata ticket 04 passed real login/TLS/query/timeout faults. |
| Runtime credential or OAuth use fails | Runtime events distinguish provider_discovery, token_exchange, credential_validation, authentication, and authorization; denial is not necessarily an outage. Inspect existing owner auth status and, for provider problems, ./dev-env local logs oidc. | A2 passed with an expired real OAuth artifact and a still-valid distinct Runtime credential: status exits 0 but the finding is failed/expired. Follow explicit reauthentication advice. Tested walkthrough records capture/evaluation times and denial controls. A1 passed first-request generic rejection after fixture-proven absence; a generic 401 or preflight not-checked finding does not identify absence to the caller. |
| Lake storage is readable but Session opening fails | Compare independent preflight storage posture with session.lake.catalog_authentication and session.lake.catalog_read. The Runtime events use lake_catalog_authentication and lake_catalog_read. | L2 passed: positively identified login rejection leaves read not checked. L3 passed: authentication succeeds and the actual read fails/category query. An arbitrary attach failure leaves authentication unestablished, not labelled bad credentials. Use config preflight --help and Runtime logs. L1–L3 commands/results also show repeated status retaining old success during a catalog fault. |
| Web is live but requests or readiness fail | ./dev-env local logs web and ./dev-env local logs trusted-runtime: locate the same request, inspect transport, compatibility, timeout or admission. /diagnostics can be HTTP 200 with a failed Runtime check. | Check the named Runtime’s status/logs and deployed service connection. Web ticket 12 and ticket 14 passed unavailable/rejected/incompatible/stalled Runtime cases while liveness stayed independent. Web gains no Datastore fallback. |
L1 successes carry catalog findings in data.session.adapter_observations;
L2/L3 failures use error.adapter_observations, HTTP 200 protocol failure,
operation_failed, CLI 3. No Session is manufactured for a failed open.
The ticket 16 authorization trace
explains why explicit owner Session opening can acquire catalog evidence while
later inspection cannot open adapters. The ticket 19 walkthrough
provides the automated disposable reproduction, fault setup, public text/JSON
commands, protected-state checks and verified cleanup. The fresh report’s
ok=true and five_case_acceptance=PASS require all five cases and cleanup.
A1 combines validated exact-owner absence with first-request HTTP 401 and CLI
exit 6, preserving generic rejection and expected security cleanup. Earlier
reports retain their original A1 BLOCKED result; they are historical evidence.
Operational event contract, version 1
ahri_tre_observability owns the event contract and bounded executable sink;
ahri_tre_security owns sensitive-material policy. CLI/Managed runtime/Web/Trusted
runtime initialize newline-delimited JSON stderr logging before bootstrap.
Stdout remains command results or the defined data stream. Human usage errors
may still appear on stderr. Libraries and the C ABI emit passive tracing events
and never replace the host subscriber. No ambient logging filter such as
RUST_LOG changes these fixed defaults; logging setup failure leaves execution
available without this subscriber.
| Field group | Version-1 fields |
|---|---|
| Every event | schema_version=1, UTC RFC3339 timestamp, severity, event, service_role, package_version, process UUID process_instance_id, release-owned summary |
| Known configuration only | deployment_id, configuration_fingerprint, configuration_schema_version, application_version copied from immutable Effective configuration |
| Request stage | UUID request_id, closed operation, UUID span_id, optional local parent_span_id, stage, truncated |
| Completion | outcome, monotonic duration_ms, optional failure_category and already-public error_code |
| Optional request evidence | UUID operation_id, existing retry attempt, owner acquisition observed_at, numeric evidence measurements returned_records, rows, columns |
| Process lifecycle / loss | phase and local span_id, optional Operation join, completion fields when completed; loss recovery uses dropped_events |
Unknown provenance is omitted, especially before configuration resolution.
Client processes do not claim an Application fingerprint. Enrichment neither
reopens documents/Secret stores nor hashes Secret material. Version 1 has no
Datastore identity field; do not infer one from a resource name. timestamp
locates event emission; observed_at records acquisition before delivery;
duration_ms uses a monotonic clock. Nested stage times can overlap and must not
be added to invent total elapsed time. Runtime total starts before credential
validation, after HTTP decoding. An attempt number describes an existing retry
under the request; telemetry never adds retries.
| Vocabulary | Values |
|---|---|
| Service | cli, managed_runtime, web, trusted_runtime |
| Event | operation_started, operation_completed, process_phase_started, process_phase_completed, process_ready, bootstrap_failed, recovery_unavailable, connection_rejected, administration_rejected, events_dropped |
| Outcome | success, accepted, skipped, rejected, unavailable, timeout, cancelled, internal_failure, interrupted |
| Failure category | legacy_environment, configuration, secret, listener, interrupted, metadata, storage, catalog, cleanup, commit_unknown, provider, credential_invalid, credential_expired, admission, adapter, authentication, authorization, connection, tls, query, compatibility, dependency, internal |
| Transport/workflow stage | cli, managed_runtime, web, transport, trusted_runtime, application, authentication, authorization, admission, session, secret_capability, workflow_entry, metadata_preflight, metadata_decision, preparation, metadata_commit, compensation |
| Adapter stage | provider_discovery, token_exchange, credential_validation, metadata, metadata_connection, metadata_query, lake, lake_storage, lake_read, lake_catalog_authentication, lake_catalog_read |
| Phase | configuration, injected_secrets, managed_secrets, dependencies, recovery, recovery_cleanup, listener, serving, shutdown, session_drain, maintenance |
Operation names are the 20 fixed Discovery/search actions,
browser_access_request_submit/list/get/withdraw and
browser_custodian_access_request_list/approve/reject (each slash denotes a
separate suffix), domain_list, session_open/status/close,
runtime_login_start/callback/poll/acknowledge,
runtime_credential_status/rotate/logout, dataset_from_datafile,
operation_list/get/cancel/result_get, runtime_capabilities, and other_request.
Unlisted request kinds use the last fixed name, never input-derived text.
The crate reference explains
stage placement and governance outcomes for each implemented workflow.
Starts, success, accepted and skipped outcomes are info; internal failures are
error; other completion outcomes are warn. process_ready is info and the other
standalone process signals are warn. Successful probe/status traffic suppresses
redundant successful child stages and emits at most one summary at its owning
boundary. Detailed workflow events are bounded by operation/stage/attempt, not
rows, chunks, Variables or search matches. A successful child never proves the
parent mutation committed. Human summaries are not the machine contract.
Only allowlisted fields are serialized. Credentials, cookies, tokens, private keys, personal identities, Session references, Restricted local references, raw targets/query strings, SQL, resource names, request/rejection text and governed content are excluded. Pipeline logs remain governed content and never enter operational telemetry. Configuration/Secret ownership, OAuth/libpq separation and owner authorization remain unchanged; there is no diagnostic login/repair, new Web datastore probe or new user-side server authority.
Delivery limits
Each JSON event is at most 8192 bytes, excluding its newline. Optional numeric
evidence is limited to 256 measurements on ingress. Oversize envelopes omit all
optional evidence and set truncated=true; required JSON and typed IDs are never
byte-truncated. The writer has 256 queued records plus at most one in flight.
Producers enqueue without blocking and drop the newest event when full or
disconnected. A blocked stderr stalls only the dedicated writer. Failed writes
are discarded without retries. Loss counts saturate at u64::MAX; after a
successful write, events_dropped reports accumulated loss in dropped_events.
A failed recovery report retains the count for a later successful recovery.
Executable normal return attempts a bounded 100 ms flush; destruction never joins a blocked writer. Abrupt termination can lose queued records. A partial sink write can damage a line; recovery terminates that prefix before further JSON, and readers must discard damaged lines. There is no durable delivery, retention, replay or audit guarantee. Delivery failures do not determine workflow or governance results. The pressure acceptance record covers blocked, failed and recovering sinks, concurrent requests, host-owned FFI logging, and unchanged CLI data stdout with constant event counts.