Recovering a Conformance Candidate After Runtime Key Loss

Use this procedure when Secret preparation stops because the Runtime private key is missing or does not match the Runtime certificate in the 0.3.14 installed-conformance candidate. It creates a fresh Runtime key, issues a new Runtime certificate beneath the retained deployment CA, regenerates the site bundle, and composes a replacement candidate archive and checksum.

This is a pre-conformance recovery. Do not use it after any server, WSL2, macOS, recovery, cleanup, or finalization phase has consumed the candidate. Once any phase has started, preserve that candidate and its evidence as one immutable attempt instead of replacing its bytes.

The procedure retains all identities and public authorities except the Runtime leaf key pair:

• kit 0.3.14 and release 0.10.8;
• host svrltreapcc02 at 192.168.31.75;
• Runtime DNS name runtime.svrltreapcc02.home.arpa;
• Deployment f2ef37c5-7430-468a-a439-b3ba1b0527c1;
• Datastore ahri-tre-test;
• the existing deployment CA and PostgreSQL CA;
• the existing server and client package bytes; and
• the fixed 0.3.12 predecessor and its checksum.

The existing clients can be retained because they authenticate the Runtime through the deployment CA rather than pinning the Runtime leaf certificate. This procedure proves that their complete generated public input surface is unchanged before reusing them.

Run WSL2 commands in the ordinary Ubuntu shell. Run Cargo commands in the persistent VS Code development container. Never display a private key, put a CA passphrase on a command line, or copy private material into the repository, candidate archive, terminal log, screenshot, or evidence directory. Keep the WSL2 shell open while using the separate development-container terminal so that its reviewed path variables remain in scope.

Guided recovery wizard

The dedicated wizard performs Sections 1 through 12 in order. Run it from the repository root in the ordinary WSL2 Ubuntu shell:

./scripts/recover-installed-conformance-candidate-wizard.sh

The wizard fixes the candidate, predecessor, host, address, Deployment, Datastore, and public identities to the values documented below. It asks for confirmation before retiring the old public leaf inputs, advancing the deployment CA serial, and activating the replacement candidate. The CA-key passphrase is read only by OpenSSL and is never stored. The wizard pauses while the operator runs the displayed package-site command in the persistent development container, then independently verifies its output.

Successful completion prints CANDIDATE READY. It does not install a package, invoke the conformance harness, or prepare the MinisForum host. Continue with the Secret-preparation wizard only after that result. The detailed procedure below remains the canonical explanation of every wizard action and the recovery reference if an interrupted run leaves guarded output paths behind. If an interruption leaves only the verified recovery source plus a protected Runtime key and matching public CSR, rerunning the wizard verifies and reuses those exact inputs instead of generating another key. If an interruption occurs after package-site but before the clients are copied, rerunning also recognizes the complete issued identity and a component root containing exactly server and site. It reverifies those artifacts, does not reopen the CA signing key, does not advance the CA serial, and resumes at the Stage 9 verification boundary.

1. Confirm the recovery boundary

In WSL2, confirm that the preparation wizard stopped before declaring the host ready and that no conformance phase used the current candidate. Do not proceed unless both statements are true.

Set the fixed paths:

cd /home/kobus/repos/ahri-tre-rs
repository_root="$PWD"
input_root="$HOME/ahri-tre-conformance/input"
archive="ahri-tre-test-datastore-deployment-kit-0.3.14.tar.gz"
checksum="$archive.sha256"
predecessor="ahri-tre-test-datastore-deployment-kit-0.3.12.tar.gz"
predecessor_checksum="$predecessor.sha256"
runtime_name="runtime.svrltreapcc02.home.arpa"

Verify the current candidate and the fixed predecessor before using either as a recovery input:

test -s "$input_root/$archive"
test -s "$input_root/$checksum"
test -s "$input_root/$predecessor"
test -s "$input_root/$predecessor_checksum"

cd "$input_root"
sha256sum --check --strict "$checksum"
grep -qxF \
  "6f5555c25d96274d7772d1b46d4409bd36a42205f21c4c715de8df5179ab18ed  $predecessor" \
  "$predecessor_checksum"
sha256sum --check --strict "$predecessor_checksum"

All checks must report OK. Return to the repository and extract the current candidate into a new private recovery directory:

cd "$repository_root"
source_parent="$repository_root/dist/conformance-candidate-recovery-source-0.3.14"
test ! -e "$source_parent"
install -d -m 0700 "$source_parent"
tar --extract --gzip \
  --file "$input_root/$archive" \
  --directory "$source_parent" \
  --no-same-owner \
  --no-same-permissions
source_root="$source_parent/ahri-tre-test-datastore-deployment-kit-0.3.14"

test -f "$source_root/SHA256SUMS"
(cd "$source_root" && sha256sum --check --strict SHA256SUMS)

Do not alter this extracted source. It supplies the already verified server and clients and the old site inputs used for byte comparisons.

2. Verify the retained deployment CA

The recovery reuses the existing deployment CA that was previously generated to issue certificates for this MinisForum deployment. It does not create or replace a CA. The CA private key remains protected in WSL2; the MinisForum uses certificates and public trust issued beneath it, not this signing key. Confirm the protected and public inputs:

umask 077
pki_root="$HOME/ahri-tre-pki"
ca_key="$pki_root/ca/ca-private-key.pem"
ca_certificate="$pki_root/ca/deployment-ca-certificate.pem"
ca_serial="$pki_root/ca/deployment-ca-certificate.srl"
ca_chain="$pki_root/public/deployment-ca-chain.pem"

test -s "$ca_key"
test -s "$ca_certificate"
test -s "$ca_serial"
test -s "$ca_chain"
test "$(grep -c 'BEGIN CERTIFICATE' "$ca_chain")" -eq 1
cmp --silent "$ca_certificate" "$ca_chain"
openssl verify -CAfile "$ca_chain" "$ca_certificate"

Confirm that the private signing key belongs to that public CA. OpenSSL will prompt for the passphrase chosen when this existing deployment CA was originally generated; it is not asking you to create a new password or enter a Runtime password. Enter the existing CA-key passphrase interactively:

ca_key_sha256="$({
  openssl pkey -in "$ca_key" -pubout -outform DER
} | sha256sum | cut -d ' ' -f 1)"
ca_certificate_sha256="$({
  openssl x509 -in "$ca_certificate" -pubkey -noout |
    openssl pkey -pubin -outform DER
} | sha256sum | cut -d ' ' -f 1)"
test "$ca_key_sha256" = "$ca_certificate_sha256"
unset ca_key_sha256 ca_certificate_sha256

The final test must return silently. A failure means this is not the signing authority embedded in the existing candidate; stop without generating a leaf certificate.

3. Retire the superseded public Runtime inputs

Preserve the old public leaf material for diagnosis while keeping it out of the active input paths:

retired_runtime="$pki_root/retired/runtime-leaf-before-conformance-$(date -u +%Y%m%dT%H%M%SZ)"
install -d -m 0700 "$retired_runtime"

for path in \
  "$pki_root/runtime-certificate.csr" \
  "$pki_root/runtime-certificate.ext" \
  "$pki_root/public/runtime-certificate.pem" \
  "$pki_root/public/runtime-certificate-chain.pem"; do
  if test -e "$path"; then
    mv -- "$path" "$retired_runtime/"
  fi
done

This does not retire or replace the deployment CA, PostgreSQL CA, or their private keys.

4. Generate the fresh Runtime key and request

The preparation wizard requires the protected Runtime key at this exact WSL2 path:

/home/kobus/ahri-tre-pki/private/runtime-private-key.pem

Create it only if it is absent:

runtime_key="$pki_root/private/runtime-private-key.pem"
runtime_csr="$pki_root/runtime-certificate.csr"
runtime_extension="$pki_root/runtime-certificate.ext"
runtime_certificate="$pki_root/public/runtime-certificate.pem"
runtime_chain="$pki_root/public/runtime-certificate-chain.pem"

install -d -m 0700 "$pki_root/private" "$pki_root/public"
test ! -e "$runtime_key"
test ! -e "$runtime_csr"
test ! -e "$runtime_extension"
test ! -e "$runtime_certificate"
test ! -e "$runtime_chain"

openssl genpkey \
  -algorithm RSA \
  -pkeyopt rsa_keygen_bits:3072 \
  -out "$runtime_key"
chmod 0600 "$runtime_key"
openssl pkey -in "$runtime_key" -check -noout

The key is deliberately not passphrase-encrypted because the Runtime service cannot answer a prompt. Its protected WSL2 storage and the wizard’s guarded Runtime-only projection provide its access boundary.

Create and verify a certificate-signing request:

openssl req \
  -new \
  -sha256 \
  -key "$runtime_key" \
  -subj "/CN=$runtime_name" \
  -addext "subjectAltName=DNS:$runtime_name" \
  -addext "keyUsage=critical,digitalSignature,keyEncipherment" \
  -addext "extendedKeyUsage=serverAuth" \
  -out "$runtime_csr"

openssl req -in "$runtime_csr" -verify -noout
openssl req -in "$runtime_csr" -noout -subject
openssl req -in "$runtime_csr" -noout -text | grep -F "DNS:$runtime_name"

The final output must show DNS:runtime.svrltreapcc02.home.arpa.

5. Sign and verify the Runtime certificate

Create the certificate extension file:

nano "$runtime_extension"

Paste exactly:

[server_certificate]
basicConstraints = critical, CA:false
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @subject_alt_names
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer

[subject_alt_names]
DNS.1 = runtime.svrltreapcc02.home.arpa

Save and exit. Issue the certificate for 365 days beneath the retained CA:

openssl x509 \
  -req \
  -sha256 \
  -days 365 \
  -in "$runtime_csr" \
  -CA "$ca_certificate" \
  -CAkey "$ca_key" \
  -CAserial "$ca_serial" \
  -extfile "$runtime_extension" \
  -extensions server_certificate \
  -out "$runtime_certificate"
chmod 0644 "$runtime_csr" "$runtime_extension" "$runtime_certificate"

Enter the CA-key passphrase only at the OpenSSL prompt. Verify the certificate chain, hostname, purpose, and dates:

openssl verify \
  -CAfile "$ca_chain" \
  -purpose sslserver \
  "$runtime_certificate"
openssl x509 \
  -in "$runtime_certificate" \
  -noout \
  -checkhost "$runtime_name"
openssl x509 \
  -in "$runtime_certificate" \
  -noout \
  -subject \
  -issuer \
  -dates \
  -ext subjectAltName

Prove that the certificate contains the public key derived from the new private key:

runtime_key_sha256="$({
  openssl pkey -in "$runtime_key" -pubout -outform DER
} | sha256sum | cut -d ' ' -f 1)"
runtime_certificate_sha256="$({
  openssl x509 -in "$runtime_certificate" -pubkey -noout |
    openssl pkey -pubin -outform DER
} | sha256sum | cut -d ' ' -f 1)"
test "$runtime_key_sha256" = "$runtime_certificate_sha256"
unset runtime_key_sha256 runtime_certificate_sha256

Because this deployment CA has no intermediate CA, the Runtime chain contains only the leaf certificate:

install -m 0644 "$runtime_certificate" "$runtime_chain"
openssl verify -CAfile "$ca_chain" "$runtime_chain"
test "$(grep -c 'BEGIN CERTIFICATE' "$runtime_chain")" -eq 1
if grep -q 'PRIVATE KEY' "$runtime_chain" "$ca_chain"; then
  echo 'STOP: a public certificate input contains private-key material'
  false
fi

6. Stage the public packager inputs

Only the public Runtime and deployment chains enter the repository working tree. They are local ignored build inputs and must not be committed:

cd "$repository_root"
install -d -m 0755 pki
install -m 0644 "$runtime_chain" pki/runtime-certificate-chain.pem
install -m 0644 "$ca_chain" pki/deployment-ca-chain.pem

openssl verify \
  -CAfile pki/deployment-ca-chain.pem \
  pki/runtime-certificate-chain.pem
cmp --silent pki/deployment-ca-chain.pem \
  "$source_root/site/public-ca-chain.pem"
git check-ignore pki/runtime-certificate-chain.pem \
  pki/deployment-ca-chain.pem

Both paths must be reported by git check-ignore. A CA comparison failure is authority drift, not Runtime leaf renewal; stop and do not reuse the existing clients.

7. Prepare a fresh component root

Still in the ordinary WSL2 shell, create a new component root and copy only the verified server publication into it:

new_component_root="$repository_root/dist/conformance-candidate-rebuild-0.3.14"
test ! -e "$new_component_root"
install -d -m 0700 "$new_component_root"
cp -a "$source_root/server" "$new_component_root/server"

test -s \
  "$repository_root/dist/private-kit-inputs-0.3.14/site-inputs.minisforum.v3.json"
test -s \
  "$repository_root/dist/private-kit-inputs-0.3.14/postgresql-ca-chain.pem"
test -s \
  "$repository_root/dist/release-v0.10.8/validator/release-metadata.json"

The component root contains no installed package and must not be transferred to a conformance host.

8. Regenerate the site bundle

Switch to the persistent VS Code development-container terminal. Run:

cd /workspaces/ahri-tre-rs

cargo run --locked -p xtask -- test-datastore-kit package-site \
  --contract deployment/test-datastore-kit/minisforum-contract-v0.3.14.json \
  --inputs dist/private-kit-inputs-0.3.14/site-inputs.minisforum.v3.json \
  --runtime-certificate-chain pki/runtime-certificate-chain.pem \
  --public-ca-chain pki/deployment-ca-chain.pem \
  --postgresql-ca-chain dist/private-kit-inputs-0.3.14/postgresql-ca-chain.pem \
  --server-component-manifest dist/conformance-candidate-rebuild-0.3.14/server/component-versions.json \
  --validator-release-root dist/release-v0.10.8/validator \
  --artifact-root dist/conformance-candidate-rebuild-0.3.14

The command must report kit 0.3.14, release 0.10.8, host svrltreapcc02, and Datastore ahri-tre-test. It must not install any package or invoke the conformance harness.

9. Prove the regenerated site matches the fixed candidate identity

Return to the ordinary WSL2 shell:

cd "$repository_root"
new_component_root="$repository_root/dist/conformance-candidate-rebuild-0.3.14"
new_site="$new_component_root/site"

jq -e '
  .kit_version == "0.3.14" and
  .server.artifact_base_release == "0.10.8" and
  .server.artifact_source_revision ==
    "a292eab3d21a02060b115927da3266f62ad982a0" and
  .hostname == "svrltreapcc02" and
  .ipv4_address == "192.168.31.75" and
  .runtime_dns_name == "runtime.svrltreapcc02.home.arpa" and
  .deployment_id == "f2ef37c5-7430-468a-a439-b3ba1b0527c1" and
  .datastore_id == "ahri-tre-test"
' "$new_site/site-inputs.json" >/dev/null

Extract only the public Runtime leaf from the regenerated Application configuration and bind it to the new key:

runtime_review="$(mktemp -d /tmp/ahri-tre-runtime-review.XXXXXX)"
chmod 0700 "$runtime_review"
python3 -c \
  'import sys,tomllib; document=tomllib.load(open(sys.argv[1], "rb")); print(document["services"]["trusted_runtime"]["certificate_chain"][0], end="")' \
  "$new_site/application.toml" \
  >"$runtime_review/runtime-certificate.pem"

openssl verify \
  -CAfile "$new_site/public-ca-chain.pem" \
  "$runtime_review/runtime-certificate.pem"
openssl x509 \
  -in "$runtime_review/runtime-certificate.pem" \
  -noout \
  -checkhost "$runtime_name"
issued_certificate_sha256="$(
  openssl x509 -in "$runtime_certificate" -outform DER |
    sha256sum | cut -d ' ' -f 1
)"
new_certificate_sha256="$(
  openssl x509 -in "$runtime_review/runtime-certificate.pem" -outform DER |
    sha256sum | cut -d ' ' -f 1
)"
test "$issued_certificate_sha256" = "$new_certificate_sha256"

new_certificate_public_key_sha256="$({
  openssl x509 -in "$runtime_review/runtime-certificate.pem" -pubkey -noout |
    openssl pkey -pubin -outform DER
} | sha256sum | cut -d ' ' -f 1)"
new_key_public_key_sha256="$({
  openssl pkey -in "$runtime_key" -pubout -outform DER
} | sha256sum | cut -d ' ' -f 1)"
test "$new_certificate_public_key_sha256" = "$new_key_public_key_sha256"
unset issued_certificate_sha256 new_certificate_sha256
unset new_certificate_public_key_sha256 new_key_public_key_sha256
rm -rf -- "$runtime_review"

The certificate digest comparison uses canonical DER bytes so that harmless PEM trailing-newline differences do not look like certificate drift. The separate public-key comparison proves that the same certificate is bound to the freshly generated Runtime private key.

The temporary directory has an explicit /tmp/ahri-tre-runtime-review. prefix and contains public certificate material only.

10. Prove the existing client packages remain exact

Compare every site input bound into the WSL2 and macOS client manifests:

for relative_path in \
  client.toml \
  public-ca-chain.pem \
  client-publication/wsl2.json \
  client-publication/macos.json; do
  cmp --silent \
    "$source_root/site/$relative_path" \
    "$new_site/$relative_path"
done

All comparisons must return silently. If any comparison fails, stop. That is broader site or trust drift and requires newly packaged clients; it is not the leaf-key recovery described here.

Copy the unchanged, checksum-verified client publications into the fresh component root and verify them again:

cp -a "$source_root/clients" "$new_component_root/clients"
(cd "$new_component_root/clients/wsl2" && \
  sha256sum --check --strict SHA256SUMS)
(cd "$new_component_root/clients/macos" && \
  sha256sum --check --strict SHA256SUMS)

test "$(find "$new_component_root" -mindepth 1 -maxdepth 1 \
  -printf '%f\n' | sort | paste -sd ' ' -)" = "clients server site"
test -z "$(find "$new_component_root" -type l -print -quit)"

11. Compose and verify the replacement candidate

Compose into a new protected staging directory. Do not overwrite the current candidate yet:

candidate_staging="$HOME/ahri-tre-conformance/candidate-staging-0.3.14-runtime-reissue"
test ! -e "$candidate_staging"
install -d -m 0700 "$candidate_staging"

cd "$repository_root"
./scripts/compose-installed-conformance-candidate.sh \
  --component-root "$new_component_root" \
  --output-directory "$candidate_staging"

cd "$candidate_staging"
sha256sum --check --strict "$checksum"

Verify the complete internal checksum inventory in a fresh temporary extraction:

candidate_review="$(mktemp -d /tmp/ahri-tre-candidate-review.XXXXXX)"
chmod 0700 "$candidate_review"
tar --extract --gzip \
  --file "$candidate_staging/$archive" \
  --directory "$candidate_review" \
  --no-same-owner \
  --no-same-permissions
candidate_root="$candidate_review/ahri-tre-test-datastore-deployment-kit-0.3.14"

(cd "$candidate_root" && sha256sum --check --strict SHA256SUMS)
test -x "$candidate_root/conformance/run.sh"
test -f "$candidate_root/clients/wsl2/ahri-tre-client.tar"
test -f "$candidate_root/clients/macos/ahri-tre-client.tar"
jq -e '
  .kit.version == "0.3.14" and
  .release.version == "0.10.8" and
  .qualification_evidence_included == false and
  .validation.outcome == "go"
' "$candidate_root/candidate-manifest.json" >/dev/null

rm -rf -- "$candidate_review"

This verification reads package files but does not install them or invoke the conformance harness.

12. Activate the exact replacement bytes

Retire the old candidate only after the replacement archive and checksum have passed both verification layers:

retired_candidate="$HOME/ahri-tre-conformance/retired/candidate-before-runtime-reissue-$(date -u +%Y%m%dT%H%M%SZ)"
install -d -m 0700 "$retired_candidate"

test -s "$input_root/$archive"
test -s "$input_root/$checksum"
(cd "$input_root" && sha256sum --check --strict "$checksum")
mv -- "$input_root/$archive" "$retired_candidate/"
mv -- "$input_root/$checksum" "$retired_candidate/"

mv -- "$candidate_staging/$archive" "$input_root/"
mv -- "$candidate_staging/$checksum" "$input_root/"

cd "$input_root"
sha256sum --check --strict "$checksum"
sha256sum --check --strict "$predecessor_checksum"

The two final checks must report OK. From this point onward, the replacement candidate archive and its adjacent checksum are the only 0.3.14 bytes used by the server, WSL2, macOS, recovery, cleanup, and finalization phases. Keep the fixed predecessor archive and checksum unchanged.

Return to the repository root and restart Secret preparation:

cd "$repository_root"
./scripts/minisforum-conformance-preparation-wizard.sh

The wizard will independently verify that the new Runtime private key matches the certificate embedded in the replacement candidate. It still stops after printing HOST READY; it never invokes the conformance harness.