the_prg_server_configuration_old/secrets.md

Secrets – managing encrypted secrets with sops (age)

This document explains how we manage secrets for the PRG server configuration using sops and the age encryption protocol. It keeps the original workflow and file layout but presents it in a more structured, readable format with examples and useful tips.

Note

This file documents repository-side secrets management only. Do not paste private keys or unencrypted secrets into the repository. Keep private key material on the host machines or in a secure vault.

Quick overview

We encrypt secrets with sops using the age tool. The repository contains a .sops.yaml that documents which public recipients can decrypt which files (via creation_rules). Encrypted files are safe to commit to git; only machines with the corresponding private age keys can decrypt them.

Files and conventions (what to look for)

• .sops.yaml - repository configuration that lists public age recipients and creation_rules that map file paths to key groups.
• nix-system-configs/secrets/... - the repository path where secrets are stored, organized by subsystem (for example database, mail, songsheet, build_machine, traefik, etc.).
• Nix module files that use sops (examples in secrets-config - those show how secrets are exported into NixOS as file or string secrets.

Inspect the repository's .sops.yaml (example shown later) and the secrets folders before creating or editing secrets.

Creating and registering age keys (on the machine that will hold the private key)

Tip

Prefer creating the age key on the machine that needs to decrypt the secret (for example the server). Keep the private key on that machine only.

Example using nix-shell (macOS or Nix-enabled machine):

# open a temporary shell with `age` available, create a dir for sops-nix keys and generate a key
nix-shell -p age --run 'sudo mkdir -p /var/lib/sops-nix && sudo chown $USER /var/lib/sops-nix && age-keygen -o /var/lib/sops-nix/key.txt'

# The generated file contains a public and private key pair. Copy the PUBLIC key (the line starting with "age1...") into the repository's .sops.yaml.
# DO NOT add the private key to the repo. Keep it under /var/lib/sops-nix on the machine.

What the generated key file contains (example):

• private key: starts with AGE-SECRET-KEY- (keep private)
• public key (recipient): looks like age1... - this is what you add to .sops.yaml

Inspecting / editing .sops.yaml

Note

.sops.yaml controls which public keys can decrypt files and where new secret files should live. When adding a public key, add the recipient under the keys: list and update creation_rules: if needed.

Example .sops.yaml entries (repository already contains similar entries):

keys:
  - &admin_one age1x...
  - &server_two age1kzsr...
  - &server_mail age1p...
creation_rules:
  - path_regex: nix-system-configs/secrets/example_project/[^/]+\.(yaml|json|env|ini)$
    key_groups:
      - age:
          - *admin_one
          - *server_two

Notes:

• The repository's actual .sops.yaml uses YAML anchors (for example &admin_christine) and path-specific creation_rules for songsheet, traefik, database, christine, wireguard, build_machine, and mail.
• path_regex is matched relative to the repository root. Keep the files under nix-system-configs/secrets/... for consistency with existing rules.

Creating a new encrypted secret file

1. Ensure the public recipient(s) for the intended end systems exist in .sops.yaml.
2. Use sops to create or edit an encrypted file. If sops cannot find an age key in the usual locations, pass SOPS_AGE_KEY_FILE pointing to the generated private key (only when editing locally with your private key present).

Example (editor micro) - note we use a temporary environment so sops can locate the key you generated locally:

# from the repository root, create/edit a new secret for the build machine
EDITOR=micro nix-shell -p sops --run 'SOPS_AGE_KEY_FILE=$HOME/.config/sops/age/keys.txt sops nix-system-configs/secrets/example_project/secrets.yaml'

What you will see inside the new file while editing:

# plain YAML content (sops will encrypt on save)
runner_token: "example-runner-token"
# any other keys as required by your service

After saving, sops will replace plaintext with encrypted blocks and add a sops: metadata section - commit this encrypted file to git.

Example secret file

The repository includes secrets.yaml (at nix-system-configs/secrets/build_machine/secrets.yaml) which contains an encrypted example plus the sops metadata. That file shows the encrypted enc blocks and recipients under sops:, thus you can use it as a reference for structure.

How to add a new secrets set and wire it into Nix (step‑by‑step)

This is a practical walkthrough that follows the pattern used in nix-system-configs/modules/secrets-config/sops-database.nix and sops-mail.nix.

1. Create the secrets file in the repo

Note

Do this only as a template: paste the examples below into the appropriate repo paths when you're ready. This file intentionally keeps everything inline so you don't need separate helper files to understand the pattern.

2. Create a small sops-based Nix module for this secret set

• Copy one of the existing modules (for example sops-database.nix) and adapt it. Example module: create nix-system-configs/modules/secrets-config/sops-project_example.nix with the following content:

{
  config,
  pkgs,
  lib,
  ...
}: {...

  # This will add secrets.yaml to the Nix store; you can instead reference a path string
  sops.defaultSopsFile = ../../secrets/project_example/secrets.yaml;

  # NOTE: sops.age.keyFile is where the private age key is expected on the target system
  sops.age.keyFile = "/var/lib/sops-nix/key.txt";  # SEE NOTE ABOUT THIS
  sops.age.generateKey = true;                      # SEE NOTE ABOUT THIS

  # declare secrets that sops-nix will expose
  sops.secrets."project_example/hello" = {
    format = "yaml";
    sopsFile = ../../secrets/project_example/secrets.yaml;
    owner = "root";
    mode = "0400";
  };

  sops.secrets."project_example/example_key" = {
    format = "yaml";
    sopsFile = ../../secrets/project_example/secrets.yaml;
    owner = "root";
    mode = "0400";
  };
}

Minimal example secrets.yaml (paste to nix-system-configs/secrets/project_example/secrets.yaml or edit in-place with sops):

# project_example secrets - plaintext template (encrypt with sops before committing)
hello: "replace-with-actual-hello-value"
example_key: "replace-with-actual-example-key"

3. Import the new module into your system

• Add the new Nix module to the imports = [ ... ] block of the system configuration that should receive the secrets. For example, in your system file (e.g. nix-system-configs/modules/system/your-system.nix) add:

imports = [
  #   ...existing imports...
  ../../modules/secrets-config/sops-project_example.nix
];

• You can also import it from a central place where you manage secrets modules; the important part is that the module is available in the imports set used for the machine(s) that need it.

Note

This section is only required if you did not already generate and register age keys for the target end system. If you pregenerated the key pair and added the public recipient to .sops.yaml, you can skip these steps.

If sops.age.generateKey = true and no key exists, the module will create /var/lib/sops-nix/key.txt on the target end system when the module activates. After the module generates a key, extract the public recipient (the age1... line) and add it to the repository .sops.yaml under keys: and to the appropriate creation_rules so future sops edits include the new recipient.

Example commands to extract the public recipient (run on the target machine; use sudo if needed):

# print the whole key file (DO NOT commit the private key)
sudo cat /var/lib/sops-nix/key.txt

Important: never commit the private key (the AGE-SECRET-KEY-... line) into the repository - only the age1... public recipient belongs in .sops.yaml.

Tip

After you add a new public age1... recipient to .sops.yaml, update any existing encrypted files so the new recipient is included in each file's recipients list. Use sops updatekeys <file> to update a single file; sops must be able to decrypt the file locally (for example by having a private key available or by setting SOPS_AGE_KEY_FILE).

Examples:

# update a single file so the new recipient can decrypt it
sops updatekeys nix-system-configs/secrets/project_example/secrets.yaml

# or update all yaml secrets in the secrets tree (run from repo root)
find nix-system-configs/secrets -type f -name "*.yaml" -exec sops updatekeys {} \;

4. Reference the secret from other Nix code and services

• After the module is imported and active, sops-nix exposes each declared secret in config.sops.secrets under the quoted secret name. The repo uses the flat, quoted key names (for example "project_example/hello") so you can reference them in Nix like this:

# use the path to the secret file in another module
someService.extraConfig = {
  passwordPath = config.sops.secrets."project_example/example_key".path;
};

• If you declared a secret that should be used by a Docker Compose conversion (compose2nix), annotate the service in docker-compose.yml with the label used by compose2nix and pass --sops_file when running compose2nix:

config.sops.secrets."project_example/hello".path
config.sops.secrets."project_example/example_key".path

Then run compose2nix with the SOPS file:

compose2nix --sops_file nix-system-configs/secrets/project_example/secrets.yaml ...other args...

This instructs the generated Nix expression to reference the secret name (which compose2nix and/or your Nix code can resolve via config.sops.secrets."project_example/hello".path).

Common pitfalls & troubleshooting

• Never commit private keys. Only public age recipients belong in .sops.yaml.
• If sops fails to encrypt/decrypt locally, check that you have the corresponding private key available and SOPS_AGE_KEY_FILE points to it when running from a shell that cannot otherwise find it.
• path_regex in .sops.yaml is matched relative to the repo root - use it carefully so creation rules apply where expected.
• If sops.age.generateKey is enabled and you rely on the module to generate keys, remember you must register the generated public recipient in the repo .sops.yaml after first activation. Extract the age1... line with one of the commands above and add it to keys: and the appropriate creation_rules.
• If the Nix module expects the private key at /var/lib/sops-nix/key.txt, make sure that is where the key is placed on the target end system and that ownership/mode are correct for the system service that reads it.