the_prg_server_configuration_old/documentation_titlepage.md

Welcome to the Polyteknisk Radiogruppe's Server Documentation

This documentation outlines how the server is built and maintained, covering both short- and long-term management practices.

Note

Important: This document should be actively maintained whenever functional changes occur - no matter how small or large. This includes compilation flag changes, password key updates, new standard packages, and any other modifications.

Documentation File Structure

As this document is in alpha release, the following package has been made with the standard Rust project template in the case of having to create custom binaries for the system of the sysadmin or any related responsibilites/positions. Thusly, the following document focus should be in the nix-system-configs folder, which is the main folder for the system configuration files and scripts.

Tip

For documentation maintainers: Update the directory tree using lsd --tree from the project root. Be careful not to expose secrets and other things when documenting from your local system.

 .
├──  Cargo.lock
├──  Cargo.toml
├──  documentation_titlepage.md
├──  dry_run.zsh
├──  flake.lock
├──  flake.nix
├──  LICENSE
├──  nix-system-configs
│   ├──  build-deprecated
│   ├──  database-deprecated
│   ├──  dns
│   ├──  example-composed.nix
│   ├──  forgejo-deprecated
│   ├──  gateway
│   ├──  modules
│   ├──  old-server-notes.md
│   ├──  prg-blank-setup
│   ├──  secrets
│   └──  traefik-deprecated
├──  README.md
├──  secrets.md
└── 󱧼 src
    └──  main.rs

Module Architecture

The system is organized into reusable modules across all NixOS machines in the PRG network. The system module serves as the core configuration for Linux systems running on Proxmox, while supporting diverse deployments through modular composition.

Configuration Pattern

Use blank_system_USE_THIS_AS_COPY.nix as a template for new system configurations. This file demonstrates the standard pattern of declaring system variables using Nix syntax to create custom local options that are callable across imported modules.

Core Modules

The imports = [ ... ] block specifies which modules are attached to your system configuration. These core modules ensure consistency and functionality:

• Bootloader - bootloader (required): Handles system boot initialization. Configuration varies by installation type (ISO-assigned or Proxmox-assigned).
• Local - local (required): Provides internet connectivity, hostname resolution, DNS configuration, and user settings.
• Standard Toolset - lix-default.nix (strongly recommended): Ensures consistent tools across the PRG network—file transfers (rsync), SSH with key-based authentication, security features (U2F and SSH agent passwordless sudo), package management, and automated daily system upgrades.
• Desktop/Window Manager - desktop-manager (optional): Simplifies VM interaction within Proxmox and supports systems requiring graphical interfaces.
• Toolsets - toolsets (optional): Contains reusable configurations for specific tools or services that may be needed across multiple systems, such as remote building configurations.
• Secrets Config secrets-config (required when needing password management): Manages sensitive information like database credentials and Traefik secrets using SOPS-encrypted Nix files (see this and that). This module is essential for systems that require secure handling of secrets.
• Docker Containers - songsheet (optional): This special module handles Docker Compose files that are converted to Nix. Currently it is used for a single VM but can be expanded to contain all compose2nix generated configurations.
• System Scripts - system_scripts (variable): system_scripts contains scripts (for example, database backup scripts) that are often easier to write and maintain in shell than in Nix.
• Styling Scripts - styling (optional): Contains styling assets such as .css themes, logo image files and fonts. This module is not required for system functionality but helps to have assets organized in one place and access later on.

Module Organization

 .
├──  bootloader
│   ├──  seabios-assigned-iso-at-birth.nix
│   └──  seabios-assigned-proxmox-at-birth.nix
├──  desktop-manager
│   ├──  gnome.nix
│   └──  sway_greetd_homemanager.nix
├──  lix-default.nix
├──  local
│   ├──  hostname_username.nix
│   └──  networking_local.nix
├──  scripts
│   ├──  compose_two_nix.zsh
│   ├──  pull.zsh
│   └──  push.zsh
├──  secrets-config
│   ├──  notes.md
│   ├──  sops-build-machine.nix
│   ├──  sops-composesongsheet.nix
│   ├──  sops-database.nix
│   ├──  sops-mail.nix
│   └──  sops-wireguard.nix
├──  songsheet
│   └──  wavelog
├──  styling
│   ├──  forgejo
│   ├──  PRG_logo.png
│   └──  PRG_logo.svg
├──  system
│   ├──  blank_system_USE_THIS_AS_COPY.nix
│   ├──  build_machine.nix
│   ├──  compose-songsheet.nix
│   ├──  database.nix
│   ├──  forgejo.nix
│   ├──  gramethus.nix
│   ├──  mail-server.nix
│   ├──  system_wishlist.md
│   ├──  teamspeak.nix
│   ├──  traefik.nix
│   └──  wireguard_server.nix
├──  system_scripts
│   ├──  backup_strategem
│   └──  gcloud_backup.nix
└──  toolsets
    ├──  grafana_metric.nix
    ├──  remote_building.nix
    └──  wireguard_peer.nix

Module Scripts

Warning

As you saw the odd one in the file structure, the scripts folder contains helper scripts to simplify maintenance and to make updating system configuration files on remote machines easier. These helper scripts live under nix-system-configs/modules/scripts.

pull.zsh

Deploys configuration changes from the repository to the local system. This script:

1. Prompts you to select a system configuration (or accepts one as a command-line argument)
2. Pulls the latest changes from the git repository
3. Syncs the modules directory to /etc/nixos/modules
4. Syncs the secrets directory to /etc/nixos/secrets (if present)
5. Syncs the system_scripts directory to /etc/nixos/system_scripts (if present)
6. Copies the selected system configuration to /etc/nixos/configuration.nix
7. Executes nixos-rebuild switch --upgrade-all to apply changes

push.zsh

Synchronizes local system changes back to the repository. This script:

1. Prompts you to select a system configuration (or accepts one as a command-line argument)
2. Rebuilds the local NixOS system with nixos-rebuild switch --upgrade-all
3. Pulls the latest changes from the git repository
4. Copies the active configuration from /etc/nixos/configuration.nix back to the repository
5. Syncs changes from /etc/nixos/modules back to the repository
6. Stages, commits, and pushes changes to the remote git repository with a timestamped commit message

Adding New Systems

To add a new system configuration to the infrastructure:

1. Create the configuration file in nix-system-configs/modules/system/:

   cp nix-system-configs/modules/system/blank_system_USE_THIS_AS_COPY.nix \
      nix-system-configs/modules/system/your-new-system.nix
   

2. Edit the configuration to set your system's unique properties:

   config = {
     local.hostname = "nixos-your-system";
     local.username = "yourusername";
     local.userDescription = "Your System Description";
     local.address = "10.XXX.XXX.XXX";  # Choose an available IP
   
     system.stateVersion = "25.11";
   };
   

3. Register in flake.nix to make it available for builds:

   nixosConfigurations = {
     # ...existing configurations...
   
     "nixos-your-system" = nixpkgs.lib.nixosSystem {
       inherit system;
       modules = [ ./nix-system-configs/modules/system/your-new-system.nix ];
     };
   };
   

4. Update the dry_run.zsh script according to variable name in the flake.nix

   configs=(
     "nixos-local-wireguard-server"
     "nixos-blank"
     ...
     "nixos-your-system"
   )
   
   

Tip

IF YOU WANT TO CHECK IT MANUALLY - Test the configuration before deploying, change the nixos-your-system accordingly:

nix build .#nixosConfigurations.nixos-your-system.config.system.build.toplevel

Or if you have already setup the dry_run.zsh

zsh dry-run.zsh nixos-your-system 

Once the build succeeds by running the zsh dry_run.zsh or by looking at the Forgejo Actions of this repository, the configuration is ready for deployment using the pull.zsh script on the target machine.

How to maintain the local machines

Note

The following maintenance scripts should be made less unwieldy to use, so they will be changed in the next possible project restructuring. It will use the ssh nixos-rebuild command, but need to setup the proper script.

From the home machine:

The remote access IP address should be reachable via the VPN service you use (Tailscale, OpenVPN, etc.). To connect to the remote machine (or other machines on the same network) with established keys, use:

ssh -A REMOTE_MACHINE_USERNAME@ACCESS_IP_ADDRESS

To connect to other machines:

ssh -A OTHER_REMOTE_MACHINE_USERNAME@LOCAL_IP_ADDRESS

After having had connected to the remote machine, run the following commands, although this assumes you have git cloned this repository into the home directory of the remote machine. Now, running through the selection of the different systems as per script and upon selection of the script, it will then try to upgrade the system.

cd the-prg-server-structure
git pull
zsh nix-system-configs/modules/scripts/pull.zsh

Secret management

See secrets.md

What to do if ran out of Disk Space on ISO borne systems

See oh-heck-ran-out-of-disk-space.md