the_prg_server_configuration_old/documentation_titlepage.md

# Welcome to the Polyteknisk Radiogruppe's Server Documentation

This documentation outlines how the server is built and maintained, covering both short- and long-term management practices.

> [!NOTE]
> **Important:** This document should be actively maintained whenever functional changes occur - no matter how small or large. 
> This includes compilation flag changes, password key updates, new standard packages, and any other modifications.


## Documentation File Structure

As this document is in alpha release, the following package has been made with the standard Rust project template in the case
of having to create custom binaries for the system of the sysadmin or any related responsibilites/positions. Thusly,
the following document focus should be in the *[nix-system-configs](nix-system-configs)* folder, which
is the main folder for the system configuration files and scripts.

> [!TIP] 
> For documentation maintainers: Update the directory tree using `lsd --tree` from the project root. Be careful not to expose secrets and other things when documenting from your local system.

```bash
 .
├──  Cargo.lock
├──  Cargo.toml
├──  documentation_titlepage.md
├──  dry_run.zsh
├──  flake.lock
├──  flake.nix
├──  LICENSE
├──  nix-system-configs
│   ├──  build-deprecated
│   ├──  database-deprecated
│   ├──  dns
│   ├──  example-composed.nix
│   ├──  forgejo-deprecated
│   ├──  gateway
│   ├──  modules
│   ├──  old-server-notes.md
│   ├──  prg-blank-setup
│   ├──  secrets
│   └──  traefik-deprecated
├──  README.md
├──  secrets.md
└── 󱧼 src
    └──  main.rs
```

## Module Architecture

The system is organized into reusable modules across all NixOS machines in the PRG network. 
The *[system](nix-system-configs/modules/system)* module serves as the core configuration for Linux systems running on Proxmox, while supporting diverse 
deployments through modular composition.

### Configuration Pattern

Use `blank_system_USE_THIS_AS_COPY.nix` as a template for new system configurations. 
This file demonstrates the standard pattern of declaring system variables using Nix syntax to create custom local options that are callable across imported modules.

### Core Modules

The `imports = [ ... ]` block specifies which modules are attached to your system configuration. These core modules ensure consistency and functionality:
 - Bootloader - [bootloader](nix-system-configs/modules/bootloader) (**required**): Handles system boot initialization. Configuration varies by installation type (ISO-assigned or Proxmox-assigned).
 - Local - [local](nix-system-configs/modules/local) (**required**): Provides internet connectivity, hostname resolution, DNS configuration, and user settings.
 - Standard Toolset - [lix-default.nix](nix-system-configs/modules/lix-default.nix) (**strongly recommended**): Ensures consistent tools across the PRG network—file transfers (rsync), SSH with key-based authentication, security features (U2F and SSH agent passwordless sudo), package management, and automated daily system upgrades.
 - Desktop/Window Manager - [desktop-manager](nix-system-configs/modules/desktop-manager) (**optional**): Simplifies VM interaction within Proxmox and supports systems requiring graphical interfaces.
 - Toolsets - [toolsets](nix-system-configs/modules/toolsets) (**optional**): Contains reusable configurations for specific tools or services that may be needed across multiple systems, such as remote building configurations.
 - Secrets Config [secrets-config](nix-system-configs/modules/secrets-config) (**required when needing password management**): Manages sensitive information like database credentials and Traefik secrets using SOPS-encrypted Nix files (see [this](https://github.com/Mic92/sops-nix) and [that](https://github.com/getsops/sops#2usage)). This module is essential for systems that require secure handling of secrets.
 - Docker Containers - [songsheet](nix-system-configs/modules/songsheet) (**optional**): This special module handles Docker Compose files that are converted to Nix. Currently it is used for a single VM but can be expanded to contain all [compose2nix](https://github.com/aksiksi/compose2nix) generated configurations.
 - System Scripts - [system_scripts](nix-system-configs/modules/system_scripts) (**variable**): `system_scripts` contains scripts (for example, database backup scripts) that are often easier to write and maintain in shell than in Nix.
 - Styling Scripts - [styling](nix-system-configs/modules/styling) (**optional**): Contains styling assets such as .css themes, logo image files and fonts. This module is not required for system functionality but helps to have assets organized in one place and access later on.

### Module Organization

```bash
 .
├──  bootloader
│   ├──  seabios-assigned-iso-at-birth.nix
│   └──  seabios-assigned-proxmox-at-birth.nix
├──  desktop-manager
│   ├──  gnome.nix
│   └──  sway_greetd_homemanager.nix
├──  lix-default.nix
├──  local
│   ├──  hostname_username.nix
│   └──  networking_local.nix
├──  scripts
│   ├──  compose_two_nix.zsh
│   ├──  pull.zsh
│   └──  push.zsh
├──  secrets-config
│   ├──  notes.md
│   ├──  sops-build-machine.nix
│   ├──  sops-composesongsheet.nix
│   ├──  sops-database.nix
│   ├──  sops-mail.nix
│   └──  sops-wireguard.nix
├──  songsheet
│   └──  wavelog
├──  styling
│   ├──  forgejo
│   ├──  PRG_logo.png
│   └──  PRG_logo.svg
├──  system
│   ├──  blank_system_USE_THIS_AS_COPY.nix
│   ├──  build_machine.nix
│   ├──  compose-songsheet.nix
│   ├──  database.nix
│   ├──  forgejo.nix
│   ├──  gramethus.nix
│   ├──  mail-server.nix
│   ├──  system_wishlist.md
│   ├──  teamspeak.nix
│   ├──  traefik.nix
│   └──  wireguard_server.nix
├──  system_scripts
│   ├──  backup_strategem
│   └──  gcloud_backup.nix
└──  toolsets
    ├──  grafana_metric.nix
    ├──  remote_building.nix
    └──  wireguard_peer.nix
```

### Module Scripts

> [!WARNING]
> As you saw the odd one in the file structure, the `scripts` folder contains helper scripts to 
> simplify maintenance and to make updating system configuration files on remote machines easier. 
> These helper scripts live under `nix-system-configs/modules/scripts`.

#### `pull.zsh`

Deploys configuration changes from the repository to the local system. This script:

1. Prompts you to select a system configuration (or accepts one as a command-line argument)
2. Pulls the latest changes from the git repository
3. Syncs the `modules` directory to `/etc/nixos/modules`
4. Syncs the `secrets` directory to `/etc/nixos/secrets` (if present)
5. Syncs the `system_scripts` directory to `/etc/nixos/system_scripts` (if present)
6. Copies the selected system configuration to `/etc/nixos/configuration.nix`
7. Executes `nixos-rebuild switch --upgrade-all` to apply changes

#### `push.zsh`

Synchronizes local system changes back to the repository. This script:

1. Prompts you to select a system configuration (or accepts one as a command-line argument)
2. Rebuilds the local NixOS system with `nixos-rebuild switch --upgrade-all`
3. Pulls the latest changes from the git repository
4. Copies the active configuration from `/etc/nixos/configuration.nix` back to the repository
5. Syncs changes from `/etc/nixos/modules` back to the repository
6. Stages, commits, and pushes changes to the remote git repository with a timestamped commit message

## Adding New Systems

To add a new system configuration to the infrastructure:

1. **Create the configuration file** in `nix-system-configs/modules/system/`:
   ```bash
   cp nix-system-configs/modules/system/blank_system_USE_THIS_AS_COPY.nix \
      nix-system-configs/modules/system/your-new-system.nix
   ```

2. **Edit the configuration** to set your system's unique properties:
   ```nix
   config = {
     local.hostname = "nixos-your-system";
     local.username = "yourusername";
     local.userDescription = "Your System Description";
     local.address = "10.XXX.XXX.XXX";  # Choose an available IP
     
     system.stateVersion = "25.11";
   };
   ```

3. **Register in *[flake.nix](flake.nix)*** to make it available for builds:
   ```nix
   nixosConfigurations = {
     # ...existing configurations...
     
     "nixos-your-system" = nixpkgs.lib.nixosSystem {
       inherit system;
       modules = [ ./nix-system-configs/modules/system/your-new-system.nix ];
     };
   };
   ```
   
4. **Update the [dry_run.zsh](dry_run.zsh) script** according to variable name in the `flake.nix`

   ```zsh
   configs=(
     "nixos-local-wireguard-server"
     "nixos-blank"
     ...
     "nixos-your-system"
   )
   
   ```

>    [!TIP]
>   ***IF YOU WANT TO CHECK IT MANUALLY*** - **Test the configuration** before deploying, change the `nixos-your-system` accordingly:
>   ```zsh
>   nix build .#nixosConfigurations.nixos-your-system.config.system.build.toplevel
>   ```
> Or if you have already setup the `dry_run.zsh`
>   ```zsh
>   zsh dry-run.zsh nixos-your-system 
>   ```

Once the build succeeds by running the `zsh dry_run.zsh` or by looking at the Forgejo Actions of this repository, the configuration is ready for deployment using the `pull.zsh` script on the target machine.

## How to maintain the local machines

> [!NOTE] 
> The following maintenance scripts should be made less unwieldy to use, so they will be changed in the next 
> possible project restructuring. It will use the ssh nixos-rebuild command, but need to setup the proper script.

From the home machine:

The remote access IP address should be reachable via the VPN service you use (Tailscale, OpenVPN, etc.).
To connect to the remote machine (or other machines on the same network) with established keys, use:

```zsh
ssh -A REMOTE_MACHINE_USERNAME@ACCESS_IP_ADDRESS
```

To connect to other machines:
```zsh
ssh -A OTHER_REMOTE_MACHINE_USERNAME@LOCAL_IP_ADDRESS
```

After having had connected to the remote machine, run the following commands, although this assumes you have git cloned 
this repository into the home directory of the remote machine. Now, running through the selection of the different 
systems as per script and upon selection of the script, it will then try to upgrade the system.

```zsh
cd the-prg-server-structure
git pull
zsh nix-system-configs/modules/scripts/pull.zsh
```

### Secret management

See [secrets.md](secrets.md)