HomeLab Infrastructure

For a detailed history of the project’s evolution, see CHANGELOG.md.

• HomeLab Infrastructure
  • Overview
  • Architecture Overview
  • Repository Structure
  • Components
    • 1. Infrastructure & Provisioning (/ansible, /machineConfigs)
  • Hardware Inventory
    • Kubernetes
      • Machines
        • Storage
      • Kubernetes Nodes
        • Ceph Performance Tests
        • GPU
  • Network Architecture
    • Physical Topology
    • Networking Machines
    • VLAN Map
    • 2. Kubernetes Cluster (/okd)
    • 3. GitOps & Applications (/kubernetes)
    • 4. Tooling (/containers, main.bash)
    • 5. Virtualization & Cloud (/vms, /terraform)
    • 6. AI Agent Tooling (AGENTS.md, .agents/)
  • Usage
    • Bootstrapping
    • Deploying a New App
      • Kubernetes Commands
        • Delete Pod Using Graceful Termination Eviction Request

Overview

This repository contains the complete Infrastructure-as-Code (IaC) and GitOps configuration for my HomeLab. It manages everything from bare-metal server provisioning and virtualization to the Kubernetes (OKD) cluster and application deployment.

Architecture Overview

The infrastructure is built in layers, starting from physical hardware management with Ansible, up to application deployment managed by ArgoCD.

graph TD
    User[User / Admin] -->|manages| MainScript[main.bash]

    subgraph "Provisioning Layer"
        MainScript -->|triggers| Ansible[Ansible Playbooks]
        Ansible -->|configures| BareMetal[Physical Servers]
        Ansible -->|configures| KVM[KVM Hypervisors]
        Ansible -->|configures| Desktops[Workstations]
    end

    subgraph "Cluster Layer (OKD)"
        AgentConfig[Agent Config] -->|bootstraps| OKD[OKD Cluster]
        InstallConfig[Install Config] -->|defines| OKD
        BareMetal -->|hosts| OKD
    end

    subgraph "GitOps Layer"
        OKD -->|runs| ArgoCD[ArgoCD]
        ArgoCD -->|syncs| Apps[Applications]

        Apps -->|source| K8sDir[./kubernetes/]
    end

    subgraph "Applications"
        K8sDir --> Monitoring[Observability]
        K8sDir --> Storage[Storage]
        K8sDir --> HomeAuto[Home Automation]
        K8sDir --> DevTools[DevOps & Security]
        K8sDir --> Media[Media]
        K8sDir --> AI[AI / LLM Inference]
    end

Repository Structure

├── .agents/             # Skill libraries consumed by AI coding agents
├── ansible/             # Ansible playbooks for node configuration (KVM, Servers, Desktops)
│   ├── playbooks/      # Reusable Ansible playbooks
│   └── inventory       # Host inventory
├── containers/         # Custom container images (Toolbox, Utilities)
├── kubernetes/         # Kubernetes manifests for all applications (GitOps source)
│   ├── argocd/         # ArgoCD bootstrap configuration
│   ├── ceph/           # Rook-Ceph storage configuration
│   ├── ...             # Individual application folders (Traefik, Vault, etc.)
├── machineConfigs/     # Ignition/Preseed configs for machine provisioning
├── okd/                # OpenShift/OKD specific installation configs
│   ├── agent-config.yaml   # Node definitions for Agent Installer
│   └── install-config.yaml # Cluster configuration
├── tekton/             # Tekton Pipelines and Tasks
│   ├── base/           # Base Tekton configurations
│   ├── overlays/       # Environment-specific overlays
│   └── tasks/          # Reusable Tekton Tasks
├── terraform/          # Terraform configurations for infrastructure
│   ├── agent/          # Infrastructure for the Agent Based Terraform KVM
│   ├── homelab/        # Main HomeLab infrastructure
│   └── sandbox/        # Infrastructure for the UPI Based Terraform KVM
├── vms/                # Virtual Machine definitions (KubeVirt)
├── AGENTS.md           # Entry point for AI coding agents (mirrors SKILL.md)
├── SKILL.md            # Canonical HomeLab agent skill
├── main.bash           # Central entrypoint script for management tasks
└── notes/              # Documentation, scratchpads, and manual scripts

Components

1. Infrastructure & Provisioning (/ansible, /machineConfigs)

• Ansible: Used to configure the base operating system (Fedora/CentOS Stream), set up KVM, and manage networking (NMState).
• Networking:
  • Subnets: Machine Network (10.101.10.0/24), Cluster Network (10.101.32.0/19).
  • Bonding: Nodes utilize bonded interfaces for redundancy and performance.
    • active-backup: Used for split-switch topologies or where LACP is unavailable.
    • 802.3ad (LACP): Used for bandwidth aggregation on supported switches.
  • MTU: Jumbo frames (9000) are enabled for storage and cluster traffic.
• Virtualization: KVM is used to host virtualized control plane or worker nodes where applicable.

Hardware Inventory

Kubernetes

Machines

CPU Benchmark

Storage

Kubernetes Nodes

Ceph Performance Tests

GPU

Replaces the previous NVIDIA GeForce GTX 1080 (8GB). See kubernetes/intel-device-plugins for xpu-smi/intel_gpu_top monitoring commands (the nvidia-smi equivalents), and kubernetes/llm for B70 LLM tuning.

Network Architecture

The network is designed for high throughput (10GbE core) and segmentation via VLANs.

• Router: Ubiquiti UDM-SE (Gateway 10.0.0.1)
  • WAN 1: 1Gbps Fiber
  • WAN 2: LTE Backup
• Core Switching:
  • USW-Pro-HD-24: 10GbE/2.5GbE Core Switch.
  • USW-Aggregation (x2): 10GbE Fiber/DAC aggregation layer.
  • USW-Pro-Max-16: 2.5GbE/1GbE distribution layer.
• Wireless: U7 Pro Max & U7 Pro Wall (WiFi 7).

Physical Topology

graph TD
 linkStyle default interpolate basis

subgraph Networking
    wan1[WAN192.168.1.254]---|2/2 GB|router{UDM Pro SE10.0.0.1}
    wan2[LTE192.168.1.1]---|750/25 Mb|router

    router---|10GbE|switch_hd[USW Pro HD 24]

    switch_hd---|10GbE|switch_agg_bot[Agg Switch Bottom]
    switch_hd---|10GbE|switch_max[USW Pro Max 16]

    switch_agg_bot---|10GbE|switch_agg_top[Agg Switch Top]
    switch_agg_top---|10GbE|switch_max

    router---|2.5GbE|ap1{U7 Pro Wall}
    switch_hd---|2.5GbE|ap2{U7 Pro Max}
end

subgraph Storage
    switch_hd---|10GbE|unas(UNAS10.101.2.6)
    switch_hd---|2.5GbE|truenas(TrueNas10.102.1.7)
    router---|1GbE|truenas
end

subgraph "OpenShift/OKD"
    %% Control Plane
    switch_agg_bot---|2x10GbE|server1(server-110.101.10.101)
    switch_agg_top---|2x10GbE|server1
    switch_max-.-|4x 1GbE|server1

    switch_agg_bot---|2x10GbE|server2(server-210.101.10.102)
    switch_agg_top---|2x10GbE|server2
    switch_max-.-|4x 1GbE|server2

    switch_agg_bot---|2x10GbE|server3(server-310.101.10.103)
    switch_agg_top---|2x10GbE|server3
    switch_max-.-|4x 1GbE|server3

    %% Workers & GPU
    switch_hd---|2.5GbE|worker1(worker-110.101.10.104)
    switch_max---|2.5GbE|worker1

    switch_hd---|2.5GbE|worker2(worker-210.101.10.105)
    switch_max---|2.5GbE|worker2

    switch_hd---|2.5GbE|worker3(worker-310.101.10.106)
    switch_max---|2.5GbE|worker3

    switch_hd---|2.5GbE|gpu1(gpu-110.101.10.107)
    switch_max---|2.5GbE|gpu1
end

subgraph Edge
    router---|1GbE|microshift[MicroShift / PiHole10.101.1.7]
end

Networking Machines

VLAN Map

2. Kubernetes Cluster (/okd)

• Distribution: OKD (OpenShift Kubernetes Distribution).
• Installation: Uses the Agent-based Installer for bare-metal deployment.
• Configuration:
  • agent-config.yaml: Defines the physical hosts, network interfaces, and static IP assignments.
  • install-config.yaml: Defines the cluster name, base domain, and networking CIDRs.

3. GitOps & Applications (/kubernetes)

• Controller: ArgoCD watches this repository and syncs the state of the cluster.
• Storage Strategy:
  • Rook-Ceph: Block, Object, and File storage for the cluster.
  • MinIO: S3-compatible object storage for applications.
  • CloudNative-PG: Cloud-native PostgreSQL operator for database workloads.
• AI / LLM Inference:
  • llama.cpp (Vulkan): Local LLM inference engine running on the Intel Arc GPU.
  • Open WebUI: Chat frontend for local LLM inference.
  • Intel Device Plugins: OLM operator exposing gpu.intel.com/xe GPU resources.
  • Node Feature Discovery: Labels GPU-capable nodes (Intel xe KMD detection).
• Observability Stack:
  • VictoriaMetrics: Long-term metrics storage (replaced Prometheus remote write).
  • Prometheus & Grafana: Metrics collection and visualization.
  • Loki: Log aggregation.
  • Wire-Tap: Network traffic inspection and analysis.
  • StackRox (ACS): Kubernetes security and compliance.
  • Network Observability: Flow logs and traffic analysis.
  • Dragonfly Operator: In-cluster image caching.
• Security:
  • Vault: Secret management.
  • Cert-Manager: Automated certificate management.
  • External Secrets: Syncing secrets from external providers.
  • Coraza WAF: Web application firewall.
  • Kyverno: Kubernetes policy engine.
• Home Automation:
  • Home Assistant: Central automation hub.
  • ESPHome: Microcontroller management.
• DevOps:
  • Gitea: Self-hosted Git service.
  • Tekton: CI/CD pipelines.
  • Renovate: Automated dependency management (GitHub App).
  • Docker Registry: In-cluster container image registry.
• DNS:
  • Technitium DNS: Primary DNS server (replaced Pi-hole), with multi-node clustering.
  • External DNS (UniFi): Automatic DNS record management for UniFi controller.

4. Tooling (/containers, main.bash)

• Toolbox: A custom container image (containers/toolbox) pre-loaded with CLI tools (oc, kubectl, ansible, tofu) for managing the environment.
• main.bash: A wrapper script to execute common tasks like running Ansible playbooks, managing VMs, and provisioning OKD clusters.
  • ansible: Runs the Ansible playbooks for server configuration.
  • stateful_workload_stop: Suspends CronJobs and scales down monitoring/stateful workloads (useful for maintenance).

5. Virtualization & Cloud (/vms, /terraform)

• KubeVirt (/vms): Defines Virtual Machines (e.g., Fedora) running on top of the OKD cluster, managed via GitOps.
• Terraform (/terraform): Manages external cloud resources (GCP) and integrates with Vault for secret management.

6. AI Agent Tooling (AGENTS.md, .agents/)

• AGENTS.md: Entry point for AI coding agents (Copilot, opencode, etc.) describing repo conventions, cluster auth, and the kubeconfig discipline.
• SKILL.md: Canonical HomeLab skill, mirrored at .agents/skills/homelab/SKILL.md.
• .agents/skills/kubernetes-skill/SKILL.md: Production-grade Kubernetes manifest, Helm, and Kustomize guidance with a failure-mode workflow.

Usage

Bootstrapping

The environment is typically bootstrapped using the main.bash script which orchestrates the Ansible runs and prepares the OKD installer.

./main.bash

Deploying a New App

1. Create a new folder in kubernetes/.
2. Add Kubernetes manifests or a Kustomization file.
3. Register the application in ArgoCD (usually via kubernetes/argocd/applications).
4. Commit and push. ArgoCD will sync the changes automatically.

Kubernetes Commands

# Watch ALl Pods
watch kubectl get pods -A -o wide --sort-by=.metadata.creationTimestamp
# Watch non Health Pods
watch -n10 $(echo "kubectl get pods  -A | grep -vE 'Running|Completed'")
# Delete Pods that Have a Restart
kubectl get pods -A | awk '$5>0' | awk '{print "kubectl delete pod -n " $1 " " $2}' | bash -
# Drain Node
 oc adm drain server-3 --delete-emptydir-data --ignore-daemonsets --force
# Nextcloud
kubectl exec -it nextcloud-0 -n nextcloud -- runuser -u www-data -- php -f /var/www/html/occ

Delete Pod Using Graceful Termination Eviction Request

NAMESPACE=homelab
POD=el-webhook-6b56cc5f84-clfc6

curl --header "Authorization: Bearer $(oc whoami -t)" -H 'Content-type: application/json' \
"$(oc whoami --show-server)/api/v1/namespaces/{$NAMESPACE}/pods/{$POD}/eviction" \
-d '{"apiVersion": "policy/v1","kind": "Eviction","metadata": {"name": "'"${POD}"'","namespace": "'"${NAMESPACE}"'"}}'