Homelab 1: Genesis

This blog. The one that you are currently reading right now. Is running on a metal box that is just barely large enough to fit the average house cat. It cost ~$800 USD to build and has been christened Megamind. The HTTP request that you initiated when you visited https://howdoicomputer.lol has returned digital bits from hardware that is more than likely a couple feet from where I’m sitting. Welcome!

yeah, but why

I have a plethora of software projects that I’ve written that need a home. Discord bots, daemons, websites, etc.
I needed a place to host a blog. Hi!
I want to replace the myriad online software services that I’m subscribed to. Spotify, Netflix, etc.
I want an experimentation platform that I could use to learn new technologies or try new ideas.

I wanted to build a self-hosted platform with logging, metric collection, service discovery, orchestration, etc; fly.io in a box, if you will.

Also, I love making things. Making things is fun.

okay, but how

homelab

If you go on the /r/homelab subreddit and sort by the most popular posts then you’ll see setups that involve quite a bit more hardware than what my homelab currently sports. I don’t have a full metal rack with multiple blades and switches. There are no neatly braided corridors of ethernet cabling. My homelab is a box that is filled with consumer grade hardware that sprouts a single cable that snakes to my off-the-shelf, eight year old consumer router.

Currently, that box is outfitted with:

Ryzen 5600X
64 GB of non-ECC memory
Intel Arc GPU
Two WD Red 12TB drives

The hardware isn’t the interesting part, however. The interesting part is what runs inside the computer. It’s essentially a Platform as a Service (PaaS) solution in a box and I love it.

nomad and nix and zfs oh my

Here are some features of my homelab:

Nomad is the container orchestrator
A ZFS mirrored pool for data redundancy
Consul exists as a service mesh
Vault exists for secret storage
Traefik is for ingress
Prometheus for monitoring and Grafana for dashboards
The operating system is NixOS and I use my laptop to do flake based deployments to the underlying OS

A secret that I’ve withheld from you so far is that Megamind isn’t my first homelab. A previous homelab that I built was a k3s cluster running on top of Raspbery Pis. It was quaint but got decommissioned when I moved apartments. I never set it up again and it currently lies in pieces in a drawer somewhere.

So why Nomad? Mostly because, at the time, I was leading an infrastructure initiative at my day job to use Nomad to orchestrate applications that are deployed to heterogenous IoT devices spread across distinct geographic regions. Nomad was a better fit than k8s for that specific use case because it was lighter-weight, easier to use, had less overhead (which was important because embedded devices don’t have a lot of resources), and could run non-containerized applications that needed deep access to hardware. And so I wanted to dog food my own proposal. Also, Nomad is a lot simpler to use than Kubernetes. Nomad starts as just an orchestrator but gives you the option to introduce complexity later on through Vault and Consul. Every concept is almost immediately grokable and it has just been a pleasure to work with.

what’s runnin

Here is what is running so far:

Jellyfin (movies, TV shows, etc)
Gitea (Git server and Docker registry)
Factorio server
FoundryVTT
Dufs (a UI for uploading files to the server)
Calibre Web (my library of ebooks)
Navidrome (music server)
Grafana and Prometheus
A Discord bot for looking up books
An Elixir/Phoenix application that I’m working on

internal paas

I write code. I like writing code. Sometimes that code becomes semi-functional pieces of software that I need to deploy. Previously, I used fly.io to host applications but have now moved those applications to my homelab.

Here is how they are developed and deployed:

Each application has a ci/ directory that contains a Dagger pipeline for testing the application and building a docker image. This is local for now.
The docker image is uploaded to git.howdoicomputer.lol. What is behind that domain is a Gitea instance that serves as a git host and a docker registry.
That application also has a deploy/ directory that contains a job.nomad file. That Nomad job definition specifies the docker image that exists in Gitea and is used for the job allocation.
If the application has a /metrics endpoint, then Prometheus will be updated to scrape that endpoint and collect those metrics. This is powered by Consul.
If the application needs secrets then they are fetched from Vault.
Traefik and Consul Registry tags are used to define the ingress for the application. Most services are behind a subdomain on the howdoicomputer.lol domain and Traefik is what maps those subdomains to running services. If the service is not ready for production, then an IP whitelist is used to make sure that only ingress from my Tailscale VPN is allowed to access the site.
Logs are displayed through the Nomad web dashboard.
Traefik manages Let’s Encrypt SSL certs through the DNS challenge method.

This blog is developed similarily. It’s Hugo based so posts are markdown files that get converted into HTML files. A Dagger pipeline executes Hugo to get those files and then builds the container image - which runs Nginx - and then uploads that image to Gitea. There is a job definition in the blog’s repo that deploys that application to Nomad.

blog_nomad_job

Here is the job file for the blog:

job "blog" {
  type        = "service"
  datacenters = ["dc1"]

  group "blog" {
    network {
      mode = "bridge"

      port "http" {
        to = 80
      }
    }

    service {
      name = "blog"
      port = "http"

      tags = [
        "traefik.enable=true",
        "traefik.http.routers.blog-https.tls=true",
        "traefik.http.routers.blog-https.rule=Host(`howdoicomputer.lol`)",
        "traefik.http.routers.blog-https.tls.certresolver=resolver",
        "traefik.http.routers.blog-https.tls.domains[0].main=howdoicomputer.lol",
        "traefik.http.routers.blog-https.entrypoints=websecure",
        "traefik.http.routers.blog-http.entrypoints=web",
        "traefik.http.routers.blog-http.rule=Host(`howdoicomputer.lol`)",
        "traefik.http.routers.blog-http.middlewares=redirecthttps",
        "traefik.http.middlewares.redirecthttps.redirectscheme.scheme=https"
      ]
    }

    task "blog" {
      driver = "docker"

      config {
        image = "git.howdoicomputer.lol/howdoicomputer/blog:2"
        ports = ["http"]
      }

      resources {
        cpu    = 500
        memory = 200
      }
    }
  }
}

nix

The OS is configured remotely using deploy-rs - a Rust based CLI that allows you to apply a Nix flake to a specific host. This is what I’ve replaced Ansible with. Nix is… okay. I’ll review Nix later but one cool feature to note here is that deploy-rs will apply a Nix configuration and if the SSH connection is broken then the previous Nix state would be rolled back to on the host.

What Nix currently owns:

The installation of Nomad, Consul, Vault and other packages on the host.
The creation of the HashiStack configuration files.
The ZFS pool.
Firewall rules.
User creation.
Kernel parameters.

I do really like the immutability of Nix. Something I want to explore down the line is either impermanence or baking an OS image and continuously reimagining the homelab server through PXE.

the future

The future is bright! It has been incredibly fluid to roll out new services to the server. There are two additional features that I want: a CI/CD solution and an auth model for friends who want to use services.

For CI/CD, I’ll most likely go with Drone. Really, I just want a UI for Dagger pipelines. For an auth model, eh, I am not sure. LDAP? There is also the possiblity of using Oauth2 and whitelisting accounts.

For now though, my homelab is now an integral part of my life. For consumption, I use it to store a digital library of books, music, and movies. For productivity, the homelab is a cornerstone of developing applications that I want to expose to the public internet. It’s… neat.

Part 2