Resizing disk of Fedora CoreOS

After provisioning Fedora CoreOS, resizing the disk might be necessary. This is a guide for you.

Situation at hand

The default disk size is 10GB, looks something like this:

[root@coreos ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk 
├─sda1   8:1    0    1M  0 part 
├─sda2   8:2    0  127M  0 part 
├─sda3   8:3    0  384M  0 part /boot
└─sda4   8:4    0  9.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot

You realise that you need more space, you grow the disk (i choose 100GB in this example) on underlying hardware.

[root@coreos ~]# echo 1 > /sys/block/sda/device/rescan
[root@coreos ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  100G  0 disk 
├─sda1   8:1    0    1M  0 part 
├─sda2   8:2    0  127M  0 part 
├─sda3   8:3    0  384M  0 part /boot
└─sda4   8:4    0  9.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot

Note that partition /dev/sda4 does not resize anymore, as it would on the first boot.

The fix

Theese manual steps made it:

[root@coreos ~]# growpart /dev/sda 4
CHANGED: partition=4 start=1050624 old: size=19920863 end=20971487 new: size=208664543 end=209715167
[root@coreos ~]# unshare --mount
[root@coreos ~]# mount -o remount,rw /sysroot
[root@coreos ~]# xfs_growfs /sysroot
meta-data=/dev/sda4              isize=512    agcount=21, agsize=124416 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=0
data     =                       bsize=4096   blocks=2490107, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 2490107 to 26083067

Situation after:

[root@coreos ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0  100G  0 disk 
├─sda1   8:1    0    1M  0 part 
├─sda2   8:2    0  127M  0 part 
├─sda3   8:3    0  384M  0 part /boot
└─sda4   8:4    0 99.5G  0 part /var
                                /sysroot/ostree/deploy/fedora-coreos/var
                                /usr
                                /etc
                                /
                                /sysroot

…and we are happy.

The article is edited to add the block device rescan to avoid the need of reboot of the server.

Do you have better solution or improvement ideas? Please comment.

Install Fedora CoreOS on Hetzner using Terraform

There is an exellent guide published by Hetzner about this subject. This article is a update on that. Instead of producing the ignition file in the rescue environment at Hetzner, I create it locally and and provision it to the server.

Create a directory for your project first, example ~/hetzner/terraform-coreos.

I am using terraform provided as a snap in ubuntu.

Step 1. Install Terraform provider for Hetzner Cloud

Create a versions.tf file with the following content:

# versions.tf

terraform {
  required_providers {
    hcloud = {
      source = "hetznercloud/hcloud"
    }
  }
  required_version = ">= 0.13"
}

Now execute terraform init to install the Terraform Hetzner Cloud Provider.

Step 2. Build coreos-installer

coreos-installer is built using rust, and recent versions require quite new version of rust compiler. It cannot be built in the rescue environment at Hetzner as there is not enough space available. The most convenient way is to build it on your own computer using the official containers of rust and transfer the binary to the rescue environment. The rescue environment at Hetzner is now using Debian bookworm, so build coreos-installer using docker.io/library/rust:slim-bookworm container using docker or podman. coreos-installer also needs some libraries to be installed in the container to build successfully.

$ podman run --rm --hostname rustbuilder -ti -v $PWD:/work docker.io/library/rust:slim-bookworm /bin/bash

root@rustbuilder:/# apt update && apt install -y pkg-config libssl-dev libzstd-dev
root@rustbuilder:/# cargo install --target-dir . coreos-installer
root@rustbuilder:/# cp /usr/local/cargo/bin/coreos-installer /work/

Step 3. Terraform configuration

To instruct Terraform how to setup and configure your servers, create a main.tf file:

# main.tf

####
# Variables
##

variable "hcloud_token" {
  description = "Hetzner Cloud API Token"
  type = string
}

variable "hcloud_server_type" {
  description = "vServer type name, lookup via `hcloud server-type list`"
  type = string
  default = "cpx11"
}

variable "hcloud_server_datacenter" {
  description = "Desired datacenter location name, lookup via `hcloud datacenter list`"
  type = string
  default = "hel1-dc2"
}

variable "hcloud_server_name" {
  description = "Name of the server"
  type = string
  default = "www1"
}

variable "hcloud_ssh_key_id" {
  description = "Name of already uploaded ssh key at Hetzner"
  type = string
}

####
# Infrastructure config
##

provider "hcloud" {
  token = var.hcloud_token
}

resource "hcloud_server" "master" {
  name = var.hcloud_server_name
  labels = { "os" = "coreos" }

  server_type = var.hcloud_server_type
  datacenter = var.hcloud_server_datacenter

  # Image is ignored, as we boot into rescue mode, but is a required field
  image = 107768015
  rescue = "linux64"
  ssh_keys = [ var.hcloud_ssh_key_id ]

  connection {
    host = hcloud_server.master.ipv4_address
    timeout = "5m"
    agent = true
    # Root is the available user in rescue mode
    user = "root"
  }

  # Copy config.ign
  provisioner "file" {
    source = "config.ign"
    destination = "/root/config.ign"
  }

  # Copy coreos-installer binary
  provisioner "file" {
    source = "coreos-installer"
    destination = "/usr/local/bin/coreos-installer"
  }

  # Install Fedora CoreOS in rescue mode
  provisioner "remote-exec" {
    inline = [
      "set -x",
      "chmod +x /usr/local/bin/coreos-installer",
      # Download and install Fedora CoreOS to /dev/sda
      "coreos-installer install /dev/sda -i /root/config.ign",
      # Exit rescue mode and boot into coreos
      "reboot"
    ]
  }

  # Configure CoreOS after installation
  provisioner "remote-exec" {
    connection {
      host = hcloud_server.master.ipv4_address
      timeout = "1m"
      agent = true
      # This user is configured in config.ign
      user = "core"
    }

    inline = [
      "sudo hostnamectl set-hostname ${hcloud_server.master.name}"
      # Add additional commands if needed
    ]
  }
}

Additionally we need a Fedore CoreOS Configuration file named config.yaml

# For docs, see: https://docs.fedoraproject.org/en-US/fedora-coreos/producing-ign/

variant: fcos
version: 1.1.0

passwd:
  users:
    - name: core
      groups:
        - docker
        - wheel
        - sudo
      ssh_authorized_keys:
        # add all keys that you need for the server
        - ssh-ed25519 AAAA.... name-of-ed25519-key
        - ssh-rsa AAAA.... name-of-rsa-key
      # If you need ssh password login, generate hash via `openssl passwd -1` and insert it below
      # password_hash: $1$1234567890abcdef...

storage:
  files:
    - path: /etc/profile.d/zz-default-editor.sh
      overwrite: true
      contents:
        inline: |
          export EDITOR=vim
  links:
    - path: /etc/localtime
      target: ../usr/share/zoneinfo/Europe/Helsinki

systemd:
  units:
    # Installing vim, open-vm-tools and rsync as a layered package with rpm-ostree
    - name: rpm-ostree-install-software.service
      enabled: true
      contents: |
        [Unit]
        Description=Layer vim with rpm-ostree
        Wants=network-online.target
        After=network-online.target
        # We run before `zincati.service` to avoid conflicting rpm-ostree
        # transactions.
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.stamp

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        # `--allow-inactive` ensures that rpm-ostree does not return an error
        # if the package is already installed. This is useful if the package is
        # added to the root image in a future Fedora CoreOS release as it will
        # prevent the service from failing.
        ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive vim open-vm-tools rsync
        ExecStart=/bin/touch /var/lib/%N.stamp

        [Install]
        WantedBy=multi-user.target

In the section Variables of the main.tf, adjust the default values for hcloud_server_type, hcloud_server_datacenter and hcloud_server_name to your needs or add a terraform.tfvars file to set your desired settings.

Optionally, have a look at the config.yaml. This file will be used by the coreos-installer to configure the OS. For the first try, you should leave it as it is. But after successful provisioning, you want to add systemd service definitions to run containers on it.

The yaml file needs to become an ignition file using butane tool. I use a container here again with podman to accomplish that:

podman run --rm --interactive --security-opt label=disable --volume ${PWD}:/pwd --workdir /pwd quay.io/coreos/butane:release --pretty --strict config.yaml > config.ign

Step 4 – Get it up and running!

Run terraform apply, enter your Hetzner Cloud API Token if prompted, and let the magic be done!

After a successful installation, you will see the message Apply complete! Resources: 2 added, 0 changed, 0 destroyed..

terraform show now outputs you all information about your new machine:

# hcloud_server.master: (tainted)
resource "hcloud_server" "master" {
    allow_deprecated_images    = false
    backups                    = false
    datacenter                 = "hel1-dc2"
    delete_protection          = false
    firewall_ids               = []
    id                         = "12345678"
    ignore_remote_firewall_ids = false
    image                      = "fedora-38"
    ipv4_address               = "xxx.xxx.xxx.xxx"
    ipv6_address               = "2a01:xxxx:xxxx:xxxx::1"
    ipv6_network               = "2a01:xxxx:xxxx:xxxx::/64"
    keep_disk                  = false
    labels                     = {
        "os" = "coreos"
    }
    location                   = "hel1"
    name                       = "www1"
    rebuild_protection         = false
    rescue                     = "linux64"
    server_type                = "cpx11"
    ssh_keys                   = [
        123435678,
    ]
    status                     = "running"
}

Step 5 – Do it again

You can now incrementally fill the two configuration files, to setup your services/containers. Run terraform destroy, confirm with yes and now adjust the files and recreate the infrastructure via terraform apply. But keep a look at the Hetzner Cloud Portal to ensure that there haven’t been spawned any “ghost machines”.

Conclusion

You have now a fully working Fedora CoreOS machine at Hetzner Cloud, set up with Terraform.