Bluish Coder: nixos

Installing NixOS on Linode with Encrypted Partitions

2014-10-22T21:00:00+13:00

A while back I updated the instructions to install NixOS on Linode on the NixOS Wiki. This post adds to this to include encrypted partitions. It's based on those wiki instructions and my previous post on installing NixOS to an encrypted drive.

Some points to keep in mind when running a Linode with an encrypted drive are:

When you reboot you will need to access the Linode console to enter the password to mount the encrypted partition.
Because the server is located remotely it's possible for Linode or a party in a similar position to trap the console input to capture your password. So while encryption prevents a malicious admin from scanning your disks it won't prevent someone located at Linode from rebooting and capturing the password you enter.
I'm sure there are other weaknesses such as keys existing in memory while the Linode is running. Make sure you are ok with the attack points in this setup.

Boot a Linode into Rescue Mode

First step is to create the Linode as usual. I tested with a $20/month Linode 2048. Create two disk images. They should be:

A boot disk that will be unencrypted. I made this 1GB which is way oversized for what it needs to be but makes math easy.
A root disk that will be encrypted. I made this the remainder of my free disk space.

From the Linode manager choose the option to boot into the new Linode in Rescue mode. Make sure the disks are setup as:

/dev/xvda is the boot disk.
/dev/xvdb is the root disk.

When the recovery image is booted you can ssh into it with the instructions in the Remote Access tab of the Linode manager under Console Access. This will get you to a root prompt on the Linode to perform the rest of the steps.

Encryption Setup

Perform the following commands to setup the disk encryption:

# cryptsetup luksFormat /dev/xvdb
# cryptsetup luksOpen /dev/xvdb enc-pv
# pvcreate /dev/mapper/enc-pv
# vgcreate vg /dev/mapper/enc-pv
# lvcreate -L 1G -n swap vg
# lvcreate -l 11525 -n root vg

Note that these operate on the /dev/xvdb disk which is the root disk we created earlier. You will be prompted for a passphrase during the luksFormat and luksOpen commands. Make sure you remember this as this is the passphrase needed when rebooting.

The lvcreate lines create the partitions for swap and root partition. The 1G means a one gigabyte swap file. The 11525 is the extent for the remainder of the disk space. I found this number by initally running lvcreate -L 99G -n root vg which is bigger than the 40GB available on the linode. This gave an error message showing the maximum extent to use which was 11525 for me.

Formatting and Mounting

Format the new partitions with:

# mkfs.ext4 -L boot /dev/xvda
# mkfs.ext4 -O dir_index -j -L root /dev/vg/root
# mkswap -L swap /dev/vg/swap
# swapon /dev/vg/swap

To install NixOS we need to mount the partitions under /mnt:

# mount /dev/vg/root /mnt
# mkdir /mnt/boot
# mount /dev/xvda /mnt/boot

Installing NixOS

The installation is relatively simple. First install the Nix package manager:

# bash <(curl https://nixos.org/nix/install)
# . /root/.nix-profile/etc/profile.d/nix.sh

Set the channel to be NixOS:

# nix-channel --remove nixpkgs
# nix-channel --add http://nixos.org/channels/nixos-14.04 nixos
# nix-channel --update

Create a default configuration file for some NixOS packages we will need for the install later:

# cat <<EOF > configuration.nix
  { fileSystems."/" = {};
    boot.loader.grub.enable = false;
  }
  EOF

Install the NixOS installation software:

# export NIX_PATH=nixpkgs=/root/.nix-defexpr/channels/nixos:nixos=/root/.nix-defexpr/channels/nixos/nixos
# export NIXOS_CONFIG=/root/configuration.nix
# nix-env -i -A config.system.build.nixos-install \
             -A config.system.build.nixos-option \
             -A config.system.build.nixos-generate-config \
             -f "<nixos>"

Generate a default configuration file for the bootable system:

# nixos-generate-config --root /mnt

This creates a /mnt/etc/nixos/configuration.nix file which should be edited to install the software you want. It also requires some changes for Grub and the disk encryption. Replace the existing section related to Grub 2 in this file with:

# Use the GRUB 1 boot loader.
boot.loader.grub = {
 enable = true;
 version = 1;
 extraPerEntryConfig = "root (hd0)";
 device = "nodev";
};

For the encryption support add:

boot.initrd.luks.devices = [
  { name = "root"; device = "/dev/xvdb"; preLVM = true; }
];

To enable OpenSSH access to the Linode add:

services.openssh.enable = true;

Now run the install:

# nixos-install

This will take some time to download and install things.

Post install

Once nixos-install completes the following commands will need to be run to fixup Grub 1 usage on Linode. This must be done before rebooting:

# mkdir -p /mnt/boot/boot/grub
# cd /mnt/boot/boot/grub
# ln -sv ../../grub/menu.lst /mnt/boot/boot/grub

First boot

Create a new Configuration Profile in the Linode Manager for the Linode. Set the kernel to pv-grub-x86_64. Set the disks as they were setup in the Rescue boot. Everything else can be left at the default.

Boot the new Configuration. Now you will need to ssh back into the Linode console so you can enter your passphrase. This will continue the booting process. Login as root. There is no initial password. Set one:

# passwd

By default you won't be able to ssh as root so you should set up a normal user:

# useadd -m myuser
# passwd myuser
# usermod -a -G wheel myuser

The latter command lets you use sudo as that user. You should now be able to ssh into the Linode with the newly created user.

Customization

These steps install the stable version of NixOS. This does not receive new packages, only updates to existing ones. I like to live on the bleeding edge so I use nixos-unstable. You can switch to this by running the following as root:

# nix-channel --add http://nixos.org/channels/nixos-unstable nixos
# nixos-rebuild switch --upgrade

If you prefer vim to nano as an editor, add the following to /etc/nixos/configuration.nix:

environment.systemPackages = with pkgs; [
  vim
];

environment.variables.EDITOR = pkgs.lib.mkOverride 0 "vim";

If you need non-free packages, add:

nixpkgs.config.allowUnfree = true;

Firefox Development on NixOS

2014-05-15T19:00:00+12:00

Now that I've got NixOS installed I needed a way to build and make changes to Firefox and Firefox OS. This post goes through the approach I've taken to work on the Firefox codebase. In a later post I'll build on this to do Firefox OS development.

Building Firefox isn't difficult as NixOS has definitions for standard Firefox builds to follow as examples. To build from a local source repository it requires all the pre-requisite packages to be installed. I don't want to pollute my local user environment with all these packages though as I develop on other things which may have version clashes. As an example, Firefox requires autoconf-2.13 whereas other systems I develop with require different verisons.

NixOS (through the Nix package manager) allows setting up build environments that contain specific packages and versions. Switching between these is easy. The file ~/.nixpkgs/config.nix can contain definitions specific for a user. I add the definitions as a packageOverride in this file. The structure of the file looks like:

{
  packageOverrides = pkgs : with pkgs; rec {
    ..new definitions here..
  };
}

My definition for a build environment for Firefox is:

firefoxEnv = pkgs.myEnvFun {
  name = "firefoxEnv";
  buildInputs = [ stdenv pkgconfig gtk glib gobjectIntrospection
                  dbus_libs dbus_glib alsaLib gcc xlibs.libXrender
                  xlibs.libX11 xlibs.libXext xlibs.libXft xlibs.libXt
                  ats pango freetype fontconfig gdk_pixbuf cairo python
                  git autoconf213 unzip zip yasm alsaLib dbus_libs which atk
                  gstreamer gst_plugins_base pulseaudio
                ];

  extraCmds = ''
   export C_INCLUDE_PATH=${dbus_libs}/include/dbus-1.0:${dbus_libs}/lib/dbus-1.0/include
   export CPLUS_INCLUDE_PATH=${dbus_libs}/include/dbus-1.0:${dbus_libs}/lib/dbus-1.0/include
   LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:${gcc.gcc}/lib64
   for i in $nativeBuildInputs; do
     LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:\$i/lib
   done
   export LD_LIBRARY_PATH
   export AUTOCONF=autoconf
  '';
};

The Nix function pkgs.myEnvFun creates a program that can be run by the user to set up the environment such that the listed packages are available. This is done using symlinks and environment variables. The resulting shell can then be used for normal development. By creating special environments for development tasks it becomes possible to build with different versions of packages. For example, replace gcc with gcc46 and the environment will use that C compiler version. Environments for different versions of pango, gstreamer and other libraries can easily be created for testing Firefox builds with those specific versions.

The buildInputs field contains an array of the packages to be avaliable. These are all the pre-requisites as listed in the Mozilla build documentation. This could be modified by adding developer tools to be used (Vim, Emacs, Mercurial, etc) if desired.

When creating definitions that have a build product Nix will arrange the dynamic loader and paths to link to the correct versions of the libraries so that they can be found at runtime. When building an environment we need to change LD_LIBRARY_PATH to include the paths to the libraries for all the packages we are using. This is what the extraCmds section does. It is a shell script that is run to setup additional things for the environment.

The extraCmds in this definition adds to LD_LIBRARY_PATH the lib directory of all the packages in buildInputs. It exports an AUTOCONF environment variable to be the autoconf executable we are using. This variable is used in the Mozilla build system to find autoconf-2.13. It also adds to the C and C++ include path to find the DBus libraries which are in a nested dbus-1.0 directory.

To build and install this new package use nix-env:

$ nix-env -i env-firefoxEnv

Running the resulting load-env-firefoxEnv command will create a shell environment that can be used to build Firefox:

$ load-env-firefoxEnv
...
env-firefoxEnv loaded
$ git clone git://github.com/mozilla/gecko-dev
...
$ cd gecko-dev
$ ./mach build

Exiting the shell will remove access to the pre-requisite libraries and tools needed to build Firefox. This keeps your global user environment free and minimizes the chance of clashes.

Installing NixOS on a ThinkPad W540 with encrypted root

2014-05-14T12:00:00+12:00

I recently got a ThinkPad W540 laptop and I'm trying out the NixOS Linux distribution:

NixOS is a GNU/Linux distribution that aims to improve the state of the art in system configuration management. In existing distributions, actions such as upgrades are dangerous: upgrading a package can cause other packages to break, upgrading an entire system is much less reliable than reinstalling from scratch, you can't safely test what the results of a configuration change will be, you cannot easily undo changes to the system, and so on.

I use the Nix package manager alongside other distributions and decided to try out the full operating system. This post outlines the steps I took to install NixOS with full disk encryption using LVM on LUKS.

Windows

The W540 comes with Windows 8.1 pre-installed and recovery partitions to enable rebuilding the system. I followed the install procedure to get Windows working and proceeding to make a recovery USB drive so I could get back to the starting state if things went wrong. Once this completed I went on with installing NixOS.

NixOS Live CD

I used the NixOS Graphical Live CD to install. I could have used the minimal CD but I went fo the graphical option to make sure the basic OS worked fine on the hardware. I installed the Live CD to a USB stick from another Linux machine using unetbootin.

To boot from this I had to change the W540 BIOS settings:

Change the USB drive in the boot sequence so it was the first boot option.
Disable Secure Boot.
Change UEFI to be UEFI/Legacy Bios from the previous UEFI only setting.

Booting from the USB drive on the W540 worked fine and got me to a login prompt. Logging in with root and no password gives a root shell. Installation can proceed from there or the GUI can be started with start display-manager.

Networking

The installation process requires a connected network. I used a wireless network. This is configured in the Live CD using wpa_supplicant. This required editing /etc/wpa_supplicant.conf to contain the settings for the network I was connecting to. For a public nework it was something like:

network={
  ssid="My Network"
  key_mgmt=NONE
}

The wpa_supplicant service needs to be restarted after this:

# systemctl restart wpa_supplicant.service

It's important to get the syntax of the wpa_supplicant.conf file correct otherwise it will fail to restart with no visible error.

Partition the disk

Partitioning is done manually using gdisk. Three partitions are needed:

A small partition to hold GPT information and provide a place for GRUB to store data. I made this 1MB in size and it must have a partition type of ef02. This was /dev/sda1.
An unencrypted boot partition used to start the initial boot, and load the encrypted partition. I made this 1GB in size (which is on the large side for what it needs to be) and left it at the partition type 8300. This was /dev/sda2.
The full disk encrypted partition. This was set to the size of the rest of the drive and partition type set to 8e00 for "Linux LVM". This was /dev/sda3.

Create encrypted partitions

Once the disk is partitioned above we need to encrypt the main root partition and use LVM to create logical partitions within it for swap and root:

# cryptsetup luksFormat /dev/sda3
# cryptsetup luksOpen /dev/sda3 enc-pv
# pvcreate /dev/mapper/enc-pv
# vgcreate vg /dev/mapper/enc-pv
# lvcreate -L 40G -n swap vg
# lvcreate -l 111591 -n root vg

The lvcreate commands create the logical partitions. The first is a 40GB swap drive. The laptop has 32GB of memory so I set this to be enough to store all of memory when hibernating plus extra. It could be made quite a bit smaller. The second creates the root partition. I use the -l switch there to set the exact number of extents for the size. I got this number by trying a -L with a larger size than the drive and used the number in the resulting error message.

Format partitions

The unencrypted boot partition is formatted with ext2 and the root partition with ext4:

# mkfs.ext2 -L boot /dev/sda2
# mkfs.ext4 -O dir_index -j -L root /dev/vg/root
# mkswap -L swap /dev/vg/swap

These should be mounted for the install process as follows:

# mount /dev/bg/root /mnt
# mkdir /mnt/boot
# mount /dev/sda2 /mnt/boot
# swapon /dev/vg/swap

Configure NixOS

NixOS uses a declarative language for the configuration file that is used to install and configure the operating system. An initial file ready to be edited should be created with:

$ nixos-generate-config --root /mnt

This creates the following files:

/mnt/etc/nixos/configuration.nix
/mnt/etc/nixos/hardware-configuration.nix

The latter file is rewritten everytime this command is run. The first file can be edited and is never rewritten. For the initial boot I had to make one change to hardware-configuration.nix. I commented out this line:

# services.xserver.videoDrivers = [ "nvidia" ];

I can re-add it later when configuring the X server if I want to use the nvidia driver.

The changes that need to be made to configuration.nix involve setting the GRUB partition, the Luks data and any additional packages to be installed. The Luks settings I added were:

boot.initrd.luks.devices = [
  { 
    name = "root"; device = "/dev/sda3"; preLVM = true;
  }
];

I changed the GRUB boot loader device to be:

boot.loader.grub.device = "/dev/sda";

To enable wireless I made sure I had:

networking.wireless.enable = true;

I added my preferred editor, vim, to the system packages:

environment.systemPackages = with pkgs; [
  vim
];

Enable OpenSSH:

services.openssh.enable = true;

I've left configuring X and other things for later.

Install NixOS

To install based on the configuration made above:

# nixos-install

If that completes successfully the system can be rebooted into the newly installed NixOS:

# reboot

You'll need to enter the encryption password that was created during cryptsetup when rebooting.

Completing installation

Once rebooted re-enable the network by performing the /etc/wpa_supplicant.conf steps done during the install.

Installation of additional packages can continue following the NixOS manual. This mostly involves adding or changing settings in /etc/nixos/configuration.nix and then running:

# nixos-rebuild switch

This is outlined in Changing the Configuration in the manual.

Troubleshooting

The most common errors I made were syntax errors in wpa_supplicant.conf and configuration.nix. The other issue I had was not creating the initial GPT partition. GRUB will give an error in this case explaining the issue. You can reboot the Live USB drive at any time and mount the encrypted drives to edit files if needed. The commands to mount the drives are:

# cryptsetup luksOpen /dev/sda3 enc-pv
# vgscan --mknodes
# vgchange -ay
# mount /dev/bg/root /mnt
# mkdir /mnt/boot
# mount /dev/sda2 /mnt/boot
# swapon /dev/vg/swap

Tips

environment.systemPackages in /etc/configuration.nix is where you add packages that are seen by all users. When this is changed you need to run the following for it to take effect:

# nixos-rebuild switch

To find the package name to use, run something like (for vim):

$ nix-env -qaP '*'|grep vim

A user can add their own packages using:

$ nix-env -i vim

And remove with:

$ nix-env -e vim

A useful GUI for connecting to wireless networks is wpa_gui. To enable this add wpa_supplicant_gui to environment.systemPackages in /etc/nixos/configuration.nix followed by a nixos-rebuild switch. Add the following line to /etc/wpa_supplicant.conf:

ctrl_interface=/var/run/wpa_supplicant

Restart wpa_supplicant and run the gui:

$ systemctl restart wpa_supplicant.service
$ sudo wpa_gui

It's possible to make custom changes to Nix packages for each user. This is controlled by adding definitions to ~/.nixpkgs/config.nix. The following config.nix will provide Firefox with the official branding:

{
  packageOverrides = pkgs : with pkgs; rec {
    firefoxPkgs = pkgs.firefoxPkgs.override { enableOfficialBranding = true; };
  };
}

Installing or re-installing for the user will use this version of Firefox:

$ nix-env -i firefox