ciarand.me another code blog

Packer, Salt, and FreeBSD on DigitalOcean

I spent a large part of my weekend setting up a FreeBSD server on DigitalOcean (referral link). I've been meaning to try their FreeBSD offering ever since it was announced, and I finally found a good excuse to experiment. While I was experimenting, I figured I'd try out some of the tools I've been meaning to get some experience with. In this case, I settled on a combination of Packer and Salt.

Packer is a tool for building machine images. It can build a bunch of different types of machine images, including VMware boxes, AMIs, Docker images, and - most important for this post - DigitalOcean snapshots. Packer delegates most of the actual provisioning (installing packages, changing configuration files, managing users, etc.) to external tools. These can be as simple as inline shell scripts, or as complex as entire Ansible playbooks. You can see a full list of the provisioners that Packer supports on the documentation site.

I was lucky enough to attend a SaltStack session at the SCALE conference in February, and I've been meaning to take a deeper look at the tool ever since. Usually Salt is run with a master and a set of minion nodes, but since Packer only deals with creating machine images (and not managing the actual nodes at runtime), it only supports running Salt in a "masterless" (i.e. single-node) state. That's perfect for our needs.

The goal of this post is to setup a FreeBSD server running Nginx. Additionally, some extra general system configuration tasks need to be performed:

  • Setup a non-root user for remote administration
  • Make sure our user is in the wheel group
  • Restrict SSH access to public key authentication (no passwords)

This is totally not a complete guide on how to setup a public-facing server. There's a lot of things this is missing - log rotation, health monitoring, regular security updates, tuning kernel parameters, etc. - but this is a good start for those new to these tools. I also ran into enough problems when trying to set this up that I thought it would be worth writing about them.

Prerequisites

You'll need Packer. You can grab it from the website, a binary release should be fine. This was tested on v0.8.0.

You'll also need a DigitalOcean Personal Access Token (PAT). You can generate one from the applications panel in your account.

Finally, you may need some patience and a willingness to debug problems as they occur. There's a fair amount of moving parts here, and it's easy to make mistakes or run into issues caused by hidden assumptions made by those same parts.

Getting started with Packer

Let's create a new directory for our work.

mkdir packed-salt && cd packed-salt

The first thing we need is a build file for Packer. You can read more about the format in the docs, but the short version is that it's a simple JSON file. For the impatient, paste this into build.json:

{
  "variables": {
    "digitalocean_pat": "{{env `DIGITALOCEAN_PAT`}}"
  },
  "builders": [{
    "type": "digitalocean",
    "api_token": "{{user `digitalocean_pat`}}",
    "region": "sfo1",
    "size": "1gb",
    "ssh_username": "freebsd",
    "image": "freebsd-10-1-x64",
    "droplet_name": "my_cool_droplet_name",
    "snapshot_name": "my_cool_droplet_name--{{timestamp}}"
  }]
}

Let's go over each of the parts.

The first section, variables, contains a listing of all the user-supplied variables. In our case there's only one - digitalocean_pat. The {{env `DIGITALOCEAN_PAT`}} part just pulls DIGITALOCEAN_PAT from the local env. That's a nice way of keeping sensitive information out of config files (and Git repos).

The second part, builders, is an array of builder configurations. In our case we're just building a single snapshot for DigitalOcean. It's important to note that the size, region, and image parameters need to match the available options retrieved from the DigitalOcean API. We also need to specify the ssh_username because, unlike all the rest of the base images on the service, the FreeBSD base image requires you login as freebsd instead of root.

To run this, make sure the DIGITALOCEAN_PAT variable is set appropriately in your environment. Something like this should work:

export DIGITALOCEAN_PAT="<PUT YOUR GENERATED TOKEN HERE>"

Once you've got your environment setup, you can run the build using Packer:

packer build build.json

If everything went well, this should setup a temporary SSH key, create a FreeBSD droplet, save it to your account under the name my_cool_droplet-- (+ a timestamp), and then shut the Droplet down (and remove the aforementioned SSH key). If something failed, now would be a really good time to try and debug it. You can increase the verbosity of what Packer prints to stdout by setting the environment variable PACKER_LOG to 1, like so:

PACKER_LOG=1 packer build build.json

Getting started with Salt

Assuming everything went well, we can start to add provisioning scripts. Let's first change the build.json file to look like this:

{
  "variables": {
    "digitalocean_pat": "{{env `DIGITALOCEAN_PAT`}}"
  },
  "builders": [{
    "type": "digitalocean",
    "api_token": "{{user `digitalocean_pat`}}",
    "region": "sfo1",
    "size": "1gb",
    "ssh_username": "freebsd",
    "image": "freebsd-10-1-x64",
    "droplet_name": "my_cool_droplet_name",
    "snapshot_name": "my_cool_droplet_name--{{timestamp}}"
  }],
  "provisioners": [{
      "type": "shell",
      "execute_command": "chmod +x {{.Path}}; env {{.Vars}} {{.Path}}",
      "inline": [
          "sudo pkg install -y py27-salt-2015.5.2",
          "sudo mkdir -p /srv /usr/local/etc/salt/states",
          "sudo ln -s /usr/local/etc/salt /etc/salt",
          "sudo ln -s /usr/local/etc/salt /srv/salt"
      ]
  }, {
      "type": "salt-masterless",
      "skip_bootstrap": true,
      "local_state_tree": "{{template_dir}}/salt"
  }]
}

That's a lot more information, so let's take a moment to digest what we've added. There's a new section, provisioners, that's a list of 2 different provisioning options. The first is a simple inline-shell provisioner, and the second is a call to the salt-masterless provisioner.

I've opted to install Salt using FreeBSD's native pkg manager instead of through Salt's bootstrap script. I like this method because it helps keep everything roughly in the same place. Unfortunately, Salt still expects to find files under /srv/salt and /etc/salt (which are the defaults). As a result, we have to include the inline commands above to create some symlinks and directory trees in order to make sure everything runs correctly.

The part that gave me the most trouble here is that execute_command line. FreeBSD's default shell (tcsh) doesn't like the way Packer specifies environment variables, so we have to adjust the template it uses. The default template is:

chmod +x {{.Path}}; {{.Vars}} {{.Path}}

That ends up creating a command that looks something like:

chmod +x myshellscript; FOO=1 BAR=baz myshellscript

Unfortunately that particular invocation is not valid tcsh (it thinks FOO=1 is the command), so we have to adjust it by adding the env command like so:

chmod +x {{.Path}}; env {{.Vars}} {{.Path}}

One other note on the inline shell provisioner: py27-salt-2015.5.2 is the most current package available at the time of writing, but that may change in the future. If you run into problems finding that package, spin up a Droplet and run pkg search salt to figure out what name you should be using.

Next, we have the salt-masterless provisioner. This looks deceptively simple. skip_bootstrap does just what you'd expect and prevents Salt from downloading the bootstrap script. Since we're using the pkg version, that's exactly the behavior we want. local_state_tree points to a local collection of Salt "states" that we'll be using to configure our machine.

That's it for build.json. Now we need to create a new directory for our Salt states:

mkdir salt

Inside the salt directory, paste the following into a file called top.sls:

# http://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html
base:
  '*':
    - webserver

This is a very simple file that says each of the nodes being configured should be setup under the webserver role. Now we need to add that role. We can do so by putting our configuration in another file, webserver.sls, inside the salt directory:

# create a new user called "msmith"
msmith:
    user.present:
        - fullname: Mary Smith
        - home: /home/msmith
        - gid_from_name: True
        - groups:
            - wheel
    ssh_auth.present:
        - user: msmith
        - source: salt://ssh_keys/id_rsa.pub

# make sure the Nginx logs exist, otherwise Nginx will cry
{% for f in ["access", "error"] %}
/var/log/nginx/{{ f }}.log:
    file.managed:
        - makedirs: True
{% endfor %}

nginx:
    # install Nginx via pkg, but first refresh the pkg database
    pkg.installed:
        - refresh: True
    # make sure the nginx service is enabled (rc.conf) and running
    service.running:
        - enable: True
        # reload the service whenever the pkg or config file changes
        - reload: True
        - watch:
            - pkg: nginx
            - file: /usr/local/etc/nginx/nginx.conf

# copy the Nginx config file up
/usr/local/etc/nginx/nginx.conf:
    file.managed:
        - source: salt://nginx/nginx.conf
        - user: root
        - group: wheel
        - mode: 644

# copy our site up
/usr/local/www/nginx:
    file.recurse:
        - source: salt://site
        - include_empty: True

# copy our new sshd_config file up
sshd:
    file.managed:
        - name: /etc/ssh/sshd_config
        - source: salt://ssh/sshd_config
        - user: root
        - group: wheel
        - mode: 644
    service.running:
        - reload: True
        - watch:
            - file: /etc/ssh/sshd_config

Let's examine this piece by piece:

# create a new user called "msmith"
msmith:
    user.present:
        - fullname: Mary Smith
        - home: /home/msmith
        - gid_from_name: True
        - groups:
            - wheel
    ssh_auth.present:
        - user: msmith
        - source: salt://ssh_keys/id_rsa.pub

This section uses Salt's user state to configure a new user. Specifically, it uses the user.present API. The salt:// prefix on the filename is a convention that Salt uses to indicate the file is local. We'll add the file in the next section.

This brings us to the next section, which illustrates a powerful component of Salt's default configuration language. Each YAML file will be run through Jinja2 first, which means you can use all the same looping constructs you may already be familiar with.

# make sure the Nginx logs exist, otherwise Nginx will cry
{% for f in ["access", "error"] %}
/var/log/nginx/{{ f }}.log:
    file.managed:
        - makedirs: True
{% endfor %}

All this section does is make sure that the log files we reference in our configuration later on (/var/log/nginx/access.log and /var/log/nginx/error.log) exist, otherwise Nginx will refuse to start.

nginx:
    # install Nginx via pkg, but first refresh the pkg database
    pkg.installed:
        - refresh: True
    # make sure the Nginx service is enabled (rc.conf) and running
    service.running:
        - enable: True
        # reload the service whenever the pkg or config file changes
        - reload: True
        - watch:
            - pkg: nginx
            - file: /usr/local/etc/nginx/nginx.conf

This is where a lot of the meat happens. The first step, pkg.installed, uses Salt's pkg state to make sure Nginx is installed. The refresh: True line just makes sure that the pkg cache is refreshed prior to the install.

The next step, service.running, makes sure that the Nginx service is running. It's using Salt's service state to do that. The enable: True line makes sure that the Nginx service is enabled in the /etc/rc.conf file (nginx_enable="YES"). Finally, the reload and watch sections are telling Salt to reload the Nginx service whenever the pkg changes (via an upgrade, for example) or the configuration file changes.

Speaking of the configuration file, we need to copy that up to our server.

# copy the Nginx config file up
/usr/local/etc/nginx/nginx.conf:
    file.managed:
        - source: salt://nginx/nginx.conf
        - user: root
        - group: wheel
        - mode: 644

This uses Salt's file state to copy the file up with the correct permissions. The most interesting part about this section is the source line. The salt://nginx/nginx.conf value is a local-file reference to nginx/nginx.conf within the salt directory (i.e. packed-salt/salt/nginx/nginx.conf). Everything else should look pretty basic. We'll add that file in a moment, as soon as we're done going over the rest of the provisioning script.

# copy our site up
/usr/local/www/nginx:
    file.recurse:
        - source: salt://site
        - include_empty: True

This part should look pretty familiar. It's copying the salt/site directory (i.e. packed-salt/salt/site) to /usr/local/www/nginx on the remote server. This is where our public web files (i.e. HTML, JS, etc.) will live.

sshd:
    file.managed:
        - name: /etc/ssh/sshd_config
        - source: salt://ssh/sshd_config
        - user: root
        - group: wheel
        - mode: 644
    service.running:
        - reload: True
        - watch:
            - file: /etc/ssh/sshd_config

Almost done. This moves a new sshd_config file into place and makes sure that sshd is reloaded whenever that file changes. You can read more about the watch option in the requisites documentation.

Configuring Nginx

Now is a good time to add that nginx.conf file we talked about earlier. First, create the directory structure:

# assuming you're in the root of the project (i.e. packed-salt)
mkdir -p salt/nginx

Now create a salt/nginx/nginx.conf file with the following contents:

user  www;
worker_processes  1;
error_log /var/log/nginx/error.log info;

events {
    worker_connections  512;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    access_log /var/log/nginx/access.log;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80 default_server;
        server_name  _;
        root /usr/local/www/nginx;
        index index.html;

        location / {
            try_files $uri $uri/ =404;
        }

        error_page      500 502 503 504  /50x.html;
        location = /50x.html {
            root /usr/local/www/nginx-dist;
        }
    }
}

This isn't an Nginx configuration tutorial, so I'm going to skip over most of this. Important things to note are the locations of the error and access logs, as well as the server_name (set to the _ wildcard for now).

Adding a site

If you've already got a static HTML site, now would be a great time to grab it. I used a built version of my Jekyll site. Just place it in salt/site and you're good to go.

Adding your public SSH key

We also need to make sure your public SSH key is on the server. Create a salt/ssh_keys directory and place your public key (usually id_rsa.pub) inside.

Setting up a new sshd_config file

This is where you get to customize the sshd_config file. You'll want to save it to salt/ssh/sshd_config in your local project directory. Here's the one I was using:

Port 22
AddressFamily any
ListenAddress 0.0.0.0
ListenAddress ::

Protocol 2

SyslogFacility AUTH
LogLevel INFO

# Authentication:
StrictModes yes
MaxAuthTries 3

RSAAuthentication yes
PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and
# .ssh/authorized_keys2
AuthorizedKeysFile .ssh/authorized_keys

# stop passwords
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
# pam is ok without passwords
UsePAM yes

# override default of no subsystems
Subsystem       sftp    /usr/libexec/sftp-server

# turn off root login
PermitRootLogin no

All together now

Let's try running it. From the root directory (i.e. packed-salt):

packer build build.json

If this didn't work, it's time to get your hands dirty. Try running PACKER_LOG=1 packer build build.json and reading through the errors as they happen. I found searching for Result: False in the output stream usually pinpointed the errors pretty quickly.

If it worked, congratulations! You've successfully created a DigitalOcean snapshot of a FreeBSD machine, complete with a minimal Nginx configuration and password-less SSH authentication. The next step is to create a Droplet from that base image. You can go to your DigitalOcean control panel and create a new droplet using the my_cool_droplet_name--TIMESTAMP image as a base.

It's also important to note that this server probably still isn't production ready. That being said, hopefully I've sparked your interest in these tools and given you enough of a platform to begin learning and using them.

Was this tutorial helpful to you? Did you run into unexpected problems? Let me know on Twitter (@ciarandowney) or App.net (@ciarand)!

Using dd and pv on OS X

I find myself burning ISOs to flash drives fairly often. This is the best and quickest way I've found of doing it (on OS X - your experience with other systems and non-BSD tools may differ):

$FILE=my_cool_new_distro.iso
$RDISK=/dev/rdisk1 # diskutil list

dd if=$FILE \
    | pv -tpreb -s $(du -k $FILE | awk '{print $1}')k \
    | sudo dd of=$RDISK bs=1m

Notes

  • $RDISK should have the r in front (i.e. /dev/rdisk1 not /dev/disk1), that tells the kernel not to buffer it. It will display as /dev/disk1 in diskutil list, but you should put the r in front. This makes it faster.

  • bs=1m tells dd to use a blocksize of 1m. This makes it faster.

  • $(du -k $FILE | awk '{print $1}')k is a hack to make sure that dd understands the units being shown in du's answer. We pass the -k flag to du, which asks it to display the size of the file in kilobyte blocks. The snippet then has a trailing 'k', which makes it clear that the result is in kilobytes. This just gives you a progress bar.

Installing Neovim

Update: This isn't necessarily up to date any more. Check the Neovim wiki for the most up-to-date instructions.

Neovim has been moving along quite steadily over the last six months, and it's finally starting to hit a level of stability where I feel comfortable using it full-time. I did have to jump through a few minor hoops to get it working, however:

Homebrew

I use Homebrew to handle my dependencies, as you'd expect, so this was the first thing I tried:

brew tap neovim/neovim
brew install neovim --HEAD

Unfortunately it failed with an unknown error and ate my error logs before I could view them. On to the next method!

Installing by hand

The instructions claim that this command will install the required Luarocks modules:

make CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=$HOME/neovim" install

But I ended up having to install them by hand:

brew install luarocks
luarocks install lpeg
luarocks install lua-messagepack
luarocks install luabitop
# for tests
luarocks install busted

Once that was done I could correctly run the above command:

make CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=$HOME/neovim" install

Making plugins work

I use quite a few plugins to make some boring tasks easier, and Vundle to manage them. Luckily moving them over was a very simple matter:

cp ~/.vimrc ~/.nvimrc
pip install neovim

Tmux navigation

This is what my old .tmux.conf file looked like:

bind -n C-h run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)vim(diff)?$' && tmux send-keys C-h) || tmux select-pane -L"
bind -n C-j run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)vim(diff)?$' && tmux send-keys C-j) || tmux select-pane -D"
bind -n C-k run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)vim(diff)?$' && tmux send-keys C-k) || tmux select-pane -U"
bind -n C-l run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)vim(diff)?$' && tmux send-keys C-l) || tmux select-pane -R"

This is what my new .tmux.conf file looks like:

bind -n C-h run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)(n)?vim(diff)?$' && tmux send-keys C-h) || tmux select-pane -L"
bind -n C-j run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)(n)?vim(diff)?$' && tmux send-keys C-j) || tmux select-pane -D"
bind -n C-k run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)(n)?vim(diff)?$' && tmux send-keys C-k) || tmux select-pane -U"
bind -n C-l run "(tmux display-message -p '#{pane_current_command}' | grep -iqE '(^|\/)(n)?vim(diff)?$' && tmux send-keys C-l) || tmux select-pane -R"

All I've done is add the (n)? prefix to the grep pattern (and reloaded the conf file via tmux source-file ~/.tmux.conf).

Conclusion

Everything works, and it seems snappier. I'm excited, and I'll post some more details when / if I encounter any more problems.

Boring is good

I just switched my blog engine again. I was previously using Hugo, which is a static site engine written in Go. It's an interesting project and there's a lot to like about it. It's fast, being written in Go means that precompiled binaries are available for any modern system, and did I mention it was fast?

If I'm being honest with myself, however, I didn't pick it for either of those reasons. I picked it because it was interesting and new. But those same attributes have caused me to have to do a lot of extra work, including migrating a theme, fixing some issues with the 404.html logic, and hand-crafting an individual-post layout system. None of that work was particularly challenging, though the layout system wasn't exactly fun. But - at least apart from the 404.html issue which may help some other users - it was very much a waste of time.

If all I needed was a blog, I could have just used Jekyll. But I didn't, because both Jekyll and Ruby are boring. They're not fun or sexy or interesting, they're just there. Chugging along, doing what they're supposed to, not really drawing a lot of attention to themselves.

But you know what? That's ok. I don't have to fight Jekyll to get it to generate my blog. I don't have to create a custom publish / deploy workflow to get it to work correctly within GitHub pages. I don't have to come up with fancy new solutions to boring, old, solved problems.

It's stable, it's proven, and it gets out of my way. I can focus on doing and expect my tools to Just Work™ without much input from me. That's the reason I use vim. That's the reason I use Mac OS X. And now, that's the reason I use Jekyll. They're boring, and they leave me to focus on solving real problems instead of reorganizing index cards as productivity porn.

Refactoring a PHP application, part 2

This is the second post in the series. The first post is available here.

In the last post we talked about exactly what we'd be working with, and more importantly what we want to end up with. In this post we'll be talking about getting a single entry point enabled.

First, let's move our project directory into a subfolder. We're going to start using a "public" directory, so there will be some project files above that. Something like this might work for you, but you should double check that only files that should be publically accessible are in the public directory. For example, your .git directory should not be in your public/ directory.

# move all the front-end assets into a public dir
find . -type f \( \
        -name "*.css" -o  \
        -name "*.html" -o \
        -name "*.js" -o \
        -name "robots.txt" -o \
        -name "humans.txt" -o \
        -name "favicon.ico" -o \
        -name ".htaccess" \) |
    while read file
    do
        mkdir -p "public/$(dirname $file)"
        mv "$file" "public/$file"
    done

# move all the PHP files into a "handlers" directory
find . -name '*.php' | while read file; do
    mkdir -p "handlers/$(dirname $file)"
    mv "$file" "handlers/$file"
done

We'll be creating our entry script (index.php) inside the public directory. This will be where all our dynamic app requests get processed, so we'll put our dispatch and routing logic here. But first we need a way of sending the requests to this file, which is where server config rules come into play.

If you're using Nginx, you can use something like this:

# change the try_files directive to
try_files $uri $uri/ /index.php?$args

The preferred method for Apache users is now the FallbackResource directive. If you're using Apache v2.2.16 or higher, you can put this in your .htaccess or httpd.conf file (make sure the .htaccess file is in your /public directory):

FallbackResource /index.php

Older Apache users are stuck with RewriteRules (placed in the same file):

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ index.php [QSA,L]

For those few Lighttpd users, I've been told this will get you up and running:

url.rewrite-if-not-file = ("(.*)" => "/index.php/$0")

To test this works, let's check the superglobals in our entry script (public/index.php):

<?php echo $_SERVER["SCRIPT_NAME"];

Visiting the root (/) entry point should output something like this:

/

Visiting an arbitrary endpoint, say /lorem/ipsum/blah/blah, should output something like this:

/lorem/ipsum/blah/blah

Cool! We've got an entry script that can effectively handle all of our future requests.

Now we need to setup some code inside the index.php to forward requests to the same files as before. For the sake of this post series I've created a quick and dirty router. You can find it here. Assuming you have Composer installed globally, setting it up will look something like this:

composer require 'ciarand/quick-n-dirty-php-router' 'dev-master'

All you need in your public/index.php file is this:

<?php // index.php

require "vendor/autoload.php";

use Ciarand\Router;

// where you keep your scripts (outside of the public web dir)
$handler_dir = __DIR__ . "/../handler";

// the script you'd like to handle 404s with
// (this isn't strictly necessary, but it's a nice touch)
$missing_script = $handler_dir . "/404.php";

$request = $handler_dir . "/" . Router::scriptForEnv($_SERVER);

return require (file_exists($request))
    ? $request
    : $missing_script;

Okay, so now we can run some tests. Try opening some random URLs that worked before and make sure they still work. This is just a quick sanity check as the router is unit tested, but everyone's configuration is different and you may encounter some project-specific bugs. Make sure to test:

  • subdirectories (i.e. /subdir/ and /subdir should show the same thing as /subdir/index.php)

  • static assets (css, js, image files)

  • 404s

Great! We've got a single entry point and we haven't broken anything! We can safely start adding features via required files to the index.php and make sure we don't repeat any effort. More importantly, we've made a huge step in the direction of moving toward a more automated, test-friendly system.

In the next post we'll go over how to add some functional tests so we can be sure our future efforts don't break anything.

Update (2015-06-28): There's a ton of really good materials on this topic that were released either while I was writing this post or right after. Instead of regurgitating all that, here's a list of helpful resources on the topics of testing and refactoring legacy PHP applications: