Since moving to exclusively to podman a ways back, the only major issue I’ve run into is that the CNI networking will break periodically with version updates.

Normally when a port-binding exists, you’ll see the dnat rules on the CNI-HOSTPORT-DNAT chain of the nat table in iptables. Similar rules should show up in nftables. For instance –

# iptables -nvL CNI-HOSTPORT-DNAT -t nat
Chain CNI-HOSTPORT-DNAT (2 references)
 pkts bytes target     prot opt in     out     source               destination
    4   240 CNI-DN-550b4bf3691bef6919331  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "podman" id: "aea317babc7f3ec7607b242227b39c90a43b50a7dd0f0b8fbccc82620370831d" */ multiport dports 8081

# iptables -nvL CNI-DN-550b4bf3691bef6919331 -t nat
Chain CNI-DN-550b4bf3691bef6919331 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.88.0.66           0.0.0.0/0            tcp dpt:8081
    4   240 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:8081
    4   240 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8081 to:10.88.0.66:80

As we can see, traffic bound for the port 0.0.0.0:8081 on the host will be sent to 10.88.0.66:80 which is the containers address on the cni-bridge.

Periodically, this setup will stop functioning altogether and the nat table chains and rules will not be created. The easiest way I’ve found to track this issue down is to run podman with logging turned way up and look for warnings and errors.

In my case, the following warning stood out to me when I ran into issues last:

WARN[0000] Error loading CNI config file /etc/cni/net.d/87-podman-bridge.conf: error parsing configuration: missing 'type'

This file was taken verbatim from the libpod repo, and it turns out that the instructions were actually incorrect – the file is not a standard configuration file but is actually considered to be a config list file, and thus should be named accordingly.

TL;DR:

The solution was dead simple – rename /etc/cni/net.d/87-podman-bridge.conf to /etc/cni/net.d/87-podman-bridge.conflist and relaunch my pods.

I recently zeroized1 a handful of EX3400s running 15.1X53-D55.5, and found that they would no longer properly boot into the kernel once the zeroizing process finished. Unable to get the USB recovery methods to work for me, I did the following to recover the device.

First, boot into the JunOS volume manually

loader> set currdev=disk0p2
loader> include /boot/loader.rc
loader> boot

Next, upgrade the firmware to 15.1X53-D56 or later. I found I couldn’t proceed without doing this first but you’re mileage may vary. In my case I used our ZTP provisioning tools to re-provision the device with a minimal configuration using 15.1X53-D56. Use whatever works for you.

After you’ve upgraded, boot again into the JunOS volume and clean up the /var/tmp (mounted on /) filesystem so you have as much space as possible. You need around 700MB but I was able to clear over 800MB by cleaning up old non-recovery snapshots as well as deleting the upgrade image out of /var/tmp.

Once you have enough space, you can run request system recover oam-volume. This process can take about 10m, but should complete successfully. If you ran into issues, see the end of the article for steps you need to take to try again.

After succesfull completion, reboot into the OAM and recover the JunOS volume by running:

request system reboot oam
# after the host boots into the OAM volume
request system recover junos-volume
request system reboot junos

If everything ran successfully, you should be booted back into a working switch.

Recovering from a failed OAM recovery attempt

If you ran out of space or some other error occurred during the recovery of the oam-volume, you’ll need to run umount /dev/gpt/oam to unmount the oam filesystem. Otherwise you’ll run into a bug in D55.5 and D56 versions that fail to umount the correct filesystem and will continue to fail until you unmount or reboot.

In one case the unmount process wasn’t enough and I found I had to reboot and start from the beginning. If it doesn’t work for you, keep at it – provided there’s no evidence of a hardware issue you should be able to recover all the devices.

  1. https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/request-system-zeroize.html 

RFC7217 describes a privacy extension that aims to improve upon RFC4941. In contrast to RFC4941 which provides random and temporary addresses, RFC7217 provides a stable address that provides much of the same benefits as the temporary addresses.

Known as ‘stable-privacy’ in Network Manager, RFC7217 is not desirable in some situations. For example, in a hosted environment where the IPv6 address is expected to be determinstic based on the MAC address, as is the case on Linode.

Linode uses SLAAC to provide IPv6 addresses to it’s nodes. With SLAAC, Linode’s routers advertise the prefix for the IPv6 network and the hosts, if configured properly, will use that information combined with the hardware address to generate an IPv6 address for use.

On Red Hat based systems like RHEL and CentOS, NetworkManager interferes with this process by defaulting to using RFC7217 to configure stable addresses. In order to disable this behavior, we need to reconfigure NetworkManager to not use stable addresses.

To check and see if your interface is running in ‘stable-privacy’ mode, run:

$ nmcli conn show "Wired interface 1" | grep ipv6.addr-gen-mode
ipv6.addr-gen-mode:                     stable-privacy

Above, we are running in stable-privacy mode. We want to disable privacy extensins altogether and run in eui64 mode. Update our connection configuration by running:

$ sudo nmcli conn edit "Wired interface 1" ipv6.addr-gen-mode eui64
# reload our connection
$ for act in down up; do sudo nmcli conn $act "Wired connection 1"; done

Our IPv6 address should now properly reflect the combination of your prefix and hardware address.

Introduction

Over the last day or so I’ve been slowly moving my ruby projects over to using rbenv instead of RVM. There’s nothing inherently wrong with RVM, but I do lots of interesting things with my shell that, when combined with my tmux setup, seems to always be giving me flak.

So at the recommendation of a friend, I sat down with rbenv for a couple of hours, and these are my notes from that experience.

Installation

Typically, I would not make a change so drastically, but I found the process of converting to be fairly painless and simple enough for me to go back to if need be without much fanfare. The process for me was to …

  1. Remove any source references to RVM scripts in my .bashrc and .bash_profile files
  2. Remove any path modifications that include the RVM bin dir
  3. Use homebrew to install rbenv (alternatively, you could clone the repository1 into ~/.rbenv
  4. Add the following into my .bashrc
if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi

Moar Rubies!

If all went well above, we should have a working rbenv installation. Now let’s take a look at the rubies currently available –

$ rbenv versions
* system (set by /Users/arusso/.rbenv/version)

I only have a single ruby version install initially, but with the help of the ruby-build2 plugin (available via hombere), I get access to the rbenv install command where I can install new versions of ruby.

In this case, my system ruby is version 2.0.0-p481 (ruby -v). This is too new for all my work, since I do a good deal with Puppet on RHEL6 which ships with 1.8.7-p374. Let’s start by installing that version –

$ rbenv install 1.8.7-p374
Downloading ruby-1.8.7-p374.tar.gz...
-> http://dqw8nmjcqpjn7.cloudfront.net/876eeeaaeeab10cbf4767833547d66d86d6717ef48fd3d89e27db8926a65276c
Installing ruby-1.8.7-p374...
Installed ruby-1.8.7-p374 to /Users/arusso/.rbenv/versions/1.8.7-p374

Downloading rubygems-1.6.2.tgz...
-> http://dqw8nmjcqpjn7.cloudfront.net/cb5261818b931b5ea2cb54bc1d583c47823543fcf9682f0d6298849091c1cea7
Installing rubygems-1.6.2...
Installed rubygems-1.6.2 to /Users/arusso/.rbenv/versions/1.8.7-p374

Now looking at the versions available to us, we see –

$ rbenv versions
* system (set by /Users/arusso/.rbenv/version)
  1.8.7-p374

Activating a Ruby

I typically activate rubies in two ways. First and foremost, when I’m switching between rubies for testing I used to use rvm use $version to get the ruby I want. With rbenv, this becomes rbenv shell $version.

The second way I choose rubies is by setting my ruby version in the .ruby-version file in my project directory. Fortunately, this does not really change and I can mostly leave it alone3.

For more information on how rbenv chooses a ruby version, see the project’s README section4 on the subject

Next Time

The next post will dive into the differences in gemset management between RVM and rbenv, as well as some useful plugins that make rbenv a better tool all around.

  1. https://github.com/sstephenson/rbenv 

  2. https://github.com/sstephenson/ruby-build 

  3. https://github.com/sstephenson/rbenv#choosing-the-ruby-version 

  4. rvm conviently allows you to select which gemset you want to use within the .ruby-version file. rbenv on the other hand does not even support gemsets without the help of the rbenv-gemset5 plugin. With it, you need only move the gemset information into the .ruby-gemset file. Part 2 will go into more detail about gemsets. 

  5. https://github.com/jf/rbenv-gemset 

…because “Poor Man’s Puppet Testing” just sounded lame…

Disclaimer

There are better, more effective and automated ways of testing your puppet manifests. However if you are in a position where setting up a CI server is not in the time budget, this article is for you.

This article also assumes you do not have some sort of orchestration tool at your disposal, and uses SSH1 to fake it until we make it. If this is not your situation fret not! You should be able to tailor it to use your orchestration tool of choice with minimal effort.

Overview

This article discusses a fairly simple idea – we will be updating some puppet code, committing our changes and having a simple process run noop runs on hosts we specify. The output of these runs will be displayed for us to inspect.

Setup

First off, we will need two bash functions. The first is a helper function that I came up with as a way to abstract away running arbirtray commands on a host called runon. Fancy, right? Fortunately, you should be able to refactor it to use your tool of choice fairly easily.

runon () {
    CLIENT=$1;
    COMMAND=$2;
    SSH_OPTS="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -a -T"

    if [[ "$1" == "" || "$2" == "" ]]; then
        echo "USAGE: runon <client> \"<command>\"    ";
    else
        ssh $SSH_OPTS ${CLIENT} "$2";
    fi
}

Next up, we are going to setup a bash function that is constantly checking a file we specify for hosts to connect and test our manifests against. I use environments to test all code changes2, so the function needs to know about the environment as well as the filename to poll for changes.

test_puppet_env () {
    if [[ -f $1 && "$2" != "" ]]; then
        while [ 1 -eq 1 ]; do
            while read line; do
                for hst in $(eval "echo $line"); do
                    echo "#### HOST: ${hst} ####";
                    runon ${hst} "sudo puppet agent -t --noop --environment ${2}";
                    echo "#######################";
                done;
                eval "sed -i '' -e '/^$line\$/d'" $1;
            done < $1;
            sleep 1;
        done
    else
       echo "USAGE: test_puppet_env <filename> <environment>"
       return
    fi
}

With that, we have all the functions we need. Now, let’s put it to use. We will assume I am doing some refactoring in a puppet environment ‘foo’, and want to test changes that would apply to hosts specified in ~/test-hosts

test_puppet_env ~/test-hosts foo

Now open up a new window (or hopefully a pane3) and run the following command:

echo arusso-dev-0{1..9}.example.com >> ~/test-hosts

Suddenly, you will see output in the original pane where we ran test_puppet_env. For a couple of hosts, this works reasonably well. But when I am testing a large number of hosts (>5) I typically do the following so I can grep through the files later rather than read them outright:

test_puppet_env ~/test-hosts foo | tee session-$(date +%Y%m%dT%H%M)

Now, in addition to having the output sent to stdout I can have a file I can grep/awk/sed through to find interesting bits of information I’m looking for.

So that’s it. Not too bad, right? Let me know what you think below.

  1. Technically this is SSH in a loop. Sleep well knowing I carry the burden of this terrible deed. Still, let’s keep it between us and not say anything to Luke. 

  2. Obviously some code cannot be tested in environments; the most obvious example being types. I assume that you know this, and use something like Vagrant to develop and modify such code in a sandbox. If not, I highly suggest this approach. 

  3. tmux (or screen) is your friend here.