v13 – random words

Linux containers (a.k.a. LXC) rock. It’s the ultimate way of having multiple Linux boxes with minimal requirements.

Here’s how I do it under Debian (and the script I’m using):

Requirements

This guide is for Debian  testing as of 25 March 2012. However it should work for other cases as well.

The procedure creates a minimal installation which can then be fully customized by hand or with puppet. The procedure installs Debian under Debian but should be easy to change for other distributions as well (especially Ubuntu).

Packages

You will need to install:

• lxc – The linux containers package
• bridge-utils – For bridging network interfaces
• uml-utilities – For tun/tap interfaces
• cdebootstrap – For the bootstrapping of the virtual machines
• puppet (optional) – for managing multiple machines

Networking

I prefer networking between lxc installations to be separate from my normal network. It is trivial however to bridge with the outside network as well.

Add the following to /etc/network/interfaces:

auto virtlxc
iface virtlxc inet manual
 tunctl_user     root
 up              ip link set virtlxc up
 down            ip link set virtlxc down

auto brvirt
iface brvirt inet static
 bridge_ports            virtlxc
 bridge_maxwait          0
 bridge_stp              off
 address                 10.3.1.1
 netmask                 255.255.255.0
 dns-search              virt.local

Then add the following to /etc/hosts:

10.3.1.1    deb0 deb0.virt deb0.virt.local
10.3.1.11    deb1 deb1.virt deb1.virt.local
10.3.1.12    deb2 deb2.virt deb2.virt.local
10.3.1.13    deb3 deb3.virt deb3.virt.local
10.3.1.14    deb4 deb4.virt deb4.virt.local

Add as many entries as you like. There should be one entry per virtual machine. It doesn’t matter if you have more entries than virtual machines since you may use them in the future. The first (deb) entry is for the local machine.

Bring up the brvirt and virtlxc interfaces and keep reading (ifup virtlxc; ifup brvirt).

You may also want to run something like this to provide network access to the virtual machines (assuming that eth0 is the interface the connects you to the rest of the world):

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -I POSTROUTING -o eth0 -s 10.3.1.0/24 -j MASQUERADE

Create the virtual machine

Get the following script and change the desired variables at the beginning as follows (assuming that you followed the network configuration):

• SUITE: The Debian suite to use (e.g. squeeze)
• MIRROR: A mirror to download debian from. If you use approx like me then you want to use the local machine (i.e. 10.3.1.1)
• VIRTUSER: A username you want to have created in the virtual machine. After that you may ssh as that user.
• LOCALUSERS: A space separated list of local users to get ssh public keys from and put then in VIRTUSER’s authorized_keys file to allow ssh.
• PUPPETMASTER: Leave it empty if you don’t have a puppet master.
• DNSSERVER: The DNS server to use. By default it is the local machine.

Each virtual machine should get a unique MAC address. The MAC addresses are auto-generated from the current y/m/d/H/M, so you should not create more than one virtual machines every minute. You’re free to change this of course.

Now run the script at the end of the page and let it create a virtual machine:

./easylxc deb1

The installation will happen under /var/lib/lxc (the default for lxc). You may visit that and fix things by hand if you (i.e.) manage to lock yourself out.

The virtual machine can be started with:

lxc-start -n deb4
- or -
lxc-start -n deb4 -d

However, a bug/feature of rxvt will prevent that for succeeding. In that case you can run:

sudo lxc-start -n deb4
- or -
sudo lxc-start -n deb4 -d

Inside the virtual machine you will be able to su to root by using “su” without password. You will be also able to ssh as root (using the same ssh keys).

Hints’n'tips

I strongly suggest using approx and changing the MIRROR variable as needed. It will speed the creation of many machines by orders of magnitude since there will be no network delays.

The script

#!/bin/bash

if [ -z "$1" ] ; then
	echo "Pass the name of the machine as the first parameter"
	exit 1
fi

# The name of the container to create. Also used as the hostname
NAME="$1"

# The name of the parent (local) machine without the domain
PARENTNAME="deb0"

# Distribution
SUITE="squeeze"

# The domain to be used by the virtual machines.
DOMAIN="virt.hell.gr"

# The network prefix (first 3 octets - it is assumed to be a /24 network)
NETPREFIX="10.3.1"

# Since we use approx, this is the approx server. If not, add a mirror.
MIRROR="http://ftp.debian.org/debian/"

# The gateway address for the virtual machine. This is most probably the
# address of the bridge interface.
GW="$NETPREFIX.1"

# The bridge interface to use for networking
BRIDGEIF="brvirt"

# The username of the user to create inside the container
VIRTUSER="v13"

# A list of local users that will have ssh access to the container
# They need to have a public key in the local machine
LOCALUSERS="v13 root"

# The puppet master. This must be the hostname of the master (not an IP addr).
# No puppet if this is empty.
PUPPETMASTER=""

# The DNS server to use.
DNSSERVER="$GW"

IPADDR2=$(getent hosts $NAME.$DOMAIN | awk '{print $1}')

if [ "x$IPADDR2" = "x169.254.1.1" ] ; then
	IPADDR2=""
fi

if [ -z "$IPADDR2" ] ; then
	echo "Could not resolve $NAME.$DOMAIN"
	exit 1
fi

IPADDR="$IPADDR2/24"

MAC=$(date "+4a:%y:%m:%d:%H:%M")

lxc-stop -n $NAME
lxc-destroy -n $NAME

export SUITE
export MIRROR

R0=/var/lib/lxc/$NAME
R=$R0/rootfs

mkdir $R0 $R

# Install base system
echo cdebootstrap -f standard $SUITE $R $MIRROR
cdebootstrap -f standard $SUITE $R $MIRROR

CFG=$R0/config

# Create config file
cat << _KOKO > $CFG
# Auto-generated by: $*
# at $(date)

## Container
lxc.utsname		= $NAME
lxc.rootfs		= $R
lxc.tty			= 6
lxc.pts			= 1024

## Network
lxc.network.type	= veth
lxc.network.hwaddr	= $MAC
lxc.network.link	= $BRIDGEIF
lxc.network.veth.pair	= veth-$NAME

## Capabilities
lxc.cap.drop		= mac_admin
lxc.cap.drop		= mac_override
lxc.cap.drop		= sys_admin
lxc.cap.drop		= sys_module

## Devices
# Allow all device
lxc.cgroup.devices.allow	= a
# Deny all device
lxc.cgroup.devices.deny		= a
# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allow	= c *:* m
lxc.cgroup.devices.allow	= b *:* m

# /dev/console
lxc.cgroup.devices.allow	= c 5:1 rwm
# /dev/fuse
lxc.cgroup.devices.allow	= c 10:229 rwm
# /dev/null
lxc.cgroup.devices.allow	= c 1:3 rwm
# /dev/ptmx
lxc.cgroup.devices.allow	= c 5:2 rwm
# /dev/pts/*
lxc.cgroup.devices.allow	= c 136:* rwm
# /dev/random
lxc.cgroup.devices.allow	= c 1:8 rwm
# /dev/rtc
lxc.cgroup.devices.allow	= c 254:0 rwm
# /dev/tty
lxc.cgroup.devices.allow	= c 5:0 rwm
# /dev/urandom
lxc.cgroup.devices.allow	= c 1:9 rwm
# /dev/zero
lxc.cgroup.devices.allow	= c 1:5 rwm
# /dev/net/tun
lxc.cgroup.devices.allow        = c 10:200 rwm

## Limits
#lxc.cgroup.cpu.shares                  = 1024
#lxc.cgroup.cpuset.cpus                 = 0
#lxc.cgroup.memory.limit_in_bytes       = 256M
#lxc.cgroup.memory.memsw.limit_in_bytes = 1G

## Filesystem
lxc.mount.entry		= proc $R/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry		= sysfs $R/sys sysfs defaults,ro 0 0

_KOKO

# fix interfaces
T=$R/etc/network/interfaces
mv $T $T.orig
(
	cat $T.orig \
		| sed "s/^iface eth0.*$//"
	echo "
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
	address			$IPADDR2
	netmask			255.255.255.0
	gateway			$GW
	dns-nameservers		$DNSSERVER
	"
) > $T
rm $T.orig

# fix resolv.conf
T=$R/etc/resolv.conf
cat << _KOKO > $T
domain $DOMAIN
search $DOMAIN
nameserver $GW
_KOKO

# add info to hosts
T=$R/etc/hosts
echo "$IPADDR2 $NAME $NAME.$DOMAIN" >> $T
echo "$GW gw gw.$DOMAIN $PARENTNAME.$DOMAIN $PARENTNAME" >> $T

# set debian_chroot (for help)
echo "lxc-$NAME" >> $R/etc/debian_chroot

# create ttys
for i in $(seq 0 6) ; do
	mknod $R/dev/tty$i c 4 $i
done

run()
{
	echo chroot $R "$@"
	LC_ALL=C chroot $R "$@"
}

run2()
{
	ssh -o StrictHostKeyChecking=no $IPADDR2 "$@"
}

runmaster()
{
	ssh -o StrictHostKeyChecking=no $PUPPETMASTER "$@"
}

# Install locales
run apt-get -y install locales

# disable init scripts
DISABLED="bootlogd bootlogs checkfs.sh checkroot.sh halt hostname.sh \
	hwclockfirst.sh hwclock.sh module-init-tools mountall.sh \
	mountdevsubfs.sh mountkernfs.sh mountnfs.sh mountoverflowtmp procps \
	reboot stop-bootlogd stop-bootlogd-single udev umountfs umountnfs.sh \
	umountroot"
for dis in $DISABLED ; do
	run update-rc.d $dis disable
done

# disable rsyslog's kernel logging
run sed -i 's/^\(.*imklog.*\)$/#\1/' /etc/rsyslog.conf

# add user
run adduser --gecos $VIRTUSER --disabled-password $VIRTUSER
run adduser $VIRTUSER root

# fix sources.list
T=$R/etc/apt/sources.list
cat << _KOKO > $T
deb $MIRROR $SUITE main
_KOKO

# Install ssh
run apt-get update
run apt-get -y install openssh-server
run /etc/init.d/ssh stop

# Fix root and su
run passwd -l root
T=$R/etc/pam.d/su
mv $T $T.old
cat $T.old \
	| sed 's/^# \(.*pam_wheel.so trust\)/\1/' \
	> $T
rm $T.old

# Add ssh keys
T=$R/home/$VIRTUSER/.ssh/authorized_keys
T2=$R/root/.ssh/authorized_keys
mkdir $R/home/$VIRTUSER/.ssh $R/root/.ssh
for u in $LOCALUSERS ; do
	H=$(getent passwd $u | cut -f 6 -d : )
	cat $H/.ssh/id_rsa.pub >> $T
	cat $H/.ssh/id_rsa.pub >> $T2
done
chown $VIRTUSER.$VIRTUSER $R/home/$VIRTUSER/.ssh $T
chown root.root $R/home/$VIRTUSER/.ssh $T2

# Start it
# Use sudo to bypass file descriptor problems
sudo lxc-start -n $NAME -d
sleep 1

if ! [ -z "$PUPPETMASTER" ] ; then
	# Install packages
	run2 apt-get -y install puppet

	# Clear any existing certificate
	runmaster puppet cert clean $NAME.$DOMAIN

	# Fix puppet config
	T=$R/etc/default/puppet
	mv $T $T.old
	cat $T.old \
		| sed 's/START=no/START=yes/' \
		| sed "s/DAEMON_OPTS=\"\"/DAEMON_OPTS=\"--server=$PUPPETMASTER --verbose\"/" \
		> $T
	rm -rf $T.old

	run2 puppet agent --server=$PUPPETMASTER --no-daemonize --onetime

	# sign the certificate
	runmaster puppet cert --sign $NAME.$DOMAIN

	run2 /etc/init.d/puppet start
fi

cat << _KOKO

LXC virtual box is ready!

Config file is at: $R0/config
Root fs is at: $R

Get a console with:
  lxc-console -n $NAME

Stop it with:
  lxc-stop -n $NAME

Start it with:
  lxc-start -n $NAME -d

_KOKO

Update: You can use the above code under the GPLv3 license.