v13 - random words

While performing some tests with a congested 10Mbps link, a strange thing happened: The link was congested only on one direction and both endpoint queues were RED queues. Based on the parameters and the queue size, the delay between those two links should be something near 170ms. However, the delay was much larger: >300ms (!).

The “problem” was the ring-buffer of the underlying driver (e1000). This one used a buffer of 128 packets which when added to the average 150 packets in the queue, resulted in >300ms delay.

You can see this buffer by running:

#ethtool -g eth0

And you can modify this buffer by running:

#ethtool -G eth0 tx 80

This is the transmit buffer which (when filled) adds to the delay of the local queue.

Of course, in normal use, this buffer is a good thing as it will allow to get higher transfer rates easier (from the POV of the operating system). But when making experiments, this little thing gets in the way.

Another thing is that there seems to be a minimum value for this number. For example, on this card:

Marvell Technology Group Ltd. 88E8001 Gigabit Ethernet Controller

the minimum value is 80 (using e1000 driver from kernel 2.6.32)

So beware and don’t loose a week looking for this thing like I did.

NOTE: This is not related to the transmit queue length that is used on the interface, as shown by:

# ifconfig eth0
# ip link show eth0


For a long time ago, a computer connected to the Internet had an RV770 ATI card and used to use the proprietary fglrx driver. Yes… That was my pc…

Then the latest fglrx (10.2) wasn’t compatible with the latest kernel (2.6.33) and that kernel supported Kernel Mode Setting (KMS) using the radeon driver. Debian also started to have appropriate libdrm and Xorg (+ driver).

So I switched to radeon/KMS driver… At first everything was not working and I blamed the radeon driver. However, as it was proved, even after uninstalling the fglrx driver it kept causing me problems. A couple of files were left behind and there was at least one file left diverted to the fglrx’s one. To fix this problem one needs to examine diversions (dpkg-divert –list), use debsusm (e.g. debsums -s libgl1-mesa-glx) and reinstall the packages with checksum problems.

Finally, after switching to radeon/KMS the following things changed:

• The used memory after startup reduced from about 2.5-3GB to less than 500MB (!!!!). I’m not talking about card’s mapped memory. I’m talking about system’s memory.
• Everything runs a lot faster. It looks like system latency is greatly reduced. There are two kind of improvements: (a) KDE’s desktop effects are smoother and (b) it looks like the latency is reduced. Somehow the effects seem to run faster because there is less delay.
• I stopped getting crashes of plasma when logging-in.
• KDE’s effects stopped being disabled every now and then.
• Tearing disappeared (!).
• Compositing effects seem to use less CPU.
• Some sound-card issues disappeared. Every now and then the sound was muted after system startup, but not any more.

So yeah… I really suggest that you switch to radeon/KMS if you’re using the fglrx driver. It sucks.

BTW, I also tested that to a laptop with an older ATI card and it had most of the above improvements as well. A colleague of mine also switched to opensource driver and show the exact same, dramatic reduce of memory usage.

This video illustrates why Linux is superb:

MP4, 80MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-3.mp4
MP4, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.mp4
AVI, 40MB: http://www.v13.gr/files/videos/kde4-1/kde4-1-2.avi

In the video you see: KDE 4.3, mplayer, tvtime, konqueror, KVM, nexuiz, wine.
The video was made with kdenlive

After doing this two times, I finally conclude that it *is* possible to convert an i386 debian installation to an amd64 one. At the end of the procedure most of the system will work. Here is how:

NOTE: Don’t do anything that is mentioned here unless you fully understand what you are doing. You will most probably need to customize this procedure a bit!

You’ll need enough free space for the procedure to work.

While you still have a working system, print this document and download the latest debian netinst image. This can be used as a rescue disk if something goes wrong. It is trivial to restore your system using a rescue CD since we will not delete anything at all.

Step 1: Backup

I cannot stress this enough. Backup all your data, especially databases and anything binary that needs an executable file to be accessed. Most probably files at your home directory don’t risk at all but better be safe than sorry.

It is absolutely essential to backup your PostgreSQL database if you have one. The i386 and amd64 disk formats are not compatible (bad postgre… bad bad) and you’ll need to restore the database. This most probably comes to:

# # Backup:
# su - postgres
$ pg_dumpall > mydump
# # Restore
$ cat mydump | psql template0


I’ve done this procedure twice without backing up anything except from the PostgreSQL database which is required. The reason I didn’t backed-up anything is that I was sure I could rollback to my old system using a rescue CD. If you don’t know how just backup.

Step 2: Install an amd64 kernel
You’ll need to boot your i386 with an amd64 kernel. Do:

# apt-get install libc6-amd64 linux-image-2.6-amd64
# # Most probably you would like to have X working while you run this procedure so install your favorite closed-source modules too:
# apt-get install nvidia-kernel-2.6-amd64

and reboot

Step 3: Install a base amd64 system.

# mkdir /amd64
# dpkg --get-selections '*' > /amd64/sel1
# debootstrap --arch=amd64 lenny /amd64 http://myproxy/

(NOTE: The above code was corrected to include “–arch=amd64″ which I forgot to include. Thanks to Slawek Wernikowski)

Step 4: Install an identical amd64 system
Since this is a time consuming procedure and many things can go bad, I suggest you run this under “screen”. This way, if the X server crashes you will not loose what you’ve done.

# chroot /amd64 /bin/bash
# vi /etc/apt/sources.list
# # Add all sources.list entries that exist in your main system but not the CDs
# vi /etc/apt/apt.conf
# # Add the line: APT::Default-Release "your-release";
# # if needed
# apt-get install locales
# mount /proc
# mount /sys
# mount -t devpts none /dev/pts
# export TERM=xterm
# dpkg --set-selections < /sel1

Now run the following lines until everything is installed. Most probably parts of the installation may fail and you’ll need to rerun those commands. It should be OK. At the end there can be some packages that will not be configured. Just ignore them for now.

# apt-get -o 'Dpkg::Options={"--force-confdef";"--force-confnew"};' dselect-upgrade
# dpkg --configure --force-confdef --force-confnew -a

I repeat: You’ll have to repeat the above commands until everything is installed. If you have trouble just fix it yourself. Don’t forget that you’re working in a chrooted amd64 system.

Step 5: Change the system
This is the part that I don’t have written so there might be some this missing. Key points are here and you should be able to figure what you need to do if something goes wrong. It is really a very simple procedure:

# init 1
# mkdir /i386

## Move other dirs
# mv /amd64/emul /
# mv /sbin /i386
# mv /amd64/sbin /

Now is the tricky part where you move /lib and /bin:

# # /bin is easy
# mv /bin /i386
# /i386/bin/mv /amd64/bin /
# # Don't forget that you can only run commands from /i386 from now on until /lib is replaced.
# # So, to do ls just write: /i386/bin/ls

# # Time for /lib
# mv /lib /i386
# # At this point you cannot execute anything at all but don't panic. Just run everything using the runtime linker. This is what happens whenever you run a command.
# export LD_LIBRARY_PATH=/i386/lib
# /i386/lib/ld-2.7.so /i386/bin/mv /amd64/lib /
# # TATA! Welcome to your amd64 system!
# ls


Step 6: Fix /var
This step is highly dependent on what you have installed. Most probably you can remove old /var and keep the new one. If you’d like to keep the old one there are some dirs that you need to replace:

# cd /var
# mv cache cache.i386
# mv /amd64/var/cache .
# # I suggest that you replace the whole /var/lib tree:
# mv lib lib.i386
# mv /amd64/var/lib .


Whatever you do you should know that you should replace /var/lib/dpkg and /var/cache/apt with the new one.

Step 7: Make system bootable
Finally you should ensure that your system is bootable. A nice approach is to clean your /boot from whatever kernel is installed (don’t forget that with your new system, no package will own existing kernels) and force-reinstall your current kernel:

# cd /boot
# mkdir i386
# mv System* initrd* vmlinuz* config* i386
# apt-get install --reinstal linux-image-2.6.26-amd64
# # Replace linux-image-2.6.26-amd64 with your current kernel


This is it. Unless I’ve forgotten something you should just reboot to your know system. After rebooting you will have to fix the unconfigured packages and possible rerun dselect-upgrade:

# dpkg --configure -a
# apt-get dselect-upgrade


Finally, restore your PostgreSQL database and you’re done.
MySQL doesn’t need dump and restore but wou’ll have to copy/move old files from /var/lib.i386/myslq to /var/lib/mysql.

For a couple of days now, VMWare Server 2 RC2 was refusing to start virtual machines. It kept waiting at 95%.

It turns out that VMWare cannot boot virtual machines (or upgrade them) when kernel modules kvm_intel, kvm etc are loaded.

Here is the error message from hostd.log:

Question info: The virtualization capability of your processor is already in use. Disable any other running hypervisors before running VMware Server.

Of course the obvious solution is to rmmod the kvm related modules:

# lsmod |grep kvm
# rmmod kvm_intel
# rmmod kvm_amd
# rmmod kvm

The modules are most probably auto-loaded by /etc/init.d/kvm (but there may be other reasons too).

Sometimes there is the need to allow normal users to run wireshark and capture packets from the network. Running wireshark with sudo is a security hole since anyone can overwrite any file.

A secure one-liner that solves this problem is:

# (sudo dumpcap -w -) | wireshark -k -i -

Assuming that sudo is configured to allow the user to run “dumpcap -w -” as root.

This should be 100% secure (except from the traffic monitoring issue) and will work well in (for example) labs.

It seems that PAT support that is included in linux kernel 2.6.26 has some problems. If you are using a recent ATI Catalst (fglrx) driver (at the time of this writting, the lattest was 8.8) with 2.6.26 kernel with PAT support you may have problems.

Problems may include freezes every couple of seconds when playing videos or full-screen 3D apps/games.

To solve this problem you will have to disable PAT support in kernel. Fortunately there is a kernel boot parameter that does with without requiring recompilation. Just add “nopat” to kernel boot parameters and you should be OK. Debian users should not worry since PAT is disabled in the default kernel.

The above is half the truth. When disabling PAT you should make sure that MTRR is being set-up correctly. Start the X server and look at Xorg.0.log for the string “Linear”:

$ grep Linear /var/log/Xorg.0.log
(--) fglrx(0): Linear framebuffer (phys) at 0xc0000000

Then look at lspci -v outpout:

01:00.0 VGA compatible controller: ATI Technologies Inc Mobilitiy Radeon HD 3600 Series
Subsystem: ASUSTeK Computer Inc. Device 01e4
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at ff6f0000 (64-bit, non-prefetchable) [size=64K]
[...]

(”size=256M” is what we need). Don’t worry if you have more than 256MB of memory in your graphics card but it only shows 256MB. This is expected behaviour.

Then look at /proc/mtrr for a line like this one:

reg04: base=0xc0000000 (3072MB), size= 256MB: write-combining, count=1

The base address is from the Xorg.0.log file and the size is from lspci output (0×10000000 is 256MB (=256*1024*1024 bytes)). For most cases this should be taken care of by the X server

If you don’t see it then you’ll have to setup MTRR by yourself. There are many documents on the web but at the end you’ll be running something like this:

echo "base=0xc0000000 size=0x10000000 type=write-combining" > /proc/mtrr