OpenStack troubleshooting basics

If you’ve been running OpenStack from the developer trunk (as per my blog) you will occasionally come across some bugs. This is the nature of the beast for running bleeding edge code.
So how do you track down a solution for them?

Step 1. Check the logs

First place to look is in /var/log/nova where you will see the logs related to OpenStack.
Some bugs will be related to changes in the software, so maybe an extra config line is needed in /etc/nova/nova.conf.
For example you may have seen this in /var/log/nova/nova-network.log:

2011-03-15 17:33:35,732 CRITICAL nova [-] failed to create /usr/lib/pymodules/python2.6/cloud2.MainThread-18360
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE:   File "/usr/bin/nova-network", line 48, in
(nova): TRACE:     service.serve()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/service.py", line 284, in serve
(nova): TRACE:     x.start()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/service.py", line 84, in start
(nova): TRACE:     self.manager.init_host()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 489, in init_host
(nova): TRACE:     super(VlanManager, self).init_host()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/network/manager.py", line 125, in init_host
(nova): TRACE:     self.driver.init_host()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/network/linux_net.py", line 394, in init_host
(nova): TRACE:     iptables_manager.apply()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/nova/utils.py", line 523, in inner
(nova): TRACE:     with lock:
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/lockfile.py", line 223, in __enter__
(nova): TRACE:     self.acquire()
(nova): TRACE:   File "/usr/lib/pymodules/python2.6/lockfile.py", line 239, in acquire
(nova): TRACE:     raise LockFailed("failed to create %s" % self.unique_name)
(nova): TRACE: LockFailed: failed to create /usr/lib/pymodules/python2.6/cloud2.MainThread-18360
(nova): TRACE:

There was a change between releases that required the following lines present in /etc/nova/nova.conf to solve this:

--state_path=/var/lib/nova
--lock_path=/var/lock/nova

Step 2. Check https://bugs.launchpad.net/nova and https://bugs.launchpad.net/swift

A recent one I came across this morning was the following:

2011-03-17 08:49:19,160 ERROR nova.api [GXEJM3P1HVP7N53IGI5J admin myproject] Unexpected error raised: invalid literal for int() with base 16: 'ami-jqxvgtmd'
(nova.api): TRACE: Traceback (most recent call last):
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/api/ec2/__init__.py", line 318, in __call__
(nova.api): TRACE:     result = api_request.invoke(context)
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/api/ec2/apirequest.py", line 150, in invoke
(nova.api): TRACE:     result = method(context, **args)
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/api/ec2/cloud.py", line 906, in describe_images
(nova.api): TRACE:     images = self.image_service.detail(context)
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/image/s3.py", line 76, in detail
(nova.api): TRACE:     images = self.service.detail(context)
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/image/local.py", line 58, in detail
(nova.api): TRACE:     for image_id in self._ids():
(nova.api): TRACE:   File "/usr/lib/pymodules/python2.6/nova/image/local.py", line 50, in _ids
(nova.api): TRACE:     return [int(i, 16) for i in os.listdir(self._path)]
(nova.api): TRACE: ValueError: invalid literal for int() with base 16: 'ami-jqxvgtmd'
(nova.api): TRACE:

I found this related bug: https://bugs.launchpad.net/nova/+bug/735641 by searching the bug database for the error. In this case the solution is to remove my images from my objectstore and re-upload them due to changes in how the images are stored and retrieved.

Step 3. Always a good place to go is on IRC @ freenode.net

(http://webchat.freenode.net/) and join #openstack where the developers and contributors will answer your questions. Have patience though, they do have work to do.

Step 4. You can also ask questions on Launchpad: https://answers.launchpad.net/nova/+addquestion (and similar for swift).

I also find its handy to not be too vague – describe your set up instead of saying “I launch an instance and it’s stuck on “Scheduling, can you help?” doesn’t give anyone any details of why that could be the case. As you can imagine, this could be anything from user-error, hardware errors, software errors or misconfigured environments, etc. – all requiring many different ways to troubleshoot so help yourself by being more specific.

For more information check out the OpenStack Wiki on contributing to the project: http://wiki.openstack.org/HowToContribute and this information on support and troubleshooting from the docs: http://docs.openstack.org/openstack-compute/admin/content/ch08.html

OpenStack Nova CentOS Instance

I’ve been working on tweaking a CentOS 5.3 image you can download from http://open.eucalyptus.com/wiki/EucalyptusUserImageCreatorGuide_v1.6 as there seems to be a big bias towards running Ubuntu under OpenStack. This is great for getting OpenStack up and running, but for us evangelists that operate a RHEL family house, its crucial to be able to demonstrate like-for-like offerings against what you currently run to help promote its use.

This guide should get you to a point where you have a usable, useful CentOS image for your environment. When I get around to it I’ll upload my version for use in your environment with the modifications laid out in this blog post.

The Guide

• Start off by downloading a compatible image from Eucalyptus: http://open.eucalyptus.com/wiki/EucalyptusUserImageCreatorGuide_v1.6. I’ll work on the 64-Bit CentOS 5.3 image for this guide.
• mkdir cloud/images and unpack the tarball here

  • mkdir -p cloud/images

  • cd cloud/images

  • tar zxvf <path_to_tarball>/euca-centos-5.3-x86_64.tar.gz

  • cd euca-centos-5.3-x86_64

  • At this stage we’d normally upload the image to OpenStack but some modifications are needed such as increasing the size of the image to accommodate some new packages so we must first mount the image (read-only because we’re not needing to make edits to this yet) as follows

    • mkdir image

    • sudo mount centos.5-3.x86-64.img image -o loop,ro

  • Increase the size of the image as follows and copy the contents

    • dd if=/dev/zero of=newcentos.img bs=1M count=2048

    • mkfs.ext3 newcentos.img

    • mkdir newcentos

    • sudo mount newcentos.img newcentos -o loop,rw

    • sudo cp -pR image/* newcentos/

    • sudo umount image

• Modify the image as follows
• IMPORTANT! (ensure you’re chrooted as described below to your mounted image and you have verified that you’re not modifying your running environment – I accept no responsibility because you can’t read)

  • sudo su -

  • chroot ~/cloud/images/euca-centos-5.3-x86_64/newcentos

  • mount proc -t proc /proc

• Now to modify the image and install some new packages…

  • yum update

  • yum install redhat-lsb sudo enhanced-vim

  • Remove /etc/udev/rules.d/* to stop the lengthy wait on boot
  • edit /etc/sysconfig/network and disable ZEROCONF (your instance will fail to download meta data from OpenStack nova-api otherwise)

    • NOZEROCONF=yes

  • Edit /etc/profile.d/vim.sh

    • if [ -n "$BASH_VERSION" -o -n "$KSH_VERSION" -o -n "$ZSH_VERSION" ]
      then
      [ -x /usr/bin/id ] || return
      tmpid=$(/usr/bin/id -u)
      [ "$tmpid" = "" ] && tmpid=0
      # for bash and zsh, only if no alias is already set
      alias vi >/dev/null 2>&1 || alias vi=vim
      alias view >/dev/null 2>&1 || alias view='vim -R'
      fi

  • Ensure /dev/null is writeable by all

    • chmod 777 /dev/null

• That’s the modifications done, but feel free to add your own to suit your own environment so to wrap it up

  • umount /proc

  • logout

  • logout

  • sudo umount newcentos

  • To make things neat rename it appropriately

    • mv newcentos.img centos-5.5-x86_64.img

Upload CentOS image to OpenStack

• Now you have a CentOS image suitable for OpenStack you need to upload it to OpenStack.
• The tarball ships with 2 lots of kernels and ramdisks. I’ll assume you’ll be using KVM, but change the instructions to suit a Xen hypervisor.
  • Upload the kernel and make note of the ami

    • euca-bundle-image -i kvm-kernel/vmlinuz-2.6.28-11-generic
      --kernel true

    • euca-upload-bundle -b mybucket
      -m /tmp/vmlinuz-2.6.28-11-generic.manifest.xml

    • euca-register mybucket/vmlinuz-2.6.28-11-generic.manifest.xml

  • Upload the ramdisk and make a note of the ami

    • euca-bundle-image -i kvm-kernel/initrd.img-2.6.28-11-generic
      --ramdisk true

    • euca-upload-bundle -b mybucket
      -m /tmp/initrd.img-2.6.28-11-generic.manifest.xml

    • euca-register mybucket/initrd.img-2.6.28-11-generic.manifest.xml

  • Upload the machine image you modifed above, specifying the ami values from the steps above to specify the kernel and ramdisk to load with this

    • euca-bundle-image -i centos-5.5-x86_64.img
      --kernel aki-XXXXXXXX --ramdisk ari-XXXXXXXX

    • euca-upload-bundle -b mybucket
      -m /tmp/centos-5.5-x86_64.img.manifest.xml

    • euca-register mybucket/centos-5.5-x86_64.img.manifest.xml

• That’s it done (you may have to wait a short while whilst it uploads to the nova-objectstore server) – you should now see your new AMI available

  • euca-describe-images

    • IMAGE    ami-reey5wk5    mybucket/centos.5-5.x86-64.img.manifest.xml   
      myproject    available    private        x86_64    machine    ami-f4ks8moj   
      ami-jqxvgtmd

• You can now use this to launch an instance

  • euca-run-instances ami-reey5wk5 -k openstack -t m1.tiny

OpenStack: Desktop Wallpaper

I’m not the most creative person on the planet but here’s a 1600×1200 OpenStack wallpaper for you to adorn your desktop!

OpenStack: useful start, stop and restart nova compute one-liners

I find the following one liners useful for starting, stopping and restarting all the nova-* compute services – handy for development when you’re modifying config files, etc.

Stop

ls /etc/init.d/nova-* | cut -d’/’ -f4 | while read N; do sudo stop ${N}; done

Start

ls /etc/init.d/nova-* | cut -d’/’ -f4 | while read N; do sudo start ${N}; done

Restart

ls /etc/init.d/nova-* | cut -d’/’ -f4 | while read N; do sudo restart ${N}; done