Home Rackspace Private Cloud / OpenStack Lab: Part 3

In the first two posts I covered the basics: what hardware is involved and the basic network services which forms the basis of my Rackspace Private Cloud install. In this post I set up Rackspace Private Cloud to give an OpenStack environment consisting of highly available controllers running as a pair with services such as the OpenStack APIs, Neutron, Glance and Keystone and 3 compute servers allowing me flexibility to do some testing.

Home Lab Network DiagramTo recap, this is my environment showing the Rackspace Private Cloud. Click on the picture to view a bigger version.

To install Rackspace Private Cloud using the Rackspace Chef Cookbooks, we need to first have Chef installed.  In this environment, to make most use of the hardware, I’ll be installing Chef onto the first of my hosts, openstack1, which we will refer to as a controller to facilitate the installation of RPC via the Chef Cookbooks. If you have Chef already installed in your environment, you can skip these steps and head straight to step 7 below.  In Rackspace Private Cloud, Controllers are the servers that run the APIs and OpenStack services such as Glance, Keystone and Neutron.  We also run MySQL and RabbitMQ on these too.  Through the use of our cookbooks, when we utilise two of these Controllers, we end up with a HA pair, giving highly available Controller services.

Rackspace Private Cloud provides scripts to make installation of an OpenStack powered private cloud very easy – including the initial setup of Chef and underlying services. Head over to the Rackspace Private Cloud website for more information on this. This blog post pulls out those steps to show complete transparency and give greater control over my installation.

Installation of RabbitMQ

As we will be running Chef alongside other OpenStack services, we need to do some initial setup and configuration of services such as RabbitMQ to ensure common services and ports don’t conflict and operate seamlessly with one another.

1. We first install some pre-requisite packages

apt-get update
apt-get install -y python-dev python-pip git erlang erlang-nox erlang-dev curl lvm2

2. Then we install and set up RabbitMQ on openstack1 (  Both Chef and OpenStack will be set up to use the same RabbitMQ service.

# Ensure our Rabbit environment doesn't lose it's settings later on
mkdir -p /var/lib/rabbitmq
echo -n "ANYRANDOMSTRING" > /var/lib/rabbitmq/.erlang.cookie
chmod 600 /var/lib/rabbitmq/.erlang.cookie
wget -O /tmp/rabbitmq.asc ${RABBITMQ_KEY};
apt-key add /tmp/rabbitmq.asc
wget -O /tmp/rabbitmq.deb ${RABBITMQ}
dpkg -i /tmp/rabbitmq.deb

3. Now that we have RabbitMQ installed, we need to configure it so that Chef can utilise it.  To do this we create a vhost and appropriate user on here as follows:

rabbitmqctl add_vhost /chef
rabbitmqctl add_user chef $CHEF_RMQ_PW
rabbitmqctl set_permissions -p /chef chef '.*' '.*' '.*'

Installation of Chef Server

4. Now that we have RabbitMQ setup on one of our nodes (openstack1) we can install Chef Server. To do this grab the package for Ubuntu 12.04 from the opscode.com website as shown below:

wget -O /tmp/chef_server.deb ${CHEF}
dpkg -i /tmp/chef_server.deb

5. We can then configure Chef Server for our environment, where we set various configuration items such as the ports to run on and where RabbitMQ is, along with the password we created for the chef user in step 2. This is done by running the following commands:

RMQ_IP=""    # openstack1

mkdir -p /etc/chef-server
cat > /etc/chef-server/chef-server.rb <<EOF
erchef["s3_url_ttl"] = 3600
nginx["ssl_port"] = 4000
nginx["non_ssl_port"] = 4080
nginx["enable_non_ssl"] = true
rabbitmq["enable"] = false
rabbitmq["password"] = "${CHEF_RMQ_PW}"
rabbitmq["vip"] = "${RMQ_IP}"
rabbitmq['node_ip_address'] = "${RMQ_IP}"
chef_server_webui["web_ui_admin_default_password"] = "openstack"
bookshelf["url"] = "https://#{node['ipaddress']}:4000"

chef-server-ctl reconfigure

Installation of Chef Client

6. After this has pulled down and packages and dependencies, we can configure the Chef client as shown below:

# Make sure knife can be found
ln -sf /opt/chef-server/embedded/bin/knife /usr/bin/knife 

SYS_IP=$(ohai ipaddress | awk '/^ / {gsub(/ *\"/, ""); print; exit}')
export CHEF_SERVER_URL=https://${SYS_IP}:4000
# Configure Knife
mkdir -p /root/.chef
cat > /root/.chef/knife.rb <<EOF
log_level :info
log_location STDOUT
node_name 'admin'
client_key '/etc/chef-server/admin.pem'
validation_client_name 'chef-validator'
validation_key '/etc/chef-server/chef-validator.pem'
chef_server_url "https://${SYS_IP}:4000"
cache_options( :path => '/root/.chef/checksums' )
cookbook_path [ '/opt/chef-cookbooks/cookbooks' ]

Upload Rackspace Private Cloud Cookbooks to Chef Server

7. With Chef Server running and the Chef client configured, we can grab the Rackspace Private Cloud Cookbooks from GitHub and upload them to our Chef Server. We do this as follows:

COOKBOOK_VERSION="v4.2.1"     # Check versions here
mkdir -p /opt/
if [ -d "/opt/chef-cookbooks" ];then
    rm -rf /opt/chef-cookbooks
git clone https://github.com/rcbops/chef-cookbooks.git /opt/chef-cookbooks
pushd /opt/chef-cookbooks
git checkout ${COOKBOOK_VERSION}
git submodule init
git submodule sync
git submodule update
# Upload all of the RCBOPS Cookbooks
knife cookbook upload -o /opt/chef-cookbooks/cookbooks -a

knife role from file /opt/chef-cookbooks/roles/*.rb

Configuration of Environment for Rackspace Private Cloud

With the cookbooks uploaded into Chef, we can now configure our environment ready for an installation. For this home lab we have 2 Controllers (openstack1 and openstack2) and 3 Computes (openstack3, openstack4 and openstack5).  Each of the servers has 2 nics:

eth0 is on the LAN Subnet of and is our Management network.

eth1 has not been assigned an IP and will be used for Neutron and is known as the Provider network.

This means we will configure our environment so that eth1 will be used for Neutron, and when we get to use our environment (GUI or CLI), we will be accessing the environment using a 192.168.1.X/24 address – just like any other server or computer on this LAN.

The /etc/network/interfaces for these servers have the following contents:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback

# Host/Management
auto eth0
iface eth0 inet dhcp
# Neutron Provider interface
auto eth1
iface eth1 inet manual
  up ip link set $IFACE up
  down ip link set $IFACE down

8. After this we can create our Environment JSON file which describes our complete setup for Rackspace Private Cloud:

VIP_PREFIX="192.168.1"    # Home lab network is

cat > /opt/base.env.json <<EOF
  "name": "rpcs",
  "description": "Environment for Openstack Private Cloud",
  "cookbook_versions": {
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
  "override_attributes": {
  "monitoring": {
  "procmon_provider": "monit",
  "metric_provider": "collectd"
  "enable_monit": true,
  "osops_networks": {
    "management": "${VIP_PREFIX}.0/24",
    "swift": "${VIP_PREFIX}.0/24",
    "public": "${VIP_PREFIX}.0/24",
    "nova": "${VIP_PREFIX}.0/24"
  "rabbitmq": {
    "open_file_limit": 4096,
    "use_distro_version": false
  "nova": {
    "config": {
      "use_single_default_gateway": false,
      "ram_allocation_ratio": 1.5,
      "disk_allocation_ratio": 1.0,
      "cpu_allocation_ratio": 8.0,
      "resume_guests_state_on_host_boot": false
    "network": {
      "provider": "neutron"
    "scheduler": {
      "default_filters": [
    "libvirt": {
      "vncserver_listen": "",
      "virt_type": "kvm"
  "keystone": {
    "pki": {
      "enabled": false
    "admin_user": "admin",
    "tenants": [
    "users": {
      "admin": {
        "password": "secrete",
        "roles": {
          "admin": [
      "demo": {
        "password": "secrete",
        "default_tenant": "demo",
        "roles": {
          "Member": [
      "demo2": {
        "password": "secrete",
        "default_tenant": "demo2",
        "roles": {
          "Member": [
  "neutron": {
    "ovs": {
      "external_bridge": "",
      "network_type": "gre",
      "provider_networks": [
          "bridge": "br-eth1",
          "vlans": "1024:1024",
          "label": "ph-eth1"
  "mysql": {
    "tunable": {
      "log_queries_not_using_index": false
    "allow_remote_root": true,
    "root_network_acl": ""
  "vips": {
    "horizon-dash": "${VIP_PREFIX}.${API_VIP}",
    "keystone-service-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-xvpvnc-proxy": "${VIP_PREFIX}.${API_VIP}",
    "nova-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-metadata-api": "${VIP_PREFIX}.${API_VIP}",
    "cinder-api": "${VIP_PREFIX}.${API_VIP}",
    "nova-ec2-public": "${VIP_PREFIX}.${API_VIP}",
    "config": {
      "${VIP_PREFIX}.${API_VIP}": {
        "vrid": 12,
        "network": "public"
      "${VIP_PREFIX}.${MYSQL_VIP}": {
        "vrid": 10,
        "network": "public"
      "${VIP_PREFIX}.${AMQP_VIP}": {
        "vrid": 11,
        "network": "public"
    "rabbitmq-queue": "${VIP_PREFIX}.${AMQP_VIP}",
    "nova-novnc-proxy": "${VIP_PREFIX}.${API_VIP}",
    "mysql-db": "${VIP_PREFIX}.${MYSQL_VIP}",
    "glance-api": "${VIP_PREFIX}.${API_VIP}",
    "keystone-internal-api": "${VIP_PREFIX}.${API_VIP}",
    "horizon-dash_ssl": "${VIP_PREFIX}.${API_VIP}",
    "glance-registry": "${VIP_PREFIX}.${API_VIP}",
    "neutron-api": "${VIP_PREFIX}.${API_VIP}",
    "ceilometer-api": "${VIP_PREFIX}.${API_VIP}",
    "ceilometer-central-agent": "${VIP_PREFIX}.${API_VIP}",
    "heat-api": "${VIP_PREFIX}.${API_VIP}",
    "heat-api-cfn": "${VIP_PREFIX}.${API_VIP}",
    "heat-api-cloudwatch": "${VIP_PREFIX}.${API_VIP}",
    "keystone-admin-api": "${VIP_PREFIX}.${API_VIP}"
  "glance": {
    "images": [
    "image": {
    "image_upload": false
  "osops": {
    "do_package_upgrades": false,
    "apply_patches": false
  "developer_mode": false

9. When we execute the above command, we end up with a file call /opt/base.env.json applicable to the home lab environment. We then load this into Chef as follows:

knife environment from file /opt/base.env.json

10. With the environment loaded, we can simply install Rackspace Private Cloud with just a few commands and a cup of coffee. In my environment I check that my proxy server is configured in /etc/apt/apt.conf.d/01squid as follows (as we’ll be installing a lot of the same packages on all the hosts):

Acquire::http { Proxy ""; };

Bootstrapping the Controllers (Installing Rackspace Private Cloud)

11. We are now ready to bootstrap the nodes (install Chef on each node), assign roles to them and install Rackspace Private Cloud to them.  The first servers to do this on are the Controllers (openstack1 and openstack2). We assign the roles of ha-controller1 to openstack1 and ha-controller2 to openstack2, a well as the role of single-neutron to each which preps these for the roles assigned.  We do this as follows:


knife bootstrap -E rpcs -r role[ha-controller1],role[single-network-node] ${CONTROLLER1}
knife bootstrap -E rpcs -r role[ha-controller2],role[single-network-node] ${CONTROLLER2}

12. This will fetch and install Chef Client and configure their roles within Chef Server. We can then run chef-client to apply the roles to the servers. As we are running a HA pair, we run chef-client in the following order: first on openstack1, then on openstack2, then finally on openstack1 again. This is because of the preparation and work to sync MySQL and RabbitMQ between the two to allow us to operate them in a HA (failover) capacity:

# On openstack1
# Configure Rabbit HA Policy
knife ssh -C1 -a ipaddress 'role:*controller*' 
    "rabbitmqctl set_policy ha-all '.*' '{\"ha-mode\":\"all\", \"ha-sync-mode\":\"automatic\"}' 0"

# SSH to openstack2 and run chef-client
ssh openstack2 chef-client

# Back on openstack1

Bootstrapping the Computes

13. For the Computes it’s even simpler. We simply assign the role single-compute to them then execute chef-client across each of them as follows

for node in {103..105}; do
 knife bootstrap -E rpcs -r role[single-compute] 192.168.1.${node}
knife ssh "role:single-compute" chef-client

Finalising the install

14. Lastly we add the eth1 device to the bridge, then reboot the cluster so everything starts up as expected each time

knife ssh "role:*" "ovs-vsctl add-port br-eth1 eth1&&reboot"

Tip: I’ve seen RabbitMQ cause a problem when first rebooting. If it fails to die due to multiple status checks running, kill it with:
ps -ef | awk ‘/rabbitmq/ {print $2}’ | while read R; do kill -9 $R; done

Congratulations! You now have a home lab ready for testing Rackspace Private Cloud! 

I tend to wrap all these commands into a single script and execute from my first node where Chef is to be installed, openstack1 (Recall that I grab a shell script (install-openstack.sh) as part of the Ubuntu installation process). This will run through all these steps fully automated. On my network and slow ADSL (and the N40L isn’t the fastest server on the planet!), I tend to have Rackspace Private Cloud up and running in about 90 mins from boot.

In the next post we’ll look at configuring some basic Neutron Networking which I set up after each installation of RPC on my home lab.  This includes private networking and a Flat Network to allow me to access my instances from laptops on my home network.

Thanks to @cloudnull (Kevin Carter, Rackspace) and @IRTermite (Dale Bracey) for help fine-tuning these steps!


4 thoughts on “Home Rackspace Private Cloud / OpenStack Lab: Part 3

  1. […] the next post I’ll cover off the steps to get an HA Rackspace Private Cloud installed using the Rackspace […]

  2. […] after following the first three posts, we now have a Rackspace Private Cloud by OpenStack running with 2 Controllers (HA) and 3 Computes. […]

  3. […] first four of these posts covered setup and installation of my home lab, including the networking, PXE booting Ubuntu […]

  4. […] Lab Part 3: Installation of Rackspace Private Cloud […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: