Category Archives: Amazon EC2

Cloud outages happen – deal with it.

Yesterday, Tuesday 28th February 2017, Amazon’s S3 service from US-EAST-1 failed had increased error rates. Whatever you want to call it, it interrupted services across the Internet for many users. From Docker to Slack, from IoT to email. The fact it affected so many services wasn’t a fault of Amazon’s – it was a fault of the decision makers, developers and architects at the companies using Amazon’s services.

In a world where magical things happen in IT, where cool startups attract investors by the $millions, you can almost be forgiven that development gives little forethought for the possibility of an outage – especially one that has so many different variations of “9s” in its SLAs across all of its different services. However, dealing with established companies that should know better, leave a bitter taste in my mouth and makes me wonder what brought the company to run important services (to the point there are paying customers that expect a “just works” mentality) in the cloud in the first place – without an understanding of what to expect when things go wrong. Didn’t we all read the memo on “design for failure”? Despite this being an everyday staple of our IT diet, it came to light more when dealing with cloud services – one where applications should tolerate potential latent connections and services (noisy neighbours) and unknown patch schedules. So how, in 2017, are we still suffering outages because of one single entity of the whole cloud ecosystem failed?

Architects, developers and people who pay the bills: it is your responsibility to ensure services for your company are available, regardless of where that service runs. Cloud is there to allow you to consume services in an easy, digestible way. It potentially saves you time and money, considering a world where cables and server specs took most of the whiteboard, and where developers couldn’t care less about whether they’re getting a HP or a Dell box. It’s just a place where code runs, right?

Half-right.

The fact is, behind the rich veneer of cloud, there are still computers, cables and people. There is still physics connecting one part of the Internet and the other. There are still physical services sitting between your end-users and your pieces of code. That’s not to say you need to know what is going on behind the scenes – it’s more of an appreciation that someone, outside of your IT department, has solved the long-standing problem of being able to give you the services you need in a timely manner; giving you access to components without having to get a signature on a purchase order and waiting for it to be racked and stacked. Learn to appreciate that cloud meets you half way and that you have choices to make. Choices revolve around architecture, costs and operations.

Going all in with public cloud is not one that is a natural yes. For many businesses it is a no-brainer. For others it is longer journey. For some, it just isn’t a decision they need to make. After all, despite what the Clouderati want to impose, everyone gets to vote whether you agree or not with the end result. However, there are numerous public cloud success stories on the Internet, but that success wasn’t because they ticked the cloud box, it is because they employ savvy people to guide them into the best place for running their services. Architects who understand that a bridge is only as strong as its weakest point, developers who are solving problems through their end-users’s eyes – and not because some cool tech gets them to an even cooler conference, and key business people (from leaders, HR and Finance) who realise that cloud is a multi-faceted engine that needs the right fuel to function as intended. Public cloud only works when you understand its weaknesses. Regions cloudwash over the detail that there are datacenters still involved. So why did we assume we now no longer need to think about putting services into multiple locations? Cloud doesn’t solve this for you. The ability to do it is there, but developers and architects – you have to make it work yourself: understand what you have to play with, understand what limitations are, and most of all – what happens in the event of a disaster? I suspect people are having these conversations today after doing an S3 outage test in public yesterday. I also suspect people are hitting the AWS Cost Calculator hard today. You see, AWS has various tools that allow you to keep costs down on their platform. Making decisions around spot instances or reserved instances, require forethought and planning. Crossing regions, with variations in costs or further up-front reserved instance costs, change your budgets. If the cost of running multi-region AWS was greater than a “few mins of outage” from AWS, then some decisions force people to run from a single place. Is it time to review your insurance cost of the hit on your business in light of extended outages? Of course it isn’t all down to cost – what if there was an assumption that services would seamlessly run from another region – there are companies that are probably scratching their heads today wondering why services went down. We’re only human, we’re still infallible. There are a number of cogs in providing services to end-users: the cloud service provider is only one part of that puzzle.

US-EAST-1 is an interesting place – the one that tends to go on fire like this. It’s the default region when you sign up to AWS services, as well as the oldest region. How many times have you been itching to get started after you had your credit card accepted? Before you know it, you’re demonstrating the value of the cloud to decision makers and the adoption rate increases. At what point will you consider anything more than the single-region, blank canvas you’ve been painting in? I dare-say, businesses outside of the US (where end-users are not US-centric) forces this hand early, but it is sometimes an after-thought until it is pointed out.

Running in a public cloud, such as AWS, is extremely attractive, but many complex decisions lead to alternative choices such as multi-cloud (AWS, Azure and Google Compute Platform), or variations of that include hybrid (a mix of on-premises and public cloud). If people are not considering multi-region, then they have to make decisions around multi-cloud. Multi-cloud remove the dependency of a single service provider. AWS services are not infallible, else there would be 100% uptime SLAs on all their services. The fact they do not implies failure at some point, no matter how big or small: writing a number on an SLA does not automatically mean the code or operational procedures are airtight. Therefore, multi-cloud is about spreading risk. Multi-cloud is made possible when tools and development are abstracted away from specifics of a particular cloud service provider. For example, specifying APIs directly with one provider will exclude those scripts from working with one that doesn’t share the same syntax. Moving to a multi-cloud world suddenly unlocks you from the clutches of a single provider. In a non-cloud world, we used to call that lock-in. We cloudwash over this when thinking AWS because it is so for up and to the right in the Gartner Magic Quadrant(tm) that we think it’s ok to be there. If I was paying the bills, I’d worry what that kind of long-term tie-in might be for a smaller voice in the cloud world. This isn’t to send FUD about AWS – far from it. Savvy companies are realising the benefits of multi-cloud, and simply not risking all their eggs in one basket. If anything, it gives ultimate choice and flexibility and that should only be highly encouraged. It attracts diverse talent. It reduces reliance on specific CSP services. IT could even attract cost savings through negotiations knowing business are using multiple clouds – each trying to win one over the other for PR gain.

A variation on the multi-cloud is hybrid – where portions of services run as a private cloud in datacenters with fixed, controlled resources, as well as utilising public cloud services where it makes sense. It is a mistake to even begin to use the likes of OpenStack alone as a way of combatting the S3 outage. It is actually embarrassing. If you’re considering enhancing current datacenter operations, looking to revert an otherwise ambitious datacenter exit strategy and require on-premises resources for efficient delivery of services, or even looking at a datecenter exit strategy, OpenStack should be high on your list of technologies to help realise this plan. OpenStack, complementing services to VMware or more legacy infrastructure is immensely popular in unlocking access to cloud services to enhance internal IT services. However, utilising services such as OpenStack on-premises doesn’t solve any problem like the S3 outage alone because the S3 outage affected people who didn’t think multi-region. OpenStack, deployed across multiple datacenters is the counter-argument but consideration still needs to be made around apparent cloud-lock-in. However, most successful implementations of OpenStack have end-users already verse in using tools that abstract away from the underlying infrastructure: a key tenet of multi-cloud. Therefore, given that my experience and wages come indirectly from OpenStack, I get to see a massive potential for established Enterprises wanting to utilise cloud tools to speed up delivery of services and development, whilst taking a risk-free approach to cloud, by using OpenStack as an alternative platform, and immediately unlocking both on-premises and public cloud. It isn’t a completely solved problem. Cloud meets you half-way. The other half is filled by the same people who would solve any cloud-resiliency problem: architects, developers, decision makers.

AWS outages will happen. Azure outages will happen. OpenStack outages will happen. Deal with it because your customers will deal with those situations too by voting with their feet.

CloudCamp Warrington, February 23rd hosted by @appsense #aws #openstack

I’ll be doing a short (well, I can get carried away but the plan is short) talk on what Autotrader.co.uk is doing in terms of cloud computing which includes efforts and plans around OpenStack and AWS.

You will find some great people at the informal event – from people who are actively involved in cloud technology, to those that are just starting their cloud journey.

If you’re in the North West of the UK and fancy coming along – it is being kindly hosted by the Warrington offices of AppSense Ltd.  The evening is turning out to be a great forum and the hope is that more events in the area will follow.  If you have an interest in the cloud bubble, please come along for a chat over some drinks and some food.

Full details here: http://cloudcamp.org/warrington

Running OpenStack under VirtualBox – A Complete Guide (Part 2)

In the previous post, we very simply got OpenStack running under VirtualBox. This next part takes this further by getting multiple compute nodes installed to spread the load of your instances. It also paves the way for Part 3 where we will then start to look at Swift, OpenStack’s storage implementation of S3.

Part 2 – OpenStack on a multiple VirtualBox VMs with OpenStack instances accessible from host

We will be using the same set up as for Part 1.

The proposed environment

  • OpenStack “Public” Network: 172.241.0.0/25
  • OpenStack “Private” Network: 10.0.0.0/8
  • Host has access to its own LAN, separate to this on 192.168.0.0/16 and not used for this guide
  • One VirtualBox VM running the software needed for the controller
  • One VirtualBox VM running the software needed for a compute node

This guide will assume you have followed Part 1. We are simply adding in a compute node now, so that will make the virtual machine you created in Part 1 will become the Cloud Controller (CC). Also, OpenStack has been designed so that any part of the environment can run on a separate server. For this guide we will have the following

  • Cloud Controller running MySQL, RabbitMQ, nova-network, nova-scheduler, nova-objectstore, nova-api, nova-compute
  • Compute node running: nova-compute

The Guide

  • Add a new VirtualBox Guest
    • Name: cloud2
      • OS Type: Linux
      • Version: Ubuntu (64-Bit)
    • 2048Mb Ram
    • Boot Hard Disk
      • Dynamically Expanding Storage
      • 8.0Gb
    • After this initial set up, continue to configure the guest
      • Storage:
        • Edit the CD-ROM so that it boots Ubuntu 10.10 Live or Server ISO
        • Ensure that the SATA controller has Host I/O Cache Enabled (recommended by VirtualBox for EXT4 filesystems)
      • Network:
        • Adapter 1
          • Host-only Adapter
          • Name: vboxnet0
        • Adapter 2
          • NAT
          • This will provide the default route to allow the VM to access the internet to get the updates, OpenStack scripts and software
      • Audio:
        • Disable (just not required)
    • Boot the guest and install Ubuntu as per normal
  • Assign static IPs to the cloud controller
    • Ensure that the Cloud Controller you created in Part 1 has static addresses for eth0 and eth1.
      • For the sake of this guide, I’m assuming you have assigned the following
        • eth0: 172.241.0.101/255.255.255.128.  This address is your Cloud Controller address (CC_ADDR) and will be the API interface address you will be communicating on from your host.
        • eth1: stays as dhcp as it is only used for NAT’d access to the real world
    • Your compute nodes don’t need to be set statically, but for the rest of this guide it is assumed the addresses are as follows
      • Cloud2
        • eth0: 172.241.0.102/255.255.255.128
        • eth1: stays as dhcp as it is only used for NAT’d access to the real world
  • Grab this script to install OpenStack
    • This will set up a repository (ppa:nova/trunk) and install MySQL server where the information regarding your cloud will be stored
    • The options specified on the command line match the environment described above
    • wget --no-check-certificate 
      https://github.com/uksysadmin/OpenStackInstaller/raw/master/OSinstall.sh
  • Run the script (as root/through sudo) specifying that you want a compute node and that you’ve specified the IP address of the Cloud Controller
    • sudo bash ./OSinstall.sh -A $(whoami) -T compute -C 172.241.0.101
  • No further configuration is required.
  • Once completed, ensure that the Cloud Controller knows about this new compute node
    • On the Cloud Controller run the following
      • mysql -uroot -pnova nova -e 'select * from services;'
      • sudo nova-manage service list
      • You should see your new compute node listed under hosts
      • If you don’t have a DNS server running that resolves these hosts add your new compute node to /etc/hosts
        • 172.241.0.102 cloud2
  • On the new compute node run the following
    • sudo nova-manage db sync
  • As you copied your credentials from the cloud controller created in Part 1, you should just be able to continue to use this from your host – but this time you can spin up more guests.
    • If you changed the eth0 address of your cloud controller, ensure your cloud/creds/novarc environment file has the correct IP.
  • Repeat the steps above to create further compute nodes in your environment, scaling seamlessly

Running OpenStack under VirtualBox – A Complete Guide (Part 1)

UPDATE: I’ve been working on a new version of the script which can be used to create an OpenStack host running on Ubuntu 12.04 Precise Pangolin and the Essex release.
I’ve now got a video to accompany this which is recommended over this guide
Head over to  ‎http://uksysadmin.wordpress.com/2012/03/28/screencast-video-of-an-install-of-openstack-essex-on-ubuntu-12-04-under-virtualbox/

Running OpenStack under VirtualBox allows you to have a complete multi-node cluster that you can access and manage from the computer running VirtualBox as if you’re accessing a region on Amazon.
This is a complete guide to setting up a VirtualBox VM running Ubuntu, with OpenStack running on this guest and an OpenStack instance running, accessible from your host.

Part 1 – OpenStack on a single VirtualBox VM with OpenStack instances accessible from host

The environment used for this guide

  • A 64-Bit Intel Core i7 Laptop, 8Gb Ram.
  • Ubuntu 10.10 Maverick AMD64 (The “host”)
  • VirtualBox 4
  • Access from host running VirtualBox only (so useful for development/proof of concept)

The proposed environment

  • OpenStack “Public” Network: 172.241.0.0/25
  • OpenStack “Private” Network: 10.0.0.0/8
  • Host has access to its own LAN, separate to this on 192.168.0.0/16 and not used for this guide

The Guide

  • Download and install VirtualBox from http://www.virtualbox.org/
  • Under Preferences… Network…
  • Add/Edit Host-only network so you have vboxnet0. This will serve as the “Public interface” to your cloud environment
    • Configure this as follows
      • Adapter
        • IPv4 Address: 172.241.0.100
        • IPv4 Network Mask: 255.255.255.128
      • DHCP Server
        • Disable Server
    • On your Linux host running VirtualBox, you will see an interface created called ‘vboxnet0’ with the address specified as 172.241.0.100. This will be the IP address your OpenStack instances will see when you access them.
    • Create a new Guest
      • Name: Cloud1
        • OS Type: Linux
        • Version: Ubuntu (64-Bit)
      • 1024Mb Ram
      • Boot Hard Disk
        • Dynamically Expanding Storage
        • 8.0Gb
      • After this initial set up, continue to configure the guest
        • Storage:
          • Edit the CD-ROM so that it boots Ubuntu 10.10 Live or Server ISO
          • Ensure that the SATA controller has Host I/O Cache Enabled (recommended by VirtualBox for EXT4 filesystems)
        • Network:
          • Adapter 1
            • Host-only Adapter
            • Name: vboxnet0
          • Adapter 2
            • NAT
            • This will provide the default route to allow the VM to access the internet to get the updates, OpenStack scripts and software
        • Audio:
          • Disable (just not required)
    • Power the guest on and install Ubuntu
    • For this guide I’ve statically assigned the guest with the IP: 172.241.0.101 for eth0 and netmask 255.255.255.128.  This will be the IP address that you will use to access the guest from your host box, as well as the IP address you can use to SSH/SCP files around.
    • Once installed, run an update (sudo apt-get update&&sudo apt-get upgrade) then reboot
    • If you’re running a desktop, install the Guest Additions (Device… Install Guest Additions, then click on Places and select the VBoxGuestAdditions CD and follow the Autorun script), then Reboot
    • Install openssh-server
      • sudo apt-get -y install openssh-server
    • Grab this script to install OpenStack
      • This will set up a repository (ppa:nova/trunk) and install MySQL server where the information regarding your cloud will be stored
      • The options specified on the command line match the environment described above
      • wget https://github.com/uksysadmin/OpenStackInstaller/raw/master/OSinstall.sh
    • Run the script (as root/through sudo)
      • sudo bash ./OSinstall.sh -A $(whoami)
    • Run the post-configuration steps
      • ADMIN=$(whoami)
        sudo nova-manage user admin ${ADMIN}
        sudo nova-manage role add ${ADMIN} cloudadmin
        sudo nova-manage project create myproject ${ADMIN}
        sudo nova-manage project zipfile myproject ${ADMIN}
        mkdir -p cloud/creds
        cd cloud/creds
        unzip ~/nova.zip
        . novarc
        cd
        euca-add-keypair openstack > ~/cloud/creds/openstack.pem
        chmod 0600 cloud/creds/*

    Congratulations, you now have a working Cloud environment waiting for its first image and instances to run, with a user you specified on the command line (yourusername), the credentials to access the cloud and a project called ‘myproject’ to host the instances.

    • You now need to ensure that you can access any instances that you launch via SSH as a minimum (as well as being able to ping) – but I add in access to a web service and port 8080 too for this environment as my “default” security group.
      • euca-authorize default -P tcp -p 22 -s 0.0.0.0/0
        euca-authorize default -P tcp -p 80 -s 0.0.0.0/0
        euca-authorize default -P tcp -p 8080 -s 0.0.0.0/0
        euca-authorize default -P icmp -t -1:-1
    • Next you need to load a UEC image into your cloud so that instances can be launched from it
      • image="ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz"
        wget http://smoser.brickies.net/ubuntu/ttylinux-uec/$image
        uec-publish-tarball $image mybucket
    • Once the uec-publish-tarball command has been run, it will present you with a line with emi=, eri= and eki= specifying the Image, Ramdisk and Kernel as shown below. Highlight this, copy and paste back in your shell
      Thu Feb 24 09:55:19 GMT 2011: ====== extracting image ======
      kernel : ttylinux-uec-amd64-12.1_2.6.35-22_1-vmlinuz
      ramdisk: ttylinux-uec-amd64-12.1_2.6.35-22_1-initrd
      image  : ttylinux-uec-amd64-12.1_2.6.35-22_1.img
      Thu Feb 24 09:55:19 GMT 2011: ====== bundle/upload kernel ======
      Thu Feb 24 09:55:21 GMT 2011: ====== bundle/upload ramdisk ======
      Thu Feb 24 09:55:22 GMT 2011: ====== bundle/upload image ======
      Thu Feb 24 09:55:25 GMT 2011: ====== done ======
      emi="ami-fnlidlmq"; eri="ami-dqliu15n"; eki="ami-66rz6vbs";
    • To launch an instance
      • euca-run-instances $emi -k openstack -t m1.tiny
    • To check its running
      • euca-describe-instances
      • You will see the Private IP that has been assigned to this instance, for example 10.0.0.3
    • To access this via SSH
      • ssh -i cloud/creds/openstack.pem root@10.0.0.3
      • (To log out of ttylinux, type: logout)
    • Congratulations, you now have an OpenStack instance running under OpenStack Nova, running under a VirtualBox VM!
    • To access this outside of the VirtualBox environment (i.e. back on your real computer, the host) you need to assign it a “public” IP
      • Associate this to the instance id (get from euca-describe-instances and will be of the format i-00000000)
        • euca-allocate-address
        • This will return an IP address that has been assigned to your project so that you can now associate to your instance, e.g. 172.241.0.3
        • euca-associate-address -i i-00000001 172.241.0.3
      • Now back on your host (so outside of VirtualBox), grab a copy of cloud/creds directory
        • scp -r user@172.241.0.101:cloud/creds .
      • You can now access that host using the Public address you associated to it above
        • ssh -i cloud/creds/openstack.pem root@172.241.0.3

    CONGRATULATIONS! You have now created a complete cloud environment under VirtualBox that you can manage from your computer (host) as if you’re managing services on Amazon. To demonstrate this you can terminate that instance you created from your computer (host)

    • sudo apt-get install euca2ools
      . cloud/creds/novarc
      euca-describe-instances
      euca-terminate-instances i-00000001

    Credits

    This guide is based on Thierry Carrez’ blog @ http://fnords.wordpress.com/2010/12/02/bleeding-edge-openstack-nova-on-maverick/

  • Next: Part 2 – OpenStack on a multiple VirtualBox VMs with OpenStack instances accessible from host