August 27, 2014

Mirantis

OpenStack Silicon Valley Featured Track: Networking Fact and Fantasy

Over these final few weeks leading up to OpenStack Silicon Valley, we will be featuring the 4 different tracks of the show, which will take place once the keynote speeches have ended. Today’s track: Networking Fact and Fantasy.

“Does Software-Defined Networking in OpenStack work?” Kinda.

After virtualizing compute resources, virtualizing the network layer of the stack is the next step in making the cloud datacenter more agile, dense, and efficient. Software-defined networking can be used to do this in OpenStack, but only in certain cases. This track will examine which networking options are most prevalent in the OpenStack ecosystem, and how to build an SDN topology when planning for hybrid cloud and scale-out scenarios. The track also examines OpenStack deployments that are currently running different SDN architectures in production, as well as production-hardened deployments that rely on the original Nova Networking framework. Additionally, the track looks at the new OpenStack project, Congress, which promises to simplify policy management. This track aims to leave participants with a balanced, informed view of SDN realities, ready to lead their organizations through the multitude of choices and tradeoffs in embracing SDN for their OpenStack deployments.

The Networking Fact and Fantasy Track is sponsored by Juniper, and is hosted by track chair Kyle Mestery. The track is divided into the following 4 sessions:

Session 1: Popular Options for SDN Deployment in OpenStack

OpenStack Networking (Neutron) is maturing, but is still a work in progress. So what options are available for polishing the rough edges of Neutron and building a production-grade solution? This session illuminates the different options.

Speakers:

  • Jonathan LaCour, Dreamhost

  • Peter Balland, VMWare

  • Mike Dvorkin, Cisco

Session 2: Hybrid Cloud and SDN: Special Considerations

SDN on OpenStack is further complicated when hybrid cloud is in your plans. This session looks at how to plan an SDN architecture without throwing a wrench in your hybrid cloud scenario.

Speakers:

  • Sean Roberts, Yahoo

  • Lane Patterson, Symantec

  • Aleksey Mints, Juniper

Session 3: Scale-Out Planning When Using SDN

Scale-out planning is key to any agile cloud planning strategy. When adding SDN into the mix, additional thought is required to avoid painting yourself into a corner through the wrong combination of switch hardware and software when scaling from a couple of racks to a couple dozen racks. This session helps attendees plan for success when scaling their virtual networks.

Speakers:

  • Pedro Marques, Juniper

  • Kyle Forster, Big Switch

  • Mark McClain, Yahoo

  • Pere Monclus, PLUMgrid

Session 4: Four SDN Case Studies

This final session of the SDN track will give attendees the chance to hear from folks who have deployed SDN in OpenStack. After each of these 4 brief case studies, there will be plenty of time for Q&A.

Speakers:

  • Lane Patterson, Symantec

  • Jonathan LaCour, DreamHost

  • Adam Johnson, Midokura

  • Ben Lin, VMWare

Don’t miss out on OpenStack Silicon Valley and its incredible lineup of speakers – register today!

The post OpenStack Silicon Valley Featured Track: Networking Fact and Fantasy appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Samuel Lee at August 27, 2014 04:23 PM

Tesora Corp

Short Stack: VMware embraces OpenStack, HP doesn't want Rackspace either and top automaker chooses OpenStack

short stack_b small_0_0.jpgWelcome to the Short Stack, our weekly feature where we search for the most intriguing OpenStack links to share with you. These links may come from traditional publications or company blogs, but if it's about OpenStack, we'll find the best links we can to share with you every week.

If you like what you see, please consider subscribing.

Here we go with this week's links:

VMware: OpenStack, Docker, Open Compute Project support announced | CITEworld

VMware announced support for OpenStack and Docker this week. OpenStack is probably just another operating system in VMware's eyes, but Docker is a VMware killer. I guess they decided sleeping with the enemy was the way to go here.

HP Is Still Not Buying Rackspace | Re/code

Nobody seems to want to buy poor Rackspace. HP shot down a recent rumor that it wanted to buy the beleaguered company. HP has made strong moves into OpenStack this year, releasing a slew of products, and it would make sense to buy Rackspace, but apparently they don't see a fit for some reason.

Rackspace doubles down on the managed cloud and VMware | TechRepublic

Meanwhile Rackspace keeps pulling levers and pushing buttons and this week came news that the company which had previously announced it was getting out of the infrastructure as a service business and moving to a so-called managed cloud, also appear to be hitching its wagon to VMware, which is an interesting strategy given VMware is under a lot of competitive pressure itself.

Why a Top Automaker Adopted OpenStack | Superuser

Why would a major car maker choose OpenStack? It needs to build a private cloud to process massive amounts of telematics data it's been collecting and OpenStack seemed like the perfect choice. Seems every week, we have another story of OpenStack adoption as it plants itself firmly in the mainstream.

Dogs and Cats Living Together?! Eucalyptus and OpenStack | CSC Engage

Of all the strange announcements we've heard about companies supporting OpenStack, none is stranger than the case of Eucalyptus, until recently a huge critic of OpenStack, suddenly deciding to be an OpenStack contributor. Is it recognition of the power of the platform or another case of keeping your enemies close?

 

by 693 at August 27, 2014 12:12 PM

Elizabeth K. Joseph

OpenStack Infrastructure August 2014 Bug Day

The OpenStack Infrastructure team has a pretty big bug collection.

1855 collection
Well, not literal bugs

We’ve slowly been moving new bugs for some projects over to StoryBoard in order to kick the tires on that new system, but today we focused back on our Launchpad Bugs to par down our list.

Interested in running a bug day? The steps we have for running a bug day can be a bit tedious, but it’s not hard, here’s the rundown:

  1. I create our etherpad: cibugreview-august2014 (see etherpad from past bug days on the wiki at: InfraTeam#Bugs)
  2. I run my simple infra_bugday.py script and populate the etherpad.
  3. Grab the bug stats from launchpad and copy them into the pad so we (hopefully) have inspiring statistics at the end of the day.
  4. Then comes the real work. I open up the old etherpad and go through all the bugs, copying over comments from the old etherpad where applicable and making my own comments as necessary about obvious updates I see (and updating my own bugs).
  5. Let the rest of the team dive in on the etherpad and bugs!

Throughout the day we chat in #openstack-infra about bug statuses, whether we should continue pursuing certain strategies outlined in bugs, reaching out to folks who have outstanding bugs in the tracker that we’d like to see movement on but haven’t in a while. Plus, we get to triage a whole pile of New bugs (thanks Clark) and close others we may have lost track of (thanks everyone).

As we wrap up, here are the stats from today:

Starting bug day count: 270

31 New bugs
39 In-progress bugs
6 Critical bugs
15 High importance bugs
8 Incomplete bugs

Ending bug day count: 233

0 New bugs
37 In-progress bugs
3 Critical bugs
10 High importance bugs
14 Incomplete bugs

Full disclosure, 4 of the bugs we “closed” were actually moved to the Zuul project on Launchpad so we can import them into StoryBoard at a later date. The rest were legitimate though!

It was a busy day, thanks to everyone who participated.

by pleia2 at August 27, 2014 12:08 AM

August 26, 2014

OpenStack @ NetApp

The results are in - Manila is an OpenStack incubated project!

The results are in - Manila is an OpenStack incubated project!

In today's meeting of the OpenStack technical committee (comprised of 13 representatives across multiple vendors, charged with enforcing OpenStack ideals (such as Openness, Transparency, Commonality, Integration and Quality), deciding on issues that impact multiple programs, providing an ultimate appeals board for technical decisions and general oversight), Manila was voted into official incubation status!

Explicitly the technical committee formally accepted the creation of shared file systems project, as well as the incubation of the Manila project. This is indicative of strategic alignment with OpenStack, acknowledgement of Manila being developed within the OpenStack community according to established processes, and in recognition of recent momentum in the project – new contributors, reviewers, and deployers.

Congratulations to the entire Manila team – we’re looking forward to lots of great work in the coming months!

Here are some links to learn more about Manila and how to deploy it:

August 26, 2014 08:00 PM

OpenStack Blog

OpenStack Community Events in September

If you want to hear some great content before the OpenStack Summit in Paris this November, attend one of these OpenStack Community events being held around the world. Join fellow community members, learn from user stories and hear directly from industry leaders. Hurry, registration deadlines are quickly approaching!

Screen Shot 2014-08-26 at 9.58.24 AM

On September 16, attend OpenStack Silicon Valley to hear from industry thought leaders on a wide range of topics, including OpenStack in the Enterprise, OpenStack for NFV, Emerging Trends in OpenStack in addition to a half-day OpenStack training. 

Review the full agenda and register before prices increase on August 28! 

When: Tuesday, September 16, 2014
Where: Computer History Museum, Mountain View, California
Tickets: If you are registered to attend the OpenStack Summit in Paris, email paris-summit@openstacksv.com to register for free.
Featured Speakers:

  • Alessandro Perilli, Red Hat
  • Marten Mickos, Eucalyptus Systems
  • Jesse Proudman, Blue Box
  • Chris Kemp, Nebula
  • Jonathan Bryce, OpenStack Foundation
  • Boris Renski, Mirantis
  • & more!

 

Screen Shot 2014-08-26 at 10.11.04 AM

OpenStack Conference Benelux 2014
Register for the first official OpenStack Conference Benelux 2014, a one-day conference for developers, users and administrators of OpenStack Cloud Software. Whether you are just getting started and want to learn more about OpenStack, or are an advanced user, there’s content planned for you, including case studies, technical talks and workshops.

View the agenda and register today!

When: Friday, September 19, 2014
Where:
‘t Spant, Bussum
Doctor Abraham Kuyperlaan 3
1402 SB Bussum
Tickets: Free
Featured Speakers:

  • Alan Clark, Board Chairman of the OpenStack Foundation
  • Ruud Harmsen, Fairbanks NV
  • Erik Monninkhof, Dupaco Distribution B.V.
  • Alessandro Vozza, Red Hat
  • Marijn de Vos, SUSE

Screen Shot 2014-08-26 at 10.52.36 AM

Please note that the content planned for OpenStack Day Taiwan will be presented in Chinese. Registration will close on September 18th, so register for free now! 

When: Tuesday, September 23, 2014
Where: Howard Plaza Hotel, Taipei, Taiwan
Tickets: Attendance is free, but registration is required to attend
Featured Speakers:

  • Tom Fifield, OpenStack Foundation
  • Chris Huang, SUSE
  • Jimmy Kao, In Win Development
  • Sam Chen, In Win Development
  • Robert Feng, Cisco

by Allison Price at August 26, 2014 04:04 PM

Aptira

OpenStack Object Storage (Swift) on FreeBSD with ZFS

Welcome to part two of the series of Aptira blogs where we explore the potential of running OpenStack on FreeBSD! At the end of part one, I mentioned that the next section would be on the OpenStack Image (Glance) service but after considering it some more I thought we would try for something a bit harder and go for OpenStack Object Storage (Swift) instead, since this is a fully fledged user facing service (actually my favorite of all the OpenStack services) which can also be used as an active-active highly available, horizontally scalable backend for Glance.

So, grab your popcorn and settle in because this job will be a bit more involved than the last one! If you aren't familiar with object storage as a concept, or how OpenStack Swift is designed, it's definitely worth doing a bit of reading before proceeding as this blog is intended more as an operators guide than introduction to either object storage or swift itself.

To start with, we are going to create a new Vagrant environment building on the work we did in part one, but with some modifications to the OpenStack Identity service Keystone definition (you can tear down your Vagrant environment from part one as we will be starting this environment from scratch).

Here is the Vagrantfile we will be using for this guide (please see part one for the command to download the vagrant box we will be using below hfm4/freebsd-10.0):

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.define "keystonebsd" do |keystonebsd|
    keystonebsd.vm.box = "hfm4/freebsd-10.0"
    keystonebsd.vm.hostname = "keystonebsd.sy3.aptira.com"
    keystonebsd.vm.network "private_network", ip: "10.0.3.15"
    keystonebsd.vm.provider "virtualbox" do |v|
      v.customize ['modifyvm', :id ,'--memory','2048']
    end
  end

  config.vm.define "swiftproxybsd01" do |swiftproxybsd01|
    swiftproxybsd01.vm.box = "hfm4/freebsd-10.0"
    swiftproxybsd01.vm.hostname = "swiftproxybsd01.sy3.aptira.com"
    swiftproxybsd01.vm.network "private_network", ip: "10.0.3.16"
    swiftproxybsd01.vm.provider "virtualbox" do |v|
      v.customize ['modifyvm', :id ,'--memory','2048']
    end
  end

  config.vm.define "swiftstoragebsd01" do |swiftstoragebsd01|
    swiftstoragebsd01.vm.box = "hfm4/freebsd-10.0"
    swiftstoragebsd01.vm.hostname = "swiftstoragebsd01.sy3.aptira.com"
    swiftstoragebsd01.vm.network "private_network", ip: "10.0.3.17"
    swiftstoragebsd01.vm.provider "virtualbox" do |v|
      v.customize ['modifyvm', :id ,'--memory','2048']
      v.customize ['createhd', '--filename', '/tmp/swiftstoragebsd01.vdi', '--size',  500]
      v.customize ['storageattach', :id, '--storagectl', 'IDE Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', '/tmp/swiftstoragebsd01.vdi']
    end
  end

  config.vm.define "swiftstoragebsd02" do |swiftstoragebsd02|
    swiftstoragebsd02.vm.box = "hfm4/freebsd-10.0"
    swiftstoragebsd02.vm.hostname = "swiftstoragebsd02.sy3.aptira.com"
    swiftstoragebsd02.vm.network "private_network", ip: "10.0.3.18"
    swiftstoragebsd02.vm.provider "virtualbox" do |v|
      v.customize ['modifyvm', :id ,'--memory','2048']
      v.customize ['createhd', '--filename', '/tmp/swiftstoragebsd02.vdi', '--size',  500]
      v.customize ['storageattach', :id, '--storagectl', 'IDE Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', '/tmp/swiftstoragebsd02.vdi']
    end
  end

  config.vm.define "swiftstoragebsd03" do |swiftstoragebsd03|
    swiftstoragebsd03.vm.box = "hfm4/freebsd-10.0"
    swiftstoragebsd03.vm.hostname = "swiftstoragebsd03.sy3.aptira.com"
    swiftstoragebsd03.vm.network "private_network", ip: "10.0.3.19"
    swiftstoragebsd03.vm.provider "virtualbox" do |v|
      v.customize ['modifyvm', :id ,'--memory','2048']
      v.customize ['createhd', '--filename', '/tmp/swiftstoragebsd03.vdi', '--size',  500]
      v.customize ['storageattach', :id, '--storagectl', 'IDE Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', '/tmp/swiftstoragebsd03.vdi']
    end
  end

end

As you can see the main differences to the keystone definition are that we have changed the vm name and added a private_network definition. Once you have created the Vagrantfile, start up the keystone vm and follow the steps from part one of the guide, except when running the endpoint-create command you should point it to the private_network IP defined above.

$ vagrant up keystonebsd
$ vagrant ssh keystonebsd

(now from inside the vm)

$ sudo -i
# ...(follow all the steps in part one of this guide except the last endpoint-create command which follows as)
# /usr/local/bin/keystone --os-token ADMIN --os-endpoint http://10.0.3.15:35357/v2.0/ endpoint-create --service=identity --publicurl=http://10.0.3.15:5000/v2.0 --internalurl=http://10.0.3.15:5000/v2.0 --adminurl=http://10.0.3.5:35357/v2.0
# ...(run test commands from part one of this guide using 10.0.3.15 instead of localhost as the --os-auth-url flag)

while we are logged into our keystone node we should prepare the swift service, user/tenant, endpoints as well and then we can exit out of our keystone node:

# /usr/local/bin/keystone --os-tenant-name admin --os-username admin --os-password test123 --os-auth-url http://10.0.3.15:5000/v2.0 service-create --name swift --type object-store
# /usr/local/bin/keystone --os-tenant-name admin --os-username admin --os-password test123 --os-auth-url http://10.0.3.15:5000/v2.0 tenant-create --name service
# /usr/local/bin/keystone --os-tenant-name admin --os-username admin --os-password test123 --os-auth-url http://10.0.3.15:5000/v2.0 user-create --name swift --tenant service --pass password
# /usr/local/bin/keystone --os-tenant-name admin --os-username admin --os-password test123 --os-auth-url http://10.0.3.15:5000/v2.0 user-role-add --user swift --tenant service --role admin
# ...(you can confirm the above commands worked by using the test commands at the end of part one of this guide)
# exit
$ exit

Now the fun really begins, we can start spinning up our swift servers! First on the list is our swift proxy. As mentioned above, if you aren't sure what any of these services are or what their function is, it's worth reading the architecture overview (at minimum) to familiarise yourself before continuing:

$ vagrant up swiftproxy01
$ vagrant ssh swiftproxy01

(now from inside the vm)

$ sudo -i

Installing memcached, enabling it as a service and starting the memcached service

# pkg install memcached
# echo 'memcached_enable="YES"' >> /etc/rc.conf
# service memcached start

Installing swift:

# pkg install python git wget
# pkg install py27-xattr libxslt
# pip install pbr six prettytable oslo.config python-keystoneclient netaddr keystonemiddleware
# git clone https://github.com/openstack/swift.git
# cd swift
# python setup.py install

Configuring swift user and required directories:

# mkdir /var/run/swift
# mkdir /etc/swift
# pw groupadd -n swift
# pw useradd -n swift -g swift -s /sbin/nologin -d /var/run/swift
# chown -R swift:swift /var/run/swift

Now we will copy over the provided example swift configuration files and modify them. swift.conf is first (below commands assume your current working directory is the cloned git repository):

# cp etc/swift.conf.sample /etc/swift/swift.conf

and modify the swift_hash_* lines in /etc/swift/swift.conf (again, same as in part one, we are not using secure values that would be required in production, only demonstrating the concepts):

swift_hash_path_suffix = suffsuffsuff
swift_hash_path_prefix = prefprefpref

then we will copy and modify the proxy-server.conf:

# cp etc/proxy-server.conf-sample /etc/swift/proxy-server.conf

Carefully modify the following sections, starting with [DEFAULT]:

bind_port = 8000
admin_key = adminkeyadminkey
account_autocreate = true

then [pipeline:main] (in this case we are modifying the application pipeline to remove the tempauth authentication module and instead use keystone authentication):

[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo proxy-logging proxy-server

so we should also comment out the tempauth filter:

#[filter:tempauth]
#use = egg:swift#tempauth

then uncomment and modify the authtoken filter to use the keystone configuration we setup at the start of this guide:

[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
auth_host = 10.0.3.15
auth_port = 35357
auth_protocol = http
auth_uri = http://10.0.3.15:5000/
admin_tenant_name = service
admin_user = swift
admin_password = password
delay_auth_decision = 1
cache = swift.cache
include_service_catalog = False

we also need to uncomment the keystoneauth filter:

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin, swiftoperator
reseller_admin_role = ResellerAdmin

Now we can write that configuration file and exit. The next step is to build our ringfiles, which provide the logical storage configuration to all the swift servers (ringfile documentation can be found here and here):

# cd /etc/swift
# swift-ring-builder container.builder create 18 3 1
# swift-ring-builder account.builder create 18 3 1
# swift-ring-builder object.builder create 18 3 1

once the ringfiles are created we can populate them with information about storage nodes and their devices (in the below configuration each storage node is confgured as a "zone"):

# swift-ring-builder object.builder add z1-10.0.3.17:6000/swiftdisk 100
# swift-ring-builder container.builder add z1-10.0.3.17:6001/swiftdisk 100
# swift-ring-builder account.builder add z1-10.0.3.17:6002/swiftdisk 100

# swift-ring-builder object.builder add z2-10.0.3.18:6000/swiftdisk 100
# swift-ring-builder container.builder add z2-10.0.3.18:6001/swiftdisk 100
# swift-ring-builder account.builder add z2-10.0.3.18:6002/swiftdisk 100

# swift-ring-builder object.builder add z3-10.0.3.19:6000/swiftdisk 100
# swift-ring-builder container.builder add z3-10.0.3.19:6001/swiftdisk 100
# swift-ring-builder account.builder add z3-10.0.3.19:6002/swiftdisk 100

and once the rings are populated the final step is to rebalance them:

# swift-ring-builder account.builder rebalance
# swift-ring-builder container.builder rebalance
# swift-ring-builder object.builder rebalance

Finally, we can set ownership of all the files in /etc/swift to the swift user and start the service! As I mentioned at the beginning of the guide, swift is an active-active, horizontally scalable service so you can add as many swift proxies as you like to your Vagrantfile and repeat the above procedure for each of them. You don't need to build the ringfiles on each node, simply copy the gzipped ringfiles (/etc/swift/*.gz) to /etc/swift on each of the nodes (we will do this on the swift storage nodes as you will see below).

# chown -R swift:swift /etc/swift
# /usr/local/bin/swift-init proxy start

Now we can start spinning up our swift storage nodes. I am only going to go through this procedure for one of the nodes, you should simply repeat it for each of the swift storage nodes defined in the Vagrantfile:

$ vagrant up swiftstoragebsd01
$ vagrant ssh swiftstoragebsd01

(now inside the VM)

$ sudo -i

Enabling and starting the ZFS service (we are using this since I couldn't find documentation on using XFS that is normally used when operating swift clusters) and configuring the second disk defined in the Vagrantfile as a ZFS pool:

# echo 'zfs_enable="YES"' >> /etc/rc.conf
# service zfs start
# zpool create swiftdisk /dev/ada1
# zfs set mountpoint=/srv/node/swiftdisk swiftdisk

Next, install rsync (which we will use as a daemon):

# pkg install rsync

and copy the following configuration to /etc/rsyncd.conf:

uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 0.0.0.0
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/spool/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/spool/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/spool/lock/object.lock

Enable the rsyncd service and start it:

# echo 'rsyncd_enable="YES"' >> /etc/rc.conf
# service rsyncd start

Installing swift:

# pkg install git wget python
# pkg install py27-xattr py27-sqlite3
# wget https://bootstrap.pypa.io/get-pip.py
# python get-pip.py
# git clone https://github.com/openstack/swift.git
# cd swift
# python setup.py install

Configuring swift user and required directories:

# mkdir /etc/swift
# mkdir /var/cache/swift
# mkdir /var/run/swift
# pw groupadd -n swift
# pw useradd -n swift -g swift -s /sbin/nologin -d /var/run/swift
# chown -R swift:swift /var/run/swift
# chown -R swift:swift /var/cache/swift
# chown -R swift:swift /srv

Now we will copy over the provided example swift configuration files and modify them. swift.conf is first (below commands assume your current working directory is the cloned git repository):

# cp etc/swift.conf.sample /etc/swift/swift.conf

and modify the swift_hash_* lines in /etc/swift/swift.conf (again, same as in part one, we are not using secure values that would be required in production, only demonstrating the concepts):

swift_hash_path_suffix = suffsuffsuff
swift_hash_path_prefix = prefprefpref

then we can copy the configuration files for the swift storage services (these should not require any modification):

# cp etc/account-server.conf-sample /etc/swift/account-server.conf
# cp etc/container-server.conf-sample /etc/swift/container-server.conf
# cp etc/object-server.conf-sample /etc/swift/object-server.conf

At this point you should copy the ringfiles created on the swiftproxybsd01 vm to /etc/swift. I set a password for the vagrant user and copied them using scp to each vm. This is definitely not a recommended method of doing things in production but is fine for demonstration purposes:

# scp vagrant@10.0.3.16:/etc/swift/*.gz /etc/swift

Finally, we can set ownership of all the files in /etc/swift to the swift user and start the all the storage node services!

# chown -R swift:swift /etc/swift
# /usr/local/bin/swift-init all start

After you have completed these steps on one node, repeat them for the remaining storage nodes defined in the Vagrantfile.

Once this is done, your swift service is up and running! You can test it now, so log back into your keystone node (or any computer that can access 10.0.3.15/10.0.3.16) and install the swift client:

# pip install python-swiftclient

Now you can run some test commands!

# swift --os-auth-url=http://10.0.3.15:5000/v2.0 --os-username=admin --os-password=test123 --os-tenant-name=admin post testcontainer
# swift --os-auth-url=http://10.0.3.15:5000/v2.0 --os-username=admin --os-password=test123 --os-tenant-name=admin upload testcontainer get-pip.py
# swift --os-auth-url=http://10.0.3.15:5000/v2.0 --os-username=admin --os-password=test123 --os-tenant-name=admin upload testcontainer get-pip.py (for some reason the first object never shows up in stat so I always run it twice)
# swift --os-auth-url=http://10.0.3.15:5000/v2.0 --os-username=admin --os-password=test123 --os-tenant-name=admin stat

which, if everything worked correctly, should show the following output:

Account: AUTH_516f9ace29294cff91316153d793bdab
    Containers: 1
       Objects: 1
         Bytes: 1340903
X-Account-Storage-Policy-Policy-0-Bytes-Used: 1340903
   X-Timestamp: 1408880268.17095
X-Account-Storage-Policy-Policy-0-Object-Count: 1
    X-Trans-Id: txfc005fa6c999449b81e7b-0053fa972f
  Content-Type: text/plain; charset=utf-8
 Accept-Ranges: bytes

and that's it! You're now running two OpenStack services on FreeBSD. Once we have setup and installed Glance, then the hard work of making OpenStack Compute (Nova) work somehow with FreeBSD begins. As I mentioned in part one, I am very interested in the idea of getting FreeBSD Jails supported in Nova! Stay tuned!

by Sina Sadeghi (sina@aptira.com) at August 26, 2014 02:25 PM

Sean Dague

OpenStack as Layers

Last week at LinuxCon I gave a presentation on DevStack which gave me the proper excuse to turn an idea that Dean Troyer floated a year ago about OpenStack Layers into pictures (I highly recommend reading that for background, I won't justify every part of that here again). This abstraction has been something that's actually served us well as we think about projects coming into DevStack.

OpenStack_Layers

Some assumptions are made here in terms of what essential services are here as we build up the model.

Layer 1: Base Compute Infrastructure

We assume that compute infrastructure is the common starting point of minimum functional OpenStack that people are deploying. The output of the last OpenStack User Survey shows that the top 3 deployed services, regardless of type of cloud (Dev/Test, POC, or Production) are Nova / Glance / Keystone. So I don't think this is a huge stretch. There are definitely users that take other slices (like Swift only) but compute seems to be what the majority of people coming to OpenStack seem to be focussed on.

Basic Compute services need 3 services to get running. Nova, Glance, and Keystone. That will give you a stateless compute cloud which is a starting point for many people getting into the space for the first time.

Layer 2: Extended Infrastructure

Once you have a basic bit of compute infrastructure in place, there are some quite common features that you do really need to do more interesting work. These are basically enhancements on the Storage, Networking, or Compute aspects of OpenStack. Looking at the User Survey these are all deployed by people, in various ways, at a pretty high rate.

This is the first place we see new projects integrating into OpenStack. Ironic extends the compute infrastructure to baremetal, and Designate adds a missing piece of the networking side with DNS management as part of your compute creation.

Hopefully nothing all that controversial here.

Layer 3: Optional Enhancements

Now we get a set of currently integrated services that integrate North bound and South bound. Horizon integrates on the North bound APIs for all the services, it requires service further down in layers (it also today integrates with pieces further up that are integrated). Ceilometer consumes South bound parts of OpenStack (notifications) and polls North bound interfaces.

From the user survey Horizon is deployed a ton. Ceilometer, not nearly as much. Part of this is due to how long things have been integrated, but even if you do analysis like take the Cinder / Neutron numbers, delete all the Folsom deploys from it (which is the first time those projects were integrated) you still see a picture where Ceilometer is behind on adoption. Recent mailing list discussions have hints at why, including some of the scaling issues, and a number of alternative communities in this space.

Let's punt on Barbican, because honestly, it's new since we came up with this map, and maybe it's really a layer 2 service.

Layer 4: Consumption Services

I actually don't like this name, but I failed to come up with something better. Layer 4 in Dean's post was "Turtles all the way down", which isn't great describing things either.

This is a set of things which consume other OpenStack services to create new services. Trove is the canonical example, create a database as a service by orchestrating Nova compute instances with mysql installed in them.

The rest of the layer 4 services all fit the same pattern, even Heat. Heat really is about taking the rest of the components in OpenStack and building a super API for their creation. It also includes auto scaling functionality based on this. In the case of all integrated services they need a guest agent to do a piece of their function, which means when testing them in OpenStack we don't get very far with the Cirros minimal guest that we use for Layer 3 and down.

But again, as we look at the user survey we can see deployment of all of these Layer 4 services is lighter again. And this is what you'd expect as you go up these layers. These are all useful services to a set of users, but they aren't all useful to all users.

I'd argue that the confusion around Marconi's place in the OpenStack ecosystem comes with the fact that by analogy it looks and feels like a Layer 4 service like Trove (where a starting point would be allocating computes), but is implemented like a Layer 2 one (straight up raw service expected to be deployed on bare metal out of band). And yet it's not consumable as the Queue service for the other Layer 1 & 2 services.

Leaky Taxonomy

This is not the end all be all of a way to look at OpenStack. However, this layered view of the world confuses people a lot less than the normal view we show them -- the giant spider diagram (aka the mandatory architecture slide for all OpenStack presentations):

OpenStack_Spider_Diagram

This picture is in every deep dive on OpenStack, and scares the crap out of people who think they might want to deploy it. There is no starting point, there is no end point. How do you bite that off in a manageable chunk as the spider grows?

I had one person come up to me after my DevStack talk giving a big thank you. He'd seen a presentation on Cloudstack and OpenStack previously and OpenStack's complexity from the outside so confused him that he'd run away from our community. Explaining this with the layer framing, and showing how you could experiment with this quickly with DevStack cleared away a ton of confusion and fear. And he's going to go dive in now.

Tents and Ecosystems

Today the OpenStack Technical Committee is in charge of deciding the size of the "tent" that is OpenStack. The approach to date has been a big tent philosophy, where anything that's related, and has a REST API, is free to apply to the TC for incubation.

But a big Tent is often detrimental to the ecosystem. A new project's first goal often seems to become incubated, to get the gold star of TC legitimacy that they believe is required to build a successful project. But as we've seen recently a TC star doesn't guarantee success, and honestly, the constraints on being inside the tent are actually pretty high.

And then there is a language question, because OpenStack's stance on everything being in Python is pretty clear. An ecosystem that only exists to spawn incubated projects, and incubated projects only being allowed to be in Python, basically means an ecosystem devoid of interesting software in other languages. That's a situation that I don't think any of us want.

So what if OpenStack were a smaller tent, and not all the layers that are in OpenStack today were part of the integrated release in the future? Projects could be considered a success based on their users and usage out of the ecosystem, and not whether they have a TC gold star. Stackforge wouldn't have some stigma of "not cool enough", it would be the normal place to exist as part of the OpenStack ecosystem.

Mesos is an interesting cloud community that functions like that today. Mesos has a small core framework, and a big ecosystem. The smaller core actually helps grow the ecosystem by not making the ecosystem 2nd class citizens.

I think that everyone that works on OpenStack itself, and all the ecosystem projects, want this whole thing to be successful. We want a future with interoperable, stable, open source cloud fabric as a given. There are lots of thoughts on how we get there, and as no one has ever created a universal open source cloud fabric that lets users have the freedom to move between providers, public and private, so it's no surprise that as a community we haven't figured everything out yet.

But here's another idea into the pool, under the assumption that we are all smarter together with all the ideas on the table, than any of us are on our own.

by Sean Dague at August 26, 2014 12:04 PM

Tesora Corp

OpenStack Trove (Juno) Mid-Cycle Meetup

trove-juno-mid-cycle-meetup-small.jpg

Last week was a whirl, first we had the Trove Day on Tuesday which was a wonderful event and well attended. In the audience we had several members of the Trove development team who were in town for the Trove (Juno) mid-cycle meetup that Tesora hosted. We had attendees from Tesora, HP, eBay, Rackspace and Percona at the meeting.

We met for three days at MIT CSAIL (our thanks to MIT for allowing us to use their facility for this event). At right is theview of the roof of the meeting room we used on the third day.

During the three days, we had some very productive conversations about a variety of topics including Replication, Clustering, monitoring, Horizon (the user interface for OpenStack), and integration with a number of other OpenStack projects such as ceilometer, heat, and oslo.

2014-08-22 08.41.44.jpg

Since coming out of incubation (pre IceHouse) to being an integrated project (in IceHouse), we've seen a number of improvements and at the MidCycle meetup, we reviewed many of these. Replication, a feature that Tesora helped develop is currently under code review and will be released as part of the Juno release of Trove. In this release, we support read-only slave replicas for MySQL datastores. A significant part of the work in the v1 was the setup of a framework that would be usable for other databases as well, and one that would be easily extensible to other more complex replication use-cases.

Also discussed at the meetup was Clustering, a feature that has just come in to review. Auston McReynolds and Mat Lowery from eBay were at the meetup and presented the code and we had a very good discussion of the rationale for many of their implementation choices, and how these could also be extended to future datastores.

Tim Simpson presented some very interesting testing and test related techniques which he has been working on. Robert Myers gave an update on a project related to refactoring and streamlining the code base and coming up with a better layout of the code for the guestagent and the trove (backend).

We spent some time discussing a variety of things related to the Trove project, the team and the dynamics, and how we would like to do reviews moving forward. As part of that conversation, we decided to update 'contributing.rst' in the Trove project and I took the action to post a review for it. It appears that Trove is not the only project that is dealing with these kinds of issues. A day after I posted my change for review, I saw this other review that was also working its way through the system! We also spent time discussing outstanding blueprints and what we would do with them moving forward.

Just as important as the meetings (which were long) were three days of hacking sessions that we had each afternoon. At these we had a chance to not only review the features and changes that were in the queue, but we also had a chance to speak with other developers and learn some of their tricks.

After a hard day at the meetings, we had a great time on both evenings hanging out and chatting till late in the night. After working for eight hours a day for several months with someone whom you have only interacted with on IRC, meeting face-to-face and hoisting a pint is a wonderful thing. Sure enough, with twenty-two such people and all the combinations thereof, several pints were in fact hoisted.

It was wonderful meeting all of you face to face, and I hope we all get to meet again in Paris in November.

 

by 10 at August 26, 2014 12:00 PM

Jamie Lennox

requests-mock

Having just release v0.5 of requests-mock and having it used by both keystoneclient and novaclient with others in the works I thought I’d finally do a post explaining what it is and how to use it.

Motivation

I was the person who brought HTTPretty into the OpenStack requirements.

The initial reason for this was that keystoneclient was transitioning from the httplib library to requests and I needed to prove that there was no changes to the HTTP requests during the transition. HTTPretty is a way to mock HTTP responses at the socket level, so it is not dependant on the HTTP library you use and for this it was fairly successful.

As part of that transition I converted all the unit tests so that they actually traversed through to the requesting layer and found a number of edge case bugs because the responses were being mocked out above this point. I have therefore advocated that the clients convert to mocking at the request layer rather than stubbing out returned values. I’m pretty sure that this doesn’t adhere strictly to the unit testing philosophy of testing small isolated changes, but our client libraries aren’t that deep and I’d honestly prefer to just test the whole way through and find those edge cases.

Having done this has made it remarkably easier to transition to using sessions in the clients as well, because we are testing the whole path down to making HTTP requests for all the resource tests so again have assurances that the HTTP requests being sent are equivalent.

At the same time we’ve had a number of problems with HTTPretty:

  • It was the lingering last requirement for getting Python 3 support. Thanks to Cyril Roelandt for finally getting that fixed.
  • For various reasons it is difficult for the distributions to package.
  • It has a bad habit of doing backwards incompatible, or simply broken releases. The current requirements string is: httpretty>=0.8.0,!=0.8.1,!=0.8.2,!=0.8.3
  • Because it acts at the socket layer it doesn’t always play nicely with other things using the socket. For example it has to be disabled for live memcache tests.
  • It pins its requirements on pypi.

Now I feel like I’m just ranting. There are additional oddities I found in trying to fix these upstream but this is not about bashing HTTPretty.

requests-mock

requests-mock follows the same concepts allowing users to stub out responses to HTTP requests, however it specifically targets the requests library rather than stubbing the socket. All the OpenStack clients have been converted to requests at this point, and for the general audience if you are writing HTTP code in Python you should be using requests.

Note: a lot of what is used in these examples is only available since the 0.5 release. The current OpenStack requirements still have 0.4 so you’ll need to wait for some of the new syntax.

The intention of requests-mock is to work in as similar way to requests itself as possible. Hence all the variable names and conventions should be as close to a requests.Response as possible. For example:

<notextile><figure class="code">
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</p>

<blockquote><blockquote><blockquote><p>import requests
import requests_mock
url = &lsquo;<a href="http://www.google.com">http://www.google.com</a>&rsquo;
with requests_mock.mock() as m:
&hellip;     m.get(url, text=&lsquo;Not really google&rsquo;, status_code=218)
&hellip;     r = requests.get(url)
&hellip;
r.text
u&#8217;Not really google&#8217;
r.status_code
218</p></blockquote></blockquote></blockquote>

<p>
</figure></notextile>

So text in the mock equates to text in the response and similarly for status_code. Some more advanced usage of the requests library:

<notextile><figure class="code">
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</p>

<blockquote><blockquote><blockquote><p>with requests_mock.mock() as m:
&hellip;     m.get(url, json={&lsquo;hello&rsquo;: &lsquo;world&rsquo;}, headers={&lsquo;test&rsquo;: &lsquo;header&rsquo;})
&hellip;     r = requests.get(url)
&hellip;
r.text
u&#8217;{&ldquo;hello&rdquo;: &ldquo;world&rdquo;}&lsquo;
r.json()
{u&#8217;hello&rsquo;: u&#8217;world&#8217;}
r.status_code
200
r.headers
{&lsquo;test&rsquo;: &lsquo;header&rsquo;}
r.headers[&lsquo;test&rsquo;]
&lsquo;header&rsquo;</p></blockquote></blockquote></blockquote>

<p>
</figure></notextile>

You can also use callbacks to create responses dynamically:

<notextile><figure class="code">
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</p>

<blockquote><blockquote><blockquote><p>def <em>request_callback(request, context):
&hellip;     context.status_code = 201
&hellip;     context.headers[&lsquo;test&rsquo;] = &lsquo;header&rsquo;
&hellip;     return {&lsquo;request&rsquo;: request.body}
&hellip;
with requests_mock.mock() as m:
&hellip;     m.post(url, json=</em>request_callback)
&hellip;     r = requests.post(url, data=&lsquo;data&rsquo;)
&hellip;
r.status_code
201
r.headers
{&lsquo;test&rsquo;: &lsquo;header&rsquo;}
r.json()
{u&#8217;request&#8217;: u&#8217;data&#8217;}</p></blockquote></blockquote></blockquote>

<p>
</figure></notextile>

Note that because the callback was passed as the json parameter the return type is expected to be the same as if you had passed it as a predefined json=blob value. If you wanted to return text the callback would be on the text parameter.

Cool tricks

So rather than give a lot of examples i’ll just highlight some of the interesting things you can do with the library and how to do it.

  • Queue mutliple responses for a url, each element of the list is interpreted as if they were **kwargs for a response. In this case every request other than the first will get a 401 error:

<notextile><figure class="code">
1
2
m.get(url, [{&lsquo;json&rsquo;: _request_callback},
            {&lsquo;text&rsquo;: &lsquo;Not Allowed&rsquo;, &lsquo;status_code&rsquo;: 401}])
</figure></notextile>

  • See the history of requests:

<notextile><figure class="code">
1
2
3
4
m.request_history  # all requests
m.last_request  # the last request
m.call_count  # number of requests
m.called  # boolean, True if called
</figure></notextile>

  • match on the only the URL path:

<notextile><figure class="code">
1
m.get(&lsquo;/path/only&rsquo;)
</figure></notextile>

  • match on any method:

<notextile><figure class="code">
1
m.request(requests_mock.ANY, url)
</figure></notextile>

  • or match on any URL:

<notextile><figure class="code">
1
m.get(requests_mock.ANY)
</figure></notextile>

  • match on headers that are part of the request (useful for distinguishing between multiple requests to the same URL):

<notextile><figure class="code">
1
m.get(url, request_headers={&lsquo;X-Auth-Token&rsquo;: &lsquo;XXXXX&rsquo;})
</figure></notextile>

  • be used as a function decorator

<notextile><figure class="code">
1
2
3
4
@requests_mock.mock()
def test_a_thing(m):
   m.get(requests_mock.ANY, text=&lsquo;resp&rsquo;)
   &hellip;
</figure></notextile>

Try it!

There is a lot more it can do and if you want to know more you can check out:

As a final selling point because it was built particularly around OpenStack needs it is:

  • Easily integrated with the fixtures library.
  • Hosted on stackforge and reviewed via Gerrit.
  • Continuously tested against at least keystoneclient and novaclient to prevent backwards incompatible changes.
  • Accepted as part of OpenStack requirements.

Patches and bug reports are welcome.

August 26, 2014 02:26 AM

August 25, 2014

Mirantis

What’s New in Mirantis OpenStack 5.0.1

The 5.0.1 maintenance release for Mirantis OpenStack, which has now reached general availability, represents a big milestone for the product.  For those who have already installed Mirantis OpenStack 5.0, this will be the first time that you can upgrade the Fuel Master Node in place and retain management of your existing 5.0 environments.  No need to redeploy!  

To upgrade the Fuel Master Node that manages an existing Mirantis OpenStack 5.0 cloud environment, just do the following:

  1. Be sure that no installations are in progress on your Fuel Master Node!

  2. Download the upgrade tarball from http://software.mirantis.com to a local system.

  1. You’ll want to choose the “Upgrade Package” since you’re upgrading from 5.0 to 5.0.1.  (If you’re a new customer, or on version 4.1.1 or earlier, can still download the ISO and IMG to install a new instance of the Fuel Master Node.)

  2. Use scp or your SSH client to copy the tarball to a location on the Fuel Master Node disk that has adequate space, such as /var/tmp.

scp MirantisOpenStack-5.0.1-upgrade.tar root@10.20.0.2:/var/tmp
  1. Log into the Fuel Master Node and extract the tarball contents:

cd /var/tmp  # Use the directory where the tarball is located
tar -xf MirantisOpenStack-5.0.1-upgrade.tar
  1. Once all the files are extracted, run the upgrade script from that same directory:

./upgrade.sh
  1. Do not interrupt the upgrade! The upgrade process can take 30-60 minutes. Some operations (such as uploading images) take some time; the listing of updated files may slow down, but this does not mean that the upgrade process has hung.

When the upgrade is complete, the following messages will appear on your screen

*** UPGRADE DONE SUCCESSFULLY

After the upgrade, you’ll notice in the Releases tab that you now have the ability to deploy and manage both the 2014.1 release (which came with Mirantis OpenStack 5.0) and the 2014.1.1-5.0.1 release (which was came with 5.0.1)

  

Other improvements

So, what else is in the 5.0.1 update?

  • We’ve synched with the OpenStack Icehouse 2014.1.1 stable branch release.  When you deploy a new environment from Mirantis OpenStack 5.0.1, you’ll be able to deploy this new version.

  • We’ve further enhanced the High Availability of Mirantis OpenStack to address even more points of failure.  For example, there’s further tolerance for the rare situation of multiple failures of the same controller or loss of the management interface on the primary controller.

  • When we released 5.0, there was a bootstrap kernel issue where the kernel was lacking the proper firmware files for some hardware types.  While a patch has been available from Support for 5.0 customers on this issue, it’s now included by default when deploying a 5.0.1 environment.

  • As a maintenance release, we’ve address other important defect fixes.  You can review the Release Notes for more information.

On the horizon

Mirantis 5.0.1 is a maintenance release to 5.0, but we’re hard at work on Mirantis OpenStack 5.1 which we hope to deliver with these new and exciting features:

  • Updating an existing OpenStack environment:  With Mirantis OpenStack 5.1, you’ll be able to automatically update an existing environment to the latest offered version of OpenStack without redeployment.  For example, if you’ve installed an environment using Mirantis OpenStack 5.0 (which is based on Icehouse 2014.1), there will be an option to update that environment to Icehouse 2014.1.1.

  • Deployment of ML2 OpenStack vSwitch by default: Starting with Havana, the legacy plugin structure for Neutron has been deprecated and replaced with a Modular Layer 2 (ML2) plugin structure.  This change will enable Neutron to utilize a variety of Layer 2 network technologies rather than being locked in the monolithic structure found in previous releases of OpenStack.

  • Access Control to the Fuel UI and API: When using either the Fuel UI or Fuel APIs, users will be challenged to provide authentication credentials (e.g. username and password).  These credentials and the authentication process will be handled by a local instance of Keystone present on the Fuel Master Node.  Users will also be able to change their password within the Fuel UI.

  • VMWare NSX for KVM hypervisors: VMWare NSX is a is a software-defined network (SDN) that uses controllers and overlay networking.  Mirantis OpenStack 5.1 will enable you to select VMWare NSX as an networking option for use with the KVM hypervisor.

  • Support Mellanox hardware: Mirantis OpenStack 5.1 will properly detect Mellanox hardware installed within discovered nodes.  This enables an operator to

    • Configure storage to work over ISER (High performance RDMA based ISCSI)
    • Install OFED drivers, firmware and configure SR-IOV kernel parameters
    • Install and configure the Mellanox SR-IOV plugin for OpenStack for high performance VM link
    • Test the environment utilizing Mirantis OpenStack Health Checks

The post What’s New in Mirantis OpenStack 5.0.1 appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by David Easter at August 25, 2014 11:30 PM

Red Hat Stack

Red Hat Enterprise Virtualization 3.4.1 Released

Principal Product Manager, Red Hat

I don’t often find myself getting overly excited about maintenance releases, however Red Hat Enterprise Virtualization 3.4.1 is an exception due to two key factors:

  • Preview support for Red Hat Enterprise Linux 7 as a hypervisor host
  • Support for up to 4,000 GB memory in a single virtual machine

Red Hat Enterprise Virtualization 3.4, originally introduced official guest operating system support for Red Hat Enterprise Linux (RHEL) 7. In continuing down the path of providing the latest Red Hat technologies to our customers, I am proud to announce that Red Hat Enterprise Virtualization 3.4.1 has preview support for RHEL 7 as a hypervisor.  Red Hat customers with active subscriptions will be able to take advantage of using RHEL 7 as a hypervisor either as a RHEL host, or by using our thin Red Hat Enterprise Virtualization Hypervisor image.

Red Hat Enterprise Virtualization continues to be the industry leader when it comes to scale and performance of virtual machine workloads (according to SPECvirt_sc2010).  Red Hat Enterprise Virtualization 3.4.1 enhances performance by enabling support for up to 4,000 GB of memory in a single virtual machine.  This is nearly four times the virtual memory capacity of other leading virtualization solutions in the market, and allows our customers to more confidently virtualize their most mission critical workloads at near-bare-level performance.  Combined with Red Hat Enterprise Virtualization’s support for up to 160 vCPUs in a single guest, I believe you could say that “Red Hat Enterprise Virtualization eats Monster VMs for breakfast”.

Red Hat Enterprise Virtualization is tightly integrated with Red Hat Enterprise Linux. Both platforms are built by the same people on open standards and share industry-leading characteristics such as performance, scalability, and security. In fact, Red Hat Enterprise Linux with Smart Virtualization, a combined Red Hat Enterprise Linux and Red Hat Enterprise Virtualization enables customers to fully virtualize their Red Hat Enterprise Linux workloads at a cost effective level.

More customers are looking at Red Hat Enterprise Virtualization as organizations seek lower-cost enterprise virtualization platforms. When aligned with Red Hat CloudForms and its OpenStack integration, Red Hat Enterprise Virtualization provides a platform that can meet the needs of today’s enterprise virtualization customers while assisting the transition into a private cloud infrastructure. Red Hat Enterprise Virtualization 3.4.1 is an important component in helping our customers to build their infrastructure platform without boundaries.

by Scott Herold at August 25, 2014 07:52 PM

Tesora Corp

OpenStack can stand on its own, whatever happens to Rackspace

5128458766_34d8f1834a_b.jpgFour years ago, Rackspace and its early partners came up with an idea for an open source private alternative to Amazon Web Services --and OpenStack was born. Today, the future of Rackspace is murky, but the open source project it helped create is strong enough to stand on its own, whatever happens to one of the founding members.

Earlier this month, reports surfaced that billionaire Daniel Loeb's Third Point Investment has invested a significant stake in Rackspace. The implications of this investment are not clear yet.

This follows earlier reports of some grumbling about Rackspace's low profile at the OpenStack Summit in Atlanta. It wasn't long before the reason became clear. Rackspace had hired investment banker Morgan Stanley to evaluate proposals that apparently ranged from strategic partnerships to being purchased. Then the rumors started to flow (and continue to do so) about possible suitors from IBM to HP to Microsoft among others, but so far nobody has bitten.

Other reports suggested Rackspace could go private. Whatever the reality is, Rackspace isn't doing great. Earlier this month they finally threw in the towel and quit the infrastructure service business, ceding it to AWS, Google and Microsoft in favor of a managed services approach (whatever that means).

In spite of all this bad (or at least confusing) news, the company had a better than expected earnings report earlier this month, yet the market still feels the uncertainty surrounding the company and the stock price dropped as a result.

With Rackspace reeling, what does it mean to OpenStack, the project it has nurtured from its earliest days? Fortunately, it's very likely it won't have a significant impact whatever happens to the company. While they remain steady contributors to the project, there are others, particularly Red Hat and more recently HP who are stepping up as OpenStack gains in popularity.

When looking at Ice House, the most recent big release of OpenStack, the biggest contributor in terms of "commits per company" wasn't a corporation at all, but the category "Others," which suggests individuals in the community, rather than people working inside a large organization. This too bodes well for OpenStack because it means, rather than one or two large companies controlling the project, there are thousands of individuals making it happen and that makes for a much healthier project overall.

ice house commits_0.jpg

Graph courtesy of Bitergia

While Rackspace is certainly still contributing to and supporting OpenStack, it's hard to know what will happen should it get bought and what that will mean to the company's priorities moving forward. The owners could have different goals for the company and that could mean dialing back the resources it puts toward OpenStack. Or conversely it could continue to be a meaningful contributor. We just don't know.

The fact is, it's impossible to know where Rackspace is going right now. But whatever happens to it, even if it stopped supporting OpenStack (and there's no evidence right now that's going to happen), the project it helped launch is in great shape and perfectly capable of standing on it its own, thank you very much. And that's a comforting feeling in a time of great unpredictability.

Photo on Flickr by Tom Coppin. Used under CC 2.0 License.

by 693 at August 25, 2014 03:17 PM

Opensource.com

The push for interoperability, an OpenStack card deck, and more

Interested in keeping track of what's happening in the open source cloud? Opensource.com is your source for what's happening right now in OpenStack, the open source cloud infrastructure project.

by Jason Baker at August 25, 2014 03:00 PM

Kenneth Hui

VMware Integrated OpenStack: An OpenStack That VMware Admins Can Love

 

vmwarelovesopenstack

OpenStack For VMware Administrators and Architects

In my previous role as the OpenStack and VMware Evangelist for Rackspace, I had the opportunity to speak with numerous companies about their OpenStack implementations or proof-of-concepts.  These interactions included customer meetings, user group meetups, and conference sessions.  A trend I noticed was the increasing number of VMware administrators and architects who were becoming more involved in their company’s OpenStack initiatives.  For example, Scott Lowe and I have given an “OpenStack For VMware Administrators” talk at the previous two OpenStack Summits in Hong Kong and in Atlanta and at each talk, I asked how many attendees also had responsibility for their company’s VMware infrastructure.  In Hong Kong, ~25% of the room raised there hands but 6 months later in Atlanta, ~50% of the attendees raised their hands.  The common theme I heard then and which I still hear today is that as Enterprises increasingly look at OpenStack, they typically assign the responsibility for doing so to their VMware SMEs who are already familiar with virtualization technologies.

As I continue to speak with these “VMware turned OpenStack” administrators, several common themes have emerged around challenges they are seeing in trying to deploy OpenStack:

  • Making OpenStack Enterprise-grade – OpenStack is a rapidly maturing cloud platform that is being used in a number of enterprises today.  However, hardening the platform and making it resilient requires considerable engineering and development skills.  As a large customer told me recently, “we did not realize how much money, manpower, and time would be required to make our OpenStack deployment production-ready.”
  • Learning New Skills – VMware administrator are being asked to become OpenStack experts in a very short amount of time.  That is a challenge when you are unable to fully leverage your existing knowledge base and experience and have to operate a new cloud platform using a completely different set of tools.
  • Gaps in Operational Capabilities – As OpenStack continues to mature, one of the big gaps that need to be filled is in the area of operational capabilities, particularly around monitoring and logging.  Today, users typically have to cobble together various tools, such as Nagios and Logstash, and customize them in order to have visibility into their cloud.
  • Vendor Support – Most vendor solutions today address some but not all the elements required to run an OpenStack cloud.  This typically means that users have to source their solution from multiple vendors and manage multiple touch-points to receive support from those vendors.

Introducing VMware Integrated OpenStack (VIO)

This brings us to the announcement that VMware is making today about their new OpenStack solution – VMware Integrated OpenStack (VIO).  As some of you are aware, VMware has been a top contributor to the OpenStack project since they joined the project’s foundation in 2012.  These contributions have included integration of vSphere and NSX technologies into the OpenStack platform so they can be managed via OpenStack’s Open APIs while continuing to provide differentiating value.  With VIO, VMware has taken the next step to create a solution that makes it easier than ever for current VMware administrators to be able to deploy and mange OpenStack.

Screen Shot 2014-08-24 at 11.51.26 AM

In a nutshell, VIO packages the stable release of OpenStack into an OVA and delivers it as a vApp so that OpenStack can be deployed as an “over-cloud” on to a VMware based “under-cloud.”  The OpenStack based “over-cloud” consists of the control plane services that will allow users and administrators to launch resources using the integrated OpenStack projects and various plugins such as third-party networking and storage technologies.  This “over-cloud” also exposes the OpenStack APIs so that developers can provision their own cloud resources without requiring infrastructure admin assistance.  Meanwhile, the VMware based “under-cloud” allows the OpenStack control services to leverage underlying vSphere capabilities such as HA, vMotion, and DRS.

Screen Shot 2014-08-24 at 12.20.32 PM

Inside VMware Integrated OpenStack (VIO)

Prior to VIO, a  typical OpenStack deployment with vSphere integration consists of 2 or more Controller nodes (assuming an HA configuration) that runs the required services to manage your cloud resources and some number of Nova compute nodes that communicate with vCenter servers and serve as the integration points to vSphere clusters.  You can find details on how that integration works in a series of blog posts I had previously written.

multi-hv havana

With the introduction of VIO, the Controller and Nova Compute nodes will be deployed as virtual machines in a vSphere cluster. leveraging the HA capabilities of the underlying cluster and simplifying the work it would normally take to make the OpenStack control plane services highly available.  The Compute nodes will manage some set of vSphere, allowing developer to create and launch VMware-backed cloud resources via OpenStack APIs.  Note that the beta and initial GA releases of VIO will only support vSphere as a hypervisor and NSX or a specially modified Virtual Distributed Switch (vDS) for Neutron Networking-as-a-Service.  While it is possible to deploy as the OpenStack control plane services on the same vSphere cluster as your VMware-back cloud resources, I would recommend using a separate management cluster for your Controller and Compute nodes.

VIO

It is also worth noting that since VIO leverages all the integration work and contributions that VMware has made to the OpenStack project, users will be able to take advantage of the Block-Storage-as-as-Service capabilities and third-party storage plugins available via vSphere’s integration with the OpenStack Cinder project.  You can find details on how that integration works in a blog post I had previously written.

vSphere Cinder

To address the challenges that exist with managing an OpenStack cloud, the “under-cloud” provides the “VMware turned OpenStack administrators” with a familiar set of tools for deploying and managing their OpenStack infrastructure.  These Opencast-integrated tools include components from the vCloud Suite, such as vCenter Operations Manager (vCOPs) and Log Insight.

Screen Shot 2014-08-24 at 12.42.17 PM

Screen Shot 2014-08-24 at 12.43.35 PM

Most readers will recognize that VMware had rolled out a precursor to VIO in the form of something called vSphere OpenStack Virtual Appliance (VOVA).  While VOVA is targeted at those who want to play with OpenStack in their labs, VIO is much more robust and intended to be used to provide a cloud with feature-rich APIs to developers.  In particular, VIO is designed to address some of the challenges I outlined earlier faced by enterprises as they look to deploy OpenStack.

Screen Shot 2014-08-24 at 12.30.09 PM

Learning More About VIO

So how can someone learn more about and get theirs hands on VIO?  As was announced today, VIO in now in private beta with GA targeted for the first half of 2015.  To request access to the beta program and/or to get more information on VIO, go to www.vmware.com/products/openstack.  You can also read more about VIO on the official VMware blog and Chris Wahl’s excellent blog post on the subject.

VIO in the EMC Federation

VIO will no doubt be a valuable solution to help drive the adoption of OpenStack, particularly in the enterprise, where there are so much existing VMware infrastructures and expertise.  In particular, I see VIO as a great solution for VMware administrators who have been tasked to provide developers with a cloud that is more agile and has more robust developer-friendly APIs.  With VIO being one of the EMC Federation’s OpenStack options, EMC II is committed to making sure that we will have the best storage technologies for VIO.  Along those lines, expect that ViPR and its rich set of data services capabilities will become a compelling choice for customers who want to run best-of-breed object storage with VIO.

Screen Shot 2014-08-24 at 1.05.16 PM

However, we will also have customers who may, for various reasons, choose not to go down the VIO route for their OpenStack initiative.  Since the Federation is committed to providing customer choice, EMC will also be doing what we can to help those customer succeed with their OpenStack deployments.  Stay tuned for more details on how we plan to provide solutions that also will address the challenges of deploying OpenStack that I outlined in this post.  Meanwhile, contact me if you want to work on building some exciting solutions as a part of  Team OpenStack @EMC.

EMC Team OS


Filed under: Cloud, Cloud Computing, EMC, OpenStack, Virtualization, VMware Tagged: Cloud, Cloud computing, EMC, OpenStack, VMware, VMware vSphere, VMworld

by Kenneth Hui at August 25, 2014 02:30 PM

Percona

OpenStack’s Trove: The benefits of this database as a service (DBaaS)

In a previous post, my colleague Dimitri Vanoverbeke discussed at a high level the concepts of database as a service (DBaaS), OpenStack and OpenStack’s implementation of a DBaaS, Trove. Today I’d like to delve a bit further into Trove and discuss where it fits in, and who benefits.

Just to recap, Trove is OpenStack’s implementation of a database as a service for its cloud infrastructure as a service (IaaS). And as the mission statement declares, the Trove project seeks to provide a scalable and reliable cloud database service providing functionality for both relational and non-relational database engines. With the current release of Icehouse, the technology has begun to show maturity providing both stability and a rich feature set.

In my opinion, there are two primary markets that will benefit from Trove: the first being service providers such as RackSpace who provide cloud-based services similar to Amazon’s AWS. These are companies that wish to expand beyond the basic cloud services of storage and networking and provide their customer base with a richer cloud experience by providing higher level services such as DBaaS functionality. The other players are those companies that wish to “cloudify” their own internal systems. The reasons for this decision are varied, ranging from the desire to maintain complete control over all the architecture and the cloud components to legal constraints limiting the use of public cloud infrastructures.

With Trove, much of the management of your database system is taken care of by automating a significant portion of the configuration and initial setup steps necessitated when launching a new server. This includes deployment, configuration, patching, backups, restores, and monitoring that can be administered from either a CLI interface, RESTful API’s or OpenStack’s Horizon dashboard. At this point, what Trove doesn’t provide is failover, replication and clustering. This functionality is slated to be implemented in the Kilo release of OpenStack due out in April/2015.

The process flow is relatively simple. The OpenStack Administrator first configures the basic infrastructure by installing the database service. He or she would then create an image for each type of database they wish to support such as MySQL or MongoDB. They would then import the images and offer them to their tenants. From the end users perspective only a few commandes are necessary to get up and running. First issuing the <trove create> command to create a database service instance, followed by <trove list> command to get the ID of the instance and finally trove show command to get the IP address of it.

For example to create a database, you first start off by creating a database instance. This is an isolated database environment with compute and storage resources in a single tenant environment on a shared physical host machine. You can run a database instance with a variety of database engines such as MySQL or MongoDB.

From the Trove client I can issue the following command to create a database instance called PS_troveinstance, with a volume size of 2 GB, a user called PS_user, a password PS_password and the MySQL datastore (or database engine):

$ trove create –size 2 –users PS_user:PS_password –datastore MySQL PS_troveinstance

Next I issue the following command to get the ID of the database instance:

$ trove list PS_troveinstance

And finally, to create a database called PS_trovedb, I execute:

$ trove database-create PS_troveinstance PS_trovedb

Alternatively, I could have just combined the above commands as:

$ trove create –size 2 —-database PS_trovedb users PS_user:PS_password –datastore MySQL PS_troveinstance

And thus we now have a MySQL database server containing a database called PS_trovedb.

In our next post on OpenStack/Trove, we’ll dig even further and discuss the software and hardware requirements, and how to actually set up Trove.

On a related note, Percona has several experts attending this week’s OpenStack Operations Summit in San Antonio, Texas. One of them is Matt Griffin, director of product management, who pointed out in a recent post that many OpenStack operators use Percona open source software including the MySQL drop-in compatible Percona Server and Galera-based Percona XtraDB Cluster as well as tools such as Percona XtraBackup and Percona Toolkit. “We see a need in the community to understand how to improve MySQL performance in OpenStack. As a result, Percona, submitted 16 presentations for November’s Paris OpenStack Summit,” Matt said. So stay tuned for related news from him, too, on that front.

The post OpenStack’s Trove: The benefits of this database as a service (DBaaS) appeared first on MySQL Performance Blog.

by Tim Sharp at August 25, 2014 12:00 PM

August 23, 2014

Nathan Kinder

Keystone is not an authentication service

In my previous post about security related updates that are coming in Juno, I mentioned that Keystone itself is a poor identity management solution.  I feel that this is a topic that deserves a more thorough discussion.

If you ask people familiar with OpenStack what Keystone’s purpose is, I’m willing to bet many of the answers include the term authentication.  In my mind, Keystone’s main purpose is authorization within an OpenStack deployment.  It achieves this by issuing authorization tokens to authenticated users.  It’s true that Keystone can perform authentication itself, but that doesn’t mean that it should be used for authentication.

Let’s consider what Keystone natively offers from an authentication perspective. At it’s simplest, Keystone uses a SQL backend for identity.  This involves storing a user entry that contains a hash of the user’s password.  When a user authenticates to Keystone to obtain a token, they provide their clear-text password, which Keystone hashes and compares with it’s stored copy.  This is a common approach for simple password based authentication.  The big downsides here are that the authentication is a single factor, and the user actually has to send their clear text password over the network.  The transport should of course be protected with SSL/TLS to prevent their password from being intercepted, but it’s still possible for an administrator on the Keystone system to get a user’s clear-text password by modifying Keystone or dumping memory contents.  If a user uses this same password for other authentication systems outside of OpenStack, the user has opened up their exposure to attack from a malicious Keystone administrator.  Most password-based authentication services also have additional capabilities that attempt to improve security.  This includes capabilities such as:

  • password syntax and dictionary checking to prevent against weak passwords
  • password expiration to force a regular password change interval
  • password history to prevent against password reuse
  • account lockout after a fixed number of failed authentication attempts

Keystone itself has none of the capabilities mentioned above, and I feel that it’s not worth adding them given that these capabilities are widely available in services that are designed to be centralized authentication services.  The typical answer for someone looking to have things like password policies with Keystone is to use LDAP as an identity backend instead of SQL.

Most people I talk to using OpenStack have their users stored in an existing centralized authentication service.  This is typically an LDAP server, or Active Directory (which offers LDAP capabilities).  Keystone has an LDAP driver for the identity backend to allow it to use LDAP for authentication and storage of users and groups.  Let’s consider what benefits we gain over using the SQL backend from an authentication perspective.

Keystone’s LDAP code currently only supports the LDAP simple bind operation.  This works exactly as Keystone’s SQL authentication does, which is described above.  A user provides Keystone with their clear-text password, and Keystone then sends this clear-text password to the LDAP server, which then hashes the password and compares it to the stored hash to authenticate the user.  This really doesn’t give us any benefit, as the user still gives up their clear-text password which is sent over the network.  LDAP has support for stronger authentication mechanisms via SASL, but Keystone doesn’t have any support for SASL bind operations.  Most LDAP servers have password policy and account lockout capabilities as described above, so this does provide some additional security over what Keystone itself offers.  From an identity management standpoint, using an LDAP server also gets Keystone out of the business of user provisioning and maintenance and allows for a centralized identity source that can be shared with other applications.

Keystone has the ability to allow for stronger forms of authentication than simple password-based authentication via it’s “external” authentication method.  When the “external” authentication method is used, Keystone expects that the web server that Keystone is running behind performs user authentication and supplies Keystone with the authenticated username via the “REMOTE_USER” environment variable.  This allows one to run Keystone within Apache HTTPD, utilizing the myriad of available Apache authentication modules to allow for strong forms of authentication such as Kerberos or X.509 client certificates.  This gets us away from the downsides of password-based authentication, as the user is no longer giving up their secret.  A user entry is still stored within Keystone (or in an LDAP server that Keystone utilizes), but a password credential does not need to be stored.  If we extract authentication out in this way, Keystone is really just responsible for authorization, which is the way it should be.

What about situations where you do not want to store all of your users within a single database that is used by Keystone?  If we think about it, all that Keystone really needs to know is the identity of the authenticated user and any information that it needs to map OpenStack specific roles and projects to that user.  Determining which roles to apply to a user is often based on group membership, though other attributes could be taken into account.  Instead of requiring Keystone to handle the many variations of LDAP schema and LDAP connection management itself, it would be ideal to have the additional user information supplied in a simple form by the web server along with “REMOTE_USER”.  This is exactly how Keystone’s federation extension works to provide SAML support.  From Keystone’s perspective, it is simply provided with all of the pertinent information about an authenticated user along with the request for a token.  Keystone is then able to map the user information it is provided with to the project and roles that apply to the user.  This is much simpler for Keystone than having to go out and look that information up itself as it does when LDAP is used.  This same model should be usable for any authentication method or identity source, not just SAML federation.  A very nice benefit of this approach  is that it makes Keystone itself agnostic of the authentication method.

There is a very useful Apache HTTPD module that was written by a colleague of mine called mod_lookup_identity that helps to eliminate the need for identity lookup functionality in Keystone itself.  This module utilizes SSSD from the underlying platform to provide user and group information from various identity providers.  If you’re not familiar with SSSD, it’s a robust set of client-side daemons designed to use identity providers such as FreeIPA, Active Directory, or LDAP for authentication and identity information.  It provides some nice features that  are a big step up from Keystone’s LDAP code, such as caching, connection pooling, and the ability to simultaneously provide identities from multiple sources.  Using SSSD offloads quite a bit of complexity from Keystone, which leaves Keystone to focus on it’s main purpose of providing authorization tokens.  This should result in a more scalable and performant Keystone.  SSSD is available on most common Linux platforms (Fedora, RHEL, Debian, Ubuntu, OpenSUSE, Gentoo, etc.), so using mod_lookup_identity is a nice general purpose approach for Keystone to use.

A picture is worth a thousand words, so let’s see what this whole approach looks like.

httpd_modulesWhat’s nice about this approach is that Keystone doesn’t know or care what authentication method was used or where the user information comes from.  This means that you can simply swap out the Apache modules to support different methods of authentication.  You want to use X.509 client-certificates for authentication?  Configure mod_ssl instead of mod_auth_kerb.  You want to use SAML?  Configure mod_shib or mod_mellon.  As far as Keystone is concerned, you would simply need to update the mappings to apply role assignments based on the way the environment variables are provided by the Apache modules.

A very common use-case for Keystone is to leverage users from Active Directory.  With the current LDAP identity driver, this requires putting service users in Active Directory and potentially needing to create additional OpenStack specific groups.  This is often a non-starter for many environments, as this additional information is not allowed to be added to Active Directory.  The Juno changes I’ve described in my previous post will solve the service user portion of this, but not the groups portion.  In addition, authentication is still restricted to LDAP simple bind.  A much better solution is to use FreeIPA’s cross-realm Kerberos trust functionality to establish a trust relationship with Active Directory.  This approach allows users from trusted Active Directory domains to access Linux/Unix resources that are managed in a FreeIPA realm.  An Active Directory user can use their Kerberos TGT that was obtained when logging into their Windows system to be used to fetch service tickets for Kerberos enabled services that are set up in FreeIPA, such as Keystone and Horizon.  OpenStack specific groups can also be defined locally in FreeIPA.  These groups can contain external users and groups from Active Directory in addition to local users that are defined in FreeIPA.

SSSD has been designed to take advantage of the cross-realm Kerberos trust functionality in FreeIPA.  SSSD is able to extract the group information contained in the PAC structure in the Kerberos tickets, performing the group lookups against FreeIPA.  This allows FreeIPA groups that have external members (such as Active Directory groups) to be resolved, which results in a new PAC being generated that represents the combined group memberships from multiple identity sources.  In effect, this functionality can be used to bridge multiple identity ‘silos’ without resorting to messy solutions that duplicate user data.

The area of cross-realm Kerberos trusts is complex, and likely a bit confusing if you are not already familiar with these areas.  What it boils down to is that you can allow Active Directory users to use Kerberos single-sign on for authentication to Keystone, with the ability to create additional groups for role assignment purposes outside of Active Directory.  In addition, you could create other accounts in FreeIPA which would also be able to authenticate via Kerberos.  You could even establish multiple cross-realm trusts to allow users from multiple Active Directory forests to utilize the same Keystone server.  Here is a revised version of the previous diagram that shows cross-realm Kerberos trusts in use with Keystone:

httpd_modules_trust

As you can see, offloading the authentication and identity lookup from Keystone can allow for some much more complex scenarios that are very useful for allowing existing identity providers to be used.  It also provides some very nice security benefits by providing stronger forms of authentication than Keystone is currently capable of, some of which provide for a much nicer user experience than passwords due to single-sign on.

by Nathan Kinder at August 23, 2014 12:59 AM

August 22, 2014

OpenStack Blog

OpenStack Community Weekly Newsletter (Aug 15 – 22)

Call for Proposals: Open Source Ecosystem Design Sessions at the Paris Summit

We’re continuing the Open Source Ecosystem Design Sessions at the OpenStack Summit Paris (Nov 3-7, 2014). It’s an opportunity to foster the projects and communities that don’t fall under the umbrella of the OpenStack Foundation, but are actively being used and developed within the greater ecosystem. We were inspired by everyone who came together to plan the next development cycle on their storage, configuration management, orchestration and networking projects. If you have an open source project that’s related to OpenStack and has a significant community of contributors, we invite you to submit a proposal.

OpenStack Ceilometer and the Gnocchi experiment

Julien Danjou reports on what’s happening within the OpenStack Telemetry program, with a retrospective on Ceilometer and its drawbacks.

Your baby is ugly! Picking which code is required for Commercial Core.

Entertaining post by Rob Hirschfeld about DefCore. “There’s no point in sugar-coating this: selecting API and code sections for core requires making hard choices and saying no. DefCore makes this fair by 1) defining principles for selection, 2) going slooooowly to limit surprises and 3) being transparent in operation.”

Who Are the Most Influential People of OpenStack®?

Fun game by the fun folks at Metacloud. They’re printing and giving away in Paris custom-printed decks of cards that include the most influential people of OpenStack. The best part? The community gets to decide who is featured in the deck!

The Road To Paris 2014 – Deadlines and Resources

During the Paris Summit there will be a working session for the Women of OpenStack to frame up more defined goals and line out a blueprint for the group moving forward. We encourage all women in the community to complete this very short surveyto provide input for the group.

Reports from Previous Events

Relevant Conversations

Security Advisories and Notes

Tips ‘n Tricks

Upcoming Events

Other News

Got Answers?

Ask OpenStack is the go-to destination for OpenStack users. Interesting questions waiting for answers:

Welcome New Reviewers and Developers

Welcome Michael Bayer to oslo-db core reviewers

Vlad Okhrimenko Yu Zhang
Simona Iuliana Toader Taku Fukushima
Sergey Lupersolsky Simon Leinen
Johnu George Shaun McDowell
Brent Roskos RedBaron
Bhavani Shankar Jesse Proudman
Alexis GUNST Jesse J. Cook
Adelina Tuvenie Edouard Outin
Doug Baer justin thomas simms
Angus Thomas Tom Holtzen
avi berger TARDIVEL
Dorin Paslaru Samer Deeb
Aishwarya Thangappa Puneet Arora
phanipawan
Venkatasubramanian Ramachandran
Tomáš Nováčik
Kyle Stevenson
Can ZHANG
Doug Baer
Clinton Knight
Игор Миловановић
Steven Tan
Rob Cresswell
yuriy brodskiy
avi berger
Oscar Romero
Louis Taylor

Latest Activity In Projects

Do you want to see at a glance the bugs filed and solved this week? Latest patches submitted for review? Check out the individual project pages on OpenStack Activity Board – Insights.

OpenStack Reactions

nocare

My reaction when someone nitpick on my reviews

The weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

by Stefano Maffulli at August 22, 2014 09:57 PM

Tesora Corp

Why We Certify Databases against the Tesora DBaaS Platform

Trove DBaaS.pngEarlier this week at the first OpenStack Trove Day, we announced our Tesora OpenStack Database Certification Program.  We launched this program to eliminate the headaches of testing individual database integrations and reducing implementation time. To do this we test all supported databases using our DBaaS Platform against the leading OpenStack distributions.

The Tesora DBaaS Platform is software that extends the Trove implementation and can be deployed within enterprise OpenStack clouds.  We have tested our DBaaS platform against most of the leading OpenStack distributions in addition to the current release from the OpenStack foundation. 

One question we¹ve received is whether we are certifying the Trove implementation available from the OpenStack Foundation or if we are only certifying our DBaaS Platform based on Trove. The answer is the latter, for a number of reasons.  At a high level, our certification process takes advantage of many of the same ease of use features that benefit the users of the platform. 

There are a number of reasons why we use the platform rather than the community version when we do the certification including:

  • Our certification uses our certified guest images that are pre-built and optimized for the databases that are supported.
  • We may provide certifications databases that have not (yet) been accepted into the community.  This may be due to:
    • The support for the database in question has not yet been reviewed and accepted into Trove proper, due to timing.
    • There is not yet sufficient demand in the community to provide support the particular database
    • The implementations may not currently conform to standard practices for OpenStack, (e.g. parts of the guest agents may require coding in languages other than Python).
    • Our continuous integration process ensures that the platform and databases can be installed into different OpenStack distributions using the native installation tools and packaging such as APT or RPM's.
    • This packaging is not available in the standard Trove implementation that is available from the OpenStack Foundation.
    • During our certification process we may discover issues that need to be addressed in the database integration. Fixes to these issues may be first implemented in our certified releases while at the same time being submitted upstream to the community.
    • Our testing is done against our pre-configured guest images. Constructing guest images can currently be a fairly manual and error prone process.  Using our images ensures that  we are launching databases into environments that are properly configured and optimize to run in an OpenStack environment.

We believe that our certification process makes it more likely that these databases will operate properly within standard Trove implementations. Since the core of our platform is Trove, our extensive testing of thesedatabase technologies in our platform will be applicable to the current release of Trove from the foundation.

In the end, this is a collaborative effort to improve the compatibility and performance of Trove. As always, please feel free write if you have any questions or comments.  

by 86 at August 22, 2014 03:32 PM

OpenStack Blog

Call for Proposals: Open Source Ecosystem Design Sessions at the Paris Summit

At the Atlanta OpenStack Summit we launched Open Source Ecosystem Design Sessions, a new track to give design session space to external open source projects related to OpenStack. It’s an opportunity to foster the projects and communities that don’t fall under the umbrella of the OpenStack Foundation, but are actively being used and developed within the greater ecosystem. We were inspired by everyone who came together to plan the next development cycle on their storage, configuration management, orchestration and networking projects.

We’re continuing this program at the OpenStack Summit Paris (Nov 3-7, 2014), and are opening the Call for Proposals. If you have an open source project that’s related to OpenStack and has a significant community of contributors, we invite you to submit a proposal.

The purpose of this space is to get developers in the same room so they can plan for the Kilo release cycle. This is not a speaker or marketing session, rather it’s meant to give a venue for focused collaboration and planning for a distributed development team. To achieve these goals, please follow these guidelines:

  • *The project needs to be directly related to OpenStack in some way.
  • It must be open source, with code freely available and actively developed.
  • The session must be non-commercial and vendor neutral, although a vendor that sponsors a project can organize and moderate the meeting.
  • Sessions must be open to anyone who is interested in contributing to the project. It’s primarily a venue for core developers to meet and plan, but also a place for new developers and those interested in the project to get more information.
  • Group organizers and participants must hold a conference pass and adhere to the OpenStack Summit Code of Conduct (passes will be available to organizers who do not currently have ATC status).
  • Sessions will be 90 minutes long.

The deadline for submissions is September 5, 2014, and we’ll follow up with decisions by September 19. Note that space is limited this year, with 6-9 slots available — similar or related projects may be asked to share space due to venue constraints.

Please apply for a session room with this form. You will be asked to provide the following information:

  • Project name
  • Project description and how it relates to OpenStack
  • Overview of how you’ll use the time and space
  • Names and contact information for up to two organizers committed hosting the session
  • Link to your project site and source code repository

A big thanks to everyone for all of the time and effort they put into supporting OpenStack!

by Chris Hoge at August 22, 2014 02:44 PM

Opensource.com

Survey says: OpenStack and Docker top cloud projects

When it comes to open source cloud projects, everybody has an opinion. A new survey attempts to take a broad look at those opinions and learn something about the state of the state of the open cloud and where it is headed.

by Jason Baker at August 22, 2014 08:00 AM

Michael Still

Juno nova mid-cycle meetup summary: conclusion

There's been a lot of content in this series about the Juno Nova mid-cycle meetup, so thanks to those who followed along with me. I've also received a lot of positive feedback about the posts, so I am thinking the exercise is worthwhile, and will try to be more organized for the next mid-cycle (and therefore get these posts out earlier). To recap quickly, here's what was covered in the series:

The first post in the series covered social issues: things like how we organized the mid-cycle meetup, how we should address core reviewer burnout, and the current state of play of the Juno release. Bug management has been an ongoing issue for Nova for a while, so we talked about bug management. We are making progress on this issue, but more needs to be done and it's going to take a lot of help for everyone to get there. There was also discussion about proposals on how to handle review workload in the Kilo release, although nothing has been finalized yet.

The second post covered the current state of play for containers in Nova, as well as our future direction. Unexpectedly, this was by far the most read post in the series if Google Analytics is to be believed. There is clear interest in support for containers in Nova. I expect this to be a hot topic at the Paris summit as well. Another new feature we're working on is the Ironic driver merge into Nova. This is progressing well, and we hope to have it fully merged by the end of the Juno release cycle.

At a superficial level the post about DB2 support in Nova is a simple tale of IBM's desire to have people use their database. However, to the skilled observer its deeper than that -- its a tale of love and loss, as well as a discussion of how to safely move our schema forward without causing undue pain for our large deployments. We also covered the state of cells support in Nova, with the main issue being that we really need cells to be feature complete. Hopefully people are working on a plan for this now. Another internal refactoring is the current scheduler work, which is important because it positions us for the future.

We also discussed the next gen Nova API, and talked through the proposed upgrade path for the transition from nova-network to neutron.

For those who are curious, there are 8,259 words (not that I am counting or anything) in this post series including this summary post. I estimate it took me about four working days to write (ED: and about two days for his trained team of technical writers to edit into mostly coherent English). I would love to get your feedback on if you found the series useful as it's a pretty big investment in time.

Tags for this post: openstack juno nova mid-cycle summary
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 22, 2014 07:47 AM

Juno nova mid-cycle meetup summary: the next generation Nova API

This is the final post in my series covering the highlights from the Juno Nova mid-cycle meetup. In this post I will cover our next generation API, which used to be called the v3 API but is largely now referred to as the v2.1 API. Getting to this point has been one of the more painful processes I think I've ever seen in Nova's development history, and I think we've learnt some important things about how large distributed projects operate along the way. My hope is that we remember these lessons next time we hit something as contentious as our API re-write has been.

Now on to the API itself. It started out as an attempt to improve our current API to be more maintainable and less confusing to our users. We deliberately decided that we would not focus on adding features, but instead attempt to reduce as much technical debt as possible. This development effort went on for about a year before we realized we'd made a mistake. The mistake we made is that we assumed that our users would agree it was trivial to move to a new API, and that they'd do that even if there weren't compelling new features, which it turned out was entirely incorrect.

I want to make it clear that this wasn't a mistake on the part of the v3 API team. They implemented what the technical leadership of Nova at the time asked for, and were very surprised when we discovered our mistake. We've now spent over a release cycle trying to recover from that mistake as gracefully as possible, but the upside is that the API we will be delivering is significantly more future proof than what we have in the current v2 API.

At the Atlanta Juno summit, it was agreed that the v3 API would never ship in its current form, and that what we would instead do is provide a v2.1 API. This API would be 99% compatible with the current v2 API, with the incompatible things being stuff like if you pass a malformed parameter to the API we will now tell you instead of silently ignoring it, which we call 'input validation'. The other thing we are going to add in the v2.1 API is a system of 'micro-versions', which allow a client to specify what version of the API it understands, and for the server to gracefully degrade to older versions if required.

This micro-version system is important, because the next step is to then start adding the v3 cleanups and fixes into the v2.1 API, but as a series of micro-versions. That way we can drag the majority of our users with us into a better future, without abandoning users of older API versions. I should note at this point that the mechanics for deciding what the minimum micro-version a version of Nova will support are largely undefined at the moment. My instinct is that we will tie to stable release versions in some way; if your client dates back to a release of Nova that we no longer support, then we might expect you to upgrade. However, that hasn't been debated yet, so don't take my thoughts on that as rigid truth.

Frustratingly, the intent of the v2.1 API has been agreed and unchanged since the Atlanta summit, yet we're late in the Juno release and most of the work isn't done yet. This is because we got bogged down in the mechanics of how micro-versions will work, and how the translation for older API versions will work inside the Nova code later on. We finally unblocked this at the mid-cycle meetup, which means this work can finally progress again.

The main concern that we needed to resolve at the mid-cycle was the belief that if the v2.1 API was implemented as a series of translations on top of the v3 code, then the translation layer would be quite thick and complicated. This raises issues of maintainability, as well as the amount of code we need to understand. The API team has now agreed to produce an API implementation that is just the v2.1 functionality, and will then layer things on top of that. This is actually invisible to users of the API, but it leaves us with an implementation where changes after v2.1 are additive, which should be easier to maintain.

One of the other changes in the original v3 code is that we stopped proxying functionality for Neutron, Cinder and Glance. With the decision to implement a v2.1 API instead, we will need to rebuild that proxying implementation. To unblock v2.1, and based on advice from the HP and Rackspace public cloud teams, we have decided to delay implementing these proxies. So, the first version of the v2.1 API we ship will not have proxies, but later versions will add them in. The current v2 API implementation will not be removed until all the proxies have been added to v2.1. This is prompted by the belief that many advanced API users don't use the Nova API proxies, and therefore could move to v2.1 without them being implemented.

Finally, I want to thank the Nova API team, especially Chris Yeoh and Kenichi Oomichi for their patience with us while we have worked through these complicated issues. It's much appreciated, and I find them a consistent pleasure to work with.

That brings us to the end of my summary of the Nova Juno mid-cycle meetup. I'll write up a quick summary post that ties all of the posts together, but apart from that this series is now finished. Thanks for following along.

Tags for this post: openstack juno nova mid-cycle summary api v3 v2.1
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues

Comment

August 22, 2014 12:52 AM

August 21, 2014

Mirantis

Meet Your OpenStack Training Instructor: Polina Petriuk

Next up in our “Meet your OpenStack Training Instructor” series, our interview subject is Polina Petriuk.

Tell us more about your background. How did you become involved in OpenStack training?

polina-petriuk-mirantis-training

In software development, I started as a QA Engineer, but I’ve never had any problems doing any other non-QA tasks. That gave me experience in a lot of areas beyond QA, such as business analysis and requirements gathering, datacenter planning, deployment and support, system administration and devops, and project and program management. When the opportunity to become an OpenStack trainer came up, it seemed like a perfect match for my background, skillset, and passion.

What do you enjoy most about training?

It never gets boring. I think I learn as much from my students as they do from me :-) I have students from different countries and companies, with different backgrounds and skills. I have an opportunity to talk to them and get to know what they think and how they do things. Also, my work involves a lot of traveling, which brings a whole different aspect of excitement. There is always something interesting to see and do in the new cities that I am exploring.

What particular challenges or opportunities will Mirantis OpenStack training help students meet?

OpenStack gets a lot of attention, and there clearly isn’t a shortage of information available online. The problem, however, is that this information is all over the place.

Mirantis OpenStack training solves this by presenting a holistic view of OpenStack. It focuses on giving the students the big picture and provides a better understanding of the logic behind how OpenStack works. I often tell my students that the goal is to learn what keywords to use in Google to narrow down the search results when looking for an answer to a particular question. :)

mirantis-bootcamp-san-jose

The course also facilitates interaction and collaboration between students. We often have people who have already tried OpenStack and even have it running in production. Such students have valuable experiences that they can share with the rest of the class. Mirantis instructors are also there to share the knowledge they have gained from various Mirantis customer projects, encouraging a truly collaborative learning environment.

What kinds of professionals are most likely to benefit from participating in this class?

Anyone who needs to understand how OpenStack ticks will find the training valuable. However, Solution Architects and DevOps are probably the ones that benefit the most from the entire training, start to end.

openstack-training

What advice would you give our readers who want to learn more about OpenStack?

Come to Mirantis training, of course. :-)

Additionally, openstack.org has a lot of useful information. For students who need hands-on experience, I also recommend installing OpenStack in VM using devstack or vagrant scripts with default options and then try to add or remove things. Google Search is your best friend there.

One interesting fact or story abut you that your students would be surprised to learn?

Hmm, I actually have two. I have a major in aerospace materials and designed the wings for a two-seat airplane manufactured by a small private company in Ukraine.

My grandmother and mother are both school teachers, and my sister and father teach in university. Growing up, I was expected to follow in their footsteps and go into education. But, I was really against the idea and chose the technology path instead. My mom often laughs at me now, saying that I couldn’t avoid my destiny of teaching. :)


Read more about our instructors on the Mirantis Training website.

The post Meet Your OpenStack Training Instructor: Polina Petriuk appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Lana Zhudina at August 21, 2014 10:48 PM

Giulio Fidente

TripleO vs OpenStack HA

One of the topics discussed during the TripleO mid-cycle meetup in RDU was our status in relation to deploying OpenStack in a highly available manner. This had been actively worked on for some time and recently reached a usable state.

Majority of problems seem to come from two factors: 1) we need to guarantee aivalability of external services too like the database and the message broker, 2) despite OpenStack services being roughly designed around a scale-out concept, while attempting to achieve that in TripleO we spotted a number of weak angles, some could be worked around, others could not and need changes in the core service. You're encouraged to try what we have today and help with the rest.

So to try this out, all one needs to do is set a number >= 3 for OVERCLOUD_CONTROLSCALE and controllers will be configured appropriately!:

export OVERCLOUD_CONTROLSCALE=3

Don't forget this only works on a few distros for now. I'd pick some Fedora 20.

On the controllers, MariaDB with Galera (for Fedora) is going to provide for a reliable SQL. There is still some work in progress to make sure the Galera cluster can be restarted correctly should all the controllers go down at the same time but, for single node failures, this should be safe to use.

RabbitMQ is covered too, nodes are clustered and balanced (via HAProxy), queues replicated.

And with regards to the OpenStack services, these are configured in a balancing manner (again, using HAProxy) except for those cases where this wouldn't have worked, notably the Neutron L3 agent and the Ceilometer Central agent, yet these are under control via Pacemaker. Cinder instead remains uncovered as volumes would require a shared storage too for proper HA. A spec has been proposed for this though.

Also, behind the scenes, the Heat template language addon shipped as merge.py and included in tripleo-heat-templates, which allows for example for scaling of the resources definition, is currently going to be removed and replaced with code living entirely in Heat.

And there is more so once you tried, join us on #tripleo @ freenode for the real fun!

by Giulio Fidente at August 21, 2014 01:01 PM

OpenStack @ NetApp

Highly Available OpenStack Deployments Built on NetApp Storage Systems

NetApp TR4323-DESIGN is now published!

Just a quick note to let you know that today, NetApp published TR4323-DESIGN: Highly Available OpenStack Deployments Built on NetApp Storage Systems. This document describes the solution design for a highly available deployment of OpenStack built on NetApp® storage systems. It includes architectural considerations, technology choices, and best practices for leveraging the functional and nonfunctional capabilities of NetApp storage systems toward an OpenStack deployment that provides self-service infrastructure-as- a-service (IaaS) capabilities to end users. It is based on the Icehouse release of OpenStack (April 2014).

Enjoy!

August 21, 2014 08:00 AM

August 20, 2014

Tesora Corp

Short Stack: Tesora's certification program and more cloud architects needed for $3.3Bn OpenStack market

short stack_b small_0.jpgWelcome to the Short Stack, our weekly feature where we search for the most intriguing OpenStack links to share with you. These links may come from traditional publications or company blogs, but if it's about OpenStack, we'll find the best links we can to share with you every week.

If you like what you see, please consider subscribing.

Here we go with this week's links:

More OpenStack Certifications - Because Interoperability Is Key | Forbes

Because the world really does need another OpenStack certification program, here’s what Tesora is doing to certify both database vendors and OpenStack distributions to eliminate the headache of enterprises having to test integration on their own.

The why and how of becoming a cloud architect | OpenSource.com

The demand is great; supply is limited. That’s one reason why becoming a cloud architect is a good career move. The money doesn’t hurt either. According to the OpenStack Foundation, on average you can earn $35,000 more a year than non-Openstack engineers.

Hey, biz bods: OpenStack will be worth $3.3bn by 2018 | The Register

There’s gold in them there hills. Anyone old enough to remember The Beverly Hillbillies will remember that line.  Lots of companies are vying for a piece of the exploding OpenStack market, said to be worth $3.3 billion by 2018.

What Eucalyptus' 'suckerfish' success means for Amazon | Tech Republic

The reincarnation of Eucalyptus? Matt Asay interviews Eucalyptus CEO Marten Mickos, and reports that the company is experiencing a renaissance with impressive product and customer traction.

Brocade's Tom Nadeau: OpenStack deserves a champion | Fierce Enterprise Communications

There has been much ink spilt lately about the jockeying for the title of OpenStack King. In this article, Tom Nadeau argues, it might be the right time in the development of OpenStack for a clear leader to emerge.

by 693 at August 20, 2014 01:46 PM

Opensource.com

Using Clocker and Apache Brooklyn to build a Docker cloud

With the growing potential of Docker, it's becoming clear that the future of at least some of the data center is going to be containerized. But there are still challenges in getting containerized applications deployed and managed across real and virtual hardware.

by Jason Baker at August 20, 2014 11:00 AM

Miguel Ángel Ajo

Neutron security_group_rules_for_devices RPC rewrite

We found during scalability tests, that the security_group_rules_for_devices RPC, which is transmitted from neutron-server to the neutron L2 agents during port changes, grew exponentially.

So we filled a spec for juno-3, the effort leaded by shihanzhang and me can be tracked here:

I have written a test and a little -dirty- benchmark (https://review.openstack.org/#/c/115575/1/neutron/tests/unit/test_security_groups_rpc.py line 418) to check the results and make sure the new RPC actually performs better.

Here are the results:

Message size (Y) vs. number of ports (X) graph:

image

RPC execution time in seconds (Y) vs. number of ports (X):

image

August 20, 2014 09:14 AM

Michael Still

Juno nova mid-cycle meetup summary: nova-network to Neutron migration

This will be my second last post about the Juno Nova mid-cycle meetup, which covers the state of play for work on the nova-network to Neutron upgrade.

First off, some background information. Neutron (formerly Quantum) was developed over a long period of time to replace nova-network, and added to the OpenStack Folsom release. The development of new features for nova-network was frozen in the Nova code base, so that users would transition to Neutron. Unfortunately the transition period took longer than expected. We ended up having to unfreeze development of nova-network, in order to fix reliability problems that were affecting our CI gating and the reliability of deployments for existing nova-network users. Also, at least two OpenStack companies were carrying significant feature patches for nova-network, which we wanted to merge into the main code base.

You can see the announcement at http://lists.openstack.org/pipermail/openstack-dev/2014-January/025824.html. The main enhancements post-freeze were a conversion to use our new objects infrastructure (and therefore conductor), as well as features that were being developed by Nebula. I can't find any contributions from the other OpenStack company in the code base at this time, so I assume they haven't been proposed.

The nova-network to Neutron migration path has come to the attention of the OpenStack Technical Committee, who have asked for a more formal plan to address Neutron feature gaps and deprecate nova-network. That plan is tracked at https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/Neutron_Gap_Coverage. As you can see, there are still some things to be merged which are targeted for juno-3. At the time of writing this includes grenade testing; Neutron being the devstack default; a replacement for nova-network multi-host; a migration plan; and some documentation. They are all making good progress, but until these action items are completed, Nova can't start the process of deprecating nova-network.

The discussion at the Nova mid-cycle meetup was around the migration planning item in the plan. There is a Nova specification that outlines one possible plan for live upgrading instances (i.e, no instance downtime) at https://review.openstack.org/#/c/101921/, but this will probably now be replaced with a simpler migration path involving cold migrations. This is prompted by not being able to find a user that absolutely has to have live upgrade. There was some confusion, because of a belief that the TC was requiring a live upgrade plan. But as Russell Bryant says in the meetup etherpad:

"Note that the TC has made no such statement on migration expectations other than a migration path must exist, both projects must agree on the plan, and that plan must be submitted to the TC as a part of the project's graduation review (or project gap review in this case). I wouldn't expect the TC to make much of a fuss about the plan if both Nova and Neutron teams are in agreement."


The current plan is to go forward with a cold upgrade path, unless a user comes forward with an absolute hard requirement for a live upgrade, and a plan to fund developers to work on it.

At this point, it looks like we are on track to get all of the functionality we need from Neutron in the Juno release. If that happens, we will start the nova-network deprecation timer in Kilo, with my expectation being that nova-network would be removed in the "M" release. There is also an option to change the default networking implementation to Neutron before the deprecation of nova-network is complete, which will mean that new deployments are defaulting to the long term supported option.

In the next (and probably final) post in this series, I'll talk about the API formerly known as Nova API v3.

Tags for this post: openstack juno nova mid-cycle summary nova-network neutron migration
Related posts: Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 20, 2014 04:37 AM

August 19, 2014

Julien Danjou

Tracking OpenStack contributions in GitHub

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

<figure class="pull-right illustration clear" style="width: 30%;"> </figure>

OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere.

It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit).

Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub.

<script src="https://gist.github.com/jd/20d2ffd4b1e6ce3a8b07.js"></script>

And voilà, my statistics are now including all my contributions to OpenStack!

<figure class="illustration"> </figure>

by Julien Danjou at August 19, 2014 05:00 PM

Rafael Knuth

Google+ Hangout: OpenStack Networking (Neutron) - 2014 Update

Following up on the 2013 Sessions on OpenStack Networking (Neutron) we will have a session on...

August 19, 2014 03:27 PM

Michael Still

Juno nova mid-cycle meetup summary: slots

If I had to guess what would be a controversial topic from the mid-cycle meetup, it would have to be this slots proposal. I was actually in a Technical Committee meeting when this proposal was first made, but I'm told there were plenty of people in the room keen to give this idea a try. Since the mid-cycle Joe Gordon has written up a more formal proposal, which can be found at https://review.openstack.org/#/c/112733.

If you look at the last few Nova releases, core reviewers have been drowning under code reviews, so we need to control the review workload. What is currently happening is that everyone throws up their thing into Gerrit, and then each core tries to identify the important things and review them. There is a list of prioritized blueprints in Launchpad, but it is not used much as a way of determining what to review. The result of this is that there are hundreds of reviews outstanding for Nova (500 when I wrote this post). Many of these will get a review, but it is hard for authors to get two cores to pay attention to a review long enough for it to be approved and merged.

If we could rate limit the number of proposed reviews in Gerrit, then cores would be able to focus their attention on the smaller number of outstanding reviews, and land more code. Because each review would merge faster, we believe this rate limiting would help us land more code rather than less, as our workload would be better managed. You could argue that this will mean we just say 'no' more often, but that's not the intent, it's more about bringing focus to what we're reviewing, so that we can get patches through the process completely. There's nothing more frustrating to a code author than having one +2 on their code and then hitting some merge freeze deadline.

The proposal is therefore to designate a number of blueprints that can be under review at any one time. The initial proposal was for ten, and the term 'slot' was coined to describe the available review capacity. If your blueprint was not allocated a slot, then it would either not be proposed in Gerrit yet, or if it was it would have a procedural -2 on it (much like code reviews associated with unapproved specifications do now).

The number of slots is arbitrary at this point. Ten is our best guess of how much we can dilute core's focus without losing efficiency. We would tweak the number as we gained experience if we went ahead with this proposal. Remember, too, that a slot isn't always a single code review. If the VMWare refactor was in a slot for example, we might find that there were also ten code reviews associated with that single slot.

How do you determine what occupies a review slot? The proposal is to groom the list of approved specifications more carefully. We would collaboratively produce a ranked list of blueprints in the order of their importance to Nova and OpenStack overall. As slots become available, the next highest ranked blueprint with code ready for review would be moved into one of the review slots. A blueprint would be considered 'ready for review' once the specification is merged, and the code is complete and ready for intensive code review.

What happens if code is in a slot and something goes wrong? Imagine if a proposer goes on vacation and stops responding to review comments. If that happened we would bump the code out of the slot, but would put it back on the backlog in the location dictated by its priority. In other words there is no penalty for being bumped, you just need to wait for a slot to reappear when you're available again.

We also talked about whether we were requiring specifications for changes which are too simple. If something is relatively uncontroversial and simple (a better tag for internationalization for example), but not a bug, it falls through the cracks of our process at the moment and ends up needing to have a specification written. There was talk of finding another way to track this work. I'm not sure I agree with this part, because a trivial specification is a relatively cheap thing to do. However, it's something I'm happy to talk about.

We also know that Nova needs to spend more time paying down its accrued technical debt, which you can see in the huge amount of bugs we have outstanding at the moment. There is no shortage of people willing to write code for Nova, but there is a shortage of people fixing bugs and working on strategic things instead of new features. If we could reserve slots for technical debt, then it would help us to get people to work on those aspects, because they wouldn't spend time on a less interesting problem and then discover they can't even get their code reviewed. We even talked about having an alternating focus for Nova releases; we could have a release focused on paying down technical debt and stability, and then the next release focused on new features. The Linux kernel does something quite similar to this and it seems to work well for them.

Using slots would allow us to land more valuable code faster. Of course, it also means that some patches will get dropped on the floor, but if the system is working properly, those features will be ones that aren't important to OpenStack. Considering that right now we're not landing many features at all, this would be an improvement.

This proposal is obviously complicated, and everyone will have an opinion. We haven't really thought through all the mechanics fully, yet, and it's certainly not a done deal at this point. The ranking process seems to be the most contentious point. We could encourage the community to help us rank things by priority, but it's not clear how that process would work. Regardless, I feel like we need to be more systematic about what code we're trying to land. It's embarrassing how little has landed in Juno for Nova, and we need to be working on that. I would like to continue discussing this as a community to make sure that we end up with something that works well and that everyone is happy with.

This series is nearly done, but in the next post I'll cover the current status of the nova-network to neutron upgrade path.

Tags for this post: openstack juno nova mid-cycle summary review slots blueprint priority project management
Related posts: Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support

Comment

August 19, 2014 08:34 AM

Mirantis

Enterprise OpenStack and DBaaS

OpenStack Trove Logo

As OpenStack matures, and as the community provides new features demanded by customers and the enterprise, one critical services required to benefit from IaaS managed by OpenStack is Database as a Service (dBaaS) on which the OpenStack Project is called Trove is focused. Most conversations about Trove center around the open source MySQL database, but other enterprise-level databases such as Oracle dB 12c can also be managed by Trove.  In fact, Oracle 12c includes features, such as multi-tenancy, that Enterprises want, and that Trove should support.  

Database as a Service (DBaaS)

The database is one of the most critical services in the enterprise. The database’s function is to store data, enable the ability to search for specific data, analyze data, abstract data for applications. Customers want to have the ability to scale both relational and non-relational  cloud database engines. The Trove project is focused on enabling OpenStack users to effortlessly and rapidly manage a database without the complexities of administrative jobs on the database itself.

Currently the Trove model looks something like this:

Current Trove Model   

Another major goal of Trove, however, is to isolate resources and automate complex operations usually done by a dBA Engineer, such as deployment of databases, patching a database, backing up a database, restoring a database, monitoring a running database, and configuring a new database.

Trove today

The ‘current’ process in creating a dB instance with the current build of Trove looks something like this:

Current Trove Workflow

This corresponds to the following steps:

Step 1: A request to create a virtual database is made
Step 2: A authentication request is made with Keystone
Step 3: Provision a Trove Instance (which also leads to the creation of a new VM on which to run the database)
Step 4: Trove returns a response that incorporates the description of the instance, including:

  1. The Instance ID
  2. The DataStore Type or Version
  3. The Volume Size
  4. The IP Address of the instance
  5. Other variables that might be requested or required

A current limitation of OpenStack’s ‘Database as a Service’ tool is that Trove currently only supports single-tenant instances. One new feature in Oracle dB 12c is multi-tenancy, which is a critical feature currently being deployed at  accounts such as Thomson Reuters and other enterprises in the financial industry. With their growing interest in OpenStack, having this feature on the roadmap is critical to gain access to financial industry accounts.  

Multi-tenancy and Trove

John Deveraux will be presenting on OpenStack Trove Day, on August 19th in Cambridge, MA.

Multi-tenancy is an architecture in which a single instance of a software application serves multiple customers. Each customer is called a tenant. Tenants may be given the ability to customize some parts of the application, such as color of the user interface, but they cannot customize the application’s code.

Multi-tenancy can be economical because software development and maintenance costs are shared. It can be contrasted with single-tenancy, an architecture in which each customer has their own software instance and may be given access to code. With a multi-tenancy architecture, the provider only has to make updates once. With a single-tenancy architecture, the provider has to touch multiple instances of the software in order to make updates.

Oracle’s current plan for OpenStack Trove is:

1. Each Tenant Gets a dedicated Database Instance

  1. Based on a preconfigured VM Image
  2. Tenant runs on a Compute and Storage Node(s)

2. Customer has complete control of services
3. Any Database Application, any Database Query Language, and Connection Methods are to be supported
4. For the first time, Oracle will change its per CPU/core licensing model and introduce monthly subscription pricing

To accommodate these changes, Trove will need a slightly modified architecture:

Proposed Trove Workflow

In this case, the workflow looks like this:

  1. A user requests a database.  (In this case, it’s a portable database (PDB).
  2. Keystone receives an authorization request
  3. The Database container gets provisioned
    1. The Guest agent asks Oracle to create the PDB
    2. Oracle creates the PDB
    3. Oracle responds to the guest with connection attributes
  4. The response, containing connection credentials, is returned to the requesting user.

To make this work, the Trove project needs the following components:

  1. API Server – The API Server receives requests and talks directly to guest agents to address quick tasks such as obtaining a list of dB Users, but asks the Task Manager to handle tortuous, serial tasks.
  2. Message Bus – A Messaging Queue Broker between API points such as the Task Manager and the Guest Agent via HTTP requests.
  3. Task Manager – This is the workhorse of the system, responsible for provisioning instances, controlling the life cycle of all instances, and performing operations on any instance. It works with the API Server by collecting and responding to messages. Remember that both the Task Manager and the API Server require HTTP calls to the OpenStack services.
  4. Guest Agent – This is a service that runs within a guest instance running a database, which also manages and performs operations on the datastore. It brings the datastore online and also transmits a heartbeat signal to the API via the conductor.
  5. Conductor – This is a service running on the host which handles accepting messages from guest instances to update information on the host. If you are running a database backup – the conductor, using the RPC Messages via the bus, performs the relevant operation as follows:
    1. trove-conductor is the entry point
    2. RpcService, configured by /etc/trove/trove-conductor.conf, specifies the conductor manager
    3. Requests are pushed to the Message Queue
    4. The heartbeat updates the status of any instance. There are three basic choices: NEW, BUILDING, and ACTIVE, though others can be added

Trove isn’t quite ready for this new structure, however.

Moving forward

The Community Roadmap for Icehouse includes:

  • Generalization of core in order to improve extensibility
  • Database multi-engine type using versiontype
  • Adding support of cluster provisioning and management
  • Extensibility forming complete architectures (using Conductor and Scheduler)
  • Amazon “parameters-group” like features
  • Automated scheduled tasks
  • Designate / Ceilometer Support
  • Automated failover
  • Tempest Test Coverage

As of now, multi-tenancy isn’t on that list, but it should be. Using services such as Murano, which enables your users to select which database service to deploy and with the proper flavor settings to determine how much resources will be allocated for that tenant is very enticing for enterprise customers who want to leverage an Orchestration Tool to combine the management and operations of their hardware and software components. Murano, for example, enforces permissions with Keystone to ensure only the predetermined users have access, and lets Trove communicate with the Guest Agents, allowing a quick creation of a database.

These are all attractive features for Enterprise users, and for many, they are enough. But for those customers who need that extra level of Enterprise functionality, the Trove roadmap should include a move in that direction.

The post Enterprise OpenStack and DBaaS appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by John Devereaux at August 19, 2014 01:00 AM

August 18, 2014

IBM OpenStack Team

Deploying TOSCA portable workloads in OpenStack

As this is my first time blogging on Thoughts on Cloud, I would like to take this opportunity to introduce myself. My name is Sahdev Zala and I am an advisory software engineer at IBM working on OpenStack for the past two years. I started my journey in OpenStack with a focus on Keystone LDAP backend enhancements.

Most recently, I have been working on OpenStack’s Heat orchestration project to work on providing support for native Heat Orchestration Template (HOT) and align it with an OASIS open standard called Topology Orchestration Specification for Cloud Applications (TOSCA). I joined the TOSCA technical committee (TC) to contribute to its development. TOSCA is a new cloud technology that I want to discuss today as it defines, for the first time, a means to describe applications, including their components, relationships and dependencies in a way that allows them to be portable to any cloud that supports the TOSCA standard.

The most interesting work is being done on the TOSCA Simple Profile in YAML Version 1.0, which just published its second working draft with a third planned to release in the next month. TOSCA has a large number of companies and organizations that actively support and contribute to its development, such as Alcatel-Lucent, Cisco, Fujitsu, GigaSpaces, Huawei, SAP and Vnomic along with IBM. IBM is active in the development of this standard and takes a leadership role where possible, including co-chairing the TOSCA TC and its Interoperability subcommittee.

One of the things that came out of early TOSCA TC discussions is that it is highly beneficial to cloud providers and users if TOSCA can be deployed via open cloud technologies like OpenStack. This was discussed between the OpenStack Heat core team and the IBM TOSCA team at the OpenStack Hong Kong Summit in 2013.

From the discussion, two things became clear. First, TOSCA needed to provide a simple way to express workload topology in YAML. It also became clear that a good way to deploy TOSCA workload initially in OpenStack is via converting TOSCA service templates to HOT. After the summit, I discussed the Heat Project Technical Lead (PTL) for the OpenStack IceHouse release and other members of the Heat core team. As an outcome, I started a StackForge project called Heat-Translator in early 2014, which is aimed to provide a translation layer on top of Heat to convert TOSCA templates to HOT and then deploy the converted template via Heat.

The TOSCA translator tool is designed to be generic enough to provide a base for translating any non-HOT templates to HOT. The Heat-Translator project is expected to be consumed by the Heat-Client. The Heat-Translator project currently has development contribution from IBM, Huawei and GigaSpaces, and more developers from companies like Cisco are expected to be on board shortly.

The Heat-Translator project is a work in progress. At present, the Heat-Translator tool takes TOSCA templates as an input and produces a HOT template which can be deployed via Heat. Development work is going on to provide support for TOSCA Cloud Services Archive (CSAR) as an input as well. The TOSCA CSAR is a container with TOSCA template and needed artifacts, such as installation and configuration scripts, by the template.

Below is a high level architecture diagram:

Heat Translator architecture

Figure 1: Heat-Translator architecture 

The TOSCA parser in the Heat-Translator is a stand-alone TOSCA parser, which produces an in-memory graph of the TOSCA template. The translator code takes this in-memory graph and maps it to Heat resources, utilizing the software configuration support that is provided in Heat with the Icehouse release of OpenStack, and produces a HOT template.  A more native support in Heat for TOSCA life cycle operations is under development. The following snippet shows an example of the WordPress TOSCA node template (Fig 2) and resulting Heat resource in HOT template (Fig 3):

TOSCA template with WordPress node

Figure 2: Snippet of TOSCA template with WordPress node

HOT template with WordPress resource

Figure 3: Snippet of HOT template with WordPress resource

What to expect in the near future

Currently the development and testing was done against two use cases demonstrating translation of a template with WordPress, MySQL, Web Server and a TOSCA compute node. The TOSCA YAML templates for those use cases can be found in the Heat-Translator StackForge project under the tests data of the TOSCA library. The development of Heat-Translator is going on in parallel to ongoing development in the TOSCA specification. Translation support for BlockStorage (mapped to Heat Cinder resource), Networking (mapped to Heat Nova and Neutron resource), Monitoring and TOSCA CSAR is underway and will be available soon.

Want to Contribute?

You are welcome to contribute to Heat-Translator community, please refer to the Launchpad project.

The post Deploying TOSCA portable workloads in OpenStack appeared first on Thoughts on Cloud.

by Sahdev Zala at August 18, 2014 05:14 PM

Julien Danjou

OpenStack Ceilometer and the Gnocchi experiment

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning.

In this article, I would like to relate what I've been doing with other Ceilometer developers in the last 5 months. I've lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it's largely time to take a break and talk about it.

Ceilometer early design

For the last years, Ceilometer didn't change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster…) constructs and emits a sample headed for the storage component that we call the collector.

This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support.

The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want.

The scalability issue

We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in order to be able to scan them at once, and there's no good "perfect" sharding solution, making data storage scalability painful.

It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the number of samples recorded (see big O notation if you're unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable.

During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

<figure class="illustration"> </figure>

Thinking outside the box

Unfortunately, the PTL job doesn't leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn't able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box.

When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics.

Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built into oslo.messaging.

Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth usage, the number of images that Glance is storing for you, etc… These are not events, since nothing is happening. These are facts, states we need to meter.

Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right.

I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there's still no silver bullet – this made it a good challenge.

The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API – as we do in all OpenStack services. This should cover the metric storage pretty well.

Cooking Gnocchi

<figure class="pull-right illustration" style="width: 45%;"> <figcaption>A cloud of gnocchis!</figcaption> </figure>

At the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a "pasta" name, right?

The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process.

The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I've named these things "entities", as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn't care and should be agnostic in this regard.

One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow.

I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both.

The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable.

Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn't depends on the number of metrics you have nor on the number of things you are metering. Which is already a huge win compared to the current Ceilometer collector design.

However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I've started to hack on that in my own fork, but… then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn't going to break everything (remember, no test coverage).

I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper's, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and… it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers.

Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It's a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way.

Handling resources

Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

<figure class="illustration"> <figcaption>Relationship of entities and resources</figcaption> </figure>

Once this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now.

Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network.

All of these requirements led to to the design of what's called the indexer. The indexer is responsible for indexing entities, resources, and link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It's easy enough to index the most requested attributes (columns), and they are also correctly typed.

We plan to establish a model for all known OpenStack resources (instances, volumes, networks, …) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It'd be up to the users to store extra attributes.

Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

<figure class="illustration"> <figcaption>The indexer classes and their relations</figcaption> </figure>

REST API

All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository; though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer.

The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

<figure class="illustration"> <figcaption>Macroscopic view of the Gnocchi architecture</figcaption> </figure>

Roadmap & Ceilometer integration

All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue.

It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to.

Early benchmarks

Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July.

The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

<figure class="illustration"> </figure>

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.

Next steps

<figure class="pull-right illustration" style="width: 30%;"> <figcaption>Clément drawing the logo</figcaption> </figure>

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now.

The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We'll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

by Julien Danjou at August 18, 2014 03:00 PM

eNovance Engineering Teams

OpenStack Ceilometer and the Gnocchi experiment

A little more than 2 years ago, the Ceilometer project was launched inside the OpenStack ecosystem. Its main objective was to measure OpenStack cloud platforms in order to provide data and mechanisms for functionalities such as billing, alarming or capacity planning.

In this article, I would like to relate what I’ve been doing with other Ceilometer developers in the last 5 months. I’ve lowered my involvement in Ceilometer itself directly to concentrate on solving one of its biggest issue at the source, and I think it’s largely time to take a break and talk about it.

Ceilometer early design

For the last years, Ceilometer didn’t change in its core architecture. Without diving too much in all its parts, one of the early design decision was to build the metering around a data structure we called samples. A sample is generated each time Ceilometer measures something. It is composed of a few fields, such as the the resource id that is metered, the user and project id owning that resources, the meter name, the measured value, a timestamp and a few free-form metadata. Each time Ceilometer measures something, one of its components (an agent, a pollster…) constructs and emits a sample headed for the storage component that we call the collector.

This collector is responsible for storing the samples into a database. The Ceilometer collector uses a pluggable storage system, meaning that you can pick any database system you prefer. Our original implementation has been based on MongoDB from the beginning, but we then added a SQL driver, and people contributed things such as HBase or DB2 support.

The REST API exposed by Ceilometer allows to execute various reading requests on this data store. It can returns you the list of resources that have been measured for a particular project, or compute some statistics on metrics. Allowing such a large panel of possibilities and having such a flexible data structure allows to do a lot of different things with Ceilometer, as you can almost query the data in any mean you want.

The scalability issue

We soon started to encounter scalability issues in many of the read requests made via the REST API. A lot of the requests requires the data storage to do full scans of all the stored samples. Indeed, the fact that the API allows you to filter on any fields and also on the free-form metadata (meaning non
indexed key/values tuples) has a terrible cost in terms of performance (as pointed before, the metadata are attached to each sample generated by Ceilometer and is stored as is). That basically means that the sample data structure is stored in most drivers in just one table or collection, in
order to be able to scan them at once, and there’s no good “perfect” sharding solution, making data storage scalability painful.

It turns out that the Ceilometer REST API is unable to handle most of the requests in a timely manner as most operations are O(n) where n is the
number of samples recorded (see big O notation if you’re unfamiliar with it). That number of samples can grow very rapidly in an environment of thousands of metered nodes and with a data retention of several weeks. There is a few optimizations to make things smoother in general cases fortunately, but as soon as you run specific queries, the API gets barely usable.

During this last year, as the Ceilometer PTL, I discovered these issues first hand since a lot of people were feeding me back with this kind of testimony. We engaged several blueprints to improve the situation, but it was soon clear to me that this was not going to be enough anyway.

unacceptable
Thinking outside the box

Unfortunately, the PTL job doesn’t leave him enough time to work on the actual code nor to play with anything new. I was coping with most of the project bureaucracy and I wasn’t able to work on any good solution to tackle the issue at its root. Still, I had a few ideas that I wanted to try and as soon as I stepped down from the PTL role, I stopped working on Ceilometer itself to try something new and to think a bit outside the box.

When one takes a look at what have been brought recently in Ceilometer, they can see the idea that Ceilometer actually needs to handle 2 types of data: events and metrics.

Events are data generated when something happens: an instance start, a volume is attached, or an HTTP request is sent to an REST API server. These
are events that Ceilometer needs to collect and store. Most OpenStack components are able to send such events using the notification system built
into oslo.messaging.

Metrics is what Ceilometer needs to store but that is not necessarily tied to an event. Think about an instance CPU usage, a router network bandwidth
usage, the number of images that Glance is storing for you, etc… These are not events, since nothing is happening. These are facts, states we need to meter.

Computing statistics for billing or capacity planning requires both of these data sources, but they should be distinct. Based on that assumption, and the fact that Ceilometer was getting support for storing events, I started to focus on getting the metric part right.

I had been a system administrator for a decade before jumping into OpenStack development, so I know a thing or two on how monitoring is done in this area, and what kind of technology operators rely on. I also know that there’s still no silver bullet – this made it a good challenge.

The first thing that came to my mind was to use some kind of time-series database, and export its access via a REST API – as we do in all OpenStack
services. This should cover the metric storage pretty well.

Cooking Gnocchi

gnocchi-logoAt the end of April 2014, this led met to start a new project code-named Gnocchi. For the record, the name was picked after confusing so many times the OpenStack Marconi project, reading OpenStack Macaroni instead. At least one OpenStack project should have a “pasta” name, right?

The point of having a new project and not send patches on Ceilometer, was that first I had no clue if it was going to make something that would be any better, and second, being able to iterate more rapidly without being strongly coupled with the release process.

The first prototype started around the following idea: what you want is to meter things. That means storing a list of tuples of (timestamp, value) for it. I’ve named these things “entities”, as no assumption are made on what they are. An entity can represent the temperature in a room or the CPU usage of an instance. The service shouldn’t care and should be agnostic in this regard.

One feature that we discussed for several OpenStack summits in the Ceilometer sessions, was the idea of doing aggregation. Meaning, aggregating samples over a period of time to only store a smaller amount of them. These are things that time-series format such as the RRDtool have been doing for a long time on the fly, and I decided it was a good trail to follow.

I assumed that this was going to be a requirement when storing metrics into Gnocchi. The user would need to provide what kind of archiving it would need: 1 second precision over a day, 1 hour precision over a year, or even both.

The first driver written to achieve that and store those metrics inside Gnocchi was based on whisper. Whisper is the file format used to store metrics for the Graphite project. For the actual storage, the driver uses Swift, which has the advantages to be part of OpenStack and scalable.

Storing metrics for each entities in a different whisper file and putting them in Swift turned out to have a fantastic algorithm complexity: it was O(1). Indeed, the complexity needed to store and retrieve metrics doesn’t depends on the number of metrics you have nor on the number of things you
are metering. Which is already a huge win compared to the current Ceilometer collector design.

However, it turned out that whisper has a few limitations that I was unable to circumvent in any manner. I needed to patch it to remove a lot of its assumption about manipulating file, or that everything is relative to now (time.time()). I’ve started to hack on that in my own fork, but… then everything broke. The whisper project code base is, well, not the state of the art, and have 0 unit test. I was starring at a huge effort to transform whisper into the time-series format I wanted, without being sure I wasn’t going to break everything (remember, no test coverage).

I decided to take a break and look into alternatives, and stumbled upon Pandas, a data manipulation and statistics library for Python. Turns out that Pandas support time-series natively, and that it could do a lot of the smart computation needed in Gnocchi. I built a new file format leveraging Pandas for computing the time-series and named it carbonara (a wink to both the Carbon project and pasta, how clever!). The code is quite small (a third of whisper‘s, 200 SLOC vs 600 SLOC), does not have many of the whisper limitations and… it has test coverage. These Carbonara files are then, in the same fashion, stored into Swift containers.

Anyway, Gnocchi storage driver system is designed in the same spirit that the rest of OpenStack and Ceilometer storage driver system. It’s a plug-in system with an API, so anyone can write their own driver. Eoghan Glynn has already started to write a InfluxDB driver, working closely with the upstream developer of that database. Dina Belova started to write an OpenTSDB driver. This helps to make sure the API is designed directly in the right way.

Handling resources

Measuring individual entities is great and needed, but you also need to link them with resources. When measuring the temperature and the number of a people in a room, it is useful to link these 2 separate entities to a resource, in that case the room, and give a name to these relations, so one is able to identify what attribute of the resource is actually measured. It is also important to provide the possibility to store attributes on these resources, such as their owners, the time they started and ended their existence, etc.

gnocchi-relationshipOnce this list of resource is collected, the next step is to list and filter them, based on any criteria. One might want to retrieve the list of resources created last week or the list of instances hosted on a particular node right now.

Resources also need to be specialized. Some resources have attributes that must be stored in order for filtering to be useful. Think about an instance name or a router network.

All of these requirements led to to the design of what’s called the indexer. The indexer is responsible for indexing entities, resources, and
link them together. The initial implementation is based on SQLAlchemy and should be pretty efficient. It’s easy enough to index the most requested attributes (columns), and they are also correctly typed.

We plan to establish a model for all known OpenStack resources (instances, volumes, networks, …) to store and index them into the Gnocchi indexer in order to request them in an efficient way from one place. The generic resource class can be used to handle generic resources that are not tied to OpenStack. It’d be up to the users to store extra attributes.

Dropping the free form metadata we used to have in Ceilometer makes sure that querying the indexer is going to be efficient and scalable.

gnocchi-classes

REST API

All of this is exported via a REST API that was partially designed and documented in the Gnocchi specification in the Ceilometer repository;
though the spec is not up-to-date yet. We plan to auto-generate the documentation from the code as we are currently doing in Ceilometer.

The REST API is pretty easy to use, and you can use it to manipulate entities and resources, and request the information back.

gnocchi-architecture

Roadmap & Ceilometer integration

All of this plan has been exposed and discussed with the Ceilometer team during the last OpenStack summit in Atlanta in May 2014, for the Juno release. I led a session about this entire concept, and convinced the team that using Gnocchi for our metric storage would be a good approach to solve the Ceilometer collector scalability issue.

It was decided to conduct this project experiment in parallel of the current Ceilometer collector for the time being, and see where that would lead the project to.

Early benchmarks

Some engineers from Mirantis did a few benchmarks around Ceilometer and also against an early version of Gnocchi, and Dina Belova presented them to us during the mid-cycle sprint we organized in Paris in early July.

The following graph sums up pretty well the current Ceilometer performance issue. The more you feed it with metrics, the more slow it becomes.

image03

For Gnocchi, while the numbers themselves are not fantastic, what is interesting is that all the graphs below show that the performances are stable without correlation with the number of resources, entities or measures. This proves that, indeed, most of the code is built around a complexity of O(1), and not O(n) anymore.

image05 image04 image06 image01 image00

Next steps

While the Juno cycle is being wrapped-up for most projects, including Ceilometer, Gnocchi development is still ongoing. Fortunately, the composite
architecture of Ceilometer allows a lot of its features to be replaced by some other code dynamically. That, for example, enables Gnocchi to provides a Ceilometer dispatcher plugin for its collector, without having to ship the actual code in Ceilometer itself. That should help the development of Gnocchi to not be slowed down by the release process for now.

The Ceilometer team aims to provide Gnocchi as a sort of technology preview with the Juno release, allowing it to be deployed along and plugged with Ceilometer. We’ll discuss how to integrate it in the project in a more permanent and strong manner probably during the OpenStack Summit for Kilo that will take place next November in Paris.

by Julien Danjou at August 18, 2014 02:50 PM

Opensource.com

Shifting a mindset, why OpenStack is written in Python, and more

Interested in keeping track of what's happening in the open source cloud? Opensource.com is your source for what's happening right now in OpenStack, the open source cloud infrastructure project.

by Jason Baker at August 18, 2014 02:00 PM

Percona

Getting my hands dirty on an OpenStack lab

Like you all may know, OpenStack is currently one of the coolest open source projects, so I was thrilled when I was asked to manage the deployment of an OpenStack lab for internal Percona use. Starting from basically zero, I created tasks in our Jira and assigned them to a pool of volunteer consultants. As usual in a service company, billing is the priority so I ended up losing the 2 senior guys but fortunately most of my time was with a customer that wasn’t very demanding and I could easily multitask with the project and fill the gap. So, here it goes…

Hardware

To deploy the OpenStack lab we were given 8 similar servers in our Durham, N.C. offices. The specs are:

  • CPU: 12 physical cores (24 with HT)
  • Disks: 1 sata 4 TB drive and one 480GB SSD drive
  • Nics: 2x GbE
  • OS: Centos 6

The hardware is recent and decent, a good start.

Deployment choices

Given the hardware we had, I picked the first to be the controller and jumphost, the second to be the network node (a bit overkill) and the remaining 6 nodes would become the compute nodes. I also wanted to use Ceph and RBD with 2 types of volumes, the default using SATA and a SSD type using the SSD drives. The servers only have a single GbE interface to use with Ceph, that’s not ideal but sufficient.

So, we basically followed: OpenStack doc for Centos and had our share of learning and fun. Overall, it went relatively well with only a few hiccups with Neutron and Ceph.

Neutron

Neutron is probably the most difficult part to understand. You need to be somewhat familiar with all the networking tools and protocols to find your way around. Neutron relies on network namespaces, virtual network switches, GRE tunnels, iptables, dnsmasq, etc. For my part, I discovered network namespaces and virtual network switches.

The tasks of providing isolated networks to different tenants with their own set of IPs and firewalling rules, on the same infrastructure, is a challenging tasks. I enjoyed a lot reading Networking in too much detail, from there, things just started to make sense.

A first issue we encountered was that the iproute package on Centos is old and it does not support the network namespaces. It just needs to be replaced by a newer version. It took me some time to understand how things are connected, each tenant has its own Gre tunnel id and vlan. I can only recommend you read the above document and look at your setup. Don’t forget, you have one set of iptables rules per network namespace… The network node is mainly dealing with Natting, SNAT and DNAT while the security group rules are set on the compute nodes. The “ip netns exec …” and the “ovs-ofctl dump-flows …” have been my best friends for debugging.

Once I got things working, I realized the network performance was, to say the least, pretty bad. I switch “gro off” on the Nics but it made very little change. Finally, I found how the MTU of the VMs were too large, an easy fix by adding a configuration file for dnsmasq with “dhcp-option-force=26,1400″. With an MTU of 1400, less than the NIC MTU + the GRE header, packets were no longer split and performance went back to normal.

More on Ceph

The integration of Ceph happened to be more challenging than I first thought. First, let’s say the documentation is not as polished as the Openstack one, there are some rough edges but nothing unmanageable. The latest version, at the time, had no rpms for Centos but that was easy to work around if you know rpmbuild. Same for the Centos RPM for qemu-kvm and nova required a patch to support rbd devices. I succeeded deploying Ceph over the SATA drives, configured Glance and Cinder to use it and that was all good. The patch for nova allows to launch instances on clones of an image. While normally the image has to be copied to the local drive, an operation that can take some time if the image is large, with a clone, barely a few second after you started a VM, it is actually starting. Very impressive, I haven’t tested but I suppose the same can be accomplished with btrfs or ZFS, shared over iscsi.

Since the servers all have a SSD drive, I also tackled the task of setting up a SSD volume type. That has been a bit tricky, you need to setup a rule so that a given storage pool uses the SSD drives. The SSD drives must be placed all in their own branch in the ceph osd tree and then, you need to decompile the current rule set (crush map), modify it by adding a rule that use the ssd drives, recompile and then define a storage pool that uses the ssd rule. Having done this, I modified the cinder configuration for the “volume-ssd” type and finally, I could mount a ssd backed volumes, replicated, to a VM, quite cool.

The only drawback I found using Ceph is when you want to create a snapshot. Ideally, Ceph should handle the snapshot and it should be kept there as is. The way it works is less interesting, a snapshot is created but it is then copied to /tmp of the compute node, uncompressed…, and then copied back to a destination volume. I don’t know why it is done like that, maybe some workaround for limitations of other Cinder backends. The compute nodes have only 30GB available for /tmp and with Ceph, you must use raw images so that’s quite limiting. I already started to look at the code, maybe this could be my first contribution to the Openstack project.

Overall impressions

My first impression is that OpenStack is a big project with many moving, many moving parts. Installing OpenStack is not a beginners project, my experience saved me quite a few times. Logging, when you activate verbose and even debug in the configuration files is abundant and if you take your time, you can figure out what is going on and why something is failing.

Regarding the distribution, I am not sure Centos was the right fit, I think we had a rougher ride than we could have had using Ubuntu. My feeling is that the packages, not necessarily the OpenStack ones, were not as up to date as they should have compared to Ubuntu.

Ceph rbd backend is certainly a good point in my experience, I always wanted to touch Ceph and this project has been a very nice opportunity. The rbd backend works very well and the ability to launch instances almost instantaneously is just awesome. Another great plus is the storage redundancy (2 replica), like EBS, and the storage saving of working only on a clone of the original image. Also, the integration of the SSD backed pool adds a lot of flexibility. There’s only the instance snapshot issue that will need to be worked on.

So, that’s the current status of our OpenStack lab. I hope to be able to add a few more servers that are currently idle, allowing me to replace the over powerful network node and recycle it as a compute node Another thing I would really like to do is to mix the hypervisor types, I’d like to be able to use a lightweight container like LXC or docker. Also, although this is just a lab for the consultants, I’d like to see how to improve its available which is currently quite low. So, more fun to come!

The post Getting my hands dirty on an OpenStack lab appeared first on MySQL Performance Blog.

by Yves Trudeau at August 18, 2014 10:00 AM

Sébastien Han

A must-have cron job on your OpenStack Cloud

Running OpenStack on production can be difficult, so every optimizations are good to take :).

Basically, this script flushes expired tokens and moves deleted instance to other tables. You want to run this script only on one of your cloud controllers.

<figure class="code"><figcaption></figcaption>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash

# 1. Purge expired tokens
# 2. Move deleted instances to another table that you MUST not backup,
# unless you have data retention policies.

MAX_ROWS=100

logger -t keystone-cleaner "Starting token cleanup"
/usr/bin/keystone-manage token_flush
logger -t keystone-cleaner "Ending token cleanup"

logger -t deleted-rows-archiver "Starting archiving deleted rows"
/usr/bin/nova-manage db archive_deleted_rows --max_rows $MAX_ROWS
logger -t deleted-rows-archiver "Ending archiving deleted rows"

exit 0
</figure>

Depending on the amount of deleted instances that you get on a daily basis you’d like to modify the MAX_ROWS variable.


Running these two tasks on a daily basis and ideally prior to running your databases backup is a good start.

August 18, 2014 09:46 AM

Rob Hirschfeld

Your baby is ugly! Picking which code is required for Commercial Core.

babyThere’s no point in sugar-coating this: selecting API and code sections for core requires making hard choices and saying no.  DefCore makes this fair by 1) defining principles for selection, 2) going slooooowly to limit surprises and 3) being transparent in operation.  When you’re telling someone who their baby is not handsome enough you’d better be able to explain why.

The truth is that from DefCore’s perspective, all babies are ugly.  If we are seeking stability and interoperability, then we’re looking for adults not babies or adolescents.

Explaining why is exactly what DefCore does by defining criteria and principles for our decisions.  When we do it right, it also drives a positive feedback loop in the community because the purpose of designated sections is to give clear guidance to commercial contributors where we expect them to be contributing upstream.  By making this code required for Core, we are incenting OpenStack vendors to collaborate on the features and quality of these sections.

This does not lessen the undesignated sections!  Contributions in those areas are vital to innovation; however, they are, by design, more dynamic, specialized or single vendor than the designated areas.

Designated SectionsThe seven principles of designated sections (see my post with TC member Michael Still) as defined by the Technical Committee are:

Should be DESIGNATED:

  1. code provides the project external REST API, or
  2. code is shared and provides common functionality for all options, or
  3. code implements logic that is critical for cross-platform operation

Should NOT be DESIGNATED:

  1. code interfaces to vendor-specific functions, or
  2. project design explicitly intended this section to be replaceable, or
  3. code extends the project external REST API in a new or different way, or
  4. code is being deprecated

While the seven principles inform our choices, DefCore needs some clarifications to ensure we can complete the work in a timely, fair and practical way.  Here are our additions:

8.     UNdesignated by Default

  • Unless code is designated, it is assumed to be undesignated.
  • This aligns with the Apache license.
  • We have a preference for smaller core.

9.      Designated by Consensus

  • If the community cannot reach a consensus about designation then it is considered undesignated.
  • Time to reach consensus will be short: days, not months
  • Except obvious trolling, this prevents endless wrangling.
  • If there’s a difference of opinion then the safe choice is undesignated.

10.      Designated is Guidance

  • Loose descriptions of designated sections are acceptable.
  • The goal is guidance on where we want upstream contributions not a code inspection police state.
  • Guidance will be revised per release as part of the DefCore process.

In my next DefCore post, I’ll review how these 10 principles are applied to the Havana release that is going through community review before Board approval.


by Rob H at August 18, 2014 07:02 AM

Matthew Treinish

QA Program: From Juno into the Future

Based on some of the comments that were posted on the recent “Which Program for Rally” ML thread I feel that there’s been some continued confusion around exactly how all the  projects work together in the QA program. So after discussing it with a wise council of my elders, I decided to start a blog so that I had a place to post more details and I could give a high level overview and clarify how everything works. I’m not really sure how much I’ll be using this blog in the future, as having one is something I’ve resisted for quite some time. But, I felt that making this post warranted me giving in to peer pressure.

Today’s the QA program :

So today in the QA program we have 3 projects, here is a high level over:

  • Tempest: The OpenStack Integrated test suite, it’s concerned with just  with having tests and running them
  • devstack: A documented shell script to build complete OpenStack development environments.
  • grenade: Upgrade testing using 2 versions of devstack to test an offline upgrade. Tempest can optionally be used after each version is deployed to verify the cloud.

Each of these projects is independent and is useful by itself. They have  defined scope (which admittedly gets blurred and constantly evolves) and  when used together along with external tools they can be used in different pipelines for certain goals.

Then there are several external projects which consume and use these for different configurations. Here are just a few examples:

Normal Gating Jobs:

Gate-before

Upgrade Test Gating Jobs:

grenade-before

Cloud burn-in/stress testing:

stress-before

Cloud Monitoring:

cloud-monitoring

These are just a few of the example use cases that I know of, and I’m sure  there are more in the wild. (for example the monitoring one I only found out  about in Atlanta)

My Current Vision of the Future:

Let me preface this with: This is just my personal view of where I want to see things moving long term. (at least for today, I’m sure my mind will change in the future) It’s just something that I’ve been spinning around in the back of  my head. Everything here is hypothetical for the most part, except where implementation has already started.

Where I see the project list moving in the long term is something along the lines of:

  • Tempest Library
  • Tempest
  • Devstack
  • Grenade
  • Subunit2sql
  • Javelin2
  • Stress test tool
  • Analysis/post-processing utility,

The increased number of utilities, and libraries allow for more diverse configurations, and combinations besides allowing us to have more targetted development for each repository. Which should hopefully lead to a faster  path to contribution for everyone by making everything simpler.

A brief overview of what all the new projects are:

  • tempest library: This is my current big TODO in the next few weeks, the spec for this can be found here
  • subunit2sql: this already actually exists  but the basic intent is to provide a utility and library for the storage of subunit streams. One of my TODOs here is to also integrate it into testrepository so it can be used directly to store testr results in a SQL DB.
  • javelin2: This will be an externalized version of this
  • Stress test tool: This is will be an externalization of the current stress test framework here and it’s entry point
  • Analysis/Post-processing utility: This is something that is used for both visualizing performance data from a subunit stream as well as providing some analysis and provide useful insight from larger sources of test output. (like a subunit2sql DB)

This leaves tempest to really be just the tests. The common functionality, like  the unified REST client, are part of the library which allows reuse by other  external tools. This allows people to use all the work we’ve been doing in tempest around building a black box integrated test suite to build their own functional test suites (which was a motivation for starting the library now) or whatever other purpose they have. As part of breaking the functionality out  we make a number of small utilities that are currently in the tempest repo  that were only there because they were tied to the common code in  tempest. Part of the current issue we’re seeing with tempest is because a  large portion of the common functionality in the tree is not really modular  we have things in tree which aren’t necessarily related to integrated testing.  (ie, javelin2, run_stress) Also, using tempest as a base for testing in a different  repo is very difficult. I have seen a couple cases where other projects want to spin up a functional test suite similar to tempest with the hopes of converging down the road. But, this is very difficult to accomplish today because nothing in tempest is really modular or provides a stable interface.

Also, having a leaner tempest should make it a bit easier to concentrate on some of the usability gaps which have been pointed out in this thread, and things that were already on our radar which we’ve been discussing. Improving the output and results view are definitely big ones on this list. Another possibility might be creating a tempest CLI instead of just calling tox or run_tempest.sh. Which is something that we’ve discussed in the past.

Another key area were I expect to see improvements is the instrumenation of tempest. Right when it comes to metrics around performance of anything that tempest is doing we have only rough idea of the execution time. For example, through the subunit stream we know how much time a test case takes to execute, but the setUpClass is not captured in the subunit data. Or for API calls we meaure the time it takes for the tempest rest_client’s request method to finish executing. But, this doesn’t necessarily give a complete picture, especially around async requests. To use tempest as a load generator for non-functional testing fixing these issues becomes important. I expect after libification of parts of tempest, primarily the rest client, making these improvements will become simpler and be prioritized.

With this new constellation of smaller more logically split projects we have additional modularity and each component is useful on its own. In conjunction they can be used to build even more customized pipelines. It’s really just an expansion of the model we currently have, but making the  logical splits a bit more clearly. I also expect that additional projects will be created and added as we move forward. Which is a major advantage with  using the Unix philosophy as that things can grow organically and build off  of each other.

Here are some example configurations:

Gating:

gate-after

Upgrade Gating:

grenade-after
Then from the gating runs we can use the data stored in the database in conjunction with the analysis tool to visualize long term trends in  performance data. As well as use it to track general testing trends over time. Having the data in a DB also allows to build tooling to query it to find other useful trends or information. Which is something we’re lacking today.

Cloud Burn-in/Stress testing:

stress-after

Cloud Monitoring:
cloud-monitoring

So, this one is basically the same, but I want to point out that in conjunction with some of the other tools this could be a bit more custom. Like using subunit2sql as I outlined for the gating examples to track trends over the lifetime of the cloud.

Local Benchmarking:

benchmark-after
The real advantage here is the flexibility this approach allows. These configurations can either use or not use any of the other pieces to build a custom pipeline for testing depending on the requirements.

I feel that we’re not all that far off from something like this. We’re starting the first steps working towards this now. But, I do think it will probably be at least 2 cycles before we have all the pieces as I outlined here.

by mtreinish at August 18, 2014 04:25 AM

Michael Still

Juno nova mid-cycle meetup summary: scheduler

This post is in a series covering the discussions at the Juno Nova mid-cycle meetup. This post will cover the current state of play of our scheduler refactoring efforts. The scheduler refactor has been running for a fair while now, dating back to at least the Hong Kong summit (so about 1.5 release cycles ago).

The original intent of the scheduler sub-team's effort was to pull the scheduling code out of Nova so that it could be rapidly iterated on its own, with the eventual goal being to support a single scheduler across the various OpenStack services. For example, the scheduler that makes placement decisions about your instances could also be making decisions about the placement of your storage resources and could therefore ensure that they are co-located as much as possible.

During this process we realized that a big bang replacement is actually much harder than we thought, and the plan has morphed into being a multi-phase effort. The first step is to make the interface for the scheduler more clearly defined inside the Nova code base. For example, in previous releases, it was the scheduler that launched instances: the API would ask the scheduler to find available hypervisor nodes, and then the scheduler would instruct those nodes to boot the instances. We need to refactor this so that the scheduler picks a set of nodes, but then the API is the one which actually does the instance launch. That way, when the scheduler does move out it's not trusted to perform actions that change hypervisor state, and the Nova code does that for it. This refactoring work is under way, along with work to isolate the SQL database accesses inside the scheduler.

I would like to set expectations that this work is what will land in Juno. It has little visible impact for users, but positions us to better solve these problems in Kilo.

We discussed the need to ensure that any new scheduler is at least as fast and accurate as the current one. Jay Pipes has volunteered to work with the scheduler sub-team to build a testing framework to validate this work. Jay also has some concerns about the resource tracker work that is being done at the moment that he is going to discuss with the scheduler sub-team. Since the mid-cycle meetup there has been a thread on the openstack-dev mailing list about similar resource tracker concerns (here), which might be of interest to people interested in scheduler work.

We also need to test our assumption at some point that other OpenStack services such as Neutron and Cinder would be even willing to share a scheduler service if a central one was implemented. We believe that Neutron is interested, but we shouldn't be surprising our fellow OpenStack projects by just appearing with a complete solution. There is a plan to propose a cross-project session at the Paris summit to cover this work.

In the next post in this series we'll discuss possibly the most controversial part of the mid-cycle meetup. The proposal for "slots" for landing blueprints during Kilo.

Tags for this post: openstack juno nova mid-cycle summary scheduler
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 18, 2014 04:06 AM

Juno nova mid-cycle meetup summary: bug management

Welcome to the next exciting installment of the Nova Juno mid-cycle meetup summary. In the previous chapter, our hero battled a partially complete cells implementation, by using his +2 smile of good intentions. In this next exciting chapter, watch him battle our seemingly never ending pile of bugs! Sorry, now that I'm on to my sixth post in this series I feel like it's time to get more adventurous in the introductions.

For at least the last cycle, and probably longer, Nova has been struggling with the number of bugs filed in Launchpad. I don't think the problem is that Nova has terrible code, it is instead that we have a lot of users filing bugs, and the team working on triaging and closing bugs is small. The complexity of the deployment options with Nova make this problem worse, and that complexity increases as we allow new drivers for things like different storage engines to land in the code base.

The increasing number of permutations possible with Nova configurations is a problem for our CI systems as well, as we don't cover all of these options and this sometimes leads us to discover that they don't work as expected in the field. CI is a tangent from the main intent of this post though, so I will reserve further discussion of our CI system until a later post.

Tracy Jones and Joe Gordon have been doing good work in this cycle trying to get a grip on the state of the bugs filed against Nova. For example, a very large number of bugs (hundreds) were for problems we'd fixed, but where the bug bot had failed to close the bug when the fix merged. Many other bugs were waiting for feedback from users, but had been waiting for longer than six months. In both those cases the response was to close the bug, with the understanding that the user can always reopen it if they come back to talk to us again. Doing "quick hit" things like this has reduced our open bug count to about one thousand bugs. You can see a dashboard that Tracy has produced that shows the state of our bugs at http://54.201.139.117/nova-bugs.html. I believe that Joe has been moving towards moving this onto OpenStack hosted infrastructure, but this hasn't happened yet.

At the mid-cycle meetup, the goal of the conversation was to try and find other ways to get our bug queue further under control. Some of the suggestions were largely mechanical, like tightening up our definitions of the confirmed (we agree this is a bug) and triaged (and we know how to fix it) bug states. Others were things like auto-abandoning bugs which are marked incomplete for more than 60 days without a reply from the person who filed the bug, or unassigning bugs when the review that proposed a fix is abandoned in Gerrit.

Unfortunately, we have more ideas for how to automate dealing with bugs than we have people writing automation. If there's someone out there who wants to have a big impact on Nova, but isn't sure where to get started, helping us out with this automation would be a super helpful way to get started. Let Tracy or I know if you're interested.

We also talked about having more targeted bug days. This was prompted by our last bug day being largely unsuccessful. Instead we're proposing that the next bug day have a really well defined theme, such as moving things from the "undecided" to the "confirmed" state, or similar. I believe the current plan is to run a bug day like this after J-3 when we're winding down from feature development and starting to focus on stabilization.

Finally, I would encourage people fixing bugs in Nova to do a quick search for duplicate bugs when they are closing a bug. I wouldn't be at all surprised to discover that there are many bugs where you can close duplicates at the same time with minimal effort.

In the next post I'll cover our discussions of the state of the current scheduler work in Nova.

Tags for this post: openstack juno nova mi-cycle summary bugs
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Michael's surprisingly unreliable predictions for the Havana Nova release; Juno nova mid-cycle meetup summary: DB2 support

Comment

August 18, 2014 03:38 AM

Aptira

Creating a Windows Server 2012 QCOW2 image (OpenStack/KVM)

Things are busy here in Aptiraville. One thing we have been working on recently is upgrading our business intelligence platform, ADAPT. We originally ran this platform for our customer, Mercurial, on our VMware backed OpenStack region but as part of the upgrade we wanted to increase its capability to run our own KVM backed region as well.

So I tried to convert the existing VMDK to a QCOW2 to file with:

qemu-img convert -f vmdk ws2012.vmdk -O qcow2 ws2012.qcow2

and a bunch of variations of that. Unfortunately, no matter what I did, WS2012 just wouldn't work correctly! After wasting a very frustrating day, I decided to rebuild an identical image from scratch in QCOW2 format to bypass the issue completely. To my further frustration, the available documentation on the internet for doing so is actually quite sparse and in some cases nonexistent. I thought it would be useful to write up exactly what I did this time as a blogpost, so that others can hopefully use it as a resource and avoid the pain points that I had to go through (since Windows doesn't use text based configuration files, making changes means booting the image, making the changes and reuploading, rather than simply mounting and modifying the image).

First thing you will need is a computer capable of running KVM, with VT extensions enabled in the BIOS. My advice is to use a CentOS 6 machine, as I found that images I created on newer distributions like Ubuntu 14.04 or CentOS 7 would not boot on RHEL/CentOS 6 machines due to changes in the version of QCOW2 used. Creating the image on CentOS 6 means that the image you create will be bootable everywhere.

Along with this, you will need to have a Windows Server 2012 ISO accessible on the computer and product key (Aptira is a Microsoft SPLA partner), and and the Fedora signed VirtIO drivers. Make sure you install virt-installer and virt-manager:

yum -y install virt-install virt-manager

Alternatively, if you don't want to install the GUI virt-manager on to the computer, you can install that locally and connect to the computer over SSH with virt-manager.

Make sure you have enabled IP forwarding in sysctl so that the VM will be able to access the networks when it comes online (if you want to copy scripts or other files onto it for example):

echo 1 > /proc/sys/net/ipv4/ip_forward

Create a preallocated QCOW2 image in /tmp (I tried making a thin provisioned one and the Windows installer thought the disk size was zero) and set permissions so libvirt can access it:

cd /tmp
qemu-img create -f qcow2 -o preallocation=full ws2012.qcow2 15g
chmod o+wx ws2012.qcow2

The 15GB number chosen above as the disk size is arbitrary, the actual install only takes up just under 7GB so if you want to keep the image as small as possible then something close to that would be optimal.

Run virt-install to boot the VM with the WS2012 installer ISO and VirtIO driver ISO attached attached, the disk set as a VirtIO disk and NIC attached to the default created virbr0. Unfortunately the most recent "os-variant" available for MS Windows on the CentOS 6 version of virt-install is "win7" so that is what I chose here for better or worse:

virt-install --connect qemu:///system --arch=x86_64 -n ws2012 -r 2048 --vcpus=2 --disk path=/tmp/ws2012.qcow2,device=disk,bus=virtio,size=15 -c /mnt/Source/en_windows_server_2012_x64_dvd_915478.iso --vnc --noautoconsole --os-type windows --os-variant win7 --network=bridge:virbr0 --disk path=/mnt/Source/en_windows_server_2012_x64_dvd_915478.iso,device=cdrom,perms=ro -c /mnt/Source/virtio-win-0.1-81.iso

Once the virt-install command has started, you can view the GUI by either firing up virt-manager locally, remotely (and connect to the computer running virt-install via SSH) or simply connect directly to the VNC port running at 127.0.0.1:5900 (assuming no other KVM virtual machines are running on the same host).

Complete the Windows Server 2012 installation as you normally would. We prefer the headless mode. During the installation Windows will complain that it can't find a disk to install to, and present a dialog to load drivers. The drivers should be in:

D:\WIN8\AMD64

and you are looking to install the VirtIO SCSI device driver (we will install the remaining VirtIO drivers after the operating system installation is complete and the virtual machine is running).
Once the installation is completed, the machine will restart and you can set an administrator password and login. The first step after successful installation is to run a command prompt (if using headless mode one will be presented as soon as you login) and use pnputil to insert the remaining VirtIO drivers:

pnputil -i -a D:\WIN8\AMD64\*.INF

We also want to enable .NET 2.0 and .NET 3.0/3.5 frameworks (E:\SOURCES\SXS is a directory on the WS2012 installer ISO):

DISM /Online /Enable-Feature /FeatureName:NetFx3 /All /LimitAccess /Source:E:\SOURCES\SXS

At this point, we want to copy over the installer for the latest .NET 4.5, any patches, scripts, etc as well as Cloudbase-Init (we recommend the automated Jenkins gated build) so we will create a directory for these files to go, map a CIFS network share, copy the files and then delete the share when we have completed the copy:

net use x: \\APTIRAFILESERVER01\
mkdir c:\source
copy x:\OpenStack\ADAPT\setup\* c:\source\
net use x: /DELETE

and then install Cloudbase-Init. If you are not familiar with this tool, it's a port of the popular Ubuntu Linux tool that configures cloud virtual machines when they boot up based on a variety of configuration sources (we use ConfigDrive but also supported is the EC2 style metadata server, MaaS and others):

c:\source\CloudbaseInitSetup_Beta_x64.msi

During the installation process you should set the user that Cloudbase-Init will manage from the default "Admin" to "Administrator", unless you have (or plan to) create a special user for cloud-init purposes.

On the last dialog of the Cloudbase-Init installation you should select the option to run Sysprep and "generify" the virtual machine image. Don't select the option to shutdown the virtual machine, as we still have one thing left to do.

After the Sysprep has finished running, to ensure that any scripts the user runs on boot through the OpenStack nova --user-data flag (referred to as "Post-Creation" in the OpenStack dashboard) we will need to set the PowerShell execution policy to be unrestricted:

powershell
Set-ExecutionPolicy Unrestricted

During testing we noticed that unfortunately line-breaks don't translate correctly when using the "Post-Creation" textbox in the "Launch Instance" dialog in OpenStack dashboard and ConfigDrive (works fine with metadata servers) so that a Post-Creation entry of:

#ps1_sysnative
net user Administrator test123!

will appear in the user_data file as

#ps1_sysnativenet user Administrator test123!

so we recommend booting instances from the commandline python-novaclient with the --user-data=userdata.txt flag set when running "nova boot", where userdata.txt is a file written with an editor like Vi that supports writing files in DOS format (:set ff=dos).

Once this is completed you can shut down the virtual machine immediately with:

shutdown -s -t 0

Now your image is ready to upload to the closest glance server for a test run (exciting!):

glance image-create --name WS2012 --disk-format=raw --container-format=bare --is-public --progress --file /tmp/ws2012.qcow2

Now you can enjoy Windows Server 2012 as a first class instance citizen on your KVM based OpenStack cloud like Aptira does!

by Sina Sadeghi (sina@aptira.com) at August 18, 2014 03:26 AM

August 17, 2014

Mirantis

Using OpenStack Database (Trove): Replication and Clustering; Looking ahead

tesora2[Editor's Note: This post is part 3 in a 3 part series about using Trove. It follows part 1 and part 2.]

In previous blog posts we described the replication feature for Trove, and the implementation in the Client and the Task Manager in detail.

In this post we describe some of the rationale for this implementation and the roadmap for features that provide performance and availability guarantees that are so critical for a database.

It is well recognized that solutions to problems of database performance and availability are closely interconnected. All of these solutions rely on the ability to have multiple copies of data, and to have multiple computers (that may or may not be at the same location) working together to provide the user with access to data. These kinds of solutions broadly depend on techniques like “replication”, “clustering” and “parallel computing”. While these three terms are often used interchangeably, there are subtle differences that are worth highlighting.

OpenStack Trove Day: A Deep Dive into DBaaS is a full-day dedicated to sharing the latest development on OpenStack Database as a Service. Find out why leading enterprises are adopting OpenStack and Trove. August 19th in Cambridge, MA.

Replication is a mechanism whereby data is available in more than one location. In the simplest case of replication, there is a master and a slave. The master and slave are first synchronized and after that, any change to the master is replicated to the slave. The changes may be replicated synchronously or asynchronously. Replication may either guarantee an ordering of change replication or not and in some cases change ordering may only be guaranteed at specific thresholds called barriers. Databases provide replication natively (semi-synchronous replication in MySQL) or one could use additional software (such as Galera or Tungsten), but one could also consider disk mirroring either in hardware or software as forms of replication.

Clustering is a class of techniques where many computers work collectively, and perform some function or functions. For example, if data is replicated between two locations then it is possible for a database to access each data set and answer queries submitted to it. In such a system, one could have a copy of data in one location and accessed by one database instance and data in a either the same or a different location accessed by a second database instance. Then these two database instances would be considered to be a database cluster. In this example, each database instance is able to completely answer queries against the data. Oracle Real Application Clusters (RAC) is an example of a system of this kind. An application that splits writes and reads between a database master and replication slaves is another example of this kind of system.

Parallel computing is a particular example of cluster computing where multiple computers work together to provide some function or functions whereby multiple computers each perform some sub-function. In the context of databases, a common example of parallel computing is a parallel database. Here multiple computers each process a subset of a query based on a subset of the data that they have access to. Sharded databases, or most NoSQL databases (MongoDB, Cassandra for example) are examples of this kind of system.

“Clustering” means something different to different data stores.  Since Trove is designed to support this diversity, its implementation of these features must use some common, basic operations to control these data stores in a consistent way. These operations are “to add a replication slave to a data store”, “to add a computer to a cluster”, “to remove a computer from a cluster”, and so on. The actual implementation of those features could then be performed in a manner that was applicable to that particular data store.

Therefore, in replication (v1) we create the framework for this feature and provide an implementation of this framework for MySQL. This implementation includes the feature set related to replicas for MySQL. For this implementation, we use the snapshot and replicate paradigm that is the standard for MySQL systems.

In a subsequent phase the community will deliver clustering which will implement primitives like “create cluster”, “add instance to cluster”, “replace instance” and “remove instance”. This scheme therefore makes it possible for us to also add extensions to these API calls that will define specialized instances for specific data stores such as the “create arbiter” call that will make sense only to MongoDB, which uses the mechanism of an arbiter.

The initial implementations of the replication and clustering feature are currently planned for release in the Juno release. Code for both of these features is under development and expected to be delivered in the juno-2 code drop. Blueprints for these features are currently available on Launchpad.

With the delivery of these features, the Trove project will have a solid framework on which one can implement replication and clustering for an arbitrary data store. We anticipate that with this framework, we will also be able to deliver the capability of a fully parallel MySQL database utilizing the clustering feature for parallelism and replication for redundancy. For this we plan to use the open source Database Virtualization Engine (DVE) capability that we have developed here at Tesora.

In parallel, we at Tesora, and other members of the Trove project community (from companies like Mirantis, HP, Rackspace and EBay) are contributing code that will support additional data stores and additional configuration capabilities in those data stores.

While Trove is already in production in a number of enterprises and public clouds, having these capabilities built in will make it dramatically easier to deliver a production-ready, enterprise-grade Database as a Service offering based on Trove in an OpenStack cloud.

The post Using OpenStack Database (Trove): Replication and Clustering; Looking ahead appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Amrith Kumar at August 17, 2014 09:15 PM

August 16, 2014

Assaf Muller

Layer 3 High Availability

L3 Agent Low Availability

Today, you can utilize multiple network nodes to achieve load sharing, but not high availability or redundancy. Assuming three network nodes, creation of new routers will be scheduled and distributed amongst those three nodes. However, if a node drops, all routers on that node will cease to exist as well as any traffic normally forwarded by those routers. Neutron, in the Icehouse release, doesn’t support any built-in solution.

A Detour to the DHCP Agent

DHCP agents are a different beast altogether – The DHCP protocol allows for the co-existence of multiple DHCP servers all serving the same pool, at the same time.

By changing:

neutron.conf:
dhcp_agents_per_network = X

You will change the DHCP scheduler to schedule X DHCP agents per network. So, for a deployment with 3 network nodes, and setting dhcp_agents_per_network to 2, every Neutron network will be served by 2 DHCP agents out of 3. How does this work?

dhcp_ha_topology

First, let’s take a look at the story from a baremetal perspective, outside of the cloud world. When the workstation is connected to a subnet in the 10.0.0.0/24 subnet, it broadcasts a DHCP discover. Both DHCP servers dnsmasq1 and dnsmasq2 (Or other implementations of a DHCP server) receive the broadcast and respond with an offer for 10.0.0.2. Assuming that the first server’s response was received by the workstation first, it will then broadcast a request for 10.0.0.2, and specify the IP address of dnsmasq1 – 10.0.0.253. Both servers receive the broadcast, but only dnsmasq1 responds with an ACK. Since all DHCP communication is via broadcasts, server 2 also receives the ACK, and can mark 10.0.0.2 as taken by AA:BB:CC:11:22:33, so as to not offer it to other workstations. To summarize, all communication between clients and servers is done via broadcasts and thus the state (What IPs are used at any given time, and by who) can be distributed across the servers correctly.

In the Neutron case, the assignment from MAC to IP is configured on each dnsmasq server beforehand, when the Neutron port is created. Thus, both dnsmasq leases file will hold the AA:BB:CC:11:22:33 to 10.0.0.2 mapping before the DHCP request is even broadcast. As you can see, DHCP HA is supported at the protocol level.

Back to the Lowly Available L3 Agent

L3 agents don’t (Currently) have any of these fancy tricks that DHCP offers, and yet the people demand high availability. So what are the people doing?

  • Pacemaker / Corosync – Use external clustering technologies to specify a standby network node for an active one. The standby node will essentially sit there looking pretty, and when a failure is detected with the active node, the L3 agent will be started on the standby node. The two nodes are configured with the same hostname so that when the secondary agent goes live and synchronizes with the server, it identifies itself with the same ID and thus manages the same routers.
  • Another type of solution writes a script that runs as a cron job. This would be a Python SDK script that would use the API to get a list of dead agents, get all the routers on that agent, and reschedule them to other agents.
  • In the Juno time frame, look for this patch https://review.openstack.org/#/c/110893/ by Kevin Benton to bake rescheduling into Neutron itself.

Rescheduling Routers Takes a Long, Long Time

All solutions listed suffer from a substantial failover time, if only for the simple fact that configuring a non-trivial amount of routers on the new node(s) takes quite a while. Thousands of routers take hours to finish the rescheduling and configuration process. The people demand fast failover!

Distributed Virtual Router

DVR has multiple documents explaining how it works:

The gist is that it moves routing to the compute nodes, rendering the L3 agent on the network nodes pointless. Or does it?

  • DVR handles only floating IPs, leaving SNAT to the L3 agents on the network nodes
  • Doesn’t work with VLANs, only works with tunnels and L2pop enabled
  • Requires external connectivity on every compute node
  • Generally speaking, is a significant departure from Havana or Icehouse Neutron based clouds, while L3 HA is a simpler change to make for your deployment

Ideally you would use DVR together with L3 HA. Floating IP traffic would be routed directly by your compute nodes, while SNAT traffic would go through the HA L3 agents on the network nodes.

Layer 3 High Availability

The Juno targeted L3 HA solution uses the popular Linux keepalived tool, which uses VRRP internally. First, then, let’s discuss VRRP.

What is VRRP, how does it work in the physical world?

Virtual Router Redundancy Protocol is a first hop redundancy protocol – It aims to provide high availability of the network’s default gateway, or the next hop of a route. What problem does it solve? In a network topology with two routers providing internet connectivity, you could assign half of the network’s default gateway to the first router’s IP address, and the other half to the second router.

router_ha_topology_before_vrrp

This would provide load sharing, but what happens if one router loses connectivity? Herein comes the idea of a virtual IP address, or a floating address, which will be configured as the network’s default gateway. During a failover, the standby routers won’t receive VRRP hello messages from the master and will thus perform an election process, with the winning router acting as the active gateway, and the others remain as standby. The active router configures the virtual IP address (Or VIP for short), on its internal, LAN facing interface, and responds to ARP requests with a virtual MAC address. The network computers already have entries in their ARP caches (For the VIP + virtual MAC address) and have no reason to resend an ARP request. Following the election process, the virtuous standby router becomes the new active instance, and sends a gratuitous ARP request – Proclaiming to the network that the VIP + MAC pair now belong to it. The switches comprising the network move the virtual MAC address from the old port to the new.

switch_moves_mac

 

By doing so, traffic to the default gateway will reach the correct (New) active router. Note that this approach does not accomplish load sharing, in the sense that all traffic is forwarded through the active router. (Note that in the Neutron use case, load sharing is not accomplished at the individual router level, but at the node level, assuming a non-trivial amount of routers). How does one accomplish load sharing at the router resolution? VRRP groups: The VRRP header includes a Virtual Router Identifier, or VRID. Half of the network hosts will configure the first VIP, and the other half the second. In the case of a failure, the VIP previously found on the failing router will transfer to another one.

router_ha_two_vrrp_groups

The observant reader will have identified a problem – What if the active router loses connectivity to the internet? Will it remain as the active router, unable to route packets? VRRP adds the capability to monitor the external link and relinquish its role as the active router in case of a failure.router_ha_external_trap

Note: As far as IP addressing goes, it’s possible to operate in two modes:

  1. Each router gets an IP address, regardless of its VRRP state. The master router is configured with the VIP as an additional  or secondary address.
  2. Only the VIP is configured. IE: The master router will hold the VIP while the slaves will have no IPs configured whatsoever.

VRRP – The Dry Facts

  • Encapsulated directly in the IP protocol
  • Active instance uses multicast address 224.0.0.18, MAC 01-00-5E-00-00-12 when sending hello messages to its standby routers
  • The virtual MAC address is of the form: 00-00-5E-00-01-{VRID}, thus only 256 different VRIDs (0 to 255) can exist in a single broadcast domain
  • The election process uses a user configurable priority, from 1 to 255, the higher the better
  • Preemptive elections, like in other network protocols, means that if a standby is configured with a higher priority, or comes back after losing its connectivity (And previously acting as the active instance) it will resume its role as the active router
  • Non-preemptive elections mean that when an active router loses its connectivity and comes back up, it will remain in a standby role
  • The hello internal is configurable (Say: Every T seconds), and standby routers perform an election process if they haven’t received a hello message from the master after 3T seconds

Back to Neutron-land

L3 HA starts a keepalived instance in every router namespace. The different router instances talk to one another via a dedicated HA network, one per tenant. This network is created under the blank tenant to hide it from the CLI and GUI. The HA network is a  Neutron tenant network, same as every other network, and uses the default segmentation technology. HA routers have an ‘HA’ device in their namespace: When a HA router is created, it is scheduled to a number of network nodes, along with a port per network node, belonging to the tenant’s HA network. keepalived traffic is forwarded through the HA device (As specified in the keepalived.conf file used by the keepalived instance in the router namespace). Here’s the output of ‘ip address’ in the router namespace:

[stack@vpn-6-88 ~]$ sudo ip netns exec qrouter-b30064f9-414e-4c98-ab42-646197c74020 ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    ...
2794: ha-45249562-ec: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 12:34:56:78:2b:5d brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.2/24 brd 169.254.0.255 scope global ha-54b92d86-4f
       valid_lft forever preferred_lft forever
    inet6 fe80::1034:56ff:fe78:2b5d/64 scope link 
       valid_lft forever preferred_lft forever
2795: qr-dc9d93c6-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether ca:fe:de:ad:be:ef brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global qr-0d51eced-0f
       valid_lft forever preferred_lft forever
    inet6 fe80::c8fe:deff:fead:beef/64 scope link 
       valid_lft forever preferred_lft forever
2796: qg-843de7e6-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether ca:fe:de:ad:be:ef brd ff:ff:ff:ff:ff:ff
    inet 19.4.4.4/24 scope global qg-75688938-8d
       valid_lft forever preferred_lft forever
    inet6 fe80::c8fe:deff:fead:beef/64 scope link 
       valid_lft forever preferred_lft forever

That is the output for the master instance. The same router on another node would have no IP address on the ha, hr, or qg devices. It would have no floating IPs or routing entries. These are persisted as configuration values in keepalived.conf, and when keepalived detects the master instance failing, these addresses (Or: VIPs) are configured by keepalived on the appropriate devices. Here’s an example of keepalived.conf, for the same router shown above:

vrrp_sync_group VG_1 {
    group {
        VR_1
    }
    notify_backup "/path/to/notify_backup.sh"
    notify_master "/path/to/notify_master.sh"
    notify_fault "/path/to/notify_fault.sh"
}
vrrp_instance VR_1 {
    state BACKUP
    interface ha-45249562-ec
    virtual_router_id 1
    priority 50
    nopreempt
    advert_int 2
    track_interface {
        ha-45249562-ec
    }
    virtual_ipaddress {
        19.4.4.4/24 dev qg-843de7e6-8f
    }
    virtual_ipaddress_excluded {
        10.0.0.1/24 dev qr-dc9d93c6-e2
    }
    virtual_routes {
        0.0.0.0/0 via 19.4.4.1 dev qg-843de7e6-8f
    }
}

What are those notify scripts? These are scripts that keepalived executes upon transition to master, backup, or fault. Here’s the contents of the master script:

#!/usr/bin/env bash
neutron-ns-metadata-proxy --pid_file=/tmp/tmpp_6Lcx/tmpllLzNs/external/pids/b30064f9-414e-4c98-ab42-646197c74020/pid --metadata_proxy_socket=/tmp/tmpp_6Lcx/tmpllLzNs/metadata_proxy --router_id=b30064f9-414e-4c98-ab42-646197c74020 --state_path=/opt/openstack/neutron --metadata_port=9697 --debug --verbose
echo -n master > /tmp/tmpp_6Lcx/tmpllLzNs/ha_confs/b30064f9-414e-4c98-ab42-646197c74020/state

The master script simply opens up the metadata proxy, and writes the state to a state file, which can be later read by the L3 agent. The backup and fault scripts kill the proxy and write their respective states to the aforementioned state file. This means that the metadata proxy will be live only on the master router instance.

* Aren’t We Forgetting the Metadata Agent?

Simply enable the agent on every network node and you’re good to go.

Future Work & Limitations

  • TCP connection tracking – With the current implementation, TCP sessions are broken on failover. The idea is to use conntrackd in order to replicate the session states across HA routers, so that when the failover finishes, TCP sessions will continue where they left off.
  • Where is the master instance hosted? As it is now it is impossible for the admin to know which network node is hosting the master instance of a HA router. The plan is for the agents to report this information and for the server to expose it via the API.
  • Evacuating an agent – Ideally bringing down a node for maintenance should cause all of the HA router instances on said node to relinquish their master states, speeding up the failover process.
  • Notifying L2pop of VIP movements – Consider the IP/MAC of the router on a tenant network. Only the master instance will actually have the IP configured, but the same Neutron port and same MAC will show up on all participating network nodes. This might have adverse effects on the L2pop mechanism driver, as it expects a MAC address in a single location in the network. The plan to solve this deficiency is to send an RPC message from the agent whenever it detects a VRRP state change, so that when a router becomes the master, the controller is notified, which can then update the L2pop state.
  • FW, VPN and LB as a service integration. Both DVR and L3 HA have issues integrating with the advanced services, and a more serious look will be taken during the Kilo cycle.
  • One HA network per tenant. This implies a limit of 255 HA routers per tenant, as each router takes up a VRID, and the VRRP protocol allows 255 distinct VRID values in a single broadcast domain.

Usage & Configuration

neutron.conf:
l3_ha = True
max_l3_agents_per_router = 2
min_l3_agents_per_router = 2
  • l3_ha = True means that all router creations will default to HA (And not legacy) routers. This is turned off by default.
  • You can set the max to a number between min and the number of network nodes in your deployment. If you deploy 4 net nodes but set max to 2, only two l3 agents will be used per HA router (One master, one slave).
  • min is used as a sanity check: If you have two network nodes and one goes out momentarily, any new routers created during that time period will fail as you need at least <min> L3 agents up when creating a HA router.

l3_ha controls the default, while the CLI allows an admin (And only admins) to override that setting on a per router basis:

neutron router-create --ha=<True | False> router1

References


by assafmuller at August 16, 2014 06:30 PM

August 15, 2014

OpenStack Blog

OpenStack Community Weekly Newsletter (Aug 8 – 15)

Patchwork Onion delivers stability & innovation: the graphics that explains how we determine OpenStack Core

The OpenStack board, through the DefCore committee, has been working to define “core” for commercial users using a combination of minimum required capabilities (APIs) and code (Designated Sections).  These minimums are decided on a per project basis so it can be difficult to visualize the impact on the overall effect on the Integrated Release. Rob Hirschfeld and Joshua McKenty have created the patchwork onion graphic to help illustrate how core relates to the integrated release.

OpenStack Upstream Training in Paris

We’re doing it again, bigger: the OpenStack Foundation is delivering a training program to accelerate the speed at which new OpenStack developers are successful at integrating their own roadmap into that of the OpenStack project. If you’re a new OpenStack contributor or plan on becoming one soon, you should sign up for the next OpenStack Upstream Training in Paris, November 1-2. Participation is strongly advised also for first time participants to OpenStack Design Summit. We’re doing it again before the Paris Summit, as we did in Atlanta, only bigger.

Using gerrymander, a client API and command line tool for gerrit

Over the past couple of months Daniel P. Berrange has worked on creating a python library and command line tool for dealing with gerrit. Gerrymander was born out of his frustration with the inefficiency of handling reviews through the gerrit web UI. Reviewers can use gerrymander to query gerrit for reviews and quickly get a list of comments filtering out comments by bots.

The Road To Paris 2014 – Deadlines and Resources

During the Paris Summit there will be a working session for the Women of OpenStack to frame up more defined goals and line out a blueprint for the group moving forward. We encourage all women in the community to complete this very short surveyto provide input for the group.

Reports from Previous Events

Tips ‘n Tricks

Security Advisories and Notices

Upcoming Events

Relevant Conversations

Other News

Got Answers?

Ask OpenStack is the go-to destination for OpenStack users. Interesting questions waiting for answers:

Welcome New Reviewers and Developers and Core Reviewers

Louis Taylor wudx05
Puneet Arora Miguel Grinberg
Sandro Tosi Piet Delaney
Sam Betts Ilia Meerovich
Philippe Jeurissen Chirag Shahani
Amit Gandhi venkata anil
Takashi Sogabe Takashi Sogabe
Veena Saro Chandra Bhooshan
Juergen Brendel selvakumar
Jin Dong sandhya
Qijing Li Tarun Jain
Mathieu Losmede
Lakshminarayanan Krishnan

Latest Activity In Projects

Do you want to see at a glance the bugs filed and solved this week? Latest patches submitted for review? Check out the individual project pages on OpenStack Activity Board – Insights.

OpenStack Reactions

ghana

Found the bad commit that messed up the gate

The weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

by Stefano Maffulli at August 15, 2014 10:10 PM

OpenStack @ NetApp

Manila: How to get it running in DevStack

Manila: How to get it running in DevStack

In our previous blog (Overview of Manila at Atlanta OpenStack Summit), Bob Callaway spoke about the Atlanta summit and the new OpenStack File Share Service project called Manila. This is a follow up with some information that will assist you getting Manila installed within a DevStack environment. We will show you some of the steps and links that are available that will provide information about the installation, and get you started with Manila - think of this as a Manila 101 :)

The test infrastructure being used is a VM with the following details:

  • Ubuntu 14.04 LTS as the OS
  • Three interfaces:
    • Management Interface
    • Interface used for the Bridge Network
    • Interface for my Storage Network
  • Devstack (the master branch)
  • Manila (the master branch) from Stackforge

Important Point about the Host OS version:

  • Ubuntu 14.04 was required due to blueprints that provided upstream changes in Neutron that required a minimum version of dnsmasq to be 2.63. Ubuntu 12.04 (Precise) has version 2.59.

Now, let’s walk thru the steps to get devstack and Manila setup on your host (almost all of the steps will be via the command line, with one confirmation of viewing shares via Horizon):

1. Clone the master branch of devstack:

stack@greg:~$ git clone -b master https://github.com/openstack-dev/devstack.git
Cloning into 'devstack'...
remote: Counting objects: 19577, done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 19577 (delta 11), reused 3 (delta 0)
Receiving objects: 100% (19577/19577), 5.51 MiB | 663.00 KiB/s, done.
Resolving deltas: 100% (13436/13436), done.
Checking connectivity... done.

2. Clone the master branch of Manila from Stackforge:

stack@greg:~/devstack$ git clone -b master https://github.com/stackforge/manila.git
Cloning into 'manila'...
remote: Counting objects: 5538, done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 5538 (delta 39), reused 15 (delta 10)
Receiving objects: 100% (5538/5538), 2.64 MiB | 0 bytes/s, done.
Resolving deltas: 100% (3633/3633), done.
Checking connectivity... done.

3. Copy Manila-specific files into the devstack environment:

stack@greg:~$ cp ./manila/contrib/devstack/lib/manila ./devstack/lib
stack@greg:~$ cp ./manila/contrib/devstack/extras.d/70-manila.sh ./devstack/extras.d/

4. Create your local.conf file with the appropriate values. More importantly, there are specific things for Manila required prior to running of stack.sh. Below is a sample local.conf with the items required for Neutron and Manila denoted with the comment #MANILA-SPECIFIC-STUFF-BELOW.

# Credentials
DATABASE_PASSWORD=devstack
ADMIN_PASSWORD=devstack
SERVICE_PASSWORD=devstack
SERVICE_TOKEN=devstack
RABBIT_PASSWORD=devstack
GUEST_PASSWORD=devstack
MYSQL_HOST=127.0.0.1
MYSQL_USER=root
MYSQL_PASSWORD=devstack
RABBIT_HOST=127.0.0.1
LIBVIRT_TYPE=qemu
GLANCE_HOSTPORT=127.0.0.1:9292
SERVICE_PASSWORD=devstack
ENABLED_SERVICES=key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,n-sch,n-novnc,n-xvnc,n-cauth
#glance
ENABLED_SERVICES+=,g-api,g-reg
#cinder
ENABLED_SERVICES+=,cinder,c-sch,c-api,c-vol,c-bak
#trove
ENABLED_SERVICES+=,trove,tr-api,tr-tmgr,tr-cond
#manila
ENABLED_SERVICES+=,manila,m-api,m-sch,m-shr
#others
ENABLED_SERVICES+=,horizon,rabbit,mysql,tempest
#neutron needs for manila
disable_service n-net
enable_service q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron
#
LOGDAYS=1
LOGFILE=$DEST/devstack.log
SCREEN_LOGDIR=$DEST/logs/screen
RECLONE=yes
API_RATE_LIMIT=False
#MANILA-SPECIFIC-STUFF-BELOW
HORIZON_REPO=https://github.com/NetApp/horizon.git
HORIZON_BRANCH=manila

Q_PLUGIN=ml2
ENABLE_TENANT_VLANS=True
ML2_VLAN_RANGES=physnet1:100:4000
PHYSICAL_NETWORK=physnet1
OVS_PHYSICAL_BRIDGE=br-eth1
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch
Q_ML2_PLUGIN_TYPE_DRIVERS=vlan,vxlan
SHARE_BACKING_FILE_SIZE=8G
FLAT_INTERFACE=eth1

Important Note for local.conf and Manila:

  • Your network configuration may be different than our example. Configure Neutron according to your network and VLAN ranges
  • Ensure that you add vxlan to the Q_ML2_PLUGIN_TYPE_DRIVERS parameter. This was a change to upstream Icehouse.
  • Note the changes for version of Horizon for the integration with Manila and Shares

5. Let’s make sure we have the necessary dependencies

sudo apt-get update && \
sudo apt-get -y install git vim-gtk libxml2-dev libxslt1-dev libpq-dev python-pip libsqlite3-dev && \
sudo apt-get -y build-dep python-mysqldb && sudo pip install git-review tox

6. Run stack.sh as you normally would after you have configured the local.conf for the manila components as well as any items specific for your environment.

7. Once stack.sh has completed, you should have the Manila service(s) running with a default install. Use the manila service-list command to ensure you are running.

stack@greg:~/devstack$ manila service-list
+------------------+---------------+------+---------+-------+----------------------------+
|      Binary      |      Host     | Zone |  Status | State |         Updated_at         |
+------------------+---------------+------+---------+-------+----------------------------+
| manila-scheduler |      greg     | nova | enabled |   up  | 2014-08-08T17:48:32.000000 |
|   manila-share   | greg@backend1 | nova | enabled |   up  | 2014-08-08T17:48:40.000000 |
+------------------+---------------+------+---------+-------+----------------------------+
stack@greg:~/devstack$

8. Check Horizon to validate the Manila plugins are available. We don’t have any shares defined yet; but we need to confirm that the Manila plugins for Horizon are available.

Horizon Screenshot

9. List your Neutron networks and subnets available for your devstack installation. Take a note of each listed prior to creating your Manila share:

stack@greg:~$ neutron net-list
+--------------------------------------+------------------------+----------------------------------------------------+
| id                                   | name                   | subnets                                            |
+--------------------------------------+------------------------+----------------------------------------------------+
| 003cdc18-cd83-4d8c-880d-73abef80ca4c | public                 | 9f0ff94d-0c17-4e68-88c4-92da33df8448 172.24.4.0/24 |
| 83b9d824-1186-4aed-82df-da84adca39b1 | private                | 5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 10.0.0.0/24   |
| e18fa903-183b-4de6-b4ad-a08e52faa9c8 | manila_service_network |                                                    |
+--------------------------------------+------------------------+----------------------------------------------------+
stack@greg:~$ neutron subnet-list
+--------------------------------------+----------------+---------------+------------------------------------------------+
| id                                   | name           | cidr          | allocation_pools                               |
+--------------------------------------+----------------+---------------+------------------------------------------------+
| 5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 | private-subnet | 10.0.0.0/24   | {"start": "10.0.0.2", "end": "10.0.0.254"}     |
| 9f0ff94d-0c17-4e68-88c4-92da33df8448 | public-subnet  | 172.24.4.0/24 | {"start": "172.24.4.2", "end": "172.24.4.254"} |
+--------------------------------------+----------------+---------------+------------------------------------------------+

10. Create a share network using the Neutron private network and subnet for your environment. We used the generic private network/subnet for this example.

stack@greg:~/devstack$ manila share-network-create --neutron-net-id 83b9d824-1186-4aed-82df-da84adca39b1 --neutron-subnet-id 5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 --name manila_share
+-------------------+--------------------------------------+
|      Property     |                Value                 |
+-------------------+--------------------------------------+
|        cidr       |                 None                 |
|     created_at    |      2014-08-11T19:30:34.698596      |
|    description    |                 None                 |
|         id        | 438032da-1147-4cc4-864a-cf691953def9 |
|     ip_version    |                 None                 |
|        name       |             manila_share             |
|    network_type   |                 None                 |
|   neutron_net_id  | 83b9d824-1186-4aed-82df-da84adca39b1 |
| neutron_subnet_id | 5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 |
|     project_id    |   ede0edb1c1e54e3eabc6e6c98b17d862   |
|  segmentation_id  |                 None                 |
|       status      |                 None                 |
|     updated_at    |                 None                 |
+-------------------+--------------------------------------+
stack@greg:~/devstack$ manila share-network-list
+--------------------------------------+---------------+--------+
|                  id                  |      name     | status |
+--------------------------------------+---------------+--------+
| 438032da-1147-4cc4-864a-cf691953def9 |  manila_share |  None  |
| b098a3cd-3b4b-4d7e-b51d-a7476dc761cc | manila_share1 |  None  |
+--------------------------------------+---------------+--------+

11. Create an NFS share, using the share network that you just created. Here we are creating the share “devstack_share” using the share network we created in the previous step.

stack@greg:~/devstack$ manila create --name devstack_share --share-network 438032da-1147-4cc4-864a-cf691953def9 NFS 1
+----------+---------------------------------------------------------------------------------------------------------------------------+
| Property |     Value                                                                                                                 |
+----------+---------------------------------------------------------------------------------------------------------------------------+
|    id    | e9b92837-ea2b-4acd-ab65-65c98199e78b                                                                                      |
|  links   | [{u'href': u'http://10.63.168.135:8786/v1/ede0edb1c1e54e3eabc6e6c98b17d862/shares/e9b92837-ea2b-4acd-ab65-65c98199e78b',  |
|          |   u'rel': u'self'},                                                                                                       |
|          |  {u'href': u'http://10.63.168.135:8786/ede0edb1c1e54e3eabc6e6c98b17d862/shares/e9b92837-ea2b-4acd-ab65-65c98199e78b',     |
|          |   u'rel': u'bookmark'}]                                                                                                   |
|   name   | devstack_share                                                                                                            |
+----------+---------------------------------------------------------------------------------------------------------------------------+
stack@greg:~/devstack$ manila list
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+
|                  ID                  |      Name      | Size | Share Proto |   Status  |                        Export location                        |
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+
| 4ed16b64-59ee-4123-b6f0-f42148280a99 |    myshare     |  1   |     NFS     | available | 10.254.0.3:/shares/share-4ed16b64-59ee-4123-b6f0-f42148280a99 |
| e9b92837-ea2b-4acd-ab65-65c98199e78b | devstack_share |  1   |     NFS     |  creating |                              None                             |
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+

12. Now that this share that has been created, let's take a quick look at the shares. We have two shares, myshare and devstack_share. Each uses the NFS protocol and the export paths for each share are listed in the 'export location' column.

stack@greg:~/devstack$ manila list
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+
|                  ID                  |      Name      | Size | Share Proto |   Status  |                        Export location                        |
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+
| 4ed16b64-59ee-4123-b6f0-f42148280a99 |    myshare     |  1   |     NFS     | available | 10.254.0.3:/shares/share-4ed16b64-59ee-4123-b6f0-f42148280a99 |
| e9b92837-ea2b-4acd-ab65-65c98199e78b | devstack_share |  1   |     NFS     | available | 10.254.0.5:/shares/share-e9b92837-ea2b-4acd-ab65-65c98199e78b |
+--------------------------------------+----------------+------+-------------+-----------+---------------------------------------------------------------+
stack@greg:~/devstack$

13. The export location is part of the Manila shared server configuration. There are two shared servers in this example because we created two different shares on two different share networks. Take a moment within your environment to follow the relationships between your share and share network.

14. Earlier we asked that you check out the lists of Neutron subnets. You should now have an additional entry that will route the subnet used for the NFS exports to your private subnet used to build the share network via Manila.

stack@greg:~/devstack$ neutron subnet-list
+--------------------------------------+------------------------------------------------+---------------+------------------------------------------------+
| id                                   | name                                           | cidr          | allocation_pools                               |
+--------------------------------------+------------------------------------------------+---------------+------------------------------------------------+
| 0d75e8af-ea21-4668-b0c1-351937bee35a | routed_to_5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 | 10.254.0.0/28 | {"start": "10.254.0.2", "end": "10.254.0.14"}  |
| 5d70ffc6-e26a-4189-a522-7fb9b1c2eb23 | private-subnet                                 | 10.0.0.0/24   | {"start": "10.0.0.2", "end": "10.0.0.254"}     |
| 9f0ff94d-0c17-4e68-88c4-92da33df8448 | public-subnet                                  | 172.24.4.0/24 | {"start": "172.24.4.2", "end": "172.24.4.254"} |
+--------------------------------------+------------------------------------------------+---------------+------------------------------------------------+

15. Let's quickly check on the share and find where the backing Cinder volume was created. In my environment, I’m using a NetApp Cinder back end in this example. The backing Cinder volume for my Manila shares can be seen via a df –hk on the single-node devstack host.

stack@greg:~/devstack$ df -kh
Filesystem                                    Size  Used Avail Use% Mounted on
/dev/sda1                                      34G  5.7G   26G  18% /
none                                          4.0K     0  4.0K   0% /sys/fs/cgroup
udev                                          7.9G  4.0K  7.9G   1% /dev
tmpfs                                         1.6G  840K  1.6G   1% /run
none                                          5.0M     0  5.0M   0% /run/lock
none                                          7.9G  4.0K  7.9G   1% /run/shm
none                                          100M     0  100M   0% /run/user
172.16.22.230:/v_devstack_mirror               15G   50M   15G   1% /opt/stack/data/nova/mnt/3edfeba0731309866b6773b9828448eb
172.16.22.230:/v_devstack_comp                9.5G  9.5M  9.5G   1% /opt/stack/data/nova/mnt/57975883130c2c9c4aab078fe8f72404

16. Now you've created a Manila NFS share! But what next? We need to allow access to this share for some hosts. In this example, I will provide access to the share devstack_share to two different hosts in my test environment. From here – we would mount the NFS shares on the NFS client specific by the IP address of the access-allow command.

stack@greg:~/devstack$ manila access-list e9b92837-ea2b-4acd-ab65-65c98199e78b
+----+-------------+-----------+-------+
| id | access type | access to | state |
+----+-------------+-----------+-------+
+----+-------------+-----------+-------+
stack@greg:~/devstack$ manila access-allow e9b92837-ea2b-4acd-ab65-65c98199e78b ip 10.63.168.135
+-------------+--------------------------------------+
|   Property  |                Value                 |
+-------------+--------------------------------------+
|  access_to  |            10.63.168.135             |
| access_type |                  ip                  |
|  created_at |      2014-08-12T12:44:28.112086      |
|   deleted   |                False                 |
|  deleted_at |                 None                 |
|      id     | 104a8ce8-bb2e-47a7-84a0-06e7b7dc148a |
|   share_id  | e9b92837-ea2b-4acd-ab65-65c98199e78b |
|    state    |                 new                  |
|  updated_at |                 None                 |
+-------------+--------------------------------------+
stack@greg:~/devstack$ manila access-allow e9b92837-ea2b-4acd-ab65-65c98199e78b ip 10.0.0.6
+-------------+--------------------------------------+
|   Property  |                Value                 |
+-------------+--------------------------------------+
|  access_to  |               10.0.0.6               |
| access_type |                  ip                  |
|  created_at |      2014-08-12T12:44:36.326327      |
|   deleted   |                False                 |
|  deleted_at |                 None                 |
|      id     | ec398f37-cbf4-4cd2-bc28-7a92fcbe0167 |
|   share_id  | e9b92837-ea2b-4acd-ab65-65c98199e78b |
|    state    |                 new                  |
|  updated_at |                 None                 |
+-------------+--------------------------------------+
stack@greg:~/devstack$ manila access-list e9b92837-ea2b-4acd-ab65-65c98199e78b
+--------------------------------------+-------------+---------------+--------+
|                  id                  | access type |   access to   | state  |
+--------------------------------------+-------------+---------------+--------+
| 104a8ce8-bb2e-47a7-84a0-06e7b7dc148a |      ip     | 10.63.168.135 | active |
| ec398f37-cbf4-4cd2-bc28-7a92fcbe0167 |      ip     |    10.0.0.6   | active |
+--------------------------------------+-------------+---------------+--------+
stack@greg:~/devstack$

Here's the recording of the demo presented in Atlanta which includes a scenario very similar to what we have described in this post:

<iframe frameborder="0" height="390" id="ytplayer" src="http://www.youtube.com/embed/fo5ObeYvU-g?autoplay=0" type="text/html" width="640"></iframe>

That's all for now; we will follow up with more on Manila in later blog posts, including a discussion of the integration with the NetApp clustered Data ONTAP driver for Manila. To learn more about Manila, check out our wiki page at http://wiki.openstack.org/wiki/Manila. Or, come join us on IRC at #openstack-manila and #openstack-netapp on freenode.

August 15, 2014 08:30 AM

Opensource.com

The why and how of becoming a cloud architect

Open source experts at the OpenStack Summit in Atlanta this year share their thoughts on what it's like to be a cloud architect.

by Jason Baker at August 15, 2014 07:00 AM

Michael Still

Juno nova mid-cycle meetup summary: cells

This is the next post summarizing the Juno Nova mid-cycle meetup. This post covers the cells functionality used by some deployments to scale Nova.

For those unfamiliar with cells, it's a way of combining smaller Nova installations into a thing which feels like a single large Nova install. So for example, Rackspace deploys Nova in cells of hundreds of machines, and these cells form a Nova availability zone which might contain thousands of machines. The cells in one of these deployments form a tree: users talk to the top level of the tree, which might only contain API services. That cell then routes requests to child cells which can actually perform the operation requested.

There are a few reasons why Rackspace does this. Firstly, it keeps the MySQL databases smaller, which can improve the performance of database operations and backups. Additionally, cells can contain different types of hardware, which are then partitioned logically. For example, OnMetal (Rackspace's Ironic-based baremetal product) instances come from a cell which contains OnMetal machines and only publishes OnMetal flavors to the parent cell.

Cells was originally written by Rackspace to meet its deployment needs, but is now used by other sites as well. However, I think it would be a stretch to say that cells is commonly used, and it is certainly not the deployment default. In fact, most deployments don't run any of the cells code, so you can't really call them even a "single cell install". One of the reasons cells isn't more widely deployed is that it doesn't implement the entire Nova API, which means some features are missing. As a simple example, you can't live-migrate an instance between two child cells.

At the meetup, the first thing we discussed regarding cells was a general desire to see cells finished and become the default deployment method for Nova. Perhaps most people end up running a single cell, but in that case at least the cells code paths are well used. The first step to get there is improving the Tempest coverage for cells. There was a recent openstack-dev mailing list thread on this topic, which was discussed at the meetup. There was commitment from several Nova developers to work on this, and notably not all of them are from Rackspace.

It's important that we improve the Tempest coverage for cells, because it positions us for the next step in the process, which is bringing feature parity to cells compared with a non-cells deployment. There is some level of frustration that the work on cells hasn't really progressed in Juno, and that it is currently incomplete. At the meetup, we made a commitment to bringing a well-researched plan to the Kilo summit for implementing feature parity for a single cell deployment compared with a current default deployment. We also made a commitment to make cells the default deployment model when this work is complete. If this doesn't happen in time for Kilo, then we will be forced to seriously consider removing cells from Nova. A half-done cells deployment has so far stopped other development teams from trying to solve the problems that cells addresses, so we either need to finish cells, or get out of the way so that someone else can have a go. I am confident that the cells team will take this feedback on board and come to the summit with a good plan. Once we have a plan we can ask the whole community to rally around and help finish this effort, which I think will benefit all of us.

In the next blog post I will cover something we've been struggling with for the last few releases: how we get our bug count down to a reasonable level.

Tags for this post: openstack juno nova mid-cycle summary cells
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues

Comment

August 15, 2014 05:20 AM

Juno nova mid-cycle meetup summary: DB2 support

This post is one part of a series discussing the OpenStack Nova Juno mid-cycle meetup. It's a bit shorter than most of the others, because the next thing on my list to talk about is DB2, and that's relatively contained.

IBM is interested in adding DB2 support as a SQL database for Nova. Theoretically, this is a relatively simple thing to do because we use SQLAlchemy to abstract away the specifics of the SQL engine. However, in reality, the abstraction is leaky. The obvious example in this case is that DB2 has different rules for foreign keys than other SQL engines we've used. So, in order to be able to make this change, we need to tighten up our schema for the database.

The change that was discussed is the requirement that the UUID column on the instances table be not null. This seems like a relatively obvious thing to allow, given that UUID is the official way to identify an instance, and has been for a really long time. However, there are a few things which make this complicated: we need to understand the state of databases that might have been through a long chain of upgrades from previous Nova releases, and we need to ensure that the schema alterations don't cause significant performance problems for existing large deployments.

As an aside, people sometimes complain that Nova development is too slow these days, and they're probably right, because things like this slow us down. A relatively simple change to our database schema requires a whole bunch of performance testing and negotiation with operators to ensure that its not going to be a problem for people. It's good that we do these things, but sometimes it's hard to explain to people why forward progress is slow in these situations.

Matt Riedemann from IBM has been doing a good job of handling this change. He's written a tool that operators can run before the change lands in Juno that checks if they have instance rows with null UUIDs. Additionally, the upgrade process has been well planned, and is documented in the specification available on the fancy pants new specs website.

We had a long discussion about this change at the meetup, and how it would impact on large deployments. Both Rackspace and HP were asked if they could run performance tests to see if the schema change would be a problem for them. Unfortunately HP's testing hardware was tied up with another project, so we only got numbers from Rackspace. For them, the schema change took 42 minutes for a large database. Almost all of that was altering the column to be non-nullable; creating the new index was only 29 seconds of runtime. However, the Rackspace database is large because they don't currently purge deleted rows, if they can get that done before running this schema upgrade then the impact will be much smaller.

So the recommendation here for operators is that it is best practice to purge deleted rows from your databases before an upgrade, especially when schema migrations need to occur at the same time. There are some other takeaways for operators as well: if we know that operators have a large deployment, then we can ask if an upgrade will be a problem. This is why being active on the openstack-operators mailing list is important. Additionally, if operators are willing to donate a dataset to Turbo-Hipster for DB CI testing, then we can use that in our automation to try and make sure these upgrades don't cause you pain in the future.

In the next post in this series I'll talk about the future of cells, and the work that needs to be done there to make it a first class citizen.

Tags for this post: openstack juno nova mid-cycle summary sql database sqlalchemy db2
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 15, 2014 03:20 AM

Jamie Lennox

Git Commands for Messy People

I am terrible at keeping my git branches in order. Particularly since I work across multiple machines and forget where things are I will often have multiple branches with different names being different versions of the same review.

On a project I work on frequently I currently have 71 local branches which are a mix of my code, some code reviews, and some branches that were for trialling ideas. git review at least prefixes branches it downloads with review/ but that doesn’t help to figure out what was happening with local branches labelled auth through auth-4.

However this post isn’t about me fixing my terrible habit it’s about two git commands which help me work with the mess.

The first is an alias which I called branch-date:

<notextile><figure class="code">
1
2
[alias]
    branch-date = &ldquo;!git for-each-ref &ndash;sort=committerdate &ndash;format=&lsquo;%1B[32m%(committerdate:iso8601) %1B[34m%(committerdate:relative) %1B[0;m%(refname:short)&rsquo; refs/heads/&rdquo;
</figure></notextile>

This gives a nicely formatted list of branches in the project sorted by the last time they were committed to and how long ago it was. So if I know I’m looking for a branch that I last worked on last week I can quickly locate those branches.

The next is a script to figure out which of my branches have made it through review and have been merged upstream which I called branch-merged.

Using git you can already call git branch --merged master to determine which branches are fully merged into the master branch. However this won’t take into account if a later version of a review was merged, in which case I can probably get rid of that branch.

We can figure this out by using the Commit-Id: field of our Gerrit reviews.

<script src="https://gist.github.com/a7f8080e42ec795945b8.js?file=git-branch-merged.sh"></script> <noscript>
#!/bin/bash

# create an associative array of all the changes
declare -A MERGED
for commit in `git log master | grep Change-Id | cut -d : -f 2`; do
    MERGED["$commit"]=1
done

for branch in `git branch | grep -v master`; do
    found=1

    # for every commit on the branch check to ensure its in master
    for commit in `git log $branch | grep Change-Id | cut -d : -f 2`; do
        if [[ ! ${MERGED["$commit"]} ]]; then
            # if it's not in master then this branch wasn't merged so we
            # want to break out and not print its name.
            found=0
            break
        fi
    done

    if [ $found = 1 ]; then
        # if all of the Commit-Ids on branch were also in master then
        # print out the branch name because we can probably get rid of it
        echo $branch
    fi
done
</noscript>

So print out the branches where all the Commit-Ids are also in master. It’s not greatly efficient and if you are working with code bases with long histories you might need to limit the depth, but given that it doesn’t run often it completes quickly enough.

There’s no guarantee that there wasn’t something new in those branches, but most likely it was an earlier review or test code that is no longer relevant. I was considering a tool that could use the Commit-Id to figure out from gerrit if a branch is an exact match to one that was previously up for review and so contained no possibly useful experimenting code, but teaching myself to clean up branches as I go is probably a better use of my time.

August 15, 2014 12:59 AM

August 14, 2014

Michael Still

Review priorities as we approach juno-3

I just send this email out to openstack-dev, but I am posting it here in case it makes it more discoverable to people drowning in email:

To: openstack-dev
Subject: [nova] Review priorities as we approach juno-3

Hi.

We're rapidly approaching j-3, so I want to remind people of the
current reviews that are high priority. The definition of high
priority I am using here is blueprints that are marked high priority
in launchpad that have outstanding code for review -- I am sure there
are other reviews that are important as well, but I want us to try to
land more blueprints than we have so far. These are listed in the
order they appear in launchpad.

== Compute Manager uses Objects (Juno Work) ==

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/compute-manager-objects-juno,n,z

This is ongoing work, but if you're after some quick code review
points they're very easy to review and help push the project forward
in an important manner.

== Move Virt Drivers to use Objects (Juno Work) ==

I couldn't actually find any code out for review for this one apart
from https://review.openstack.org/#/c/94477/, is there more out there?

== Add a virt driver for Ironic ==

This one is in progress, but we need to keep going at it or we wont
get it merged in time.

* https://review.openstack.org/#/c/111223/ was approved, but a rebased
ate it. Should be quick to re-approve.
* https://review.openstack.org/#/c/111423/
* https://review.openstack.org/#/c/111425/
* ...there are more reviews in this series, but I'd be super happy to
see even a few reviewed

== Create Scheduler Python Library ==

* https://review.openstack.org/#/c/82778/
* https://review.openstack.org/#/c/104556/

(There are a few abandoned patches in this series, I think those two
are the active ones but please correct me if I am wrong).

== VMware: spawn refactor ==

* https://review.openstack.org/#/c/104145/
* https://review.openstack.org/#/c/104147/ (Dan Smith's -2 on this one
seems procedural to me)
* https://review.openstack.org/#/c/105738/
* ...another chain with many more patches to review

Thanks,
Michael


The actual email thread is at http://lists.openstack.org/pipermail/openstack-dev/2014-August/043098.html.

Tags for this post: openstack juno review nova ptl
Related posts: Juno Nova PTL Candidacy; Thoughts from the PTL; Havana Nova PTL elections; Expectations of core reviewers; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 14, 2014 09:01 PM

OpenStack Blog

Open Mic Spotlight: Rossella Sblendido

Rossella.OpenStackThis post is part of the OpenStack Open Mic series to spotlight the people who have helped make OpenStack successful. Each week, a new contributor will step up to the mic and answer five questions about OpenStack, cloud, careers and what they do for fun.  If you’re interested in being featured, please choose five questions from this form and submit!

Rossella is a software engineer at SUSE. She has studied in three countries (Italy, Germany and Spain) and holds a MSc in Telecommunications Engineering from the Politecnico di Milano. She enjoys developing software, solving complex problems and learning new technologies. 

1. Finish the sentences. OpenStack is great for _______. OpenStack is bad for ______.

OpenStack is great for its community and challenging environment. OpenStack is bad for the amount of information I process every day.

2. Get creative – create an OpenStack haiku! 

It is virtual, 

But OpenStack is real too. 

Long live OpenStack! 

3. How would you explain this job to your grandmother?

Grandma, I don’t produce anything that you can touch. I am a kind of poet, but my words are not about my feelings or thoughts. My words talk to the computer and make it do what I want. I am a kind of computer tamer.

4. What does “open source” mean to you?

Open source means transparency and community. You can learn from other people’s code and you can give your opinion. You can fix bugs when you encounter them. There’s a community, you interact with people from all over the world and work for different companies to get things done.

5. How did you learn to code? Were you self-taught, or did you learn in college? On the job?

I had to learn some code in college, but I didn’t like it and I was not very good at it, to be honest. I started appreciating coding during my master thesis, which extended to the job, where I learned most of my skills.

by OpenStack at August 14, 2014 08:51 PM

Tesora Corp

Count Down to OpenStack Trove Day

trove-diver.pngOpen source is a business model that works when it works for business. It’s no longer good enough to declare something “open source” and think the world will flock to whatever solution is being offered. That novelty wore off a decade ago.

One thing that has remained constant about open source is the need for a vibrant, active community and that is certainly true of OpenStack. In its four years, OpenStack has gone from zero to production, thanks to a community of contributors that is dedicated to its technical, economic and practical success.

That is why we’re so pleased to be part of the community that is building around Trove, the database as a service component of OpenStack and to be hosting next week’s first “OpenStack Trove Day.”

OpenStack has come into its own as an enterprise-grade platform for building a cloud infrastructure. Compute, storage and networking capabilities all live comfortably in the OpenStack cloud. Now, database as a service is here, thanks to the OpenStack Trove.

You may have seen the results of the survey we released in May that shows that OpenStack is catching up to VMware. And of course, database as a service is a huge area of focus for those of us in the OpenStack community that are looking to bring real-world solutions to businesses. In fact, we did a short survey of attendees of next week’s event and more than 20 percent responded that they intend to deploy Trove in 2014-2015.

This growing adoption of OpenStack and the pent up demand for DBaaS led us to the notion that the time was right for us to get together as a community. Next week’s event will bring together users, developers and vendors that are all part of the Trove ecosystem to share experiences, challenges, opportunities and victories.

The response to this inaugural event has been nothing short of stunning. We’re sold out as of right now, but we do have an active waiting list, so let us know if you are interested in attending and we’ll do our best to accommodate you.

If you can’t make it, keep an eye out for some pretty important news that we plan to announce at the conference. You can follow this blog and on Twitter at #TroveDay. And hopefully we’ll see you next year.

by 86 at August 14, 2014 03:37 PM

Michael Still

Juno nova mid-cycle meetup summary: ironic

Welcome to the third in my set of posts covering discussion topics at the nova juno mid-cycle meetup. The series might never end to be honest.

This post will cover the progress of the ironic nova driver. This driver is interesting as an example of a large contribution to the nova code base for a couple of reasons -- its an official OpenStack project instead of a vendor driver, which means we should already have well aligned goals. The driver has been written entirely using our development process, so its already been reviewed to OpenStack standards, instead of being a large code dump from a separate development process. Finally, its forced us to think through what merging a non-trivial code contribution should look like, and I think that formula will be useful for later similar efforts, the Docker driver for example.

One of the sticking points with getting the ironic driver landed is exactly how upgrade for baremetal driver users will work. The nova team has been unwilling to just remove the baremetal driver, as we know that it has been deployed by at least a few OpenStack users -- the largest deployment I am aware of is over 1,000 machines. Now, this unfortunate because the baremetal driver was always intended to be experimental. I think what we've learnt from this is that any driver which merges into the nova code base has to be supported for a reasonable period of time -- nova isn't the right place for experiments. Now that we have the stackforge driver model I don't think that's too terrible, because people can iterate quickly in stackforge, and when they have something stable and supportable they can merge it into nova. This gives us the best of both worlds, while providing a strong signal to deployers about what the nova team is willing to support for long periods of time.

The solution we came up with for upgrades from baremetal to ironic is that the deployer will upgrade to juno, and then run a script which converts their baremetal nodes to ironic nodes. This script is "off line" in the sense that we do not expect new baremetal nodes to be launchable during this process, nor after it is completed. All further launches would be via the ironic driver.

These nodes that are upgraded to ironic will exist in a degraded state. We are not requiring ironic to support their full set of functionality on these nodes, just the bare minimum that baremetal did, which is listing instances, rebooting them, and deleting them. Launch is excluded for the reasoning described above.

We have also asked the ironic team to help us provide a baremetal API extension which knows how to talk to ironic, but this was identified as a need fairly late in the cycle and I expect it to be a request for a feature freeze exception when the time comes.

The current plan is to remove the baremetal driver in the Kilo release.

Previously in this post I alluded to the review mechanism we're using for the ironic driver. What does that actually look like? Well, what we've done is ask the ironic team to propose the driver as a series of smallish (500 line) changes. These changes are broken up by functionality, for example the code to boot an instance might be in one of these changes. However, because of the complexity of splitting existing code up, we're not requiring a tempest pass on each step in the chain of reviews. We're instead only requiring this for the final member in the chain. This means that we're not compromising our CI requirements, while maximizing the readability of what would otherwise be a very large review. To stop the reviews from merging before we're comfortable with them, there's a marker review at the beginning of the chain which is currently -2'ed. When all the code is ready to go, I remove the -2 and approve that first review and they should all merge together.

In the next post I'll cover the state of adding DB2 support to nova.

Tags for this post: openstack juno nova mid-cycle summary ironic
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues; Juno nova mid-cycle meetup summary: slots

Comment

August 14, 2014 09:49 AM

Juno nova mid-cycle meetup summary: containers

This is the second in my set of posts discussing the outcomes from the OpenStack nova juno mid-cycle meetup. I want to focus in this post on things related to container technologies.

Nova has had container support for a while in the form of libvirt LXC. While it can be argued that this support isn't feature complete and needs more testing, its certainly been around for a while. There is renewed interest in testing libvirt LXC in the gate, and a team at Rackspace appears to be working on this as I write this. We have already seen patches from this team as they fix issues they find on the way. There are no plans to remove libvirt LXC from nova at this time.

The plan going forward for LXC tempest testing is to add it as an experimental job, so that people reviewing libvirt changes can request the CI system to test LXC by using "check experimental". This hasn't been implemented yet, but will be advertised when it is ready. Once we've seen good stable results from this experimental check we will talk about promoting it to be a full blown check job in our CI system.

We have also had prototype support for Docker for some time, and by all reports Eric Windisch has been doing good work at getting this driver into a good place since it moved to stackforge. We haven't started talking about specifics for when this driver will return to the nova code base, but I think at this stage we're talking about Kilo at the earliest. The driver has CI now (although its still working through stability issues to my understanding) and progresses well. I expect there to be a session at the Kilo summit in the nova track on the current state of this driver, and we'll decide whether to merge it back into nova then.

There was also representation from the containers sub-team at the meetup, and they spent most of their time in a break out room coming up with a concrete proposal for what container support should look like going forward. The plan looks a bit like this:

Nova will continue to support "lowest common denominator containers": by this I mean that things like the libvirt LXC and docker driver will be allowed to exist, and will expose the parts of containers that can be made to look like virtual machines. That is, a caller to the nova API should not need to know if they are interacting with a virtual machine or a container, it should be opaque to them as much as possible. There is some ongoing discussion about the minimum functionality we should expect from a hypervisor driver, so we can expect this minimum level of functionality to move over time.

The containers sub-team will also write a separate service which exposes a more full featured container experience. This service will work by taking a nova instance UUID, and interacting with an agent within that instance to create containers and manage them. This is interesting because it is the first time that a compute project will have an in operating system agent, although other projects have had these for a while. There was also talk about the service being able to start an instance if the user didn't already have one, or being able to declare an existing instance to be "full" and then create a new one for the next incremental container. These are interesting design issues, and I'd like to see them explored more in a specification.

This plan met with general approval within the room at the meetup, with the suggestion being that it move forward as a stackforge project as part of the compute program. I don't think much code has been implemented yet, but I hope to see something come of these plans soon. The first step here is to create some specifications for the containers service, which we will presumably create in the nova-specs repository for want of a better place.

Thanks for reading my second post in this series. In the next post I will cover progress with the Ironic nova driver.

Tags for this post: openstack juno nova mid-cycle summary containers docker lxc
Related posts: Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic; Juno nova mid-cycle meetup summary: conclusion; Juno nova mid-cycle meetup summary: DB2 support; Juno nova mid-cycle meetup summary: social issues

Comment

August 14, 2014 09:17 AM