May 26, 2017


OpenStack Australia Day Melbourne: Innovation, Donuts and the chance to win a FREE pass to the Sydney OpenStack Summit!

Aptira - OpenStack Australia Day Melbourne

This time next week we will be halfway through OpenStack Australia Day Melbourne. And more than likely we will have demolished the donut wall!

This OpenStack Australia Day will be fast paced with more features than ever before. This includes three simultaneous tracks, covering business, tech and innovation. There will also be two panels: an innovation panel to discuss global research on open platforms before the main panel kicks off at 5pm.

We have a stack of prizes and giveaways available on the day, with the chance to win a free pass to the Sydney OpenStack Summit in November, a free registration for the OpenStack COA exam, 2 x Nespresso coffee machines, a 3D printer and many more.

And just in case you’d forgotten, there will be a donut wall, networking drinks and the chance to network with some of the most talented members of the OpenStack community – including the OpenStack Foundation’s executive Director Jonathan Bryce and spearhead of the user survey Heidi Joy Tretheway.

Register today at:

See you in Melbourne!

The post OpenStack Australia Day Melbourne: Innovation, Donuts and the chance to win a FREE pass to the Sydney OpenStack Summit! appeared first on Aptira Cloud Solutions.

by Jessica Field at May 26, 2017 06:44 PM

Ihar Hrachyshka

The Failure, part 4: Summary

This post is the sixth, and the last, in a series that covers gratuitous ARP and its relation to OpenStack. In the previous posts, we discussed what gratuitous ARP is and how it’s implemented in OpenStack Neutron L3 agent. We also introduced the Failure, covered its initial triaging, looked at traffic captured during failed test runs, and digged through Linux kernel code only to find multiple bugs in its ARP layer.

In this final post we will summarize what we have learned about the Failure. In addition, we will also discuss a possible fix that could help alleviate the impact while we are waiting for new kernels.

It is advised that you make yourself comfortable with previous posts in the series before proceeding with reading this last “episode”. Otherwise you loose all the fun of war stories.


In the previous post of the series, we finally learned what went wrong with the Failure, and we figured there is no OpenStack fault in it. Instead, it is Linux kernel that ignores all gratuitous ARP packets Neutron L3 agent eagerly sends its way; and it is the same Linux kernel that is spinning bogus ARP entries in perpetual STALE – DELAY – REACHABLE state change loop without a way out.

And that’s where we are at. We have the following alternatives to tackle the test failure problem.


  • Include the patch that avoids touching n->updated if an ARP packet is ignored;


  • Include the patch that forces override for all gratuitous ARP packets irrespective of arp_accept;


(Of course, ideally all of those pieces would find their way into your setup.)

All of those solutions currently require kernel backports, at least for RHEL7. In the meantime, could we do something just on Neutron side? On first sight, it sounds like a challenge. But when you think about it, we can use the knowledge about the failure mode to come up with a workaround that may work in most cases.

We know that the issue would not show itself up in tests if only any of gratuitous ARP replies sent by Neutron would be honored by kernel, and they are all ignored because of arrival during locktime window.

We know that the default value for locktime is 1 second, and the reason why all three gratuitous ARP packets are ignored is because each of them land into the moving locktime window, which happens because of the way we issue those packets using the arping tool. The default interval between gratuitous ARP packets issued by the tool is 1 second, but if we could make it longer, it could help the kernel to get out of the moving locktime window loop and honor one of packets sent in burst.

From looking at arping(8) and its code, it doesn’t seem like it supports picking an alternative interval with any of its command line options (I have sent a pull request to add the feature but it will take time until it gets into distributions). If we want to spread gratuitous updates in time using arping, we may need to call the tool multiple times from inside Neutron L3 agent and maintain time interval between packets ourselves.

Here is the Neutron patch to use this approach. This would of course work only with hosts that don’t change locktime sysctl setting from its default value. Moreover, it’s very brittle, and may still not give us 100% test pass guarantee.

The chance of success can of course be elevated by setting arp_accept on all nodes. The good news is that at least some OpenStack installers already do it. For example, see this patch for Fuel. While the original reasoning behind the patch is twisted (gratuitous ARP packets are accepted irrespective of arp_accept, it’s just that locktime may get in their way in bad timing), the change itself is still helpful to overcome limitations of released kernels. To achieve the same effect for TripleO, I posted a similar patch.

Finally, note that while all the workarounds listed above may somewhat help with passing tempest tests, without the patch series tackling the very real spurious confirmation issue, you still risk getting your ARP table broken without a way for it to self-heal.

The best advice I can give is, make sure you use a kernel that includes the patch series (all official kernels starting from 4.11 do). If you consume kernel from a distribution, talk to your vendor to get the patches included. Without them, you risk with your service availability. (To note, while I was digging this test only issue, we got reports from the field some ARP entries were staying invalid for hours after failover.)

And finally… it’s not always an OpenStack fault. Platform matters.

by ihrachyshka at May 26, 2017 05:12 PM

OpenStack Superuser

Deploying and redeploying OpenStack in a physical server


Openstack is deployed for multiple purposes — such as teaching or training, exploring certain features, developing or enhancing Openstack projects, integrating Openstack with other systems, running test or production workloads etc.

We often need to deploy and redeploy Openstack in the same physical server for multiple purposes including:
1. Upgrades
2. Problems with deployment
3. Perfecting configurations
4. Changes in the hypervisor
5. Trying other configuration options for fine tuning
6. Openstack deployment training

The challenge with redeployment is that we need to trace back all the packages that were installed as part of Openstack and uninstall all of them. It’s a tedious and cumbersome process. The other option is to reinstall the operating system but that’s time consuming. The best way is to restore the physical server to previous its state using the Snapper tool. This tutorial outlines the procedure for installing Snapper, creating a Snapshot of the physical server similar to a virtual image, deploying Openstack and restoring the physical server to previous state of deployment.


OpenStack is a leading open-source software platform for cloud computing, mostly deployed as an infrastructure-as-a-service (IaaS). The software platform consists of interrelated components that control diverse, multi-vendor hardware pools of processing, storage and networking resources throughout a data center.


Snapper is a Linux command ­line tool to create and manage snapshots of your filesystems. It allows users to create read­-only snapshots of a physical machines which can be used to restore the state of the server during any disaster-type situation.

Steps for Deploying Openstack with Snapper and restoring to original state

This block diagram outlines the various layer involved in the setup.


  • File system should be a BTRFS file system.
  • OS: Ubuntu14.04,16.04
  • Snapper Version 0.1.8
  • Resource requirement 4GB RAM 20GB HDD

Note: Snapper is fully supported in the Open SUSE platform

Step-1 : Install Snapper

Install Snapper using the below command

apt-get install snapper -y

Step-2 : Prepare script for creating Config and Mount point

Create below script with name

#Snapper with btrfs filesystem in ubuntu 14.04

ARGS=$(getopt -o a:b -l "configname:,mountpoint:" -- "$@");
eval set -- "$ARGS";
while true; do
case "$1" in
if [ -n "$1" ]; then
if [ -n "$1" ]; then

#creating a config file 
echo $configname
echo $mountpoint
snapper -c $configname create-config $mountpoint
snapper list-configs

Step-3 : Run script for creating Config and Mount point

Run script using the below command. Use the same configname and mount point till the end.

bash –configname –mountpoint

bash –config name snapconfig –mountpoint /
Where, – script name
Snapconfig- config name
/- mount point

Step-4 : Prepare script for creating snapshot

Create below script with name

ARGS=$(getopt -o a:b -l "configname:,snapshotname:" -- "$@");
eval set -- "$ARGS";
while true; do
case "$1" in

if [ -n "$1" ]; then
if [ -n "$1" ]; then

echo $configname
echo $snapshotname

snapper --config $configname create --description "$snapshotname "
snapper --config $configname  list

Step-5 : Run script to create a Snapshot using Snapper

Run script using the below command

bash –configname –snapshotname
Eg :
bash –configname snapconfig –snapshotname snap1
Where, – script name
Snapconfig- config name
snap1- snapshot name

The execution with create a snapshot and list the created snapshots. Create the snapshot name of your choice but remember to provide the same config name used earlier.

Step-6 : Deploy Openstack

Deploy Openstack Mitaka by following the instruction available at:

Step-7 : Prepare a Script to restore the snapshot

Create below script with name for restoring the physical server to older version.

ARGS=$(getopt -o a:b:c -l "configname:,state_to_revert:,state_to_modify:," -- "$@");
eval set -- "$ARGS";
while true; do
case "$1" in
if [ -n "$1" ]; then
if [ -n "$1" ]; then
if [ -n "$1" ]; then

echo $configname
echo $state_to_revert
echo $state_to_modify

snapper -c $configname -v undochange $state_to_revert..$state_to_modify

Step-8 : Run script to restore older snapshot using Snapper

Run the script using the below command specifying the config name and Snapshot number to which we need to restore.

bash –configname –state_to_revert –state_to_modify

bash –configname snapconfig –state_to_revert 1 –state_to_modify 2

Congratulate yourself!

If everything works correctly, you will able to deploy Openstack and restore the server to its original state and redeploy over and over again as needed.

Cover Photo // CC BY NC

This post first appeared on Cloud Enablers. Superuser is always interested in community content, email to find out more. 



The post Deploying and redeploying OpenStack in a physical server appeared first on OpenStack Superuser.

by Superuser at May 26, 2017 11:49 AM

May 25, 2017

Ihar Hrachyshka

The Failure, part 3: Diggin’ the Kernel

This post is the fifth in a series that covers gratuitous ARP and its relation to OpenStack. In the previous posts, we discussed what gratuitous ARP is and how it’s implemented in OpenStack Neutron L3 agent. We also introduced the Failure, covered its initial triaging, and looked at traffic captured during failed test runs.

All previous attempts to figure out what went wrong with the Failure hasn’t succeeded. In this post we will look closer at the Linux kernel code to learn about ARP table states, and to see if we can find any quirks in the kernel ARP layer that could explain the observations.

It is advised that you make yourself comfortable with previous posts in the series before proceeding with reading this “episode”.

Diggin’ the Kernel

Some of you may know that in my previous life, I was a Linux kernel developer (nothing fancy, mostly enabling embedded hardware). Though this short experience made myself more or less comfortable with reading the code, I figured I could use some help from the vast pool of Red Hat kernel developers. So I reached to Lance Richardson who, I was told, could help me figure out what’s going on with the Failure. And indeed, his help was enormous. In next several days, we discussed the kernel code on IRC, were digging old kernel mailing list archives, built and tested a bunch of custom kernels with local modifications to its networking layer. Here is what we’ve found.

Gratuitous ARP with arp_accept

Since RHEL7 kernel is quite old (3.10.0-514.22.1.el7 at the time of writing), we decided to start our search by looking at patches in Linus master branch and see if there were any that could be of relevance to the Failure, and that were not backported yet into RHEL7 kernel. The primary files of interest in the kernel source tree were net/ipv4/arp.c (the ARP layer) and net/core/neighbour.c (the neighbour layer which is an abstract representation of address-to-MAC mappings used for IPv4 as well as for IPv6).

Digging through the master branch history, the very first patch that drew our attention was: “ipv4: arp: update neighbour address when a gratuitous arp is received…” What the patch does is it forces override of an existing ARP table entry when a gratuitous ARP packet is received irrespective of whether it was received in locktime time interval. It is effective only when arp_accept is enabled, which is not the default. Anyway, that ringed some bell, and also suggested that maybe we dealt with a timing issue. The patch assumed arp_accept enabled and temporarily disabled the locktime behavior for gratuitous ARP packets, so let’s have a closer look at those two sysctl knobs.

Here is the documentation for the arp_accept sysctl knob. The setting controls whether the kernel should populate its ARP table with new entries on receiving ARP packets if the IP addresses are not registered in the table yet. Enabling the setting may be useful if you want to “warm up” the ARP table on system startup without waiting for the node to send its very first datagram to an IP address. The idea is that the kernel will listen for any ARP packets flying by, and it will create new table entries for previously unseen IP addresses. The default for the option is 0 (meaning off), and that’s for a reason. Enabling the feature may have unexpected consequences because the size of the kernel ARP table is limited, and in large network segments it may happen that the kernel will overflow the table with irrelevant entries due to the “warming up”. If that ever happens, the kernel may then start dropping some table entries that may still be useful. If that happens, you can see slowdown for some upper layer protocol connections for the time needed to restore the needed ARP entries using a round-trip of ARP probe packets. Long story short, the arp_accept setting is not for everyone.

As for locktime, there seems to be no in-tree documentation for the sysctl parameter, so the best source of information is probably arp(7). Quoting: “[locktime is] the minimum number of jiffies to keep an ARP entry in the cache. This prevents ARP cache thrashing if there is more than one potential mapping (generally due to network misconfiguration). Defaults to 1 second.” What it means is that if an ARP packet arrives during a 1 second interval since the previous ARP packet, it will be ignored. This is helpful when using ARP proxies where multiple network endpoints can reply to the same ARP REQUEST. In this case, you may want to ignore those replies that arrive later (to avoid so called ARP thrashing, as well as to stick to the node that is allegedly quicker/closer to the node).

With the above mentioned kernel patch, and arp_accept set to 1, the kernel should always update its ARP table if a gratuitous ARP packet is received, even if the entry is still in the locktime time interval.

Though arp_accept is not applicable for everyone, it was still worth exploring. I backported the patch into RHEL7 kernel, rebooted the tempest node, enabled arp_accept for eth2, and rerun the tests. Result? Same failures. So why hasn’t it worked?

Code inspection of neigh_update hasn’t revealed anything interesting. Everything suggested that override was still false. It took me awhile, but then it struck me: the code to determine whether an ARP packet is gratuitous considered frames of Request type, but not Reply. And Neutron L3 agent sends Replies (note the -A option instead of -U using in the arping command line)!
Here is how arping(8) defines those options:

-A     The same as -U, but ARP REPLY packets used instead of ARP REQUEST.
-U     Unsolicited ARP mode to update neighbours' ARP caches.  No replies are expected.

The next step was clear: let’s try to switch all Neutron L3 agents to gratuitous ARP requests and see if it helps. So I applied a one-liner to all controller nodes, restarted neutron-l3-agent services, and repeated the test run. It passed. I even passed multiple times in a row, first time in a long time I was banging my head over the issue!

OK, now I had a workaround. To pass tests, all I needed was:

  • Get a kernel that includes the patch (officially released as 3.14 on Mar 30, 2014);
  • Enable arp_accept for the external (eth2) interface;
  • Restart neutron-l3-agent services with the one-liner included.

But does it make sense that the kernel accepts gratuitous REQUESTs but not REPLYs? Is there anything in RFCs defining ARP that would suggest REPLYs are a different beast? Let’s have a look.

As we’ve learned in the very first post in the series, gratuitous ARP packets are defined in RFC 2002. Let’s quote the definition here in full.

-  A Gratuitous ARP [23] is an ARP packet sent by a node in order to
 spontaneously cause other nodes to update an entry in their ARP
 cache.  A gratuitous ARP MAY use either an ARP Request or an ARP
 Reply packet.  In either case, the ARP Sender Protocol Address
 and ARP Target Protocol Address are both set to the IP address
 of the cache entry to be updated, and the ARP Sender Hardware
 Address is set to the link-layer address to which this cache
 entry should be updated.  When using an ARP Reply packet, the
 Target Hardware Address is also set to the link-layer address to
 which this cache entry should be updated (this field is not used
 in an ARP Request packet).

So clearly both gratuitous ARP “flavors”, REQUEST and REPLY, are defined by the standard. There should be no excuse for the kernel to handle valid gratuitous REPLY packets in any other way than REQUESTs. To fix the wrongdoing, I posted a patch that makes the kernel to honor gratuitous REPLYs the same way as it does REQUESTs. (The patch is now merged in netdev master.)

Even though the kernel fix landed and will probably be part of the next 4.12 release, OpenStack Neutron still needs to deal with the situation somehow for the sake of older kernel, so it probably makes sense to issue REQUESTs from Neutron L3 agents to help those who rely on arp_accept while using an official kernel release. The only question is, should we issue both REQUESTs and REPLYs, or just REQUESTs? For Linux network peers, REQUESTs work just fine, but is there a risk that some other networking software stack honors REPLYs but not REQUESTs?.. To stay on safe side, we decided to issue both.

Anyhow, we discussed before that arp_accept is not for everyone, and there is a good reason why it’s not enabled by default. OpenStack should work irrespective of the sysctl knob value set on other network hosts, that’s why the patches mentioned above couldn’t be considered a final solution.

Besides, arp_accept only disabled locktime mechanism for gratuitous ARP packets, but we haven’t seen any ARP packets before the first gratuitous packet arrived. So why hasn’t the kernel honored it anyway?

Locktime gets in the way of gratuitous ARP packets

As we’ve already mentioned, without arp_accept enforcement, neigh_update hasn’t touched the MAC address for the corresponding ARP table entry. Code inspection suggested that the only case when it could happen was if arp_process passes flags=0 and not flags=NEIGH_UPDATE_F_OVERRIDE into neigh_update. And in the RHEL 7.3 kernel, the only possibility for that to happen is when all three gratuitous ARP replies would arrive in locktime time interval.

But they are sent with a 1-second interval between them, and the default locktime value is 1 second too, so at least the last, or even the second packet in the 3-set should have affected the kernel. Why hasn’t it?..

Let’s look again at how we determine whether an update arrived during the locktime:

override = time_after(jiffies, n->updated + n->parms->locktime);

What the quoted code does is it checks whether an ARP packet is received in [n->updated; n->updated + n->params->locktime] time interval, where n->params->locktime = 100. And what does n->updated represent?

If override is false, call to neigh_update will bail out without updating the MAC address or the ARP entry state. But see what it does just before bailing out: it sets n->updated to current time!

So what happens is that the first gratuitous ARP in the 3-series arrives in locktime interval; it calls neigh_update with flags=0 that updates n->updated and bails out. By moving n->updated forward, it also effectively moves forward the locktime window without actually handling a single frame that would justify that! The next time the second gratuitous ARP packet arrives, it’s again in the locktime window, so it again calls neigh_update with flags=0, which again moves the locktime window forward, and bails out. The exact same story happens for the third gratuitous ARP packet we send from Neutron L3 agent.

So where are we at the end of the scenario? The ARP entry never changed its MAC address to reflect what we advertised with gratuitous ARP replies, and the kernel networking stack is not aware of the change.

This moving window business didn’t seem right to me. There is a reason for locktime, but its effect should not take longer than its value (1 second), and this was clearly not what we’ve seen. So I poked the kernel a bit more and came up with a patch that avoids updating n->updated if neither entry state nor its MAC address would change on neigh_update. With the patch applied to RHEL kernel, I was able to pass previously failing test runs without setting arp_accept to 1. Great, seems like now I had a proper fix!

(The patch is merged, and will be part of the next kernel release. And in case you care, here is the bug for RHEL kernel to fix the unfortunate scenario.)

But why would kernel even handle gratuitous ARP differently for existing ARP table entries depending on arp_accept value? The sysctl setting was initially designed to only control behavior when an ARP packet for a previously unseen IP address was processed. So why the difference? In all honesty, there is no reason. It’s just a bug that sneaked into the kernel in the past. We figured it makes sense to fix it while we are at it, so I posted another kernel patch (this required some reshuffling and code optimization, hence the patch series). With the patch applied, all gratuitous ARP packets will now always update existing entries. (And yes, the patch is also merged and will be part of the next release.)

Of course, a careful reader may wonder why locktime even considers entry state transitions and not just actual ARP packets that are received on the wire, gratuitous or not. That’s a fair question, and I believe that the answer here is, “that’s another kernel bug”. That being said, brief look at the kernel code suggests that it won’t be too easy to make it work the way it should. It would require major rework of kernel neigh subsystem to make it track state transitions independently of MAC/IP transitions. I figured I better leave it to later (also known as never).

How do ARP table states work?

So at this point it seems like I have a set of solutions for the problem.

But one may ask, why has the very first gratuitous ARP packet arrived during locktime? If we look at the captured traffic, we don’t see any ARP packets before the gratuitous ARP burst.

But it turns out that you can get n->updated bumped even without a single ARP packet received! But how?

The thing is, the neighbour.c state machine will update the timestamp not just when a new ARP packet arrives (which happens through arp_process calling to neigh_update), but also when an entry transitions between states, and state transitions may be triggered from inside the kernel itself.

As we already mentioned before, the failing ARP entry cycles through STALE – DELAY – REACHABLE states. So how do entries transition to DELAY?


As it turned out, the DELAY state is used when an existing STALE entry is consumed by an upper layer protocol. Though it’s STALE, it still can be used to connect to the IP address. What kernel does is, on the first upper layer protocol packet sent using a STALE entry, the entry is switched to DELAY, and a timer is set for +delay_first_probe_time in the future (5 seconds by default). When the timer is fired, the kernel then checks whether any upper layer protocol confirmed the entry as reachable. If it is confirmed, the kernel merely switches the state of the entry to REACHABLE; if it’s not confirmed, the kernel issues an ARP probe and updates its table with the result.

Since we haven’t seen a single probe sent during the failing test run, the working hypothesis became – the entry was always confirmed in those 5 seconds before the ARP probe, so the kernel never needed to send a single packet to refresh the entry.

But what is this confirmation anyway?

ARP confirmation

The thing is, it’s probably not very effective to immediately drop existing ARP entries when they become STALE. In most cases, those IP-to-MAC mappings are still valid even after the aging time: it’s not too often that IP addresses move from one device to another. So it would be not ideal if we would need to repeat ARP learning process each time an entry expires (each minute by default). It would be even worse if we would need to pause all other connections to an IP address whenever an ARP entry currently in use becomes STALE, to wait until the ARP table is updated. Since upper layer protocols (TCP, UDP, SCTP, …) may already successfully communicate with the IP address, we can use their knowledge about host availability and avoid unneeded probes, connectivity flips and pauses.

For that matter, Linux kernel has the confirmation mechanism. A lot of upper layer protocols support it, among those are TCP, UDP, and SCTP. Here is a TCP example. Whenever the confirmation aware protocol sees an incoming datagram from the MAC address using the IP address, it confirms the mapping to ARP layer, which then bails out of ARP probing and silently moves the entry to REACHABLE whenever the delay timer fires up.

Scary revelations

And what is this dst that is confirmed by calling to dst_confirm? It’s a pointer to a struct dst_entry. This structure defines a single cached routing entry. I won’t describe in details what it is, and how it’s different from struct fib_info that is an uncached routing entry (better explained in other sources).

What’s important for us to understand is that the entry may be not unique for a particular IP address. As long as outgoing packets take the same routing path, they may share the same dst_entry. And all the traffic directed to the same network subnet reuses the same routing path.

Which means that all the traffic directed from a controller node to the tempest node using any floating IP may potentially “confirm” any of ARP entries that belong to other IP addresses from the same range!

Since tempest tests are executed in parallel, and a lot of them send packets using a confirmation aware upper layer protocol (specifically, TCP for SSH sessions), ARP entries can effectively live throughout the whole test case run cycling through STALE – DELAY – REACHABLE states without issuing a single probe OR receiving any matching traffic for the IP/MAC pair.

And that’s how old MAC addresses proliferate. First we ignore all gratuitous ARP replies because of locktime; then we erroneously confirm wrong ARP entries.

And finally, my Red Hat kernel fellows pointed me to the following patch series that landed in 4.11 (released on May 1, 2017, just while I was investigating the failure). The description of the series and individual patches really hits the nail, for it talks about how dst_entry is shared between sockets, and how we can mistakenly confirm wrong ARP entries because of that.

I tried the patch series out. I reverted all my other kernel patches, then cherry-picked the series (something that is not particularly easy considering RHEL is still largely at 3.10, so for reference I posted the result on github), rebooted the system, and retried tests.

They passed. They passed over and over.

Of course, the node still missed all gratuitous ARP replies because of moving locktime window, but at least the kernel later realized that the entry is broken and requires a new ARP probe, which was correctly issued by the kernel, at which point the reply to the probe healed the cache and allowed tests to pass.

Great, now I had another alternative fix, and it was already part of an official kernel release!

The only problem with backporting the series is that the patches touch some data structures considered part of kernel ABI, so just putting the patches from the series into the package as-is triggered a legit KABI breakage during RPM build. For testing purposes, it was enough to disable the check but if we were going to backport the series into RHEL7, we needed some tweaks to the patches to retain binary compatibility (which is one of RHEL long term support guarantees).

For the reference, I opened a bug against RHEL7 kernel to deal with bogus ARP confirmations. At the time of writing, we hope to see it fixed in RHEL 7.4.

And that’s where we are. A set of kernel bugs combined – some new, some old, some recently fixed – produced the test breakage. If any of those bugs were not present in the environment, tests would have a chance to pass. Only the combination of small kernel mistakes and major screw-ups hit us hard enough to dig deep into the kernel.

In the next final post of the series, we will summarize all the workarounds and solutions that were found as part of the investigation of the Failure. We will also look at a potential OpenStack Neutron only workaround for the Failure that is based on our fresh understanding of the failing scenario.

by ihrachyshka at May 25, 2017 07:46 PM

OpenStack Superuser

OpenStack Verizon case study: The illusion of infinite capacity

The total cost of ownership for an OpenStack private cloud can be half of other infrastructure alternatives.  The key to achieving such a goal is to capture relevant metrics and watch them closely say the team at Verizon Wireless.  At the recent OpenStack Summit Boston, Billy Felton and Andrew Hendrickson of Verizon along with Sanjay Mishra CEO of Talligent offered a look into the platform, its architecture and what’s next as they deal with the challenges of edge computing.

Deploying in hyperscale

Andy Hendrickson, senior manager, Verizon Wireless, offers a look inside the Verizon Cloud Platform (VCP) and shows what’s under the hood, architecture-wise.

“We like to define it as a large-scale cloud. It’s built with OpenStack,” Hendrickson says. He outlines six guiding principles around VCP: Version control, machine readable, no physical configuration, serialization/delivery, declarative and bots.

“VCO is defined by the services, not the tenants. We have to serve multiple customers with this cloud…We need to give our users the ability to quickly deploy and support their applications and multiple lines of business we’re supporting. It’s on-demand and self-service to support the ability to do things in an attended and unattended manner…When we’re deploying at our kind of scale we’re deploying at, we must version control everything,” he adds. Other important features include that everything must be machine readable (documentation, deployment guides), no physical configurations are necessary, and serialization in delivery as much as possible. “This has been a mind shift for us as a deployment organization- declaring everything in advance. So, building out our architectures, declaring those in advance in a place where we can go grab them an actually build them,” he says. The final guiding principal: use bots wherever possible —automation, but also leveraging them to go out and do things and answer questions. “This is a big part of the way that we’re supporting our platform as well,” he says.

Regional architecture

“We haven’t adopted the region zero concept with OpenStack, but  we have adopted regions as our minimum mapping unit,” Hendrickson says. Each box in the slide below is actually a region.

“This current view is just showing the live date – when things are coming online and actually what provider networks we have as part of this. But I can quickly switch persona and look at something like capacity within the same grid.”

“What we’re looking at is information that’s been drawn from live systems, showing the consumption of our virtual CPUs, our memory, and available bandwidth,” he notes. Regions have become a big part of how we deal with things. We like to refer to it as ‘regional diversity.’

The team has multiple versions of OpenStack running at the same time. “This is how we deal with our upgrade and patching strategy. We have a lot of responsibility falling upon our tenants to move workloads between the different versions of OpenStack that are running. We currently have a destructive mode for doing upgrades where we move workloads from one region to another, we destroy a region and we rebuild it. We are messing around with non-destructive ways of doing this as well in terms of leveraging live migration underneath the hood, but that’s something we’re moving slowly into. We’re not beginning there. It’s real data. It’s live.”

“The interesting thing here is anybody can come and get this information. Our illusion of infinite capacity when we’re talking about internal use- there is no illusion. You can see exactly what’s there. You can see how much is left. You see how performant it might be.”

“To solve for this illusion of infinite capacity, we’ve done things like that this that are not necessarily core or part of OpenStack itself. We call them – I don’t know if it’s appropriate, but “second-class citizens” to the platform itself. So we do this in a non-invasive manner so when we deploy and upgrade, we’re not interrupting. We’re not causing a dependency there.”

“This has become paramount for us to be able to support multiple regions, multiple types of orchestration on top.
there’s another unique problem that we needed to solve. Having moved toward that region architecture that I had described was sort of the maintenance for our users of all of the Horizon and services dashboard links and things like that. There’s multiple ways to handle that, but this actually single front end to come in fronting all of those regions that I showed earlier, which by the end of the year will be well over 100. So, this kind of correlates that and collates it together in one places.”

Edging towards the future

Billy Felton, director of technology at Verizon Wireless, offered a look into where they’re headed next. “We’ve really done a pretty good job addressing the perception of infinite capacity. We’ve recognized that we do things differently in a cloud than we’ve done them the legacy networks that we’ve built before.” He admits that the latency aspect of this is “a little bit more of a challenge.”

Mobile-edge computing is “kind of the big buzzword running around” where companies are pushing stuff out to the edge of their networks. “One of the first issues I have is been defining what the edge is. Because technically, the edge is defined by where your consumption is – whether that’s at the actual edge of your network or in the core of your network, so it comes down to where you need workload to run to best suit the consumer of that workload wherever that might be.” To solve the issue, he says, look at the behavior of the user. Is the user going to be driving across town? Is the user going to be something that’s another peered machine-to-machine connection that might be co-located in one of your core locations? And what are the application needs. What type of services? Some services are more sensitive to latency than others. It also makes sense to look at workload needs from the perspective of storage, compute, and what’s needed to support that workload. what it all boils down to is you have a spatial problem as well as a logical problem that you have to correlate.”

Certain applications require very low latency – say 5 milliseconds or 1 millisecond- and it makes sense to push that workload to that location where you can defy the speed of light. “As much as we would enjoy it, we still haven’t figured out how to accelerate packets beyond the speed of light,” Felton quips. “Because we can’t do that, there’s only one thing we can do. That’s to change physics.” That changed the formula — moving the workload to the source where it needs to be, through orchestration and container mechanisms, he says. (For more details, check out Beth Cohen’s keynote presentation.) “It’s a challenge for us to solve this — it involves deploying compute resources, storage resources and SDN network resources at a bunch of locations. It’s definitely not an easy problem to solve.”

It’s one of a series of challenges that Felton says they hope to keep solving with OpenStack and the OpenStack community. “The future is bright for us with OpenStack. It’s a core part of our plan moving forward,” he says.

You can catch the 42-minute talk here.

Cover Photo // CC BY NC

The post OpenStack Verizon case study: The illusion of infinite capacity appeared first on OpenStack Superuser.

by Superuser at May 25, 2017 11:37 AM

May 24, 2017

Ihar Hrachyshka

The Failure, part 2: Beyond OpenStack

This post is the fourth in a series that covers gratuitous ARP and its relation to OpenStack. In the previous posts, we discussed what gratuitous ARP is and how it’s implemented in OpenStack Neutron L3 agent. We also set the stage for the Failure, and dismissed easier causes of it.

This post will continue discussion of the Failure beyond OpenStack components. It is advised that you make yourself comfortable with previous posts in the series before proceeding with reading this “episode”.

To recap, a bunch of tempest connectivity tests were failing in RH-OSP 11 environment when connecting to a floating IP address. Quick log triage suggested that internal connectivity for affected instances worked fine, and that Neutron L3 agent called to arping tool at the right time. Still “undercloud” node (the node executing tempest) got timeout failures.


Before diving deeper, let’s make a step back and explore the deployment more closely. How are those floating IP addresses even supposed to work in this particular setup? Is the tempest node on the same L2 network segment with the floating IP range, or is it connected to it via a L3 router?

The failing jobs deploy Red Hat OpenStack using Director also known as TripleO. It’s not a very easy task to deploy a cloud using bare TripleO, and for this reason there are several tools that prepare development and testing environments. One of those tools is tripleo-quickstart, more popular among upstream developers, while another one is Infrared, more popular among CI engineers. The failing jobs were all deployed with Infrared.

Infrared is a very powerful tool. I won’t get into details, but to give you a taste of it, it supports multiple compute providers allowing to deploy the cloud in libvirt or on top of another OpenStack cloud. It can deploy both RH-OSP and RDO onto provisioned nodes. It can use different installers (TripleO, packstack, …). It can also execute tempest tests for you, collect logs, configure SSH tunnels to provisioned nodes to access them directly… Lots of other cool features come with it. You can find more details about the tool in its documentation.

As I said, the failing jobs were all deployed using Infrared on top of a powerful remote libvirt hypervisor where a bunch of nodes with distinct roles were created:

  • a single “undercloud” node that is used to provision the actual “overcloud” multinode setup (this node also runs tempest);
  • three “overcloud” controllers all hosting Neutron L3 agents;
  • a single “overcloud” compute node hosting Nova instances.

All the nodes were running as KVM guests on the same hypervisor, connected to each other with multiple isolated libvirt networks, each carrying a distinct type of traffic. After Infrared deployed “undercloud” and “overcloud” nodes, it also executed “neutron” set of tests that contains both api and scenario tests, from both tempest and neutron trees.

As I already mentioned, the “undercloud” node is the one that also executes tempest tests. This node is connected to an external network that hosts all floating IP addresses for the preconfigured public network, with eth2 of the node directly plugged into it. The virtual interface is consequently plugged into the hypervisor external Linux kernel bridge, where all other external (eth2) interfaces for all controllers are plugged too.

What it means for us is that the tempest node is on the same network segment with all floating IP addresses and gateway ports of all Neutron routers. There is no router between the floating IP network and the tempest node. Whenever a tempest test case attempts to establish an SSH connection to a floating IP address, it first consults the local ARP table to possibly find an appropriate IP-to-MAC mapping there, and if the mapping is missing, it will use the regular ARP procedure to retrieve the mapping.

Now that we understand the deployment, let’s jump into the rabbit hole.

Traffic capture

The initial debugging steps on OpenStack Neutron side haven’t revealed anything useful. We identified that Neutron L3 agent correctly called to arping with the right arguments. So in the end, maybe it’s not OpenStack fault?

First thing to check would be to determine whether arping actually sent gratuitous ARP packets, and that they reached the tempest node, at which point we could expect that the “undercloud” kernel would honor them and update its ARP table. I figured it’s easier to only capture external (eth2) traffic on the tempest node. I expected to see those gratuitous ARP packets there, which would mean that the kernel received (and processed) them.

Once I got my hands on hardware capable of standing up the needed TripleO installation, I quickly deployed the needed topology using Infrared.

Of course, it’s a lot easier to capture the traffic on tempest node but analyze it later in Wireshark. So that’s what I did.

$ sudo tcpdump -i eth2 -w external.pcap

I also decided it may be worth capturing ARP table state during a failed test run.

$ while true; do date >> arptable; ip neigh >> arptable; sleep 1; done

And finally, I executed the tests. After 40 minutes of waiting for results, one of tests failed with the expected SSH timeout error. Good, time to load the .pcap into Wireshark.

The failing IP address was, so I used the following expression to filter the relevant traffic:

ip.addr == or arp.dst.proto_ipv4 ==

The result looked like this:


Here, we see the following: first, SSH session is started (frame 181869), it of course initially fails (frame 181947) because gratuitous ARP packets start to arrive later (frames 181910, 182023, 182110). But then for some reason consequent TCP packets are still sent using the old 52:54:00:ac:fb:f4 destination MAC address. It seemed like arriving gratuitous ARP packets were completely ignored by the kernel. More importantly, the node continued sending TCP packets to using the old MAC address even past expected aging time (60 seconds), and it never issued a single ARP REQUEST packet to update its cache throughout the whole test case execution (!). Eventually, after ~5 minutes of banging the wrong door the test case failed with a SSH timeout.

How could it happen? The kernel is supposed to honor the new MAC address right after it receives an ARP packet!

Now, if we would compare the traffic dump to a successful test case run, we could see the traffic capture that looked more like below (in this case, the IP address of interest is


Here we see TCP retransmission of SSH, port 22, packets (74573 and 74678), which failed to deliver because the kernel didn’t know the new fa:16:3e:bd:c1:97 MAC address just yet. Later, we see a burst of gratuitous ARP packets sent from the router serving, advertising the new MAC address (frames 7471574801, and 74804). Though it doesn’t immediately suggest that these were gratuitous ARP packets that healed the connectivity, it’s clear that the tempest node quickly learned about the new MAC address and continued with its SSH session (frames 74805 and forward).

One thing that I noticed while looking at multiple traffic captures from different test runs is that whenever a test failed, it always failed on a reused floating IP address. Those would show up in Wireshark with the following warning message:


State transitions

Then maybe there was a difference in ARP entry state between successful and failing runs? Looking at the ARP table state dumps I collected during a failing run, the following could be said:

  • Before the start of the failed test case, the corresponding ARP entry was in STALE state.
  • Around the time when gratuitous ARP packets were received and the first TCP packet was sent to the failing IP address, the entry transitioned to DELAY state.
  • After 5 seconds, it transitioned to REACHABLE without changing its MAC address. No ARP REQUEST packets were issued in between.
  • The same STALE – DELAY – REACHABLE transitions were happening for the affected IP address over and over. The tempest node hasn’t issued a single ARP REQUEST during the whole test case execution. Neither it received any new traffic that would use the old MAC address.


If we compare this to ARP entries in “good” runs, we see that there they also start in STALE state, then transition to DELAY, but after 5 seconds, instead of transitioning to REACHABLE, it switched to PROBE state. The node then issued a single ARP REQUEST (could be seen in the captured traffic dump), quickly received a reply from the correct Neutron router, updated the ARP entry with the new MAC address, and only then finally transitioned to REACHABLE. At this point the connectivity healed itself.


What made the node behave differently in those two seemingly identical cases? Why hasn’t it issued a ARP probe in the first case? What are those ARP table states anyway? I figured it was time to put my kernel hat on and read some Linux code.

In the next post in the series, I am going to cover my journey into kernel ARP layer. Stay tuned.


by ihrachyshka at May 24, 2017 04:30 PM

Major Hayden

Enable AppArmor on a Debian Jessie cloud image

Armor from the middle ages makes sense when you're talking about AppArmor, right?I merged some initial Debian support into the openstack-ansible-security role and ran into an issue enabling AppArmor. The apparmor service failed to start and I found this output in the system journal:

kernel: AppArmor: AppArmor disabled by boot time parameter

Digging in

That was unexpected. I was using the Debian jessie cloud image and it uses extlinux as the bootloader. The file didn’t reference AppArmor at all:

# cat /boot/extlinux/extlinux.conf 
default linux
timeout 1
label linux
kernel boot/vmlinuz-3.16.0-4-amd64
append initrd=boot/initrd.img-3.16.0-4-amd64 root=/dev/vda1 console=tty0 console=ttyS0,115200 ro quiet

I learned that AppArmor is disabled by default in Debian unless you explicitly enable it. In contrast, SELinux is enabled unless you turn it off. To make matters worse, Debian’s cloud image doesn’t have any facilities or scripts to automatically update the extlinux configuration file when new kernels are installed.

Making a repeatable fix

My two goals here were to:

  1. Ensure AppArmor is enabled on the next boot
  2. Ensure that AppArmor remains enabled when new kernels are installed

The first step is to install grub2:

apt-get -y install grub2

During the installation, a package configuration window will appear that asks about where grub should be installed. I selected /dev/vda from the list and waited for apt to finish the package installation.

The next step is to edit /etc/default/grub and add in the AppArmor configuration. Adjust the GRUB_CMDLINE_LINUX_DEFAULT line to look like the one below:

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor"

Ensure that the required AppArmor packages are installed:

apt-get -y install apparmor apparmor-profiles apparmor-utils

Enable the AppArmor service upon reboot:

systemctl enable apparmor

Run update-grub and reboot. After the reboot, run apparmor_status and you should see lots of AppArmor profiles loaded:

# apparmor_status 
apparmor module is loaded.
38 profiles are loaded.
3 profiles are in enforce mode.
35 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Final thoughts

I’m still unsure about why AppArmor is disabled by default. There aren’t that many profiles shipped by default (38 on my freshly installed jessie system versus 417 SELinux policies in Fedora 25) and many of them affect services that wouldn’t cause significant disruptions on the system.

There is a discussion that ended last year around how to automate the AppArmor enablement process when the AppArmor packages are installed. This would be a great first step to make the process easier, but it would probably make more sense to take the step of enabling it by default.

Photo credit: Max Pixel

The post Enable AppArmor on a Debian Jessie cloud image appeared first on

by Major Hayden at May 24, 2017 04:14 PM

OpenStack Superuser

Shuttleworth on what it takes for open source startups to succeed now

—SAN FRANCISCO When Mark Shuttleworth founded Ubuntu, he says that being open source felt like the right thing at the time, if “a little bit rebellious.”  Now, he says, it’s the way we expect things to be done.

Shuttleworth shared his thoughts at a recent event about the “explosive growth” of open source startups organized by the Cloud Native Computing Foundation (CNCF).  With some 15-years hindsight, he has a couple of observations for how startups can harness open source and hit the bottom line.

“What surprises me to a certain extent is the gap between open source and what I think of as entrepreneurship and innovation,” he told a group of VCs, developers and community members. “Open source has become a canonical weapon for commoditization — you can commoditize the other guy’s stuff and, in a sense, we all benefit from that. It’s amazing how far that’s spread — all the way to Redmond…” But there’s a gap around startups, he notes, adding that recently there have been more negative than positive stories in the news. RethinkDB, for example, was “phenomenal technology” all done as open source and the company didn’t make it.

Shuttleworth launched a live demo – deploying Kubernetes on bare metal even though it threatened to wake his dogs sleeping in the garage where he keeps the machines — to make a point about simplicity.

“Many of you are innovating at this base/root level – and getting your innovation to be consumable by the enterprise is the really hard part,” he says. “That’s one of the points I’d like to drive home with startups in Silicon Valley: Your technology should be woven in. If you’re innovating at that level (storage, network, etc) if it’s woven in, you remove all the friction traditionally associated with selling to the enterprise.”

Across any kind of substrate cloud – VMs or bare metal – startups can simplify operations for complex enterprise operations so that it becomes consumable, he added. “Hopefully that helps get rid of the friction of enterprise startups in the open source space.”



Cover Photo // CC BY NC

The post Shuttleworth on what it takes for open source startups to succeed now appeared first on OpenStack Superuser.

by Nicole Martinelli at May 24, 2017 11:35 AM


hastexo Helps SUSE Roll Out Open edX-Based Learning Platform for Employee and Partner Training

SUSE, Global Player in Open Source Software, Collaborates with hastexo to leverage OpenStack for Learning Management and Courseware Development

Madrid, Spain (2017 Open edX Conference)

May 24, 2017

Coinciding with the annual gathering of users and developers of the Open edX Platform, the 2017 Open edX Conference (, hastexo is happy to announce that SUSE ( has selected Open edX as the basis of its platform for learning systems management and courseware development.

The announcement comes at a time when hundreds of information technology companies are dealing with the rapidly progressing development of increasingly complex platforms across the industry. Technologies like OpenStack, Ceph, Kubernetes, and many others are inherently built for speed, scale, and cost savings — but the most important challenge that organizations face in relation to these technologies is skill building.

Innovative companies like SUSE realize that skill development and training — of employees and partners alike — must match the pace of technology. As such, organizations are trending toward fully autonomous, self-paced, on-line training, and Open edX is a perfect platform for that purpose.

hastexo, SUSE's partner in this endeavor, is an active contributor to the Open edX project and has developed an open source solution that enables SUSE learners to spin up arbitrarily complex distributed lab environments, on demand, in an OpenStack private or public cloud. SUSE has launched the on-demand platform to both its employees and partners.

About hastexo

hastexo is an independent professional services organization specializing in open source solutions for cloud computing, distributed storage, and self-paced on-line training. The company is heavily involved in the OpenStack, Ceph, and Open edX communities and offers its services world-wide. hastexo currently serves customers on five continents, and continuously expands its customer base. hastexo’s web site is, and its self-paced learning courses are offered from

by hastexo at May 24, 2017 12:00 AM

May 23, 2017

Chris Dent

TC Report 21

With the advent Thierry's weekly status reports1 on the proposals currently under review by the TC and the optionality of the weekly TC meetings, this report becomes less about meeting minutes and more about reporting on the things that crossed my TC radar that seemed important and/or that seemed like they could do with more input.

This week has no TC meeting. The plan is that discussion will occur either asynchronously in mailing list threads on the "opentack-dev" list or in gerrit reviews in the governance project2 or for casual chats use IRC and the #openstack-dev channel3.

Pending Stuff

The need to talk about postgreSQL

There's ongoing discussion about how to deal with the position of postgreSQL in the attention of the community. There are deployments that use it and the documentation mentions it, but the attention of most developers and all tests is not upon it. It is upon MySQL (and its variants) instead.

There's agreement that this needs to be dealt with, but the degree of change is debated, if not hotly then at least verbosely. An initial review was posted proposing we clear up the document and indicate a path forward that recognized an existing MySQL orientation:

I felt this was too wordy, too MySQL oriented, and left out an important step: agitate with the board. It was easier to explain this in an alternative version resulting in:

Meanwhile discussion had begun (and still continues) in an email thread:

Observing all this, Monty noticed that there is a philosophical chasm that must be bridged before we can truly resolve this issue, so he started yet another thread:

The outcome of that thread and these resolutions is likely to have a fairly significant impact on how we think about managing dependent services in OpenStack. There's a lot to digest behind those links but on the scale of "stuff the TC is doing that will have impact" this is probably one of them.

Draft Vision for the TC

The draft vision for the TC4 got feedback on the review, via survey5 and at the forum6. Effort is now in progress to incorporate that feedback and create something that is easier to comprehend and will make the actual vision more clear. One common bit of feedback was that the document needs a preamble and other structural cues so that people get what it is trying to do. johnthetubaguy, dtroyer and I (cdent) are on the hook for doing this next phase of work. Feel free to contact one of us (or leave a comment on the review, or send some email) if you feel like you have something to add.

Dropped Stuff

A section with reminders of things that were happening or were going to happen then either stopped without resolution or never started in the first place.

OpenStack moving too fast and too slow

A thread was started on this7. It got huge. While there were many subtopics, one of the larger ones was the desire for there to be a long term support release. There were a few different reactions to this, inaccurately paraphrased as:

  • That we have any stable releases at all in the upstream is pretty amazing, some global projects don't bother, it's usually a downstream problem.
  • Great idea, please provide some of the resources required to make it happen, the OpenStack community is not an unlimited supply of free labor.

Then summit happened, people moved on to other things and there wasn't much in the way of resolution. Is there anything we could or should be doing here?

If having LTS is that much of a big deal, then it is something which the Foundation Board of Directors must be convinced is a priority. Early in this process I had suggested we at least write a resolution that repeats (in nicer form) the second bullet point above. We could do that. There's also a new plan to create a top 5 help wanted list8. Doing LTS is probably too big for that, but "stable branch reviews" is not.

by Chris Dent at May 23, 2017 08:00 PM


RDO Contributor Survey

We recently ran a contributor survey in the RDO community, and while the participation was fairly small (21 respondants), there's a lot of important insight we can glean from it.

First, and unsurprisingly:

Of the 20 people who answered the "corporate affiliation" question, 18 were Red Hat employees. While we are already aware that this is a place where we need to improve, it's good to know just how much room for improvement there is. What's useful here will be figuring out why people outside of Red Hat are not participating more. This is touched on in later questions.

Next, we have the frequency of contributions:

Here we see that while 14% of our contributors are pretty much working on RDO all the time, the majority of contributors only touch it a few times per release - probably updating a single package, or addressing a single bug, for that particular cycle.

This, too, is mostly in line with what we expected. With most of the RDO pipeline being automated, there's little that most participants would need to do beyond a handful of updates each release. Meanwhile, a core team works on the infrastructure and the tools every week to keep it all moving.

We asked contributors where they participate:

Most of the contributors - 75% - indicate that they are involved in packaging. (Respondants could choose more than one area in which they participate.) Test day participation was a distant second place (35%), followed by documentation (25%) and end user support (25%)

I've personally seen way more people than that participate in end user support, on the IRC channel, mailing list, and Possibly these people don't think of what they're doing as support, but it is still a very important way that we grow our user community.

The rest of the survey delves into deeper details about the contribution process.

When asked about the ease of contribution, 80% said that it was ok, with just 10% saying that the contribution process was too hard.

When asked about difficulties encountered in the contribution process:

Answers were split fairly evenly between "Confusing or outdated documentation", "Complexity of process", and "Lack of documentation". Encouragingly, "lack of community support" placed far behind these other responses.

It sounds like we have a need to update the documentation, and greatly clarify it. Having a first-time contributor's view of the documentation, and what unwarranted assumptions it makes, would be very beneficial in this area.

When asked how these difficulties were overcome, 60% responded that they got help on IRC, 15% indicated that they just kept at it until they figured it out, and another 15% indicated that they gave up and focused their attentions elsewhere.

Asked for general comments about the contribution process, almost all comments focused on the documentation - it's confusing, outdated, and lacks useful examples. A number of people complained about the way that the process seems to change almost every time they come to contribute. Remember: Most of our contributors only touch RDO once or twice a release, and they feel that they have to learn the process from scratch every time. Finally, two people complained that the review process for changes is too slow, perhaps due to the small number of reviewers.

I'll be sharing the full responses on the RDO-List mailing list later today.

Thank you to everyone who participated in the survey. Your insight is greatly appreciated, and we hope to use it to improve the contributor process in the future.

by Rich Bowen at May 23, 2017 04:25 PM

Ihar Hrachyshka

The Failure, part 1: It’s OpenStack fault

I’m not actually good at computers I’m just bad at giving up. // mjg59

This post is the third in a series that covers gratuitous ARP and its relation to OpenStack. In the previous posts, we discussed what gratuitous ARP is, and how it’s implemented in OpenStack Neutron L3 agent.

This post is short and merely sets the stage for a story that started a month ago inside Red Hat Networking team and that ultimately led me into learning how ARP actually works in OpenStack and in Linux. In next posts, I will expand on the story and discuss possible solutions to the problem that I will introduce below. (The plan is to post all four remaining pieces in next four days.)

But let’s not get dragged too far.

The stage

Failure mode

It was two months before the next shiny Ocata-based release of Red Hat OpenStack Platform (RH-OSP 11). Now that the team focus shifted from upstream development to polishing the product, we started looking more closely at downstream CI jobs. As usual with new releases, there were several failures in our tempest jobs. For most of them we figured out a possible culprit and landed fixes. For most of them, except The Failure.

Well, it was not a single test case that was failing, more like a whole class of them. In those affected test jobs, we execute all tempest tests, both api and scenario, and what we noticed is that a lot of scenario test cases were failing on connectivity checks when using a floating IP (but, importantly, never a fixed IP). Only positive connectivity checks were failing (meaning, cases where connectivity was expected but failed; never a case where lack of connectivity was expected).

There are two types of connectivity checks in tempest: ping check and SSH check. The former sends ICMP (or ICMPv6) datagrams to an IP address under test for 120 seconds and expects a single reply, while the latter establishes a SSH session to the IP address and waits for successful authentication.

In the failing jobs, whenever a ping check failed, the following could be seen in the tempest log file:

2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager Traceback (most recent call last):
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager   File "/usr/lib/python2.7/site-packages/tempest/scenario/", line 624, in check_public_network_connectivity
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager     mtu=mtu)
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager   File "/usr/lib/python2.7/site-packages/tempest/scenario/", line 607, in check_vm_connectivity
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager   File "/usr/lib/python2.7/site-packages/unittest2/", line 666, in fail
2017-05-06 00:02:32.563 3467 ERROR tempest.scenario.manager     raise self.failureException(msg)

When it was a SSH check that failed, then the error looked a bit different:

2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh Traceback (most recent call last):
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh   File "/usr/lib/python2.7/site-packages/tempest/lib/common/", line 107, in _get_ssh_connection
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh     sock=proxy_chan)
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh   File "/usr/lib/python2.7/site-packages/paramiko/", line 305, in connect
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh     retry_on_signal(lambda: sock.connect(addr))
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh   File "/usr/lib/python2.7/site-packages/paramiko/", line 269, in retry_on_signal
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh     return function()
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh   File "/usr/lib/python2.7/site-packages/paramiko/", line 305, in
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh     retry_on_signal(lambda: sock.connect(addr))
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh   File "/usr/lib64/python2.7/", line 224, in meth
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh     return getattr(self._sock,name)(*args)
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh timeout: timed out
2017-05-05 21:29:41.996 4456 ERROR tempest.lib.common.ssh
2017-05-05 21:29:41.997 4456 ERROR tempest.scenario.manager [-] (TestGettingAddress:test_multi_prefix_dhcpv6_stateless) Initializing SSH connection to failed. Error: Connection to the via SSH timed out.

If you would pick a single test case that sometimes failed, it didn’t have a high risk to impact job results, but once you aggregate all failures from all affected test cases, the chance of successfully passing the whole test run became abysmal, around 10%, which was clearly not ideal.

So I figured I may have a look at that, naively assuming it will take a day or two to find the root cause, fix it and move on with my life. Boy I was wrong! It took me a month to get to the bottom of it (though in all honesty, most of the time was spent trying to setup environment that would consistently reproduce the issue).

First steps

Initially, I figured failures are most often happening in our L3 HA jobs, so I focused on one of those. Reading through tempest, neutron-server, neutron-openvswitch-agent, and neutron-l3-agent log files hasn’t revealed much.

When looking at a particular test failure, we could see that the instance that was carrying the failing floating IP successfully booted and received a DHCPv4 lease for its fixed IP, as seen in its console log that tempest gladly dumps on connectivity check failures:

Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending select for
Lease of obtained, lease time 86400

To  cross-check, we could also find the relevant lease allocation event in the system journal:

May 06 06:02:57 controller-1.localdomain dnsmasq-dhcp[655233]: DHCPDISCOVER(tap7876deff-b8) fa:16:3e:cc:2c:30
May 06 06:02:57 controller-1.localdomain dnsmasq-dhcp[655233]: DHCPOFFER(tap7876deff-b8) fa:16:3e:cc:2c:30

The tempest log file clearly suggested that the failure was not because of a SSH key pair misbehaving for the failing instance. If that would be a public SSH key pair not deployed to the instance, we would not see SSH timeouts but authentication failures. Neither we saw any SSH timeouts for tests that established SSH sessions using internal fixed IP addresses of instances.

It all suggested that internal (“tenant”) network connectivity worked fine. The problem was probably isolated somewhere in Neutron L3 agent. But looking into Neutron L3 agent and neutron-server logs hasn’t surfaced any problem either: we could easily find relevant arping calls in Neutron L3 agent (for L3 legacy routers) or the system journal (for Neutron L3 HA routers).

Of course, you never believe that a legit fault may be a result of software that is external to your immediate expertise. It’s never compiler or other development tool fault; 99% of whatever you hit each day is probably your fault. And in my particular case, it was, obviously, OpenStack Neutron that was guilty. So it took me awhile to start looking at other places.

But after two weeks of unproductive code and log reading and adding debug statements in Neutron code, it was finally time to move forward to unknown places.

In the next post of the series, we will explore what I revealed once I shifted my accusatory gaze towards platform residing below OpenStack.

by ihrachyshka at May 23, 2017 03:53 PM

OpenStack Superuser

K8s or die: Skating into the future

When it comes to containers, you need to master the right moves.
There’s Kubernetes this way, that way — and then there’s the “skater way,” say Ryan Beisner and Marco Ceppi. Both have deployment experience on clouds and Kubes and “all sorts of fun things on different architectures,” Beisner on the OpenStack engineering team at Canonical and Ceppi currently at TheSilphRoad. Riffing off the classic 80s video game “Skate or Die!” the pair explored different ways to kick it, flip it and ride it home with OpenStack in their talk at the recent Summit.

Come see @ryanbeisner and I talk about about “K8S or DIE”: When, why how, or if you even should use Kubernetes.

— Marco Ceppi (@marcoceppi) May 10, 2017

Why Kubernetes?

Kubernetes is surprisingly — and shockingly — just software, Ceppi says. “It’s nothing special. It’s not a magic hat. It’s not a cape that you can wrap around your neck and fly off into the free world proclaiming, ‘DevOps is done!’”

Kubernetes is software and it’s actually from an architecture perspective since Kubernetes is a containers coordinator, “what you’re really getting is actually the abstraction of those machine resource- those VM primitives- those things where Kubernetes is running, you’re getting an abstraction for those for your Docker containers. And what it really amounts to is effectively giving you the abstraction of compute, networking, and storage for Docker-style containers,” Ceppi adds.
This sounds strikingly familiar to a couple other projects you’re probably familiar with. Kubernetes is a platform, providing that mechanism to run containers, where you say “I’m going to run them here. They’re going to be on this network. They’re going to have potentially this storage attached, and they’re going to be running on these classes of machines or these types of processors with these compute resources.”
Because Docker containers are, generally speaking, a small surface area (a single process running on a virtually immutable disk with some networking) it’s a platform that grows a lot of capabilities that were tougher in previous generations of computing, doing things like rollout and rollbacks of applications. “While absolutely it’s something you could do today on traditional VMs, but it took quite a lot of time to produce the tooling to get to that place,” Ceppi says.
In addition to that, you get things like load balancing, self healing, service discovery…Because it’s so small and small strongly defined as a surface area, it’s easy for platforms to start building these primitives where previously it was quite difficult.
“What’s gotten a lot of buzz around things like Kubernetes and these platforms is because they have these features took arguably a long time how to solve, and there are still some areas where they can improve with operating traditional-style VMs,” he says.

Kubernetes architecture 101

It breaks down to a few components, Ceppi notes. There’s Kubernetes core software itself, etcd (its data-backing store) and some form of SSL/TSL certificates. Those components are basically arranged in the slide below.


There’s a master control plane service an API server, that you or your clients or users talk to. There’s a scheduler that balances workloads and auto healing and auto scaling. Then there’s a controller manager to coordinate all of the different moving pieces inside a deployment. Finally, there are workers.

“This is where you’re actually running your workloads, and there you’re running a thing called “Kubelets.” You’re usually running a Kube proxy,” Ceppi explains. Your container runtime, Docker, Rocket, OCID, many others are supported now. There’s also some sort of networking SDN, and a CNI- container networking interface- compatible component, like Flannel, like Calico, Weave and a bunch of others as well.

That’s typically what an architecture Kubernetes deployment looks like. “Because a lot of this stuff is Dockerized/containerized, it lends itself to running in a very converged fashion,” Ceppi says. “A lot of times you’ll see with really just one machine, you can run a single Kubernetes’s cluster, but really with three machines, you can run a high availability cluster where you’re scheduling workloads. You’re also running the API control plane services, and the data store services.”

Kubernetes + OpenStack = 50-50 grindscreen-shot-2017-05-22-at-11-00-22

Ceppi and Beisner outlined a few killer cases for K8 and OpenStack:

⁃ You have an existing OpenStack. You have a production cluster, perhaps you have a DevCloud, maybe you have both staging and production…Your tenants can certainly stand up Kubernetes and have individual clusters that they consume. “If I go to Amazon to deploy a Kubernetes cluster, I basically just do it on top of OpenStack, as if I had a private cloud,” Ceppi says.
⁃ You’ve got an existing OpenStack, you have some spare bare metal and you stand up your cluster of Kubernetes alongside OpenStack, Beisner says.
⁃ You are heavy into data science, love GPUs, you’re clamoring for more metal. Dockerizing on Kubernetes on bare metal probably provides access to virtualized hardware.
⁃ You’ve got metal. You’ve got Kubernetes sitting on that metal. That Kubernetes cluster may be delivering multiple applications for your company. One of those applications might be the OpenStack control plane or other OpenStack services. Again, maybe your tenants want their own Kubernetes cluster. Dockerizing the OpenStack control plane and other services, and distributing that and managing it with Kubernetes is “gaining traction,” Beisner says.

How to get there

Ceppi and Beisner then outline the projects that can help you achieve get there – whether it’s you’re exploring or ready to go into production.screen-shot-2017-05-22-at-11-00-46

In your toolbox, they advise having Heat templates, and Charms, Kubeadm, and Kargo. “All of those speak the OpenStack API natively, so if you’re looking to turn up a Kubernetes cluster, those are kind of upstream Kubernetes ways of tackling that problem,” Ceppi says.screen-shot-2017-05-22-at-11-00-36

The state of stateful, stateless and the future

I was always very happy to say, “You know what? The future is containers,” Ceppi says. “But there’s a different story here now. It’s not Kubernetes versus OpenStack, but it’s really about architecture — whether it’s stateless or stateful.

‘Stateful’ is a pretty nebulous term. What does it actually mean to be something stateful? ’Stateless’ is pretty clear, but stateful is a really awkward term. There’s definitely a generation divide between software being built. [For example,] Cockroachdb was designed to be resilient out of the box. It was designed to do clustering, peering — it’s designed for cloud native architecture. An architecture where if one machine goes down, it’s expected to happen. So, while there’s a consistent state present across all those nodes, no one machine or VMm or Docker container has state that will, if it goes missing, will break the cluster. So, it’s a distributed stateful application and because of that, it actually survives quite well and thrives as we saw in the Interop challenge inside of a Dockerized containers/Kubernetes deployment.

Where we start to see the divide is not just all stateful applications as a general blanket statement that you can’t, or shouldn’t, run them inside Kubernetes. It’s about judging which application can I run in Kubernetes? Which application should I be running in Kubernetes? A lot of times there’s a good mix of things where developers want to go really quickly, and they have an application that suits that architecture. But you’ve also got database operators and you’ve general sys admins who want to handcraft VMs and that they manage those VMs. And that they do things to persist that data a infrastructure level- whether it’s snapshotting that data, whether it’s making sure it’s backed up and robust, or running multiple services like cron jobs and other things on that. I think that’s fine to sit in a VM — that’s where you’ll find the intersection of Kubernetes-infrastructure management.

“Because of that, in the next 5-6 years, we’ll see a nice heterogeneous deployment of things managing VMs and things managing Docker containers. Then who knows by then. We’ll hit server-less? We’ll probably hit quantum containers by then as well, so the whole world will change,” Ceppi says. “It will be an ever-evolving landscape.”

You can catch the 39-minute talk here and download the slides here.

Cover Photo // CC BY NC

The post K8s or die: Skating into the future appeared first on OpenStack Superuser.

by Superuser at May 23, 2017 11:10 AM

Adam Young

Fixing Bug 96869

Bug 968696

The word Admin is used all over the place. To administer was originally something servants did to their masters. In one of the greater inversions of linguistic history, we now use Admin as a way to indicate authority. In OpenStack, the admin role is used for almost all operations that are reserved for someone with a higher level of authority. These actions are not expected to be performed by people with the plebean Member role.

Global versus Scoped

We have some objects that are global, and some that are scoped to projects. Global objects are typically things used to run the cloud, such as the set of hypervisor machines that Nova knows about. Everyday members are not allowed to “Enable Scheduling For A Compute Service” via the HTTP Call PUT /os-services/enable.

Keystone does not have a way to do global roles. All roles are scoped to a project. This by itself is not a problem. The problem is that a resource like a hypervisor does not have a project associated with it. If keystone can only hand out tokens scoped to projects, there is still no way to match the scoped token to the unscoped resource.

So, what Nova and many other services do is just look for the Role. And thus our bug. How do we go about fixing this?

Use cases

Let me see if I can show this.

In our initial state, we have two users.  Annie is the cloud admin, responsible for maintaining the over all infrastructure, such as “Enable Scheduling For A Compute Service”.  Pablo is a project manager. As such, he has to do admin level things, but only with his project, such as setting the Metadata used for servers inside this project.  Both operations are currently protected by the “admin” role.

Role Assignments

Lets look at the role assignment object diagram.  For this discussion, we are going to assume everything is inside a domain called “Default” which I will leave out of the diagrams to simplify them.

In both cases, our users are explicitly assigned roles on a project: Annie has the Admin role on the Infra project, and Pablo has the Admin role on the Devel project.


The API call to Add Hypervisor only checks the role on the token, and enforces that it must be “Admin.”  Thus, both Pablo and Annie’s scoped tokens will pass the policy check for the Add Hypervisor call.

How do we fix this?

Scope everything

Lets assume, for the moment, that we were able instantly run a migration that added a project_id to every database table that holds a resource, and to every API that manages those resources.  What would we use to populate that project_id?  What value would we give it?

Lets say we add an admin project value to Keystone.  When a new admin-level resource is made, it gets assigned to this admin project.  All of those resources we have already should get this value, too. How would we communicate this project ID?  We don’t have a keystone instance available when running the Nova Database migrations.

Turns out Nova does not need to know the actual project_id.  Nova just needs to know that Keystone considers the token valid for global resources.

Admin Projects

We’ve added a couple values to the Keystone configuration file: admin_domain_name and admin_project_name.  These two values are how Keystone specifies which project is represents and admin project.  When these two values are set, all token validation responses contain a value for is_admin_project.  If the project requested matches the domain and project name, that value is True, otherwise false.


instead, we want the create_cell call to use a different rule.  Instead of the scope check performed by admin_or_owner, it should confirm the admin role, as it did before, and also that the token has the is_admin_project Flag set.


Keystone already has support for setting is_admin_project, but none of the remote service are honoring it yet. Why?  In part because, in order for it to make sense for one to do so, they all must do so.  But also, because we cannot predict what project would be the admin project.

If we select a project based on name (e.g. Admin) we might be selecting a project that does not exist.

If we force that project to exist, we still do not know what users to assign to it.  We would have effectively broken their cloud, as no users could execute Global admin level tasks.

In the long run, the trick is to provide a transition plan for when the configuration options are unset.]

The Hack

If no admin project is set, then every project is admin project.  This is enforced by oslo-context, which is used in policy enforcement.

Yeah, that seems surprising, but tt turns out that we have just codified what every deployment has already.  Look ad the bug description again:

Problem: Granting a user an “admin” role on ANY tenant grants them unlimited “admin”-ness throughout the system because there is no differentiation between a scoped “admin”-ness and a global “admin”-ness.

Adding in the field is a necessary per-cursor to solving it, but the real problem is in the enforcement in Nova, Glance, and Cinder.  Until they enforce on the flag, the bug still exists.

Fixing things

There is a phased plan to fix things.

  1. enable the is_admin_project mechanism in Keystone but leave it disabled.
  2. Add is_admin_project enforcement in the policy file for all of the services
  3. Enable an actual admin_project in devstack and Tempest
  4. After a few releases, when we are sure that people are using admin_project, remove the hack from oslo-context.

This plan was discussed and agreed upon by the policy team within Keystone, and vetted by several of the developers in the other projects, but it seems it was never fully disseminated, and thus the patches have sat in a barely reviewed state for a long while…over half a year.  Meanwhile, the developers focused on this have shifted tasks.

Now’s The Time

We’ve got a renewed effort, and some new, energetic developers committed to making this happen.  The changes have been rewritten with advice from earlier code reviews and resubmitted.  This bug has been around for a long time: Bug #968696 was reported by Gabriel Hurley on 2012-03-29.  Its been a hard task to come up with and execute a plan to solve it.  If you are a core project reviewer, please look for the reviews for your project, or, even better, talk with us on IRC (Freenode #openstack-keystone) and help us figure out how to best adjust the default policy for your service. 


by Adam Young at May 23, 2017 03:47 AM

May 22, 2017

Chris Dent

Ocata Multi-Node Devstack Play

One of my main concerns these days with OpenStack development, as a culture, is how infrequently some developers, notably me, actually do anything remotely real with OpenStack. This is a clear indicator (to me) of two issues we could work to resolve:

  • the corporate sponsorship style of OpenStack development can lead to some developers who are over-tasked but under-experienced making design and development decisions that may be divorced from reality

  • it can sometimes require too much effort to play with OpenStack in a concrete yet exploratory fashion

Or maybe that's just me.

In any case, I decided today was a day where I should do some multinode devstack as a bit of a refresher and should write what I did down for future reference.

Doing this is not to suggest that devstack counts as real OpenStack, but it does provide an opportunity to play and have visibility of many of the working parts.

I chose a blog posting that google revealed to me as my starting point: Devstack multinode with neutron and vagrant is from a few years ago, but provides a reasonable conceptual framework to get the ball rolling. The intent here is not to automate the process, but to observe it (of course that observation makes it clear why people want to automate it).

I'm working from a 16GB Mac Mini with VirtualBox and Vagrant. The following steps are not meant to be authoritative, they just happen to be what I used that manged to work. There may very well be better ways to do things. This is often the case. Please let me if you have ideas.

Step One: Have Some VMs

Following the model in Ajaya's blog post, we build two VMs from a Vagrantfile. (There's an additional Vagrantfile which automates the building devstack parts described below, but not the network adjustments.) One as a controller + compute, one as a compute node only. The third host is not required, at least the way my VirtualBox is set up.

$ vagrant up
$ vagrant ssh controller

Step Two: Get the Controller Hosting

Get devstack and stack a reasonable basic local.conf. I disable horizon, cinder, tempest and dstat because we don't need those things for these experiments.

$ git clone
$ cd devstack
$ <copy local.conf here>
$ ./

One way to check that things are up in a sane state is to see if there are appropriate inventories reported by the resource tracker:

$ echo "select * from inventories;" | mysql -t nova_api

Step Two and a Half: Diddle the Networking

(This is the part that I for some reason think should "just work" but manual intervention is required. Much of it is non-obvious if you aren't experienced in this area or if you are in a hurry and don't want to concern yourself with these details. Unfortunately by not doing so you often end up with stuff that doesn't work for no apparent reason.)

We created this host with two NICs, we need to get the second one involved as our external network. In these modern times the name of that interface isn't predictable so:

$ ip -a -4 address |grep -B 1 '200.10' # find the ethernet interface
$ sudo ovs-vsctl add-port br-ex enp0s9 # let ovs know about it

Then we need to add some security rules, but need to add them to the default security group for the project that we are currently in. What that is depends on how you've authenticated yourself to your devstack cloud. I always do . openrc admin admin, which may not be most secure but gives me the most capabilities. It means my "project" is "admin". The other common option in devstack is "demo".

We need to find the id of the security group for the project:

$ SECURITY_GROUP=$(openstack security group list \
    --project admin -f value -c ID)

and then add rules for ping and ssh:

$ openstack security group rule create --proto icmp \
    --dst-port 0 $SECURITY_GROUP
$ openstack security group rule create --proto tcp \
    --dst-port 22 $SECURITY_GROUP

Devstack created a private-subnet and a public-subnet. Let's boot a vm on the private-subnet to make sure that stuff is working:

$ IMAGE=$(openstack image list -f value -c Name) \
    # there's only one image in default devstack these days
$ NET_ID=$(openstack subnet show private-subnet -f value -c network_id)
$ openstack server create --flavor c1 --image $IMAGE \
    --nic net-id=$NET_ID x1

Check that it is running:

$ openstack server show x1 # remember the assigned internal IP address

Add a floating IP:

$ openstack port list # retrieve the port id of that address
$ openstack floating ip create public \
    # create a floating ip on the public net
$ openstack floating ip set \
    --port 39c9b1fc-39e8-42e0-97aa-5a84bdfd550a

The controller host currently has no route to that IP, but we should be able to see it from the physical host:

$ ssh cirros@ # password is `cubswin:)`

That ought to work.

(From a placement standpoint you can do echo "select * from allocations;" | mysql -t nova_api and gaze upon the allocations that have been made for the instance you created. If you used the c1 flavor there will be no disk allocations.)

Step Three: Add the Compute Node

Now that we have a working mini-cloud, we can add an additional compute node. From the physical host:

$ vagrant ssh compute1

Here were going to set up a simpler devstack which uses the controller as its service host. Thus we need a different local.conf:

$ git clone
$ cd devstack
$ <copy local.conf here>
$ ./

When this is done, if all went well there are a few ways to know that it had some degree of success. From the controller node:

$ echo 'select * from resource_providers;' |mysql -t nova_api \
    # will have two resource providers
$ nova service-list # should list two nova-compute
$ nova hypervisor-list # this lists one but it should be two!

cells v2 is now default in OpenStack so we need an additional step (also on the controller node):

$ nova-manage cell_v2 discover_hosts
$ nova hypervisor-list # now it's two!

Now let's launch multiple new instances. Depending on how many you've already started, you may hit a quota problem here. If so, lower the number for min and max:

$ openstack server create --flavor c1 --image $IMAGE \
    --nic net-id=$NET_ID --min 9 --max 9 m

At this point neutron will spank your controller host, especially if memory is tight. Once things have quieted down you can list those hosts that ended up on the new compute node:

$ openstack server list --host compute1

And we can repeat the floating ip dance from above to make one of those hosts remotely reachable:

$ openstack port list \
    # retrieve the port id of one of the hosts
$ openstack floating ip create public \
    # create a floating ip on the public net
$ openstack floating ip set \
    --port d1daa7ab-3f2d-432d-977e-5bd8d2d4337b

From the physical host we can ssh to it:

$ ssh cirros@

But there's still one thing that's not working. The VMs hosted by the devstack can't reach out to the rest of the internet. Were I on a Linux physical host this is relatively straightforward to fix with an iptables rule that allows SNAT but since I'm on OS X at the moment it's complicated enough that I'll leave that as an exercise for some other time (or some generous soul might explain in the comments).

You can find the files I used for this, including a Vagrantfile that automates the devstack build parts on GitHub.

by Chris Dent at May 22, 2017 03:30 PM


Recent blog posts - May 22nd

Here's some of the recent blog posts from our community:

Some lessons an IT department can learn from OpenStack by jpena

I have spent a lot of my professional career working as an IT Consultant/Architect. In those positions, you talk to many customers with different backgrounds, and see companies that run their IT in many different ways. Back in 2014, I joined the OpenStack Engineering team at Red Hat, and started being involved with the OpenStack community. And guess what, I found yet another way of managing IT.


When is it not cool to add a new OpenStack configuration option? by assafmuller

Adding new configuration options has a cost, and makes already complex projects (Hi Neutron!) even more so. Double so when we speak of architecture choices, it means that we have to test and document all permutations. Of course, we don’t always do that, nor do we test all interactions between deployment options and other advanced features, leaving users with fun surprises. With some projects seeing an increased rotation of contributors, we’re seeing wastelands of unmaintained code behind left behind, increasing the importance of being strategic about introducing new complexity.


Running (and recording) fully automated GUI tests in the cloud by Matthieu Huin

The problem Software Factory is a full-stack software development platform: it hosts repositories, a bug tracker and CI/CD pipelines. It is the engine behind RDO's CI pipeline, but it is also very versatile and suited for all kinds of software projects. Also, I happen to be one of Software Factory's main contributors. :)


by Rich Bowen at May 22, 2017 02:37 PM

Some lessons an IT department can learn from OpenStack

I have spent a lot of my professional career working as an IT Consultant/Architect. In those positions, you talk to many customers with different backgrounds, and see companies that run their IT in many different ways. Back in 2014, I joined the OpenStack Engineering team at Red Hat, and started being involved with the OpenStack community. And guess what, I found yet another way of managing IT.

These last 3 years have taught me a lot about how to efficiently run an IT infrastructure at scale, and what's better, they proved that many of the concepts I had been previously preaching to customers (automate, automate, automate!) are not only viable, but also required to handle ever-growing requirements with a limited team and budget.

So, would you like to know what I have learnt so far in this 3-year journey?


The OpenStack community relies on several processes to develop a cloud operating system. Most of these processes have evolved over time, and together they allow a very large contributor base to collaborate effectively. Also, we need to manage a complex infrastructure to support this our processes.

  • Infrastructure as code: there are several important servers in the OpenStack infrastructure, providing service to thousands of users every day: the Git repositories, the Gerrit code review infrastructure, the CI bits, etc. The deployment and configuration of all those pieces is automated, as you would expect, and the Puppet modules and Ansible playbooks used to do so are available at their Git repository. There can be no snowflakes, no "this server requires a very specific configuration, so I have to log on and do it manually" excuses. If it cannot be automated, it is not efficient enough. Also, storing our infrastructure definitions as code allows us to take changes through peer-review and CI before applying in production. More about that later.

  • Development practices: each OpenStack project follows the same structure:

    • There is a Project Team Leader (PTL), elected from the project contributors every six months. A PTL acts as a project coordinator, rather than a manager in the traditional sense, and is usually expected to rotate every few cycles.
    • There are several core reviewers, people with enough knowledge on the project to judge if a change is correct or not.
    • And then we have multiple project contributors, who can create patches and peer-review other people's patches.

    Whenever a patch is created, it is sent to review using a code review system, and then:

    • It is checked by multiple CI jobs, that ensure the patch is not breaking any existing functionality.
    • It is reviewed by other contributors.

    Peer review is done by core reviewers and other project contributors. Each of them have the rights to provide different votes:

    • A +2 vote can only be set by a core reviewer, and means that the code looks ok for that core reviewer, and he/she thinks it can be merged as-is.
    • Any project contributor can set a +1 or -1 vote. +1 means "code looks ok to me" while -1 means "this code needs some adjustments". A vote by itself does not provide a lot of feedback, so it is expanded by some comments on what should be changed, if needed.
    • A -2 vote can only be set by a core reviewer, and means that the code cannot be merged until that vote is lifted. -2 votes can be caused by code that goes against some of the project design goals, or just because the project is currently in feature freeze and the patch has to wait for a while.

    When the patch passes all CI jobs, and received enough +2 votes from the core reviewers (usually two), it goes through another round of CI jobs and is finally merged in the repository.

    This may seem as a complex process, but it has several advantages:

    • It ensures a certain level of quality on the master branch, since we have to ensure that CI jobs are passing.
    • It encourages peer reviews, so code should always be checked by more than one person before merging.
    • It engages core reviewers, because they need to have enough knowledge of the project codebase to decide if a patch deserves a +2 vote.
  • Use the cloud: it would not make much sense to develop a cloud operating system if we could not use the cloud ourselves, would it? As expected, all the OpenStack infrastructure is hosted in OpenStack-based clouds donated by different companies. Since the infrastructure deployment and configuration is automated, it is quite easy to manage in a cloud environment. And as we will see later, it is also a perfect match for our continuous integration processes.

  • Automated continuous integration: this is a key part of the development process in the OpenStack community. Each month, 5000 to 8000 commits are reviewed in all the OpenStack projects. This requires a large degree of automation in testing, otherwise it would not be possible to review all those patches manually.

    • Each project defines a number of CI jobs, covering unit and integration tests. These projects are defined as code using Jenkins Job Builder, and reviewed just like any other code contribution.
    • For each commit:
      • Our CI automation tooling will spawn short-lived VMs in one of the OpenStack-based clouds, and add them to the test pool
      • The CI jobs will be executed on those short-lived VMs, and the test results will be fed back as part of the code review
      • The VM will be deleted at the end of the CI job execution

    This process, together with the requirement for CI jobs to pass before merging any code, minimizes the amount of regressions in our codebase.

  • Use (and contribute to) Open Source: one of the "Four Opens" that drive the OpenStack community is Open Source. As such, all of the development and infrastructure processes happen using Open Source software. And not just that, the OpenStack community has created several libraries and applications with great potential for reuse outside the OpenStack use case. Applications like Zuul and nodepool, general-purpose libraries like pbr, or the contributions to the SQLAlchemy library are good examples of this.


So, which tools do we use to make all of this happen? As stated above, the OpenStack community relies on several open source tools to do its work:

  • Infrastructure as code
    • Git to store the infrastructure definitions
    • Puppet and Ansible as configuration management and orchestration tools
  • Development
    • Git as a code repository
    • Gerrit as a code review and repository management tool
    • Etherpad as a collaborative editing tool
  • Continuous integration
    • Zuul as an orchestrator of the gate checks
    • Nodepool to automate the creation and deletion of short-lived VMs for CI jobs across multiple clouds
    • Jenkins to execute CI jobs (actually, it has now been replaced by Zuul itself)
    • Jenkins Job Builder as a tool to define CI jobs as code

Replicating this outside OpenStack

It is perfectly possible to replicate this model outside the OpenStack community. We use it in RDO, too! Although we are very closely related to OpenStack, we have our own infrastructure and tools, following a very similar process for development and infrastructure maintenance.

We use an integrated solution, SoftwareFactory, which includes most of the common tools described earlier (and then some other interesting ones). This allows us to simplify our toolset and have:

  • Infrastructure as code
  • Development and continuous integration
    •, our SoftwareFactory instance, to integrate our development and CI workflow
    • Our own RDO Cloud as an infrastructure provider

You can do it, too

Implementing this way of working in an established organization is probably a non-straightforward task. It requires your IT department and application owners to become as cloud-conscious as possible, reduce the amount of micro-managed systems to a minimum, and establish a whole new way of managing your development… But the results speak for themselves, and the OpenStack community (also RDO!) is a proof that it works.

by jpena at May 22, 2017 01:33 PM

OpenStack Superuser

A checklist for securing OpenStack networking

Cloud security is currently top of mind for IT.

In this post, I’ll provide a 10-point checklist to help you secure your OpenStack Neutron networking service. For starters, ensure that you use “defense in depth” as the primary security strategy and deploy a series of defensive mechanisms, as no single method exists for successfully protecting your entire service. This strategy ensures that even if one mechanism is compromised, another will be there to defend against the attack. Design network security in layers. So, instead of creating a single layer of network security protection at the perimeter, apply network security to every layer of the network, every component of the networking service and the communication channel

Here’s the security checklist. Do keep in mind that this is not a complete list of every possible defensive mechanism that you could employ. The purpose is to provide some key security checks that you could use.

  1. Are all interactions with the networking service isolated into security domains with proper network segmentation?
  2. Does your ML2 mechanism driver mitigate ARP spoofing?
  3. Have you considered the pros and cons of supporting various tenant network types such as Flat, VLAN, VXLAN etc.?
  4. Have you hardened the Host OS, vSwitch, Message Queue and SQL Database?
  5. Have you patched all reported Neutron security vulnerabilities?
  6. Are you using neutron security-groups and enabled port-security?
  7. Are all communications using SSL encryption?
  8. Has API access been secured using role-based access control (RBAC) by using the concept of least privilege?
  9. Have you investigated the maturity and security features of the various pluggable neutron components used?
  10. Are you using quotas to limit project resources?
  11. For more in-depth information, check out my 38-minute talk on this topic at recent Boston OpenStack Summit.

    I value your feedback. Feel free to comment and share your thoughts on this topic.


    Naveen Joy, who calls himself a “Neutron hacker,”  is also technical leader at Cisco Systems in the chief technology and architecture office. This post first appeared on Cisco’s Cloud Blog.

    Superuser is always interested in community content, contact

    The post A checklist for securing OpenStack networking appeared first on OpenStack Superuser.

by Superuser at May 22, 2017 11:07 AM

James Page

Ubuntu OpenStack Dev Summary – 22nd May 2017

Welcome to the first ever Ubuntu OpenStack development summary!

This summary is intended to be a regular communication of activities and plans happening in and around Ubuntu OpenStack, covering but not limited to the distribution and deployment of OpenStack on Ubuntu.

If there is something that you would like to see covered in future summaries, or you have general feedback on content please feel free to reach out to me (jamespage on Freenode IRC) or any of the OpenStack Engineering team at Canonical!

OpenStack Distribution

Stable Releases

Ceph 10.2.7 for Xenial, Yakkety, Zesty and Trusty-Mitaka UCA:

Open vSwitch updates (2.5.2 and 2.6.1) for Xenial and Yakkety plus associated UCA pockets:

Point releases for Horizon (9.1.2) and Keystone (9.3.0) for Xenial and Trusty-Mitaka UCA:

And the current set of OpenStack Newton point releases have just entered testing:

Development Release

OpenStack Pike b1 is available in Xenial-Pike UCA (working through proposed testing in Artful).

Open vSwitch 2.7.0 is available in Artful and Xenial-Pike UCA.

Expect some focus on development previews for Ceph Luminous (the next stable release) for Artful and the Xenial-Pike UCA in the next month.

OpenStack Snaps

Progress on producing snap packages for OpenStack components continues; snaps for glance, keystone, nova, neutron and nova-hypervisor are available in the snap store in the edge channel – for example:

sudo snap install --edge --classic keystone

Snaps are currently Ocata aligned; once the team have a set of snaps that we’re all comfortable are a good base, we’ll be working towards publication of snaps across tracks for OpenStack Ocata and OpenStack Pike as well as expanding the scope of projects covered with snap packages.

The edge channel for each track will contain the tip of the associated branch for each OpenStack project, with the beta, candidate and release channels being reserved for released versions. These three channels will be used to drive the CI process for validation of snap updates. This should result in an experience something like:

sudo snap install --classic --channel=ocata/stable keystone


sudo snap install --classic --channel=pike/edge keystone

As the snaps mature, the team will be focusing on enabling deployment of OpenStack using snaps in the OpenStack Charms (which will support CI/CD testing) and migration from deb based installs to snap based installs.

Nova LXD

Support for different Cinder block device backends for Nova-LXD has landed into driver (and the supporting os-brick library), allowing Ceph Cinder storage backends to be used with LXD containers; this is available in the Pike development release only.

Work on support for new LXD features to allow multiple storage backends to be used is currently in-flight, allowing the driver to use dedicated storage for its LXD instances alongside any use of LXD via other tools on the same servers.

OpenStack Charms

6 monthly release cycle

The OpenStack Charms project is moving to a 6 monthly release cadence (rather than the 3 month cadence we’ve followed for the last few years); This reflects the reduced rate of new features across OpenStack and the charms, and the improved process for backporting fixes to the stable charm set between releases. The next charm release will be in August, aligned with the release of OpenStack Pike and the Xenial Pike UCA.

If you have bugs that you’d like to see backported to the current stable charm set, please tag them with the ‘stable-backport’ tag (and they will pop-up in the right place in Launchpad) – you can see the current stable bug pipeline here.

Ubuntu Artful and OpenStack Pike Support

Required changes into the OpenStack Charms to support deployment of Ubuntu Artful (the current development release) and OpenStack Pike are landing into the development branches for all charms, alongside the release of Pike b1 into Artful and the Xenial-Pike UCA.

You can consume these charms (as always) via the ~openstack-charmers-next team, for example:

juju deploy cs:~openstack-charmers-next/keystone

IRC (and meetings)

You can participate in the OpenStack charm development and discussion by joining the #openstack-charms channel on Freenode IRC; we also have a weekly development meeting in #openstack-meeting-4 at either 1000 UTC (odd weeks) or 1700 UTC (even weeks) – see for more details.


by JavaCruft at May 22, 2017 11:00 AM


Kubernetes Community Report — May 22, 2016

With all the interest in the Kubernetes community we saw at the OpenStack Summit, we want to keep you up to date with all the goings-on.

by Ihor Dvoretskyi at May 22, 2017 05:12 AM

May 21, 2017


Multi-cloud application orchestration on Mirantis Cloud Platform using Spinnaker

Let's look at a hybrid app running Big Data analysis using a combination of components running on bare metal, in virtual machines, and in containers.

by Guest Post at May 21, 2017 07:09 PM

May 19, 2017


Redefining TV Infrastructure: Shifting Media Processing into the Cloud Q&A

While demand for TV services is being disrupted and subscribers are expecting more personalized experiences, operators need to satisfy customer video demands to stay competitive.

by Ryan Day at May 19, 2017 07:07 PM

Assaf Muller

When is it not cool to add a new OpenStack configuration option?

Adding new configuration options has a cost, and makes already complex projects (Hi Neutron!) even more so. Double so when we speak of architecture choices, it means that we have to test and document all permutations. Of course, we don’t always do that, nor do we test all interactions between deployment options and other advanced features, leaving users with fun surprises. With some projects seeing an increased rotation of contributors, we’re seeing wastelands of unmaintained code behind left behind, increasing the importance of being strategic about introducing new complexity.

I categorize the introduction of new OpenStack configuration options to two:

  1. There’s two or more classes of operators that have legitimate use cases and a configuration option would enable those use cases without hurting cloud interoperability
    • For example: Neutron DVR, is essentially a driver for router implementations, that changes your L3 architecture substantially while abstracting details via an API. DVR has various costs and benefits and letting operators make that choice, especially as the feature matures, makes sense to me.
  2. Developers that don’t fully understand the choice they are making and pass the complexity down to the operators to figure out. This results in options that are often never changed from their defaults because operators don’t have access to sufficient documentation that explains the rationale for choosing a specific value, as well as a misuse of time and energy because developers sometimes focus on use cases that are off center or don’t exist.
    • For example: neutron.conf:DEFAULT:send_events_interval: Number of seconds between sending events to nova if there are any events to send. Unless you grep through the code, how is an operator supposed to know if they should increase or decrease the value? Even then, shouldn’t developers take responsibility of that choice, and test for the best value? If problems are found under load, shouldn’t the value be calculated as a function of some variable? Instead of distributing the work to thousands of operators, wouldn’t we as a community like to to do the work in one place?

When contemplating adding a new option, ask yourself:

  1. Is it possible that you don’t fully understand the use case, and in lieu of making a choice, you’re letting the operator bear the burden?
  2. Are you, and your replacement, prepared to own the cost of the new option indefinitely?

by assafmuller at May 19, 2017 06:42 PM

OpenStack Superuser

Why blockchains are good for the cloud, too

Blockchain is one of those buzzword bingo terms that has been as hot as it has overhyped. Unlike some of these words — “cloudbursting” anyone? — after hearing Brian Behlendorf speak a couple of times and talking to him, I’m convinced that distributed-ledger technology deserves its fame.

Behlendorf is the executive director of Hyperledger, an open-source collaborative effort hosted by the Linux Foundation that launched in 2015. Behlendorf calls himself “chief diplomat” of the project, noting that 140 member organizations count on him to manage the budget and the governance around the developer community and vendor community.

He sat down with Superuser at OSCON to talk about the impact of distributed-ledger technology on open source and society as a whole. Here’s an edited version of our conversation.

How did you get involved with blockchain?

For about 10 years, I’ve had this disturbing sense that as the internet matured, more and more governments and businesses digitized all of their services and their systems. That was leading to a greater centralization rather than greater decentralization. Instead the internet started out as a very decentralized thing — people ran their own mail servers, they ran their own WordPress servers and Apache servers. That almost seems archaic now, but it still felt like a good principle:  it created a competitive and highly diverse ecosystem.  It always disturbed me to think about how much we were concentrating on one or a few services. I wondered, was it necessary to actually build something like an Uber or an Airbnb that was actually distributed? Did you have to be centralized?

I started to hear about blockchain around the same time, 2008 – 2009, but didn’t pay much attention to it.  Then I became a VC for a couple of years and started talking to some of these Bitcoin companies — and I was still very skeptical.  It was when I heard about the use case of land titles in emerging markets that I went, “Okay, wait.  There’s something different here.”

When there’s nobody you can trust to run a land title database you can run something like a Github repository, where every change that comes in gets seen by everyone else and that’s where the source of trust comes. That reminds me of open-source software.  That’s how we trust open source software is we know when stuff gets committed to a repo, everyone else can see it, right? Or should be able to see it. (If they’re not looking at it- if you’re a developer at an open source project and you’re not watching the changes, that’s a bad thing.)

So now think about automating that process. Think about a database that inherently has witnesses. Not looking to whether this is good code or not, but simply saying, “Yes, this is transaction #43. This is transaction #44,  #45.”  And if somebody tries to change history and change it, that would be obvious…

That’s what pulled me in. As I’ve started to pull it apart, I asked the question, “Do you need a cryptocurrency to make that work?” Because I was still skeptical…As I started to look at this, I realized that permissioned ledgers, are very much like how the domain name system works. The domain name system is a permission system.  We have registrars who manage them, right? In a permission system, you don’t have to have a cryptocurrency to fuel these consensus mechanisms- these distributed databases.  That’s when I woke up and said, “Okay, I want to know more and I want to be a part of helping make that happen.” The Linux Foundation launched Hyperledger around the same time, so I spent time with Jim Zemlin and others, immersing myself in that...

Sorry, it was a long story!

Where are the main contributions coming from?

IBM is a major contributor to Hyperledger Fabric, which is one of the eight projects. Intel is a major contributor to Hyperledger Sawtooth, which is a modular platform for building, deploying, and running distributed ledgers.  We have a startup called Soramitsu that also brought a project to us based in Japan. They contributed a distributed ledger technology called Hyperledger Iroha.  Another startup called Monax that recently brought an Ethereum-related project called Hyperledger Burrow to the organization.

Everyone of these projects, in our opinion and by our rules, has to become multi-vendor, multi-stakeholder projects.  We’re more than happy to have these major companies participating and involved, but everything they do has to be public. Every touch point has to allow for external- other companies to get engaged. There’s no “We built this inside, and then we pushed it out.” That’s not allowed.  They have to think of themselves as part of a larger thing now.  That takes time to engineer.

For some companies, like for IBM, this is a major technology benefit.  They are taking this to all their customers and saying, “We think you should reinvent how you do this and this through blockchain technology.” So they are investing heavily in it. They’ve got that vision and we’re working on getting that vision on other companies. A couple others have that, but this is a 20-year project.

That’s interesting – so the cycle is to get source code public. How much emphasis do you put on having the open source bits being functional projects?  It’s always hard to draw the line between which part is ready to run in your data center and which in the open-source world?  And how much of the value-added bits are for sale or fund the next cycle of release?

Part of that question is, “What’s the state of maturity of the different projects?” And then the other part is, “How much is it a complete solution to being able to build blockchain applications?”

So, to answer the first, many of the projects are still pre-1.0.  Hyperledger Fabric is on course to be 1.0 sometime in the next few months. Hyperledger Iroha and Hyperledger Sawtooth Lake are also trending towards a 1.0 release of their own.  Some of the projects are actually fairly mature, so we just brought in “Hyperledger Composer”  a visual tool to build blockchain networks. But, it’s still very much early days. This is like 1995 and the World Wide Web again…

There are different points of maturity, but with these systems, you have basically a way to express interesting functionality. There’s still a little bit of end-user interface you’d want to write when you’re talking to average consumers, but as a backend integration technology, this stuff is ready to go. This is not the kind of thing that’s only useful if you’re also using products from a certain vendor or certain other proprietary products. This is stuff that is as independent as the Linux kernel was. Linux kernel is still dependent on a lot of work to be done on top of it, but you could put it into your products and move on.

What’s the most surprising application you have seen so far?

There’s a couple of companies. One just got bought by Spotify that uses blockchain technology as a way to track performance rights in music.  When you play a track on Spotify or something like that, in theory, some fraction of a penny makes its way to the musician, some fraction of a penny makes its way to maybe the composer or other people who played a role in recording it. But these are not as formally tracked as you would imagine because it would actually require that the establishment of a rather large database. Some of the major organizations managing author’s rights, like, ASCAP/BMI are often very criticized because their algorithms for deciding who receives the payout is not very transparent. YouTube is not very transparent about the algorithm and how that maps to how much artists get paid.

So people are experimenting with embedding those rights directly into the works. And recording that in a blockchain – in a distributed ledger, so that anytime somebody is playing a track, it’s easy to look up and see where should whatever payment was derived for the performance of this, where should that be sent to? So that’s intriguing to me, though it’s still in an earlier state.  Something like the diamond-use use case is a lot more advanced…The diamond supply chain is being reinvented today through the use of a blockchain application. And that’s really cool. That’s really fun.

And how does that work?

That’s something that may be surprising to people when they understand this is how we fight the slave labor and conflict-free diamond is by a track and trace system like this. Moving to blockchain will make that so much more resilient- so much more hardened.

Do you see or have you seen anything that couldn’t be tied with a cloud or infrastructure as a service, or-?

Blockchain is a service. It’s definitely a thing…  The idea is that every one of these is a free-standing network. If the three of us decide we have a system of record we want to build, we can either run our own nodes on our own hardware, or you can hire IBM. I can hire Amazon, she could hire Azure. We should be able to run a network between us. The important thing is independent control of these nodes. You have your business. I have my business. She has hers. There’s no one party that can corrupt that.

Now we’re probably also still get those benefits even if we all use Amazon. Because as long as these are independently owned and managed through Amazon, then we probably would have pretty high security. But we’d have the highest security if we knew that this was actually truly distributed.  These are issues that are still being sorted out. We’ll probably have ratings agencies for these different business networks that will tell you how trustworthy is it. And that will be a function of the diversity of the network.  In that scenario, having multiple blockchain-as-a-service providers and not just one will be actually pretty valuable.

Do you feel like the barrier to entry for Hyperledger and, in general, blockchain technology is higher than in the Apache days?

Well, there’s so much more money out there in the world for technology. I mean software has eaten the world, right? Five of the ten largest companies in the world are pure play technology companies. And you could argue the other five actually are.  They just masquerade as a financial firm or something like that.

Or manufacturing.

Yeah. So, everything done.  If we were to say this is like 1995 and the web, well you didn’t see a billion dollars of investment into the web in 1995.

That was probably a good thing.  It probably would have been very awkward to take a billion dollars in at http 1.0. And with Mac Mosaic and the state of play at the time because it would have meant a lot of overlap, a lot of competing efforts.  I think we have competing efforts today partly because of that.  So, separating out from the noise- it’s almost a good thing that it’s so easy to start open source projects now…

Open source is very healthy right now. We’re a very fortunate spot in that the world has kind of moved to the principles we’ve been fighting for for 20 years. It’s not time to go to sleep on those though, it’s time to continue fighting for them.

There’s so much more open-source code available. Being open first seems to be common nowadays, like the Netflix’s approach with Spinnaker of “Okay, let’s see who else has the same problem. Let’s start by collaborating from scratch.” How do you see companies accepting and adopting- or adapting- to the open source development models?

You have to have open source competency skills to survive today. I don’t think there are many companies left that don’t have some major part of their business and revenue dependent upon or engaging with the open source…

Even inside larger companies, there are some teams who understand it very deeply because they’ve been there for 20 years, and other teams for whom this might still be their first.. So there is still a very strong need to continue to explain what the principles are –how you work in an open-source community…

What’s interesting about our membership, if you look at our roster, is we have companies like Airbus. Is Airbus an open-source company?  It’d be hard to say. Or Daimler. But talking to people there internally, they get it. They know that there’s this transformation for how software’s built, and how IP is built, and they need to have much more sophisticated viewpoint on it. They don’t necessarily have the muscle, like the reflexes we have now about to engage, but they want to build those.  So, part of what we do at Hyperledger is help them understand what that means.

Cover photo by: Emily

The post Why blockchains are good for the cloud, too appeared first on OpenStack Superuser.

by Stefano Maffulli at May 19, 2017 11:49 AM

Cisco Cloud Blog

Securing OpenStack Networking

Cloud Security is currently top of mind for IT. In this blog post, I will provide a 10 point checklist to help you secure your OpenStack Neutron networking service. Ensure that you use “Defense in depth” as the primary security strategy and deploy a series of defensive mechanisms, as no single method exists for successfully protecting your entire service. This strategy ensures that even if one mechanism is compromised, another will be there to defend against the attack. Design network security in layers. So, instead of creating a single layer of network security protection at the perimeter, apply network security to every layer of the network, every component of the networking service and the communication channel.

by Naveen Joy at May 19, 2017 11:00 AM


OpenStack Australia Day Melbourne – Agenda Announced

Aptira - OpenStack Australia Day Melbourne

OpenStack Australia Day Melbourne is now less than 2 weeks away – have you registered yet?

The agenda has been announced, and will feature sessions across three simultaneous tracks.

In the business track, you will hear case studies and strategies from the OpenStack Foundation, the Commonwealth Bank of Australia, Hostworks, Puppet and more. The technical track features in-depth sessions and demonstrations from Gigaspaces and Dimension Data just to name a few. The new Innovation track will be showcasing education, research and innovation from organisations including NCTAR, the University of Melbourne and Monash University.

There will be great food (have we already mentioned the donut wall??), great company, and as always there will be networking drinks held in the evening.

Tickets are limited! Register today at:

The post OpenStack Australia Day Melbourne – Agenda Announced appeared first on Aptira Cloud Solutions.

by Jessica Field at May 19, 2017 08:18 AM

Dougal Matthews

How I Gerrit

I use Gerrit every day, and it often feels like I am using it all day too. Over time I have started to develop a workflow that helps me more effectively manage the large volume of OpenStack reviews.

tl;dr - skip the the bottom to see the dashboards I use.

The general approach I take is to prioritise and filter reviews. I start filtering like this:

  1. Look at starred reviews. These are reviews I have manually starred and want to keep track of.
  2. Find patches passing CI and with an existing +2. These should be in good shape and ready for review.
  3. Find patches passing CI but with no reviews. These are waiting for feedback.

After that, it becomes a bit wild-west. I just look at everthing. I want to try and find a better way to group these, but the signal to noise ratio isn't great.


Gerrit has a neat Dashboard feature, that I think isn't very well exposed to users.

That is a very simple example - the dashboard shows you your open patches, then below your closed patches. You can quickly write dashboards like this with some URL hacking.

There seem to be two special query parameters, then the rest are the searches that you want. We have a base URL of and then foreach can be used to apply a search to all searches in the dashboard, this can be useful if you want to limit to a project (or a user, as we did above). title lets you set a title for the dashboard, pretty obvious!

After that you can arbitrary query parameters, where the name of the parameter is the section title and the value is the search itself.

Gerrit Dash Creator

There is a nice Python project that helps you generate dashboards. I don't know all that much about it, I just forked the project, added three dashboards and ran tox to verify them.

The best thing about this is the library of examples you can learn from.

I have three dashboards, one for my patches and one for the two primary projects I am involved in; TripleO and Mistral. Now I have started storing them here, I expect over time I'll end up with a few others.

My Dashboards

As these depend on your account - i.e. patches you have submitted, or if you have starred patches, your results will vary.

  • My Dashboard - mostly patches I have submitted, but also a few others I want
  • TripleO - TripleO patches. Limited to the projects I am core in and with some prioritisation.
  • Mistral - Mistral patches, prioritised by how far they have moved.

My main takeaway from this is using the Gerrit starring feature to track specific reviews. This helps me follow along with features that are important or interesting to me. This has really helped my workflow, as I previously lost or forgot about important reviews in the noise.

by Dougal Matthews at May 19, 2017 07:32 AM

May 18, 2017

NFVPE @ Red Hat

Sailing the 7 seas with Kubernetes Helm

Helm is THE Kubernetes Package Manager. Think of it like a way to yum install your applications, but, in Kubernetes. You might ask, “Heck, a deployment will give me most of what I need. Why don’t I just create a pod spec?” The thing is that Helm will give you a way to template those specs, and give you a workflow for managing those templates by what it calls “charts”. Today we’ll go ahead and step through the process of installing Helm, installing a pre-built pod given an existing chart, and then we’ll make an existing chart of our own and deploy it – and we’ll change some template values so that it runs in a couple different ways. Ready? Anchors aweigh…

by Doug Smith at May 18, 2017 11:30 PM


Introducing Mirantis Cloud Platform — Webinar Q&A

Just before the OpenStack Summit we gave a webinar introducing Mirantis Cloud Platform. Here are the answers to your questions.

by Shaun O'Meara at May 18, 2017 11:01 PM


Running (and recording) fully automated GUI tests in the cloud

The problem

Software Factory is a full-stack software development platform: it hosts repositories, a bug tracker and CI/CD pipelines. It is the engine behind RDO's CI pipeline, but it is also very versatile and suited for all kinds of software projects. Also, I happen to be one of Software Factory's main contributors. :)

Software Factory has many cool features that I won't list here, but among these is a unified web interface that helps navigating through its components. Obviously we want this interface thoroughly tested; ideally within Software Factory's own CI system, which runs on test nodes being provisioned on demand on an OpenStack cloud (If you have read Tristan's previous article, you might already know that Software Factory's nodes are managed and built by Nodepool).

When it comes to testing web GUIs, Selenium is quite ubiquitous because of its many features, among which:

  • it works with most major browsers, on every operating system
  • it has bindings for every major language, making it easy to write GUI tests in your language of choice.¹

¹ Our language of choice, today, will be python.

Due to the very nature of GUI tests, however, it is not easy to fully automate Selenium tests into a CI pipeline:

  • usually these tests are run on dedicated physical machines for each operating system to test, making them choke points and sacrificing resources that could be used somewhere else.
  • a failing test usually means that there is a problem of a graphical nature; if the developer or the QA engineer does not see what happens it is difficult to qualify and solve the problem. Therefore human eyes and validation are still needed to an extent.

Legal issues preventing running Mac OS-based virtual machines on non-Apple hardware aside, it is possible to run Selenium tests on virtual machines without need for a physical display (aka "headless") and also capture what is going on during these tests for later human analysis.

This article will explain how to achieve this on linux-based distributions, more specifically on CentOS.

Running headless (or "Look Ma! No screen!")

The secret here is to install Xvfb (X virtual framebuffer) to emulate a display in memory on our headless machine …

My fellow Software Factory dev team and I have configured Nodepool to provide us with customized images based on CentOS on which to run any kind of jobs. This makes sure that our test nodes are always "fresh", in other words that our test environments are well defined, reproducible at will and not tainted by repeated tests.

The customization occurs through post-install scripts: if you look at our configuration repository, you will find the image we use for our CI tests is sfstack-centos-7 and its customization script is

We added the following commands to this script in order to install the dependencies we need:

sudo yum install -y firefox Xvfb libXfont Xorg jre
sudo mkdir /usr/lib/selenium /var/log/selenium /var/log/Xvfb
sudo wget -O /usr/lib/selenium/selenium-server.jar
sudo pip install selenium```

The dependencies are:

* __Firefox__, the browser on which we will run the GUI tests
* __libXfont__ and __Xorg__ to manage displays
* __Xvfb__
* __JRE__ to run the __selenium server__
* the __python selenium bindings__

Then when the test environment is set up, we start the selenium server and Xvfb
in the background:

/usr/bin/java -jar /usr/lib/selenium/selenium-server.jar -host >/var/log/selenium/selenium.log 2>/var/log/selenium/error.log
Xvfb :99 -ac -screen 0 1920x1080x24 >/var/log/Xvfb/Xvfb.log 2>/var/log/Xvfb/error.log```

Finally, set the display environment variable to :99 (the Xvfb display) and run your tests:

export DISPLAY=:99

The tests will run as if the VM was plugged to a display.

## Taking screenshots

With this headless setup, we can now run GUI tests on virtual machines within our
automated CI; but we need a way to visualize what happens in the GUI if a test

It turns out that the selenium bindings have a screenshot feature that we can use
for that. Here is how to define a decorator in python that will save a screenshot
if a test fails.

import functools
import os
import unittest
from selenium import webdriver


def snapshot_if_failure(func):
    def f(self, *args, **kwargs):
            func(self, *args, **kwargs)
        except Exception as e:
            path = '/tmp/gui/'
            if not os.path.isdir(path):
            screenshot = os.path.join(path, '%s.png' % func.__name__)
            raise e
    return f

class MyGUITests(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()

    def test_login_page(self):

If test_login_page fails, a screenshot of the browser at the time of the exception will be saved under /tmp/gui/test_login_page.png.

Video recording

We can go even further and record a video of the whole testing session, as it turns out that ffmpeg can capture X sessions with the "x11grab" option. This is interesting beyond simply test debugging, as the video can be used to illustrate the use cases that you are testing, for demos or fancy video documentations.

In order to have ffmpeg on your test node, you can either add compilation steps to the node's post-install script or go the easy way and use an external repository:

# install ffmpeg
sudo rpm --import
sudo rpm -Uvh
sudo yum update
sudo yum install -y ffmpeg

To record the Xfvb buffer, you'd simply run bash export FFREPORT=file=/tmp/gui/ffmpeg-$(date +%Y%m%s).log && ffmpeg -f x11grab -video_size 1920x1080 -i$DISPLAY -codec:v mpeg4 -r 16 -vtag xvid -q:v 8 /tmp/gui/tests.avi

The catch is that ffmpeg expects the user to press q to stop the recording and save the video (killing the process will corrupt the video). We can use tmux to save the day; run your GUI tests like so:

export DISPLAY=:99
tmux new-session -d -s guiTestRecording 'export FFREPORT=file=/tmp/gui/ffmpeg-$(date +%Y%m%s).log && ffmpeg -f x11grab -video_size 1920x1080 -i'$DISPLAY' -codec:v mpeg4 -r 16 -vtag xvid -q:v 8 /tmp/gui/tests.avi && sleep 5'
tmux send-keys -t guiTestRecording q

Accessing the artifacts

Nodepool destroys VMs when their job is done in order to free resources (that is, after all, the spirit of the cloud). That means that our pictures and videos will be lost unless they're uploaded to an external storage.

Fortunately Software Factory handles this: predefined publishers can be appended to our jobs definitions; one of which allows to push any artifact to a Swift object store. We can then retrieve our videos and screenshots easily.


With little effort, you can now run your selenium tests on virtual hardware as well to further automate your CI pipeline, while still ensuring human supervision.

Further reading

by Matthieu Huin at May 18, 2017 05:16 PM

IBM OpenTech Team

Cloud Computing for Better Science – Recap of EGI Conference and INDIGO-DataCloud Summit 2017

Miguel Caballer, PhD @micafe77
Alvaro Lopez Garcia, PhD @alvaretas

The European Grid Infrastructure (EGI) Conference 2017 and INDIGO-DataCloud (INtegrating Distributed data Infrastructures for Global ExplOitation) Summit 2017 recently took place in Catania, Italy from May 9-12 2017. I had the honor of being a Keynote speaker, and for me this was the first time attending any EGI or INDIGO event. The co-authors, Dr. Miguel Caballer and Dr. Alvaro Lopez Garcia, are INDIGO lead architects and attending EGI and INDIGO Summits on a regular basis. The in-depth schedule of the event can be found here.

EGI is a federated e-Infrastructure set up to provide advanced computing services for research and innovation. The EGI e-infrastructure is publicly-funded and comprises over 300 data centers and cloud providers spread across Europe and worldwide.

The INDIGO-DataCloud project develops an open source data and computing platform targeted at scientific communities, deployable on multiple hardware and provisioned over hybrid, private or public, e-infrastructures. It brings innovation in computing, storage and network aspects of the cloud. It introduces a cloud middleware framework based on OpenStack and OpenNebula. The INDIGO-DataCloud project is mainly focused on addressing requirements of scientific communities in four domains:

  1. Biological and Medical science
  2. Social science and Humanities
  3. Environmental and Earth science
  4. Physics and Astrophysics

Summary of Summit
The EGI Conference 2017 was the EGI Community’s main event of 2017 and the last meeting organized in the context of the EGI-Engage project. The conference was focused on the technical roadmap of EGI, with dedication to authorization and authentication, compute services to both HTC and cloud, as well as storage and data services and uptake of the services in scientific communities.
The INDIGO Summit 2017 was the flagship event of the INDIGO-DataCloud project, with a focus on user engagement and the INDIGO service catalogue. This event was centered on exploring the solutions provided by the INDIGO software, applying them to concrete use cases brought forward by scientific communities and resource providers. Demos, training and hands-on implementation sessions were also provided at the event.
The event was hosted by INFN-Catania, part of the Italian National Institute for Nuclear Physics and have the support of the Metropolitan City of Catania. The event was powered with Keynote speech from renowned speakers, several technical sessions and poster sessions. It was attended by a few hundred computer scientists and developers from all over the Europe, United States, Canada, Taiwan and South Africa. Most attendees had PhD or MSc degree in Computer Science.

Our Takeaway
We found the event very successful with over 5 Keynote sessions, 32 numbers of technical sessions, 29 numbers of Posters and Panel discussions. The Keynote speakers included Dr. Tiziana Ferrari (Technical Director, EGI Foundation), Dr. Davide Salomoni (Coordinator, INDIGO-DataCloud), Prof. Jesús Marco de Lucas (Work Package Leader, INDIGO-DataCloud), Edit Herczog (Ex-Member of European Parliament), Dr. Jan Korbel (Senior Scientist, EMBL Heidelberg) and Sahdev Pratapsinh Zala (IBM).

It was simply amazing to see how technology especially Cloud Computing is influencing science. Several sessions and demonstration were presented towards it. One of the most impressive sessions was by senior scientist Dr. Jan Korbel titled “Cancer Genomes on the Cloud: The Pan-Cancer Initiative” demonstrating how cloud is making possible advanced research on Cancer. Dr. Korbel also discussed the future of cancer research and how science cloud will play an important role.

In his Keynote speech, Sahdev P. Zala of IBM presented IBM’s leadership and commitment in open source software development. He demonstrated the collaborative efforts that project INDIGO-DataCloud and IBM team worked together to enhance OpenStack TOSCA projects like OpenStack Heat Translator and OpenStack TOSCA Parser. He also demonstrated how INDIGO-DataCloud use those projects in production. Part of his briefing on collaborative efforts were recap of INDIGO team and IBM’s joint sessions at the OpenStack summits in Austin and Barcelona 2017, and a joint submission of research paper in a prestigious Journal of Grid Computing. At the end, he provided an overview of IBM Bluemix Platform.

Dr. Korbel on the future of Cancer research and Sahdev Zala talking on IBM Bluemix.

Dr. Miguel Caballer is an architect of the Infrastructure Manager (IM). The IM is a tool that deploys complex and customized virtual infrastructures on multiple back-ends. It enables the automation of the Virtual Machine Image (VMI) selection, deployment, configuration, software installation, monitoring and update of virtual infrastructures. It supports a wide variety of back-ends, thus making user applications Cloud agnostic. In addition it features DevOps capabilities, based on Ansible to enable the installation and configuration of all the user required applications providing the user with a fully functional infrastructure. The IM is used by INDIGO-DataCloud at two levels: in the PaaS core to access Cloud providers external to the INDIGO-DataCloud project and as the TOSCA orchestration layer of the OpenNebula sites of the INDIGO-DataCloud infrastructure .

Dr. Alvaro Lopez from CSIC is coordinating the development efforts in the compute virtualization area of the INDIGO-DataCloud project, as well as the coordinator of the task responsible for the integration modules needed for the EGI Federated Cloud. Those activities were showcased in a joint session moderated by Alvaro, focused at mature user communities that are able to exploit IaaS resources directly.

Dr. Miguel talking about IM and Dr. Alvaro moderating session.

Dr. Davide Salomoni (coordinator of the INDIGO-DataCloud project) and Dr. Tiziana Ferrari (Technical Director of the EGI Foundation) provided details about progress of INDIGO-DataCloud and EGI respectively. They also provided audience with lots of data about both the projects. It was absolutely impressive to learn how big is EGI infrastructure, as it can be seen in the presentation. EGI currently federates more than 300 HTC centers, more than 20 cloud providers and providing over 2.6 billion CPU hours per year to more than 50000 users distributed worldwide. Professor Jesus Marco of CSIC (work package leader of INDIGO-DataCloud) acted as the moderator for the INDIGO-DataCloud sessions. Prof. Jesus also moderated a panel with seven Panel Experts including Dr. Tiziana Ferrari, Dr. Davide Salomoni and Sahdev Zala.

Panel discussion and a slide from EGI Federation talk.

At this vibrant event, the INDIGO and EGI community demonstrated many initiatives that are going on for a better science. It is clear that INDIGO-DataCloud can play a vital role for a sustainable cloud infrastructure for scientific communities in Europe and beyond. The detail of event and sessions can be found at the conference site. To learn more about event moments visit #egiconf17 and #indigosummit17 on Twitter.

The post Cloud Computing for Better Science – Recap of EGI Conference and INDIGO-DataCloud Summit 2017 appeared first on IBM OpenTech.

by Sahdev Zala at May 18, 2017 05:01 PM

Major Hayden

Fixing OpenStack noVNC consoles that ignore keyboard input

Televideo consoleI opened up a noVNC console to a virtual machine today in my OpenStack cloud but found that the console wouldn’t take keyboard input. The Send Ctrl-Alt-Del button in the top right of the window worked just fine, but I couldn’t type anywhere in the console. This happened on an Ocata OpenStack cloud deployed with OpenStack-Ansible on CentOS 7.

Test the network path

The network path to the console is a little deep for this deployment, but here’s a quick explanation:

  • My laptop connects to HAProxy
  • HAProxy sends the traffic to the nova-novncproxy service
  • nova-novncproxy connects to the correct VNC port on the right hypervisor

If all of that works, I get a working console! I knew the network path was set up correctly because I could see the console in my browser.

My next troubleshooting step was to dump network traffic with tcpdump on the hypervisor itself. I dumped the traffic on port 5900 (which was the VNC port for this particular instance) and watched the output. Whenever I wiggled the mouse over the noVNC console in my browser, I saw a flurry of network traffic. The same thing happened if I punched lots of keys on the keyboard. At this point, it was clear that the keyboard input was making it to the hypervisor, but it wasn’t being handled correctly.

Test the console

Next, I opened up virt-manager, connected to the hypervisor, and opened a connection to the instance. The keyboard input worked fine there. I opened up remmina and connected via plain old VNC. The keyboard input worked fine there, too!

Investigate in the virtual machine

The system journal in the virtual machine had some interesting output:

kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key pressed (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key pressed (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.
kernel: atkbd serio0: Unknown key released (translated set 2, code 0x0 on isa0060/serio0).
kernel: atkbd serio0: Use 'setkeycodes 00 <keycode>' to make it known.

It seems like my keyboard input was being lost in translation — literally. I have a US layout keyboard (Thinkpad X1 Carbon) and the virtual machine was configured with the en-us keymap:

# virsh dumpxml 4 | grep vnc
    <graphics type='vnc' port='5900' autoport='yes' listen='' keymap='en-us'>

A thorough Googling session revealed that it is not recommended to set a keymap for virtual machines in libvirt in most situations. I set the nova_console_keymap variable in /etc/openstack_deploy/user_variables.yml to an empty string:

nova_console_keymap: ''

I redeployed the nova service using the OpenStack-Ansible playbooks:

openstack-ansible os-nova-install.yml

Once that was done, I powered off the virtual machine and powered it back on. (This is needed to ensure that the libvirt changes go into effect for the virtual machine.)

Great success! The keyboard was working in the noVNC console once again!

Photo credit: Wikipedia

The post Fixing OpenStack noVNC consoles that ignore keyboard input appeared first on

by Major Hayden at May 18, 2017 04:58 PM


Steve Hardy: OpenStack TripleO in Ocata, from the OpenStack PTG in Atlanta

Steve Hardy talks about TripleO in the Ocata release, at the Openstack PTG in Atlanta.

Steve: My name is Steve Hardy. I work primarily on the TripleO project, which is an OpenStack deployment project. What makes TripleO interesting is that it uses OpenStack components primarily in order to deploy a production OpenStack cloud. It uses OpenStack Ironic to do bare metal provisioning. It uses Heat orchestration in order to drive the configuration workflow. And we also recently started using Mistral, which is an OpenStack workflow component.

So it's kind of different from some of the other deployment initiatives. And it's a nice feedback loop where we're making use of the OpenStack services in the deployment story, as well as in the deployed cloud.

This last couple of cycles we've been working towards more composability. That basically means allowing operators more flexibility with service placement, and also allowing them to define groups of node in a more flexible way so that you could either specify different configurations - perhaps you have multiple types of hardware for different compute configurations for Nova, or perhaps you want to scale services into particular groups of clusters for particular services.

It's basically about giving more choice and flexibility into how they deploy their architecture.

Rich: Upgrades have long been a pain point. I understand there's some improvement in this cycle there as well?

Steve: Yes. Having delivered composable services and composable roles for the Newton OpenStack release, the next big challenge was giving operators the flexibility to deploy services on arbitrary nodes in your OpenStack environment, you need some way to upgrade, and you can't necessarily make assumptions about which service is running on which group of nodes. So we've implented the new feature which is called composable upgrades. That uses some Heat functionality combined with Ansible tasks, in order to allow very flexible dynamic definition of what upgrade actions need to take place when you're upgrading some specific group of nodes within your environment. That's part of the new Ocata release. It's hopefully going to provide a better upgrade experience, for end-to-end upgrades of all the OpenStack services that TripleO supports.

Rich: It was a very short cycle. Did you get done what you wanted to get done, or are things pushed off to Pike now.

Steve: I think there's a few remaining improvements around operator-driven upgrades, which we'll be looking at during the Pike cycle. It certainly has been a bit of a challenge with the short development timeframe during Ocata. But the architecture has landed, and we've got composable upgrade support for all the services in Heat upstream, so I feel like we've done what we set out to do in this cycle, and there will be further improvements around operator-drive upgrade workflow and also containerization during the Pike timeframe.

Rich: This week we're at the PTG. Have you already had your team meetings, or are they still to come.

Steve: The TripleO team meetings start tomorrow, which is Wednesday. The previous two days have mostly been cross-project discussion. Some of which related to collaborations which may impact TripleO features, some of which was very interesting. But the TripleO schedule starts tomorrow - Wednesday and Thursday. We've got a fairly packed agenda, which is going to focus around - primarily the next steps for upgrades, containerization, and ways that we can potentially collaborate more closely with some of the other deployment projects within the OpenStack community.

Rich: Is Kolla something that TripleO uses to deploy, or is that completely unrelated?

Steve: The two projects are collaborating. Kolla provides a number of components, one of which is container definitions for the OpenStack services themselves, and the containerized TripleO architecture actually consumes those. There are some other pieces which are different between the two projects. We use Heat to orchestrate container deployment, and there's an emphasis on Ansible and Kubernetes on the Kolla side, where we're having discussions around future collaboration.

There's a session planned on our agenda for a meeting between the Kolla Kubernetes folks and TripleO folks to figure out of there's long-term collaboration there. But at the moment there's good collaboration around the container definitions and we just orchestrate deploying those containers.

We'll see what happens in the next couple of days of sessions, and getting on with the work we have planned for Pike.

Rich: Thank you very much.

by Rich Bowen at May 18, 2017 01:33 PM

Recent blog posts

I haven't done an update in a few weeks. Here are some of the blog posts from our community in the last few weeks.

Red Hat joins the DPDK Project by Marcos Garcia - Principal Technical Marketing Manager

Today, the DPDK community announced during the Open Networking Summit that they are moving the project to the Linux Foundation, and creating a new governance structure to enable companies to engage with the project, and pool resources to promote the DPDK community. As a long-time contributor to DPDK, Red Hat is proud to be a founding Gold member of the new DPDK Project initiative under the Linux Foundation.


What’s new in OpenStack Ocata by rbowen

OpenStack Ocata has now been out for a little over a month – – and we’re about to see the first milestone of the Pike release. Past cycles show that now’s about the time when people start looking at the new release to see if they should consider moving to it. So here’s a quick overview of what’s new in this release.


Steve Hardy: OpenStack TripleO in Ocata, from the OpenStack PTG in Atlanta by Rich Bowen

Steve Hardy talks about TripleO in the Ocata release, at the Openstack PTG in Atlanta.


Using a standalone Nodepool service to manage cloud instances by tristanC

Nodepool is a service used by the OpenStack CI team to deploy and manage a pool of devstack images on a cloud server for use in OpenStack project testing.


Red Hat Summit 2017 – Planning your OpenStack labs by Eric D. Schabell

This year in Boston, MA you can attend the Red Hat Summit 2017, the event to get your updates on open source technologies and meet with all the experts you follow throughout the year.


Stephen Finucane - OpenStack Nova - What's new in Ocata by Rich Bowen

At the OpenStack PTG in February, Stephen Finucane speaks about what's new in Nova in the Ocata release of OpenStack.


Zane Bitter - OpenStack Heat, OpenStack PTG, Atlanta by Rich Bowen

At the OpenStack PTG last month, Zane Bitter speaks about his work on OpenStack Heat in the Ocata cycle, and what comes next.


The journey of a new OpenStack service in RDO by amoralej

When new contributors join RDO, they ask for recommendations about how to add new services and help RDO users to adopt it. This post is not a official policy document nor a detailed description about how to carry out some activities, but provides some high level recommendations to newcomers based on what I have learned and observed in the last year working in RDO.


InfraRed: Deploying and Testing Openstack just made easier! by bregman

Deploying and testing OpenStack is very easy. If you read the headline and your eyebrows raised, you are at the right place. I believe that most of us, who experienced at least one deployment of OpenStack, will agree that deploying OpenStack can be a quite frustrating experience. It doesn’t matter if you are using it for […]


by Rich Bowen at May 18, 2017 01:33 PM

Introducing Rain Leander

Dear RDO Enthusiast,

We have some important news this week about what’s shifting in the RDO community.

As you may know, Rich Bowen has been serving in the role of Community Liaison for the last 4 years. In that capacity, he’s done a variety of things for the community, including event coordination, social media, podcasts and videos, managing the website, and so on.

Starting next week, this role is going to include Rain Leander as TripleO Community Liaison. Rain has been working with the RDO community, and, more generally, with Red Hat’s upstream OpenStack development efforts, for the past 18 months. She’s helped out at a number of events, including two OpenStack Summits, numerous OpenStack Days and Meetups. And she’s been a passionate advocate of TripleO in the community at large.

You may have seen her at some of these events and you’ve probably seen her on IRC as leanderthal. Please give her all the support that you’ve given Rich as she moves into this role.

If you have any questions about how this is going to work, what Rain’s priorities will be in the coming months, or concerns about getting stuff done in the next few months, please don’t hesitate to contact either one of us via email ( and, on IRC (leanderthal and rbowen) or via Twitter (@rainleander and @rbowen).


Rich Bowen and Rain Leander

by Rich Bowen at May 18, 2017 01:33 PM

Zane Bitter - OpenStack Heat, OpenStack PTG, Atlanta

At the OpenStack PTG last month, Zane Bitter speaks about his work on OpenStack Heat in the Ocata cycle, and what comes next.

Rich: Tell us who you are and what you work on.

Zane: My name is Zane Bitter, and I work at Red Hat on Heat … mostly on Heat. I'm one of the original Heat developers. I've been working on the project since 2012 when it started.

Heat is the orchestration service for OpenStack. It's about managing how you create and maintain your resources that you're using in your OpenStack cloud over time. It manages dependencies between various things you have to spin up, like servers, volumes, networks, ports, all those kinds of things. It allows you to define in a declarative way what resources you want and it does the job of figuring out how to create them in the right order and do it reasonably efficiently. Not waiting too long between creating stuff, but also making sure you have all the dependencies, in the right order.

And then it can manage those deployments over time as well. If you want to change your thing, it can figure out what you need do to change, if you need to replace a resource, what it needs to do to replace a resource, and get everything pointed to the right things again.

Rich: What is new in Ocata? What have you been working on in this cycle?

Zane: What I've been working on in Ocata is having a way of auto-healing services. If your service dies for some reason, you'd like that to recover by itself, rather than having to page someone and say, hey, my service is down, and then go in there and manually fix things up. So I've been working on integration between a bunch of different services, some of which started during the previous cycle.

I was working with Fei Long Wang from Catalyst IT who is PTL of Zaqar, getting some integration work between Zaqar and Mistral, so you can now trigger a Mistral workflow from a message on the Zaqar queue. So if you set that up as a subscription in Zaqar, it can fire off a thing when it gets a message on that queue, saying, hey, Mistral, run this workflow.

That in turn is integrated with Aodh - ("A.O.D.H". as, some people call it. I'm told the correct pronunciation is Aodh.) - which is the alarming service for OpenStack. It can …

Rich: For some reason, I thought it was an acronym.

Zane: No, it's an Irish name.

Rich: That's good to know.

Zane: Eoghan Glynn was responsible for that one.

You can set up the alarm action for an alarm in Aodh to be to post a message to this queue. When you combine these together, that means that when an alarm goes off, it posts a message to a queue, and that can trigger a workflow.

What I've been working on in Ocata is getting that all packaged up into Heat templates so we have all the resources to create the alarm in Aodh, hook it up with the subscription … hook up the Zaqar queue to a Mistral subscription, and have that all configured in a template along with the workflow action, which is going to call Heat, and say, this server is unhealthy now. We know from external to Heat, we know that this server is bad, and then kick off the action which is to mark the server unhealthy. We then create a replacement, and then when that service is back up, we remove the old one.

Rich: Is that done, or do you still have stuff to do in Pike.

Zane: It's done. It's all working. It's in the Heat templates repository, there's an example in there, so you can try that out. There's a couple caveats. There's a missfeature in Aodh - there's a delay between when you create the alarm and when … there's a short period where, when an event comes in, it may not trigger an alarm. That's one caveat. But other than that, once it's up and working it works pretty reliably.

The other thing I should mention is that you have to turn on event alarms in Aodh, which is basically triggering alarms off of events in the … on the Oslo messaging notification bus, which is not on by default, but it's a one line configuration change.

Rich: What can we look forward to in Pike, or is it too early in the week to say yet?

Zane: We have a few ideas for Pike. I'm planning to work on a template where … so, Zaqar has pre-signed URLs, so you can drop a pre-signed URL into an instance, and allow that instance … node server, in other words … to post to that Zaqar queue without having any Keystone credentials, and basically all it can do with that URL is post to that one queue. Similar to signed URLs in ____. What that should enable us to do is create a template where we're putting signed URLs, with an expiry, into a server, and then we can, before that expires, we can re-create it, so we can have updating credentials, and hook that up to a Mistral subscription, and that allows the service to kick off a Mistral work flow to do something the application needs to do, without having credentials for anything else in OpenStack. So you can let both Mistral and Heat use Keystone trusts, to say, I will offer it on behalf of the user who created this workflow. So if we can allow them to trigger that through Zaqar, there's a pretty secure way of giving applications access to modify stuff in the OpenStack cloud, but locking it down to only the stuff you want modified, and not risking that if someone breaks into your VM, they've got your Keystone credentials and can do whatever they want withour account.

That's one of the things I'm hoping to work on.

As well, we're continuing with Heat development. We've switched over to the new convergence architecture. In Newton, I think, was the first release to have that on by default. We're looking at improving performance with that now. We've got the right architecture for scaling out to a lot of Heat engines. Right now, it's a little heavy on database, a little heavy on memory, which is the tradeoff you make when you go from a monolithic architecture, which can be quite efficient, but doesn't scale out well, to, you scale out but there's potentially performance problems. I think there's some low-hanging fruit there, we should be able to crank up performance. Memory use, and database accesses. Look for better performance out of the convergence architecture in Heat, coming up in Pike.

by Rich Bowen at May 18, 2017 01:33 PM

The journey of a new OpenStack service in RDO

When new contributors join RDO, they ask for recommendations about how to add new services and help RDO users to adopt it. This post is not a official policy document nor a detailed description about how to carry out some activities, but provides some high level recommendations to newcomers based on what I have learned and observed in the last year working in RDO.

Note that you are not required to follow all these steps and even you can have your own ideas about it. If you want to discuss it, let us know your thoughts, we are always open to improvements.

1. Adding the package to RDO

The first step is to add the package(s) to RDO repositories as shown in RDO documentation. This tipically includes the main service package, client library and maybe a package with a plugin for horizon.

In some cases new packages require some general purpose libraries. If they are not in CentOS base channels, RDO imports them from Fedora packages into a dependencies repository. If you need a new dependency which already exists in Fedora, just let us know and we'll import it into the repo. If it doesn't exist, you'll have to add the new package into Fedora following the existing process.

2. Create a puppet module

Although there are multiple deployment tools for OpenStack based on several frameworks, puppet is widely used by different tools or even directly by operators so we recommend to create a puppet module to deploy your new service following the Puppet OpenStack Guide. Once the puppet module is ready, remember to follow the RDO new package process to get it packaged in the repos.

3. Make sure the new service is tested in RDO-CI

As explained in a previous post we run several jobs in RDO CI to validate the content of our repos. Most of the times the first way to get it tested is by adding the new service to one of the puppet-openstack-integration scenarios which is also recommended to get the puppet module tested in upstream gates. An example of how to add a new service into p-o-i is in this review.

4. Adding deployment support in Packstack

If you want to make it easier for RDO users to evaluate a new service, adding it to Packstack is a good idea. Packstack is a puppet-based deployment tool used by RDO users to deploy small proof of concept (PoC) environments to evaluate new services or configurations before deploying it in their production clouds. If you are interested you can take a look to these two reviews which added support for Panko and Magnum in Ocata cycle.

5. Add it to TripleO

TripleO is a powerful OpenStack management tool able to provision and manage cloud environments with production-ready features, as high availability, extended security, etc… Adding support for new services in TripleO will help the users to adopt it for their cloud deployments. The TripleO composable roles tutorial can guide you about how to do it.

6. Build containers for new services

Kolla is the upstream project providing container images and deployment tools to operate OpenStack clouds using container technologies. Kolla supports building images for CentOS distro using binary method which uses packages from RDO. Operators using containers will have it easier it if you add containers for new services.

Other recomendations

Follow OpenStack governance policies

RDO methodology and tooling is conceived according to OpenStack upstream release model, so following policies about release management and requirements is a big help to maintain packages in RDO. It's specially important to create branches and version tags as defined by the releases team.

Making potential users aware of availability of new services or other improvements is a good practice. RDO provides several ways to do this as sending mails to our mailing lists, writing a post in the blog, adding references in our documentation, creating screencast demos, etc… You can also join the RDO weekly meeting to let us know about your work.

Join RDO Test Days

RDO organizes test days at several milestones during each OpenStack release cycle. Although we do Continuous Integration testing in RDO, it's good to test that it can be deployed following the instructions in the documentation. You can propose new services or configurations in the test matrix and add a link to the documented instructions about how to do it.

Upstream documentation

RDO relies on upstream OpenStack Installation Guide for deployment instructions. Keeping it up to date is recommended.

by amoralej at May 18, 2017 01:33 PM

Stephen Finucane - OpenStack Nova - What's new in Ocata

At the OpenStack PTG in February, Stephen Finucane speaks about what's new in Nova in the Ocata release of OpenStack.

Stephen: I'm Stephen Finucane, and I work on Nova for Red Hat.

I've previously worked at Intel. During most of my time working on Nova I've been focused on the same kind of feature set, which is what Intel liked to call EPA - Enhanced Platform Awareness - or NFV applications. Making Nova smarter from the perspective of Telco applications. You have all this amazing hardware, how do you expose that up and take full advantage of that when you're running virtualized applications?

The Ocata cycle was a bit of an odd one for me, and probably for the project itself, because it was really short. The normal cycle runs for about six months. This one ran for about four.

During the Ocata cycle I actually got core status. That was probably as a result of doing a lot of reviews. Lot of reviews, pretty much every waking hour, I had to do reviews. And that was made possible by the fact that I didn't actually get any specs in for that cycle.

So my work on Nova during that cycle was mostly around reviewing Python 3 fixes. It's still very much a community goal to get support in Python 3. 3.5 in this case. Also a lot of work around improving how we do configuration - making it so that administrators can actually understand what different knobs and dials Nova exposes, what they actually mean, and what the implications of changing or enabling them actually are.

Both of these have been going in since before the Ocata cycle, and we made really good progress during the Ocata cycle to continue to get ourselves 70 or 80% of the way there, and in the case of config options, the work is essentially done there at this point.

Outside of that, the community as a whole, most of what went on this cycle was again a continuation of work that has been going on the last couple cycles. A lot of focus on the maturity of Nova. Not so much new features, but improving how we did existing features. A lot of work on resource providers, which are a way that we can keep track of the various resources that Nova's aware of, be they storage, or cpu, or things like that.

Coming forward, as far as Pike goes, it's still very much up in the air. That's what we're here for this week discussing. There would be, from my perspective, a lot of the features that I want to see, doubling down on the NFV functionality that Nova supports. Making things like SR-IOV easier to use, and more performant, where possible. There's also going to be some work around resource providers again for SR-IOV and NFV features and resources that we have.

The other stuff that the community is looking at, pretty much up in the air. The idea of exposing capabilities, something that we've had a lot of discussion about already this week, and I epxect we'll have a lot more. And then, again, evolution of the Nova code base - what more features the community wants, and various customers want - going and providing those.

This promises to be a very exciting cycle, on account of the fact that we're back into the full six month mode. There's a couple of new cores on board, and Nova itself is full steam ahead.

by Rich Bowen at May 18, 2017 01:33 PM

Using a standalone Nodepool service to manage cloud instances

Nodepool is a service used by the OpenStack CI team to deploy and manage a pool of devstack images on a cloud server for use in OpenStack project testing.

This article presents how to use Nodepool to manage cloud instances.


For the purpose of this demonstration, we'll use a CentOS system and the Software Factory distribution to get all the requirements:

sudo yum install -y --nogpgcheck
sudo yum install -y nodepoold nodepool-builder gearmand
sudo -u nodepool ssh-keygen -N '' -f /var/lib/nodepool/.ssh/id_rsa

Note that this installs nodepool version 0.4.0, which relies on Gearman and still supports snapshot based images. More recent versions of Nodepool require a Zookeeper service and only support diskimage builder images. Even though the usage is similar and easy to adapt.


Configure a cloud provider

Nodepool uses os-client-config to define cloud providers and it needs a clouds.yaml file like this:

cat > /var/lib/nodepool/.config/openstack/clouds.yaml <<EOF
      username: "${OS_USERNAME}"
      password: "${OS_PASSWORD}"
      auth_url: "${OS_AUTH_URL}"
    project_name: "${OS_PROJECT_NAME}"
      - "${OS_REGION_NAME}"

Using the OpenStack client, we can verify that the configuration is correct and get the available network names:

sudo -u nodepool env OS_CLOUD=le-cloud openstack network list

Diskimage builder elements

Nodepool uses disk-image-builder to create images locally so that the exact same image can be used across multiple clouds. For this demonstration we'll use a minimal element to setup basic ssh access:

mkdir -p /etc/nodepool/elements/nodepool-minimal/{extra-data.d,install.d}

In extra-data.d, scripts are executed outside of the image and the one bellow is used to authorize ssh access:

cat > /etc/nodepool/elements/nodepool-minimal/extra-data.d/01-user-key <<'EOF'
set -ex
cat /var/lib/nodepool/.ssh/ > $TMP_HOOKS_PATH/
chmod +x /etc/nodepool/elements/nodepool-minimal/extra-data.d/01-user-key

In install.d, scripts are executed inside the image and the following is used to create a user and install the authorized_key file:

cat > /etc/nodepool/elements/nodepool-minimal/install.d/50-jenkins <<'EOF'
set -ex
useradd -m -d /home/jenkins jenkins
mkdir /home/jenkins/.ssh
mv /tmp/in_target.d/ /home/jenkins/.ssh/authorized_keys
chown -R jenkins:jenkins /home/jenkins

# Nodepool expects this dir to exist when it boots slaves.
mkdir /etc/nodepool
chmod 0777 /etc/nodepool
chmod +x /etc/nodepool/elements/nodepool-minimal/install.d/50-jenkins

Note: all the examples in this articles are available in this repository: sf-elements. More information to create elements is available here.

Nodepool configuration

Nodepool main configuration is /etc/nodepool/nodepool.yaml:

elements-dir: /etc/nodepool/elements
images-dir: /var/lib/nodepool/dib

  cleanup: '*/30 * * * *'
  check: '*/15 * * * *'

  - name: default

  - host: localhost

  - name: dib-centos-7
      - centos-minimal
      - vm
      - dhcp-all-interfaces
      - growroot
      - openssh-server
      - nodepool-minimal

  - name: default
    cloud: le-cloud
      - name: centos-7
        diskimage: dib-centos-7
        username: jenkins
        private-key: /var/lib/nodepool/.ssh/id_rsa
        min-ram: 2048
      - name: defaultnet
    max-servers: 10
    boot-timeout: 120
    clean-floating-ips: true
    image-type: raw
    pool: nova
    rate: 10.0

  - name: centos-7
    image: centos-7
    min-ready: 1
      - name: default

Nodepool uses a gearman server to get node requests and to dispatch image rebuild jobs. We'll uses a local gearmand server on localhost. Thus, Nodepool will only respect the min-ready value and it won't dynamically start node.

Diskimages define images' names and dib elements. All the elements provided by dib, such as centos-minimal, are available, here is the full list.

Providers define specific cloud provider settings such as the network name or boot timeout. Lastly, labels define generic names for cloud images to be used by jobs definition.

To sum up, labels reference images in providers that are constructed with disk-image-builder.

Create the first node

Start the services:

sudo systemctl start gearmand nodepool nodepool-builder

Nodepool will automatically initiate the image build, as shown in /var/log/nodepool/nodepool.log: WARNING nodepool.NodePool: Missing disk image centos-7. Image building logs are available in /var/log/nodepool/builder-image.log.

Check the building process:

# nodepool dib-image-list
| ID | Image        | Filename                                      | Version    | State    | Age         |
| 1  | dib-centos-7 | /var/lib/nodepool/dib/dib-centos-7-1490688700 | 1490702806 | building | 00:00:00:05 |

Once the dib image is ready, nodepool will upload the image: nodepool.NodePool: Missing image centos-7 on default When the image fails to build, nodepool will try again indefinitely, look for "after-error" in builder-image.log.

Check the upload process:

# nodepool image-list
| ID | Provider | Image    | Hostname | Version    | Image ID | Server ID | State    | Age         |
| 1  | default  | centos-7 | centos-7 | 1490703207 | None     | None      | building | 00:00:00:43 |

Once the image is ready, nodepool will create an instance nodepool.NodePool: Need to launch 1 centos-7 nodes for default on default:

# nodepool list
| ID | Provider | AZ   | Label    | Target  | Manager | Hostname           | NodeName           | Server ID | IP   | State    | Age         |
| 1  | default  | None | centos-7 | default | None    | centos-7-default-1 | centos-7-default-1 | XXX       | None | building | 00:00:01:37 |

Once the node is ready, you have completed the first part of the process described in this article and the Nodepool service should be working properly. If the node goes directly from the building to the delete state, Nodepool will try to recreate the node indefinitely. Look for errors in nodepool.log. One common mistake is to have an incorrect provider network configuration, you need to set a valid network name in nodepool.yaml.

Nodepool operations

Here is a summary of the most common operations:

  • Force the rebuild of an image: nodepool image-build image-name
  • Force the upload of an image: nodepool image-upload provider-name image-name
  • Delete a node: nodepool delete node-id
  • Delete a local dib image: nodepool dib-image-delete image-id
  • Delete a glance image: nodepool image-delete image-id

Nodepool "check" cron periodically verifies that nodes are available. When a node is shutdown, it will automatically recreate it.

Ready to use application deployment with Nodepool

As a Cloud developper, it is convenient to always have access to a fresh OpenStack deployment for testing purpose. It's easy to break things and it takes time to recreate a test environment, so let's use Nodepool.

First we'll add a new elements to pre-install the typical rdo requirements:

  - name: dib-rdo-newton
      - centos-minimal
      - nodepool-minimal
      - rdo-requirements
      RDO_RELEASE: "ocata"

  - name: default
      - name: rdo-newton
        diskimage: dib-rdo-newton
        username: jenkins
        min-ram: 8192
        private-key: /var/lib/nodepool/.ssh/id_rsa

Then using a ready-script, we can execute packstack to deploy services after the node has been created:

  - name: rdo-ocata
    image: rdo-ocata
    min-ready: 1
      - name: default

Once the node is ready, use nodepool list to get the IP address:

# ssh -i /var/lib/nodepool/.ssh/id_rsa jenkins@node
jenkins$ . keystonerc_admin
jenkins (keystone_admin)$ openstack catalog list
| Name      | Type      | Endpoints                     |
| keystone  | identity  | RegionOne                     |
|           |           |   public: http://node:5000/v3 |

To get a new instance, either terminate the current one, or manually delete it using nodepool delete node-id. A few minutes later you will have a fresh and pristine environment!

by tristanC at May 18, 2017 01:33 PM

Joe Talerico and OpenStack Performance at the OpenStack PTG in Atlanta

Last month at the OpenStack PTG in Atlanta, Joe Talerico spoke about his work on OpenStack Performance in the Ocata cycle.

Subscribe to our YouTube channel for more videos like this.

Joe: Hi, I'm Joe Talerico. I work on OpenStack at Red Hat, doing OpenStack performance. In Ocata, we're going to be looking at doing API and dataplane performance and performance CI. In Pike we're looking at doing mix/match workloads of Rally, Shaker, and perfkit benchmarker, and different styles, different workloads running concurrently. That's what we're looking forward to in Pike.

Rich: How long have you been working on this stuff?

Joe: OpenStack performance, probably right around four years now. I started with doing Spec Cloud development, and Spec Cloud development turned into doing performance work at Red Hat for OpenStack … actually, it was Spec Virt, then Spec Cloud, then performance at OpenStack.

Rich: What kind of things were in Ocata that you find interesting?

Joe: In Ocata … for us … well, in Newton, composable roles, but building upon that, in TripleO, being able to do … breaking out the control plane even further, being able to scale out our deployments to much larger clouds. In Ocata, we're looking to work with CNCF, and do a 500 node deployment, and then put OpenShift on top of that, and find some more potential performance issues, or performance gains, going from Newton to Ocata. We've done this previously with Newton, we're going to redo it with Ocata.

by Rich Bowen at May 18, 2017 01:33 PM

Blog posts, week of March 20

Here's what the RDO community has been blogging about in the last week.

Joe Talerico and OpenStack Performance at the OpenStack PTG in Atlanta by Rich Bowen

Last month at the OpenStack PTG in Atlanta, Joe Talerico spoke about his work on OpenStack Performance in the Ocata cycle.


RDO CI promotion pipelines in a nutshell by amoralej

One of the key goals in RDO is to provide a set of well tested and up-to-date repositories that can be smoothly used by our users:


A tale of Tempest rpm with Installers by chandankumar

Tempest is a set of integration tests to run against OpenStack Cloud. Delivering robust and working OpenStack cloud is always challenging. To make sure what we deliver in RDO is rock-solid, we use Tempest to perform a set of API and scenario tests against a running cloud using different installers like puppet-openstack-integration, packstack, and tripleo-quickstart. And, it is the story of how we integrated RDO Tempest RPM package with installers so it can be consumed by various CI rather than using raw upstream sources.


An even better Ansible reporting interface with ARA 0.12 by dmsimard

Not even a month ago, I announced the release of ARA 0.11 with a bunch of new features and improvements.


Let rdopkg manage your RPM package by

rdopkg is a RPM packaging automation tool which was written to efortlessly keep packages in sync with (fast moving) upstream.


Using Software Factory to manage Red Hat OpenStack Platform lifecycle by Maria Bracho, Senior Product Manager OpenStack

by Nicolas Hicher, Senior Software Engineer – Continuous Integration and Delivery Software-Factory Software-Factory is a collection of services that provides a powerful platform to build software. It enables the same workflow used to develop OpenStack: using Gerrit for code reviews, Zuul/Nodepool/Jenkins as a CI system, and Storyboard for stories and issues tracker. Also, it ensures a reproducible test environment with ephemeral Jenkins slaves.


by Rich Bowen at May 18, 2017 01:33 PM

Red Hat Stack

What’s new in Red Hat OpenStack Platform 11?

We are happy to announce that Red Hat OpenStack Platform 11 is now Generally Available (GA).

Version 11 is based on the upstream OpenStack release, Ocata, the 15th release of OpenStack. It brings a plethora of features, enhancements, bugfixes, documentation improvements and security updates. Red Hat OpenStack Platform 11 contains the additional usability, hardening and support that all Red Hat releases are known for. And with key enhancements to Red Hat OpenStack Platform’s deployment tool, Red Hat OpenStack Director, deploying and upgrading enterprise, production-ready private clouds has never been easier. 

So grab a nice cup of coffee or other tasty beverage and sit back as we introduce some of the most exciting new features in Red Hat OpenStack Platform 11!

Composable Upgrades

By far, the most exciting addition brought by Red Hat OpenStack Platform 11 is the extension of composable roles to now include composable upgrades.

But first, composable roles

As a refresher, a composable role is a collection of services that are grouped together to deploy the Overcloud’s main components. There are five default roles (Controller, Compute, BlockStorage, ObjectStorage, and CephStorage) allowing most common architectural scenarios to be achieved out of the box. Each service in a composable role is defined by an individual Heat template following a standardised approach that ensures services implement a basic set of input parameters and output values. With this approach these service templates can be more easily moved around, or composed, into a custom role. This creates greater flexibility around service placement and management.

And now, composable upgrades …

Before composable roles, upgrades were managed via a large set of complex code to ensure all steps were executed properly. By decomposing the services into smaller, standardized modules, the upgrade logic can be moved out of the monolithic and complex script into the service template directly. This is done by a complete refactoring of the upgrade procedure into modular snippets of Ansible code which can then be integrated and orchestrated by Heat. To do this each service’s template has a collection of Ansible plays to handle the upgrade steps and actions. Each Ansible play has a tagged value to allow heat to step through the code and execute in a precise and controlled order. This is the same methodology used by puppet and the “step_config” parameter already found in the “outputs” section of each service template.

Heat iterates through the roles and services and joins the services’ upgrade plays together into a larger playbook. It then executes the plays, by tag, moving through the upgrade procedure.

For example, take a look at Pacemaker’s upgrade_tasks section (from tripleo-heat-templates/puppet/services/pacemaker.yaml):

        - name: Check pacemaker cluster running before upgrade
          tags: step0,validation
          pacemaker_cluster: state=online check_and_fail=true
          async: 30
          poll: 4
        - name: Stop pacemaker cluster
          tags: step2
          pacemaker_cluster: state=offline
        - name: Start pacemaker cluster
          tags: step4
          pacemaker_cluster: state=online
        - name: Check pacemaker resource
          tags: step4
            resource: "{{ item }}"
            max_wait: 500
          with_items: {get_param: PacemakerResources}
        - name: Check pacemaker haproxy resource
          tags: step4
            resource: haproxy
            max_wait: 500
          when: {get_param: EnableLoadBalancer}

Heat executes the play for step0, then step1, then step2 and so on. This is just like running ansible-playbook with the -t or –tags option to only run plays tagged with these values.

Composable upgrades help to support trustworthy lifecycle management of deployments by providing a stable upgrade path between supported releases. They offer simplicity and reliability to the upgrade process and the ability to easily control, run and customize upgrade logic in a modular and straightforward way.

Increased “Day 0” HA (Pacemaker) Service placement flexibility

New in version 11, deployments can use composable roles for all services. This means the remaining pacemaker-managed services, such as RabbitMQ and Galera, traditionally required to be collocated on a single controller node, can now be deployed as custom roles to any nodes. This allows operators to move core service layers to dedicated nodes increasing security, scale, and service design flexibility.

Please note: Due to the complex-nature of changing the pacemaker-managed services in an already running Overcloud we recommend consulting Red Hat support services before attempting to do so.

Improvements for NFV

Co-location of Ceph on Compute now supported in production (GA)

Co-locating Ceph on Nova is done by placing the Ceph Object Storage Daemons (OSDs) directly on the compute nodes. Co-location lowers many cost and complexity barriers for workloads that have minimal and/or predictable storage I/O requirements by reducing the number of total nodes required for an OpenStack deployment. Hardware previously dedicated for storage-specific requirements can now be utilized by the compute footprint for increased scale. With version 11 co-located storage is also now fully supported for deployment by director as a composable role. Operators can more easily perform detailed and targeted deployments of co-located storage, including technologies such as SR-IOV, all from a custom role. The process is fully supported with comprehensive documentation and tuning support (track this BZ for version 11 specifics).

For Telcos, support for co-locating storage can be helpful for optimizing workloads and deployment architectures on a varied range of hardware and networking technologies within a single OpenStack deployment.

VLAN-Aware VMs now supported in production (GA)

A VLAN-aware VM, or more specifically, “Neutron Trunkports,” is how an OpenStack instance can support VLAN tagged frames across a single vNIC. This allows an operator to use fewer vNICs to access many separate networks, significantly reducing complexity by reducing the need for one vNIC for each network. Neutron does this by allowing subports off the original parent, effectively turning the main parent port into a virtual trunk. These subports can have their own segmentation id’s assigned directly to them allowing an operator to assign each port its own VLAN.

(Image courtesy of; used under Creative Commons)

Version bumps for key virtual networking technologies

DPDK now version 16.11

DPDK 16.11 brings non-uniform memory access (NUMA) awareness to openvswitch-dpdk deployments. Virtual host devices comprise of multiple different types of memory which should all be allocated to the same physical node. 16.11 uses NUMA awareness to achieve this in some of the following ways:

  • 16.11 removes the requirement for a single device-tracking node which often creates performance issues by splitting memory allocations when VMs are not on that node
  • NUMA ID’s can now be dynamically derived and that information used by DPDK to correctly place all memory types on the same node
  • DPDK now sends NUMA node information for a guest directly to Open vSwitch (OVS) allowing OVS to allocate memory more easily on the correct node
  • 16.11 removes the requirement for poll mode driver (PMD) threads to be on cores of the same NUMA node. PMDs can now be on the same node as a device’s memory allocations

Open vSwitch now version 2.6

OVS 2.6 lays the groundwork for future performance and virtual network requirements required for NFV deployments, specifically in the ovs-dpdk deployment space. Immediate benefits are gained by currency of features and initial, basic OVN support. See the upstream release notes for full details.

CloudForms Integration

Red Hat OpenStack Platform 11 remains tightly integrated with CloudForms. It has been fully tested and supports features such as:

  • Tenant Mapping: finds and lists all OpenStack tenants as CloudForms tenants and they remain in synch. Create, update and delete of CloudForms tenants are reflected in OpenStack and vice-versa
  • Multisite support where one OpenStack region is represented as one cloud provider in CloudForms
  • Multiple domains support where one domain is represented as one cloud provider in CloudForms
  • Cinder Volume Snapshot Management can be done at volume or instance level. A snapshot is a whole new volume and you can instantiate a new instance from it, all from Cloudforms

OpenStack Lifecycle: Our First “Sequential” Release

Long Life review …

With OSP 10 we introduced the concept of the Long Life release. Long Life releases allow customers who are happy with their current release and without any pressing need for specific feature updates to remain supported for up to five years. We have designated every 3rd release as Long Life. For instance, versions 10, 13, and 16 are Long Life, while versions 11, 12, 14 and 15 are sequential. Long Life releases allow for upgrades to subsequent Long Life releases (for example, 10 to 13 without stepping through 11 and 12). Long Life releases generally have an 18 month cadence (three upstream cycles) and do require additional hardware for the upgrade process. Also, while procedures and tooling will be provided for this type of upgrade, it is important to note that some outages will occur.

Now, Introducing … Sequential!

Red Hat OpenStack Platform 11 is the first “sequential” release (i.e. N+1). It is supported for one year and is released immediately into a “Production Phase 2” release classification. All upgrades for this type of release must be done sequentially (i.e. N+1). Sequential releases feature tighter integration with upstream projects and allow customers to quickly test new features and to deploy using their own knowledge of continuous integration and agile principles. Upgrades are generally done without major workload interruption and customers typically have multiple datacenters and/or highly demanding performance requirements. For more details see Red Hat OpenStack Platform Lifecycle (detailed FAQ as pdf) and Red Hat OpenStack Platform Director Life Cycle.

Additional notable new features of version 11

A new Ironic inspector plugin can process Link Layer Discovery Protocol (LLDP) packets received from network switches during deployment. This can significantly help deployers to understand the existing network topology during a deployment and reduces trial-and-error by helping to validate the actual physical network setup presented to a deployment. All data is collected automatically and stored in an accessible format in the Undercloud’s Swift install.

There is now full support for collectd agents to be deployed to the Overcloud from director using composable roles. Performance monitoring is now easier to do as collectd joins the other fully supported OpsTools services for availability monitoring (sensu) and log management (fluentd) present starting with version 10.

And please remember, this are agents, not the full server-side implementations. Check out how to implement the server components easily with Ansible by going to the CentOS OpsTools Special Interest Group for all the details.

Additional features landing as Tech Preview

Tech Preview Features should not be implemented in production. For full details please see:


Octavia brings a robust and mature LBaaS v2 API driver to OpenStack and will eventually replace the legacy HAProxy namespace driver currently found in Newton. It will become not only a load balancing driver but also the load balancing API hosting all the other drivers. Octavia is a now a top level project outside of Neutron; for more details see this excellent update talk from the recent OpenStack Summit in Boston.

Octavia implements load balancing via a group of virtual machines (or containers or bare metal servers) controlled via a controller called “Amphora.” It manages, among other things, the images used for the balancing engine. In Ocata, Amphora introduces image support for Red Hat Enterprise Linux, Centos and Fedora. Amphora images (collectively known as amphorae) utilize HAProxy to implement load balancing. For full details of the design, consult the Component Design document.

To allow Red Hat OpenStack Platform users to try out this new implementation in a non-production environment operators can deploy a Technology Preview with director starting with version 11.

Please Note: Octavia’s director-based implementation is currently scheduled for a z-stream release for Red Hat OpenStack Platform Version 11. This means that while it won’t be available on the day of the release it will be added to it shortly. However, please track the following bugzilla, as things may change at the last moment and affect this timing.


Red Hat OpenStack Platform 11 increases ODL support in version 10 by adding deployment of the OpenDaylight Boron SR2 release to director using a composable role.

Ceph block storage replication

The Cinder RADOS block driver (RBD) was updated to support RBD mirroring (promote/demote location) in order to allow customers to support essential concepts in disaster recovery by more easily managing and replicating their data using RBD-mirroring via the Cinder API.

Cinder Service HA 

Until now the cinder-volume service could run only in Active/Passive HA fashion. In version 11, the Cinder service received numerous internal fixes around locks, job distribution, cleanup, and data corruption protection to allow for an Active/Active implementation. Having a highly available Cinder implementation may be useful for uptime reliability and throughput requirements.

To sum it all up

Red Hat OpenStack Platform 11 brings important enhancements to all facets of cloud deployment, operations, and management. With solid and reliable upgrade logic enterprises will find moving to the next version of OpenStack is easier and smoother with a lower chance for disruption. The promotion of important features to full production support (GA) keeps installs current and supported while the introduction of new Technology Preview features gives an accessible glimpse into the immediate future of the Red Hat OpenStack Platform.

More info

For more information about Red Hat OpenStack Platform please visit the technology overview page, product documentation, release notes and release annoucement.

To see what others are doing with Red Hat OpenStack Platform check out these use cases

And don’t forget you can evaluate Red Hat OpenStack Platform for free for 60 days to see all these features in action.

by August Simonelli, Technical Marketing Manager, Red Hat OpenStack Platform at May 18, 2017 12:33 PM

OpenStack Superuser

How to: Docker containerized OpenStack Heat Translator

The OpenStack Heat Translator is one of the projects under the main OpenStack Heat orchestration project. It facilitates translation of OASIS TOSCA service template to Heat Orchestration Template (HOT) and the deployment of translated template into an OpenStack cloud. To use it, you simply download a stable release through PyPI or by forking the master branch to use the latest source code. Docker containers are now widely used to consume applications and tools, and such usage are just growing. Building a container is fun and using it can be even more fun and highly productive. We have created a container using the latest stable release of the Heat-Translator available at the PyPI. This blog post will show where the image is located and how it can be used.

Get the image

The Heat Translator Docker container image is available on the Docker hub and it’s called patrocinio/h-t-container-stable.

Use the container

You can invoke the Heat Translator at the same time you run the container. The Heat Translator is commonly used to translate TOSCA service template available in the local file system or providing it via an URL. We will walk you through invoking Heat Translator in both ways.

Translate TOSCA service template from local file system

Let’s say you have TOSCA service template called tosca_helloworld.yamlon your local machine. Assume that the tosca_helloworld.yaml is located in your /tmp/tosca_testfiles. To translate it to HOT, run the container as follows:

$ docker run -v /tmp/tosca_testfiles:/tosca patrocinio/h-t-container-stable --template-file /tosca/tosca_helloworld.yaml

The patrocinio/h-t-container-stable will pull the image if it is not already available in your environment. Also note that the container requires that you provide /tmp/tosca_testfiles directory as a volume which will be mapped as the /tosca directory inside the container. You can think of /tmp/tosca_testfiles as a host source directory. It can be an arbitrary name but must be provided as an absolute path. The rest is simple: you provide the arguments that are understood by the Heat Translator. In this case, we provided the required –template-file argument with your TOSCA service template as a value to the argument. The service template can be a YAML file as provided above or it can be a TOSCA CSAR file. If you are new to CSAR, learn more about it from Sahdev Zala’s post Package Cloud Workloads with TOSCA Cloud Service Archive. Assuming your tosca_helloworld.yaml looks as below:

tosca_definitions_version: tosca_simple_yaml_1_0

description: Template for deploying a single server with predefined properties.

      type: tosca.nodes.Compute
        # Host container properties
           num_cpus: 2
           disk_size: 10 GB
           mem_size: 512 MB
        # Guest Operating System properties
            # host Operating System image properties
            architecture: x86_64
            type: Linux
            distribution: RHEL
            version: 6.5

The output of the above command will produce following HOT template,

heat_template_version: 2013-05-23

description: >
  Template for deploying a single server with predefined properties.
parameters: {}
    type: OS::Nova::Server
      flavor: m1.medium
      image: rhel-6.5-test-image
      user_data_format: SOFTWARE_CONFIG
outputs: {}

Translate TOSCA service template via an URL

Translating by providing service template or CSAR as URL is very simple. All you need to do is provide a URL as a value to the –template-file argument. For example, tosca_helloworld.yaml located on the TOSCA Parsergithub project can be translated using the following command,

$ docker run patrocinio/h-t-container-stable

Heat Translator help

You can get to the Heat Translator help by simply running:

docker run patrocinio/h-t-container-stable --help


Given below is the content of Dockerfile used to build the h-t-container-stable image. The file is also available in the Heat Translator project.

FROM ubuntu

MAINTAINER Eduardo Patrocinio and Sahdev P. Zala

RUN apt-get -y update && apt-get install -y \ 

RUN pip install heat-translator

COPY heat_translator_logging.conf /usr/local/lib/python2.7/dist-packages/translator/conf/

# Have some test TOSCA templates in my_tosca directory to copy to the container as an example.
# This is an optional step and can be removed.
COPY my_tosca /tmp/my_tosca

ENTRYPOINT ["heat-translator"]


In this article, we showed you how to use the OpenStack Heat Translator using a Docker image by passing the source file during the docker run command. We have also shared Dockerfile to use as a base to create a modified Docker container of the Heat Translator.

This post first appeared on the IBM OpenTech blog. Superuser is always interested in community content, email:

Cover Photo // CC BY NC

The post How to: Docker containerized OpenStack Heat Translator appeared first on OpenStack Superuser.

by Sahdev Zala and Eduardo Patrocinio at May 18, 2017 11:31 AM

May 17, 2017

Ed Leafe

Claims in the Scheduler

One of the shortcomings of the current scheduler in OpenStack Nova is that there is a long interval from when the scheduler selects a suitable host for a new instance until the resources on that host are claimed so that they are no longer available. Now that resources are tracked in the Placement service, we want to move the claim closer to the time of host selection, in order to avoid (or eliminate) the race condition. I’m not going to explain the race condition here; if you’re reading this, I’m assuming this is well understood, so let me just summarize my concern: the current proposed design, as seen in the series starting with, could be made much better with some design changes.

At the recent Boston Summit, which I was unable to attend due to lack of funding by my employer, the design for this change was discussed, and the consensus was to have the scheduler return a list of hosts for each instance to the super conductor, and then have the super conductor attempt to claim the resources for the first host returned. If the allocation fails, the super conductor discards that host and tries to claim the resources on the second host. When it finally succeeds in a claim, it sends a message to that host to start building the instance, and that message will include the list of alternative hosts. If something happens that causes the build to fail, the compute node sends it back to its local conductor, which will unclaim the resources, and then try each of the alternates in order by first claiming the resources on that host, and if successful, sending the build request to that host. Only if all of the alternates fail will the request fail.

I believe that while this is an improvement, it could be better. I’d like to do two things differently:

  1. Have the scheduler claim the resources on the first selected host. If it fails, discard it and try the next. When it succeeds, find other hosts in the list of weighed hosts that are in the same cell as the selected host in order to provide the number of alternates, and return that list.
  2. Have the process asking the scheduler to select a host also provide the number of alternates, instead of having the scheduler use the current max_attempts config option value.

On the first point: the scheduler already has a representation of the resources that need to be claimed. If the super conductor does the claiming, it will have to re-generate that representation. Sure, that’s not all that demanding, but it sure makes for cleaner design to not repeat things. It also ensures that the super conductor gets a good host from the start. Let me give an example. If the scheduler returns a chosen host (without claiming) and two alternates (which is the standard behavior using the config option default), the conductor has no guarantee of getting a good host. In the event of a race, the first host may fail to allocate resources, and now there are only the two alternates to try. If the claim was done in the scheduler, though, when that first host failed it would have been discarded, and the the next host tried, until the allocation succeeded. Only then would the alternates be determined, and the super conductor could confidently pass on that build request to the chosen host. Simply put: by having the scheduler do the initial claim, the super conductor is guaranteed to get a good host.

Another problem, although much less critical, is that the scheduler still has the host do consume_from_request(). With the claim done in the conductor, there is no way to keep this working if the initial host fails. We will have consumed on that host, even though we aren’t building on it, and have not consumed on the host we actually select.

On the second point: we have spent a lot of time over the past few years trying to clean up the interface between Nova and the scheduler, and have made a great deal of progress on that front. Now I know that the dream of an independent scheduler is still just that: a dream. But I also know that the scheduler code has been greatly improved by defining a cleaner interface between it an Nova. One of the items that has been discussed is that the config option max_attempts doesn’t belong in the scheduler; instead, it really belongs in the conductor, and now that the conductor will be getting a list of hosts from the scheduler, the scheduler is out of the picture when it comes to retrying a failed build. The current proposal to not only leave that config option in the scheduler, but to make it dependent on it for its functioning, is something that once again makes the scheduler Nova-centric (and Nova-exclusive). It would be a much cleaner design to simply have the conductor ask for the number of hosts (chosen + alternates), and have the scheduler’s behavior use that number. Yes, it requires a change to the RPC interface, but that is to be expected if you are changing a fundamental behavior of the scheduler. And if the scheduler is ever moved into a module, all it is is another parameter. Really, that’s not a good reason to follow a poor design.

Since some of the principal people involved in this discussion are not available now, and I’m going to be away at PyCon for the next few days, Dan Smith suggested that I post a summary of my concerns so that all can read it and have an idea what the issues are. Then next week sometime when we are all around and have the time to discuss this, we can hash it out on #openstack-nova, or maybe in a hangout. I also have pushed a series that has all of the steps needed to make this happen, since it’s one thing to talk about a design, and it’s another to see the actual code. The series starts here: For some of the later patches I haven’t finished updating the tests to match the change in method signatures and returned value structures, but you should be able to get a good idea of the code changes I’m proposing.

by ed at May 17, 2017 11:54 PM

Julio Villarreal Pelegrino

Don't Fail at Scale- How to Plan for, Build, and Operate a Successful OpenStack Cloud - Video - OpenStack Summit2017

Here is the presentation that I delivered with fellow Red Hatter Dave Costakos at OpenStack Summit Boston 2017.

Here the abstract:

"OpenStack is a unique set of cross-functional, fast-moving technologies that challenge the status quo in any IT organization. It has been easy to read about and buy into the interweaving of compute, storage, and networking technologies in a single platform. Despite such amazing promise and technology, technology is littered with companies who have tried and failed to deliver a successful, scaleable, supportable OpenStack cloud to their customers. Architects and IT teams tasked with designing"


by Julio Villarreal Pelegrino at May 17, 2017 09:31 PM

Best practices for successfully deploying NFV -Video - Red Hat Summit 2017

Here is a presentation that we (Rimma Iontel) and me delivered at Red Hat Summit 2017. The topic of the presentation is "Best practices for successfully deploying NFV". Enjoy!

by Julio Villarreal Pelegrino at May 17, 2017 09:25 PM

Terry Wilson

How not to write Python unit tests

In Openstack projects, I see tests like this, entirely too often:

    def test_main_sync_success(self):
        self.cmd_sync.sync_routers_and_rports.assert_called_once_with(mock.ANY)'Sync completed')

In the name of 100% unit test coverage, tests are written that just assert that a bunch of functions are called. It doesn't matter that any of those functions could be broken. Or stupid. Or just not needed. The test just makes sure that those functions are called.

Consider the following horrible implementation of an add() function:

def add(x, y):  
    val = 0
    for i in range(x + y):
        val += 1
    return val

This, like a lot of code that makes it into production, is stupid. The only thing that could be worse would be a test that relies on the stupidity:

import mock, unittest  
from adder import add

class TestAdd(unittest.TestCase):  
    def test_add(self):
        x, y = 1, 3
        # for the sake of example, assume range is an external that has to be mocked
        with mock.patch.object(add, 'range') as mock_range:
            add.add(x, y)
            mock_range.assert_called_once_with(x + y)

The truly crazy thing is that I see tests like this all the time. The tests just blindly call assert_called_once_with() over several methods called by the function under test. The internal details of add() are stupid and should change, but now when the function is fixed, the test also has to change. Congrats, you've tested absolutely nothing and created twice the work to fix the problem.

Any time you feel like calling Mock's assert_called* methods, please make sure that you aren't tying the test to a specific implementation. If you can't write a unit test without it, maybe the test you are writing shouldn't be a unit test. Write an integration test.

by Terry Wilson at May 17, 2017 07:55 PM

Assaf Muller

Upstream Contribution – Give Up or Double Down? The Boston Edition

In continuation to a previous blog post, I presented a version with new content at the recent OpenStack Boston summit. The session had a fair amount of participation and discussions, where we talked about the journey of a new contributor in OpenStack through the lens of data gathered from Gerrit and Stackalytics. We even had members of the Technical Committee that were interested in concrete action items they could take on their part – How do we make it easier for new contributors? What behaviors work in the quest to get your patches merged quicker?

The video, slides and code used to generate the data are below:

by assafmuller at May 17, 2017 07:46 PM

Ihar Hrachyshka

Gratuitous ARP for OpenStack Neutron

This post is the second in a series that covers gratuitous ARP and its relation to OpenStack. In the previous post, we discussed what gratuitous ARP is, and how it helps in IT operations. We also briefly touched on how it’s used in OpenStack Neutron L3 agent.

This post will expand on gratuitous ARP packets’ usage in OpenStack Neutron L3 agent. We will also dive deep into how they are implemented.

Gratuitous ARP in OpenStack Neutron


In the previous post, we already briefly touched on where gratuitous ARP is used in OpenStack Neutron.

To recollect, the primary use for gratuitous ARP in OpenStack Neutron L3 agent is to update network peers about the new location of a “floating” IP address (“elastic” in AWS-speak) when it’s disassociated from one port and then associated to another port with a different MAC address. Without issuing a gratuitous ARP on new association, it may take significant time before a reused floating IP address mapping is updated as a result of the “aging” process.

Gratuitous ARP is also used by the L3 agent to implement HA for Neutron routers. Whenever a new HA router instance becomes “master”, it adds IP addresses managed by Neutron to its interfaces and issues a set of gratuitous ARP packets into attached networks to advertise the new location. Network peers then update their ARP tables with new MAC addresses from those packets and in this way don’t need to wait for old entries to expire before connectivity would be restored. The switch to the new router instance is then a lot smoother.


There are two distinct implementations for gratuitous ARP in OpenStack Neutron, one for each distinct router deployment mode: legacy and HA. The difference comes primarily from the fact that legacy router data plane is fully realized by the L3 agent, while HA routers “outsource” IP address management to keepalived daemon spawned by the agent. (The third deployment mode – DVR – is largely covered by those two, where specific implementation depends on whether DVR routers are also HA or not; for this reason I won’t mention DVR going forward).

Let’s consider each distinct deployment mode separately, starting with legacy.

Legacy routers

Legacy mode is what once was the only mode supported by OpenStack Neutron. In this mode, the L3 agent itself implements the whole data plane, creating network namespaces for routers, creating ports, plugging them into the external br-ex bridge, and adding fixed and floating IP addresses to router ports. Besides that, the agent also issues gratuitous ARP packets into attached networks when a new IP address is added to one of its ports. This is to update network peers about the new mappings. Peers may use those unsolicited updates either to update any existing ARP entries with a new MAC address, or to “warm up” their tables with IP-to-MAC mappings even before the very first IP datagram is issued to the router IP address (this is something that Linux kernel does when arp_accept sysctl setting is enabled for the receiving interface).

When the L3 agent sends gratuitous ARP packets for an IP address, this is what you can find in the agent log file:

2017-04-28 20:53:11.264 14176 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-726095be-5916-489b-be05-860e2f19d556', 'ip', '-4', 'addr', 'add', '', 'scope', 'global', 'dev', 'qr-864545b9-5f', 'brd', ''] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/

And then later:

2017-04-28 20:53:11.425 14176 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-726095be-5916-489b-be05-860e2f19d556', 'arping', '-A', '-I', 'qr-864545b9-5f', '-c', '3', '-w', '4.5', ''] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/

As you have probably figured out, the first snippet shows the agent adding a new IPv4 address to an internal router qr-864545b9-5f port, and the second snippet is where the agent sends gratuitous ARP packets advertising the new IP address into the network to which the qr-864545b9-5f port is attached to (this is achieved by calling the arping tool from iputils package with the right arguments).

Let’s have a look at each of the arguments passed into arping tool.

  • The very first option is -A, and it’s used to issue gratuitous (broadcast) ARP packets. Without the option, the tool would send unicast ARP REQUEST packets for the IP address, and would wait for a REPLY (the unicast mode may be useful when you need to check if there are any other hosts in the network carrying the same IP address, or to sanity check an existing IP-to-MAC mapping). The packets sent are of REPLY type. (If we would use -U instead, it would send REQUEST packets.)
  • The next option is -I, and it specifies the interface to issue the packets on.
  • The -c option defines the number of ARP packets to issue into the network. There is always a 1 second interval between the packets. Since we call it with -c 3, it issues three packets in two second time span.
  • The next option is -w 4.5 and it means that we will wait for 4.5 seconds (or better, 4 seconds because the tool doesn’t recognize floating part of the argument) before exiting it. In general, the tool will exit after two seconds, but when the interface used to send packets is gone while the tool is running, it may block its execution since it will never be able to successfully send all three packets. The option guarantees that the thread running the tool will eventually make progress.
  • The very last argument is the IP address to advertise. A single port may carry multiple IPv4 addresses, so it’s crucial to define which of those addresses should be advertised.

HA routers

HA support is a relatively new addition to OpenStack Neutron routers. To use HA for Neutron routers, one should configure Neutron API controller to expose l3-ha API extension, at which point users are able to create highly available routers.

For those routers, data plane is managed both by the L3 agent as well as the keepalived daemon that the agent spawns for every HA router it manages. The agent first prepares the router namespace, its ports, rules for NAT translation; but then it falls back to the keepalived daemon which manages IP addresses on ports. For this matter, the agent generates a configuration file listing all managed IP addresses and passes it into keepalived. The daemon then starts, negotiates with other keepalived processes implementing the HA router who is going to be its “master” (for this matter, VRRP is used), and if it’s indeed “master”, it triggers state transition machinery, which, among other things, will add managed IP addresses specified in the configuration file to appropriate router ports. It will also send gratuitous ARP packets into the network to update peers about the location of those IP addresses. If you then inspect your system log, you may find the following messages there:

May  2 13:19:47 host-192-168-24-12 Keepalived[307081]: Starting Keepalived v1.2.13 (07/01,2016)
May  2 13:19:47 host-192-168-24-12 Keepalived[307082]: Starting VRRP child process, pid=307083
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Netlink reflector reports IP added
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Netlink reflector reports IP fe80::f816:3eff:fe5f:d44b added
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Registering Kernel netlink reflector
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Registering Kernel netlink command channel
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Registering gratuitous ARP shared channel
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Opening file '/var/lib/neutron/ha_confs/b7fece4b-ea95-4eb6-b7b8-dc060325d1bc/keepalived.conf'.
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Configuration is using : 64829 Bytes
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: Using LinkWatch kernel netlink reflector...
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Entering BACKUP STATE
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) removing protocol Virtual Routes
May  2 13:19:47 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP sockpool: [ifindex(16), proto(112), unicast(0), fd(10,11)]
May  2 13:19:54 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Transition to MASTER STATE
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Entering MASTER STATE
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) setting protocol VIPs.
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) setting protocol E-VIPs.
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) setting protocol Virtual Routes
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Sending gratuitous ARPs on ha-e09aa535-6f for
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Sending gratuitous ARPs on qg-6cf347df-28 for
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Sending gratuitous ARPs on qr-3ee577eb-4f for
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Sending Unsolicited Neighbour Adverts on qr-3ee577eb-4f for fe80::f816:3eff:fe9a:c17
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: VRRP_Instance(VR_1) Sending Unsolicited Neighbour Adverts on qg-6cf347df-28 for fe80::f816:3eff:fec7:861a
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: Netlink reflector reports IP fe80::f816:3eff:fe9a:c17 added
May  2 13:19:56 host-192-168-24-12 Keepalived_vrrp[307083]: Netlink reflector reports IP fe80::f816:3eff:fec7:861a added

Here we can see keepalived transitioning to master state and immediately issuing gratuitous updates after VIP addresses are set for managed interfaces. (A careful reader will also notice that it also issues something called Unsolicited Neighbour Adverts which is a similar mechanism for IPv6 addresses, but I won’t go there.)

It would seem like it’s good for the job. Sadly, the reality is uglier than one could hope.

WTF#1: HA router reload doesn’t issue gratuitous ARP packets

As we’ve learned during our testing of the HA feature, sometimes keepalived forgot to send gratuitous ARP packets. It always happened when an existing keepalived instance was asked to reload its configuration file because some Neutron API operations triggered router updates that affected the file contents. An example of an update could be e.g. adding a new floating IP address to a port, or disassociating one. In this case, Neutron L3 agent would generate a new configuration file and then send SIGHUP signal to the running keepalived instance, hoping that it will catch the changes, converge the data plane to latest configuration, and finally issue gratuitous ARP updates. It did not.

Investigation, largely carried by John Schwarz, uncovered it was not an issue with latest keepalived releases, but the one from RHEL7 repositories. Bisecting releases, we’ve found out that the very first keepalived release that was not exposing the buggy behavior was 1.3.20. Popular distributions (RHEL7, Ubuntu Xenial) were still shipping older versions of the daemon (1.2.13 for RHEL7 and 1.2.19 for Xenial).

Though the issue was technically in keepalived, we needed to adopt OpenStack to the buggy releases shipped with platforms we support. First considered option was just fully restarting keepalived, which would correctly trigger the gratuitous ARP machinery. The problem with this approach was that full restart temporarily stops the VRRP thread that sends master health checks, and with unfortunate timing, it sometimes results in an unnecessary “master” to “backup” flip, operation that is both computationally costly as well as data plane disruptive.

Since we couldn’t just upgrade keepalived, it meant that Neutron L3 agent would need to play some role in issuing gratuitous ARP packets, not relying on the daemon to do the right job. For this matter, Neutron patch was introduced. What the patch does is it calls to arping tool whenever a new IPv4 address is added to an interface managed by keepalived. A new address added indicates that VRRP negotiation resulted in the locally running keepalived instance transitioning to “master”; or it means a new floating IP address was added in the configuration file just reloaded by the daemon. At this point it makes sense to advertise the newly added addresses on the wire using gratuitous ARP, something that in an ideal world keepalived would do for us.

We already had the neutron-keepalived-state-change helper daemon running inside HA router network namespaces that monitors router interfaces for new IP addresses to detect transitions between keepalived states and then sends the information back to neutron-server. To avoid introducing a new daemon just to issue gratuitous ARP packets, we figured it’s easier to reuse the existing one.

Of course, issuing gratuitous ARP packets from outside of keepalived introduced some complications.

For one, the whole setup became slightly racy. For example, what happens when keepalived decides to forfeit its mastership in the middle of neutron-keepalived-state-change sending gratuitous ARP packets? Will we continue sending those packets into the network even after keepalived removed VIP addresses from its interfaces? Thanks to net.ipv4.ip_nonlocal_bind sysctl knob, it shouldn’t be a concern. Its default value (0) means that userspace tools (including arping) won’t be able to send an ARP packet for an IPv4 address that is not on the interface. If we hit the race, the worst that could happen is that arping would hang, failing to send more gratuitous ARP packets into the network, logging the “bind: Cannot assign requested address” error message on its stderr. Since we set a hard time limit for the tool execution (remember the -w 4.5 CLI arguments discussed above), it should be fine. To stay on safe side, we would just set the sysctl knob inside each new router namespace to 0 to override whatever custom value the platform may have for the setting.

There are still two complications with that though.

First, as it turned out, the ip_nonlocal_bind knob was set to 1 for DVR fip namespaces, and for a reason. So we needed to make sure that it’s set to 0 in all router namespaces except fip. Another issue that we surfaced was specific to RHEL7 kernel where the ip_nonlocal_bind knob was not network namespace aware, so changing it in one of namespaces affected all other routers. It was fixed in later RHEL7 kernels, and in the meantime, we could only hope that no one ever hosts both DVR fip and HA qrouter namespaces on the same node, for they would clash.

WTF#2: keepalived forfeits mastership on multiple SIGHUPs sent in quick succession

Not completely related to gratuitous ARP, but since it’s also about SIGHUP handler, I figured I will mention this issue here too.

Some testing revealed that when multiple HA router updates arrived to Neutron L3 agent in quick succession, keepalived sometimes forfeits its mastership, flipping to “backup” with no apparent reason. Consequent network disruption until a new keepalived “master” instance is elected included.

Further investigation, also led by John Schwarz, revealed that it always happens when you would send multiple SIGHUP signals to keepalived, irrespective to whether there were any changes to its configuration files.

It was clearly a bug in the daemon, but at this point we were used to work around its quirks, so it hasn’t taken a lot of time to come up with a special signal throttler for keepalived. What it does is it introduces 3 second delays between consequent SIGHUP signals sent to keepalived instances. Why 3 seconds? No particular reason, except that it worked (anything below 2 seconds didn’t), and it seemed like a good idea to give keepalived a chance to send at least a single health check VRRP message between reload requests, so we made it slightly longer than the default health check interval which is 2 seconds for Neutron.

Reading logs

So how do I know that an HA router actually sent gratuitous ARP packets without having access to a live machine? Let’s say all I have is log files for Neutron services.

For those packets that are sent by keepalived itself, it logs a message per advertised IP address in syslog, as seen in a snippet provided earlier.

As for packets issued by neutron-keepalived-state-change daemon, corresponding messages were originally logged in a file that was located in a directory that also contained other files needed for the router, including keepalived configuration and state files. The problem here is that once a HA router is unscheduled from an L3 agent, it stops keepalived and cleans up both the router namespace as well as all files used by the router, including log files for neutron-keepalived-state-change. It means that after the router is gone, you can’t get your hands on the daemon log file. You are left in darkness as to whether it even called to arping.

To facilitate post-cleanup debugging, in Pike release cycle we’ve made the daemon to log to system log in addition to its own log file. With the patch, we can now see the daemon messages in system journal, including those corresponding to arping execution.

Apr 28 20:56:00 ubuntu-xenial-rax-ord-8650506 neutron-keepalived-state-change[20945]: 2017-04-28 20:56:00.338 20945 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-433765a8-f084-4fbd-9aea-447835c32b09@testceeee6ac', 'arping', '-A', '-I', 'qg-c317683_6ac', '-c', '3', '-w', '4.5', ''] create_process /opt/stack/new/neutron/neutron/agent/linux/
Apr 28 20:56:00 ubuntu-xenial-rax-ord-8650506 sudo[24549]:    stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/ip netns exec qrouter-433765a8-f084-4fbd-9aea-447835c32b09@testceeee6ac arping -A -I qg-c317683_6ac -c 3 -w 4.5
Apr 28 20:56:02 ubuntu-xenial-rax-ord-8650506 neutron-keepalived-state-change[20945]: 2017-04-28 20:56:02.430 20945 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /opt/stack/new/neutron/neutron/agent/linux/

Now whenever you have a doubt whether gratuitous ARP packets were sent by a Neutron HA router, just inspect syslog. You should hopefully find there relevant messages, either from keepalived itself or from neutron-keepalived-state-change calling to arping.

In the next post of the series, we will start looking at a particular ARP related failure Red Hat Networking Team hit lately in RH-OSP 11 (Ocata) CI environment and that, I figured, could be of general interest to Neutron developers and operators.

by ihrachyshka at May 17, 2017 07:37 PM

Alessandro Pilotti

Windows Server 2016 OpenStack Images

Windows Server 2016 is gaining significant traction in OpenStack and other clouds, thanks to the support for Windows Docker containers and lots of other new features.

While there’s no OpenStack Windows Server 2016 image directly available for download, the good news is that our automated build scripts will do all the work for you. All you need is a Windows Server 2016 ISO.

The automated build tools are publicly available on GitHub, allowing the generation of virtual (Hyper-V, KVM, VMware ESXi) or bare metal (MAAS, Ironic) images, including Cloudbase-Init, VirtIO drivers (KVM), latest Windows updates, etc.

You can kickstart the image generation on any Windows host (e.g. Windows 10, Windows Server 2016, etc) with Hyper-V enabled and the Windows ADK installed.

git clone
cd windows-openstack-imaging-tools

Edit the create-windows-online-cloud-image.ps1 in the Examples directory to match your environment and requirements.

If you need to make any changes to the image generation (e.g. adding storage or networking drivers for bare metal servers), you have an extensive Readme which will guide through the entire process.

# This needs an elevated PowerShell prompt

For KVM, a frequent use case, the tool supports the latest Fedora VirtIO drivers, improving considerably the stability and performance of the OS.

You are now all set to generate your Windows Server 2016 images, let us know if you have any questions!


The post Windows Server 2016 OpenStack Images appeared first on Cloudbase Solutions.

by Adrian Vladu at May 17, 2017 02:23 PM

Chris Dent

OpenStack Pike Board Meeting Notes

These are my notes from the OpenStack Foundation Board meetings, held at the beginning of the recent Summit held in Boston, held on May 7th. These are just notes and completely unofficial. Hopefully the typos and lack of structure help to make that clear.

Board Meeting

The Board meets all day. After 2pm it is a join session with the User Committee and Technical Committee. Until 12 the session is public. After that, until 2pm, there is a closed door session to vote for new members and do other private business. I was there for everything but the private business.

This was my first board meeting so a lot of it was quite surprising or illuminating.

Introduction of new board members. Some of the members have titles related to marketing, which surprised me. Some board members were not in attendance.

Minutes from the last meeting were accepted.

Summary from the Executive

Jonathan Bryce gave a summary of the state of the universe:

  • 2nd time in boston (600 people last time, in 2011)
  • 5000 pre-registered attendees
  • 63 countries (most countries ever)
  • 1014 different organizations
  • 41% 1st time
  • 11% attendees identify as upstream developers, about same as Austin, bit less than Barcelona

There were issues with slow visa processing that prevented some people from attending. Apparently a lot of effort is involved in this but the road blocks are very high, especially for people from China.

Overview of stats

A summary of OpenStack statistics and recent patterns:

  • 44% year on year growth in number of deployments
  • 5,000,000 openstack cores of compute in over 80 countries
  • Emerging trend is "openstack at the edge" (fog)
    • variety of different approaches, in slightly different ways, so goal is to get these people on the same page
    • openfog, openedge, people want to use openstack
    • consolidation of cloud into datacenter raises latency, so now we need edge cloud
      • people want services in the cell tower, openstack in the cell tower
      • need a "thin version of openstack", with seamless install and upgrade because it is hands off (zero patch provisioning)
      • huge driver of openstack growth (but what about original use cases?)
      • retailers want store-based edge infrastructure
    • unclear why this is in the overview, but perhaps it is simply because of the emerging trend-ness
  • What does 2nd generation of private cloud mean?
  • Composable infrastructure, openstack has issue
    • things are too hard
    • avoid NIH issues

New category in the marketplace: remotely managed private cloud.

Update on Finances

$90,000 behind plan, due to "gold member shortfall". Increased expenses for marketing and PR. Intent is to exit year with around $3mil in the bank.

(Same problems in this group with most other groups: People haven't read the email before the meeting! Sigh.)

Expectation is that Sydney might be small because it is so far away from usual North American and European crowd.

Overview from the Interop-WG

A summary of recent activity from the Interop-WG.

Plan to have schema 2.0 that supports verticals (different applications like NFV). New trademarks in 2018 which support "openstack X with Y". The old stuff sticks around. The new stuff supports new use cases.

(It was around about this time in the meeting that I realized my experience of OpenStack from inside the development community is considerably different from some other communities. Much less catastrophizing. It's like a completely different world.)

Concern from markmc about whether OpenStack with NFV is still OpenStack (in the trademark sense). There's some talk about it being an independent vertical, which strikes me as an invitation to chaos, but the excuse is that "interop" (at the public level) isn't really an issue for NFV (which is distinctly not public). NFV may not be a superset of OpenStack.

Board members expressed that it needs to be possible for a cloud to be "OpenStack NFV" and "OpenStack Powered" at the same time. If the standards (for verticals) introduce conflict that would be very bad.

Adjacent technologies are a big deal with regard to Interop. Concern expressed about the impact there. There's an adjacent technologies working group that should be worked with too.

Product Working Group

A summary from the product working group on their activities and issues they've identified.

There is good work at getting requirements but not much execution. In part this is because there is a gap between PWG members and contributors. My initial reaction to this was that there needs to be more overlap between PWG and contributors and organizations driving the PWG also need to drive the creation of more contributors.

During this session I felt a bit alienated; as if lots of people in the room were really rather confused about the open source process. There was an undercurrent of party X asking third party Y (usually the board) to give the devs a kick in the pants as if that kick would make all the difference.

I'm fairly certain motivation and interest are not the limiting factors.

Membership Stuff

Two new candidates for gold membership gave presentations about themselves. After their presentations (some notes of which, below) the public session was closed, voting happened, and both candidates were accepted.

China Unicom

China Unicom wants to be a Foundation member, so they introduce themselves and their goal. They are the third largest chinese telecom, state owned, presence in 69 countries. 700,000 employees. POPs all over the world.

"Wo" is a brand used by China Unicom. The want to make it the best choice for OpenStack with their system "Wo Cloud". Already running 5000 servers, Havana 10%, Kilo 90%. They want to push common requirements up to upstream.

"Proud to use open source", try to be very active.

They are intending to start contributing upstream but haven't figured out how to do it well. There are cultural limitations. This idea of cultural barriers is a common theme throughout the day.


Fiber optic cable business and optical transmission. $2.5 billion revenue. Wants to provide cloud services. They have a thing called FitOS which is based on OpenStack. Province-level e-Government platform.

They are SDN oriented.

Their presentation was different from the other because a large chunk of it was oriented around how their upstream contribution record has improved over some cycles.

Joint Meeting with TC and UC

Introductions, new TC members didn't say anything about companies (win!).

A joint meeting held in March was considered very productive, so considering having a new one. Maybe do it around the PTG or the Ops Meetup. Separate is liked by some, but not by others.

The reset of the meeting was spent reviewing proposed and current strategic goals (set at the previous meeting).

Proposal: Clamping the App Gaps

There are a huge numbers of things being called "connectors". Tools which fit somewhere as SDKs, provisioning tools, C/PaaS. There are so many tools, OpenStack can't support all of them, how can we limit those things?

Solution Step 1: API Improvements

Need to get agreement on what's right so that different deployments can migrate towards it. There will be talk about this in the API-WG group.

Solution Step 2: SDK Provision Improvements

Need humans!

(zuul 3 will be able to operate as 3rd party CI against github pull requests) OpenStack community wants to be able to offer testing resources to other connecting tools to help them.

The Strategic Focus Areas

Communicate about OpenStack

Work out a map that explains how all of OpenStack fits together and what pieces can be used and how they can be composed. A visual aid that shows how it matters.

Also need a map to understand what technolgies are involved and what technologies are enabled. And another map for maturity and adoption. Multiple dimensions of view required.

Unanswered Requirements

Plan for discussing (here at the forum) how to more effectively deal with unanswered requirements: the PWG pipeline hasn't worked all that well.

Adjacent Technologies

Getting involved with other communities that are also in the biz of cloud, notably k8s.

Would be useful for projects to think about what they might look like if they were standing alone or existing independently.

(During this discussion there was continued messaging of "getting things to the TC" as if somehow that will magically mean "getting projects to do it". It sometimes feels like perhaps having that kind of executive power might be useful, but it is not how we do things now and everyone seems to understand that.)

Changes to the Technology

OpenStack needs to be simplified. There are features that aren't being used and configuration that are out of date. There are a lot of deprecated things which are past the cycle where they could be removed, so they can be cleared out. There may even be some stale projects that can be removed. Care needs to be taken: a project can be used and useful even when it is not it is not getting contributions. Contributors won't come from thin air though.

Community Health

Efforts to grow community leaders. Wanting to make clear ways to level up.

Do more efforts to create videos of various tasks that are common (including reviewing).

The TC needs to participate in publishing a list of priorities, effectively karma bounties, that companies can use to help with their contribution.


It was an illuminating day. The thing that was most interesting and useful for me was the degree of optimism present in the room. The best idea (from my perspective) was making sure that priorities are highly visible (via publishing karma bounties).

by Chris Dent at May 17, 2017 12:00 PM

OpenStack Superuser

My take on the first OpenStack Forum

At the recent OpenStack summit, we took our first test-drive of a new collaboration track, the OpenStack Forum. Set up similarly to the Design Sessions that were previously held at the Summit (and are now run at the Project Team Gathering, or PTG), the Forum gave developers, operators and users the chance to get together in a single space, share problems and solutions, and put their heads together for high-level planning for future releases.

From my first Forum session, where we discussed the future of base OpenStack service dependencies, it was clear to me that the Forum sessions would be highly informative and productive. In that particular session, we helped to reduce the complexity of maintaining database connectors by deciding to only support MySQL as the database for OpenStack projects. (You can catch up on all the Forum sessions via Etherpad, too.) Additionally, we added etcd version 3 as a dependency for managing global locks. These decisions were reached after a cross- community discussion between developers, operators, and users.

In other satisfying Forum sessions, we worked out road maps for future interoperability programs, publishing OpenStack container artifacts to external locations like Docker Hub and established an initial plan for building out our community relations and outreach with the Kubernetes community.

When I wasn’t attending the Forum, I went to several excellent sessions both in the OpenStack Summit programming and in the adjacent Open Source Days sessions. I came to Boston with nervous anticipation about the format changes and left with a deep sense of satisfaction. Indeed, the Boston OpenStack Summit and Forum were some of the best Summits I’ve ever attended.

I headed home full of excitement about the work ahead for myself and the rest of the community. I feel that with this event we’ve curated an environment where every representative, no matter what their role, has a chance to impact the future and quality of the software and our community.

A big thank you to everyone who attended. I’ll see you in Denver for the PTG and at the next Summit!

Chris Hoge is the OpenStack Foundation’s Interop engineer.


The post My take on the first OpenStack Forum appeared first on OpenStack Superuser.

by Chris Hoge at May 17, 2017 11:41 AM

May 16, 2017

Chris Dent

TC Report 20

I'm not fully recovered from summit yet, so this will not cover everything. It will pick back up next week once I'm back in the full flow.

Notes from Summit

Elsewhere I'll create a more complete write up of the Foundation board meeting that happened on the Sunday before summit, but some comments from that feel relevant to the purpose of these reports:

When new people were being introduced around the table, TC members did not include their company affiliations, board members did. This is good.

Repeated (but not frequent) theme of different members of the board expressing a need for requirements to be given to the TC to give to "the devs", sometimes via the product working group. Sometimes this felt a bit like "someone please give the devs a kick".

At the same time new board members were accepted wthout the new members giving concrete statements about how many resources (human or hardware) they were going to promise "upstream".

Cultural barriers to engagement for new participants is a huge deal. Lots of employees at lots of companies do not have an easy way to grok open source in general or the OpenStack way.

It's a complicated situation resolution of which likely requires greater communication between the TC, the UC (and related working groups) and the Board, but as usual, many people are over-tasked.

Meeting Minutes


Summit Followup

Some discussion on expanding the breadth of Forum sessions beyond three at a time. Most people did not want to expand as double booking can be a significant problem.

Some people (notably mriedem, nova PTL) said that the sessions seemed too dev-centric and did not provide as much user and ops feedback as was hoped. That is: the number of users and operators present was not greatly increased and the set of users and operators who spoke were the usual suspects.

Forum attendees felt obliged to be in forum rooms so missed out on opportunities to witness conference presentations and see what the real world is up to. This is unfortunate because, well, it's the real world where real stuff is happening.

But, in general, it felt like good improvement. For next time:

  • recording the on-boarding sessions
  • improving (making more visible) last minute scheduling of un-used rooms
  • greater preparation/agenda-making for some sessions

Next Steps TC Vision

There was a feedback session at summit regarding the vision:

The feedback from that needs to be integrated with the feedback from the recent survey, digested, and a new version of the review created. Apparently a lot of people at the summit session weren't aware of it, which says something about the info flow.

Deprecating postgresql in OpenStack

This is a move to document that postgresql is not actively tested in the gate nor considered a good choice for deployment.

Since this is my opinionated view of events: I don't think this is the right approach (because it is effectively changing a deployed API and we don't allow that other places), but there are many legitimate reasons for the described approach, mostly to do with not misleading people about the degree of support that upstream OpenStack is providing for postgresql.

Document voting process for formal-vote

With the change to be more mailing list oriented, changes are needed to the TC voting process to make sure that resolutions are visible to everyone and have sufficient time for review. This turns out to be more complicated than it might initially sound: some votes? all votes? count by business days or absolute days?

Thoughts from the Meeting and Summit

It felt like we spent a lot of the meeting expressing strategies for how to constrain options and actions in reaction to unavailable resources, and not enough time on considering strategies for gaining more resources. This while the board is expressing a relatively healthy budget and welcoming one new platinum and two new gold members and not being fully cognizant of how OpenStack development really works. The TC needs to be more active in expressing the needs of the technical community to the board that represents the companies that provides many resources. It's a two way street.

The TC is currently doing a lot of work to improve its process: potentially killing meetings, changing how pending resolutions are communicated to the community. This is important stuff because it will eventually help to increase participation and diversity of participants, but presumably it needs to be alongside ensuring that we (as an entire community) are making good stuff in a sustainable fashion. Only a very small amount of the discussion at today's TC meeting or at last week's board meeting was related to concrete technical planning. Hopefully the enhanced visibility that will result from increased use of the mailing list will drive some of that.

dhellman and ttx are going to start creating a presentation to the board reminding them about how things work with regard to things getting done. It's not just throwing requirements over the wall. This is good.

by Chris Dent at May 16, 2017 10:00 PM

Ihar Hrachyshka

Introduction into gratuitous ARP

The world is a jungle in general, and the networking game contributes many animals. // RFC 826

This post is the very first in a series that will cover gratuitous ARP and its relation to OpenStack. There will be six posts in the series. My plan is to post them all in span of the next two weeks. You can find the list of “episodes” already published below.

This post will start the series with a discussion of what gratuitous ARP is, and how it helps in IT operations. Later posts will touch on how it applies to OpenStack and Linux kernel, and also discuss several issues that you may want to be aware of before building your OpenStack operations on the protocol.

Let’s dive in.

ARP (Address Resolution Protocol)


ARP is one of the most widely used protocols in modern networks. Its history goes back to early 80s into the times of DARPA backed internetworking experiments. The very first RFC 826 that defined the protocol is dated November 1982 (it’s 35 years old at the time of writing). Despite the age, it’s still a backbone of local IPv4 network connectivity. Even in 2017 (the year I draft this post), it’s still very hard to find a IPv6-only network node, especially outside cloud environments. But that’s IPv4, so how does ARP fit into the picture?

To understand the goal of ARP, let’s first look at how network nodes are connected. The general model can be described as a set of hosts, each having one or more Network Interface Controller (NIC) cards connected to a common data link fabric. This fabric comes in different flavors (Ethernet, IEEE 802.11 aka WiFi, or FireWire). Irrespective of particular fabric flavor, all of them provide similar capabilities. One of the features that are expected from all of them is some form of endpoint addressing, ideally globally unique, so that network hosts connected to a shared medium can distinguish each other and address transferred data to specific peers. Ethernet and IEEE 802.11 are probably the most popular data link layers in the world, and since they are largely identical in terms of NIC addressing, in next discussions we will assume Ethernet fabric unless explicitly said otherwise.

For Ethernet, each NIC card produced in the world gets a unique 48-bit long hardware address allocated by a vendor under IEEE supervision that guarantees that no hardware address is allocated to two NIC cards. Uniqueness is to ensure that whichever hardware you plug into your network, it will never clash in address space with any other card also attached to the network. An example of a EUI-48 address would be, in commonly used notation, f4:5c:89:89:cd:54. These addresses are widely known as MAC addresses, and so I will also use this term moving forward.

It all means that your NIC already has a unique address, so why do you even need IP addresses? Sadly, people are bad at memorizing 48 randomized bits, so an easier scheme would be handy. Another problem is whenever your NIC dies and you replace it with a new one, the new card will have another unique address, and so you would need to advertise the new MAC address to all your network peers that may need to access your host.

And so engineers were looking for a better scheme to address network hosts. One of those successful alternative addressing proposals was IPv4. In this scheme, IPv4 addresses are defined 32-bit long. Still a lot, but the crucial point is that now you could pick addresses for your NIC cards. With that freedom, you could pick the same bit prefix for all your hosts, distinguishing them by a shorter number of trailing bits, and memorize just those unique bits, and configure your networking software to use the same prefix for network communication with other hosts. Then whenever you want to address a host, you pass unique trailing bits assigned to the host into your networking stack and allow it to produce the resulting address by prepending the common prefix.

The only problem with this approach is that now you have two address schemes: MAC addresses and IP addresses, with no established mapping between them. Of course, in small networks, you could maintain static IP-to-MAC mappings in sync on every host, but that is error prone and doesn’t scale well.

And that’s where ARP comes in to the stage. Instead of maintaining static mappings across hosts, the protocol allows to dynamically disseminate the information on the wire.

Quoting the abstract of RFC 826:

Presented here is a protocol that allows dynamic distribution of the information needed to build tables to translate an address A in protocol P’s address space into a 48.bit Ethernet address.

And that’s exactly what we need.

While the abstract and even the RFC title talk about Ethernet, the mechanism rendered so successful that it was later expanded to other data links, including e.g. FireWire.


The protocol introduces both ARP packet format as well as its state machine. Sadly, the RFC doesn’t contain a visual scheme for ARP packets, but we can consult the protocol Wikipedia page.

The RFC describes an address translation (ARP) table for each host storing IP-to-MAC mappings. It also defines two operations: a REQUEST and a REPLY. Whenever a host wants to contact an IP address for which there is no mapping in the local ARP table, the host sends a REQUEST ARP packet to broadcast destination MAC address asking the question “Who has the IP address?” Then it’s expected that the host carrying the IP address will send a REPLY ARP packet back with its own MAC address set in “Sender hardware address” field. The original host will then update its ARP table with a new IP-to-MAC mapping and will use the newly learned value as a destination MAC address for all communication with the IP address.

One thing to clarify before we move forward: this is all true assuming both interacting hosts are on the same layer-2 network segment, without an IP gateway (router) in between. If hosts are located in different segments, then connection between them is established through a router. In this case, a host willing to communicate with a host in another segment will determine that fact by inspecting its IP routing table. Since the destination IP address then would not belong to the local network IP prefix, the host will instead send the data to the default router IP address. (Of course, at this point the host may also determine that its ARP table doesn’t contain an entry for the gateway IP address yet, in which case it will use ARP to learn about the router MAC address.)

ARP table invalidation

One interesting aspect of the original RFC is that it doesn’t define a mechanism to update existing ARP table entries with new MAC addresses. Back in 1982, it was probably widely assumed that mobile IP stations roaming across network segments changing devices used to connect to outside world on the fly (think about how your smartphone seamlessly switches from WiFi to LTE) were not a too realistic use case. But even then, in “Related issue” section of the document, some ideas on how it could be implemented if needed were captured.

One suggestion was for every host to define “aging time” for its ARP entries. If a peer host is detected as unreachable (probably because there was no incoming traffic using both the MAC and IP addresses stored in ARP table), the originating host could remove the corresponding ARP entry from its table after it’s “aged”. This mechanism is indeed used in most modern ARP implementations, with 60 seconds being the common default for Linux systems (can be overridden using gc_stale_time sysctl setting).

It means that your connectivity to a roaming IP host will heal itself after a minute of temporary down time. While that’s great, some use cases would benefit from a more rapid reaction of hosts to network changes.

And that’s where gratuitous ARP comes into play.

Gratuitous ARP

Gratuitous ARP is an ARP packet that was never asked for (hence its alternative name – unsolicited ARP). RFC 826, “Related issue” section, mentions an algorithm to update existing ARP table entries in the network based on unsolicited ARP packets. But it’s only RFC 2002, “IP Mobility support” from year 1996 that made it part of a standard and introduced the very term “gratuitous ARP”.

RFC 2002 discusses protocol enhancements for IP networks to allow for IP devices roaming across networks without introducing significant connectivity delays or disruptions. Among other things, it defines the algorithm to be used to update existing ARP table entries with new MAC addresses. For this matter, it adopts the proposal from RFC 826, where a host can broadcast a gratuitous ARP packet into a network, and its peers then update their tables with the new MAC address sent, restoring connectivity even before old ARP entries expire.

There are two main use cases for gratuitous ARP. One is to quickly switch between multiple devices on the same host. Another is to move services exposed through an IP address from one host to another transparently to network peers.

This last scenario may happen either as part of a planned action on behalf of an Ops team managing a service, or triggered by a self-healing mechanism used in networks to guarantee availability of services in case of software or network failures. One piece of popular software that allows to fail over IP addresses from one host to another is keepalived which uses the VRRP protocol to negotiate between hosts which node should carry IP addresses managed by the software.

In OpenStack Neutron, gratuitous ARP is how floating IP addresses roam between ports; they also help with failing over IP addresses between HA router instances.

In the next post, I will expand on how OpenStack Neutron uses and implements gratuitous ARP.

by ihrachyshka at May 16, 2017 02:18 PM


OpenStack Summit Boston Replay

From the latest OpenStack news, to baseballs puns, to the surprise appearance of Edward Snowden, the the Boston OpenStack Summit was a hub of excitement.

by Diana Dvorkin at May 16, 2017 01:49 PM

OpenStack Superuser

Check out the OpenStack project updates

If you’re interested in getting up to speed on what’s next for OpenStack software in the current development cycle, the project update videos from the recent OpenStack Summit Boston are available now.

There are over 30 of these project updates, where you’ll hear from the Project Team Leaders (PTLs) and core contributors about what they’ve accomplished, where they’re heading for future releases and how you can get involved.

You can find the complete list of them on the OpenStack video page. You can also get a complete overview of them (with the videos) on the project navigator.

Some that you won’t want to miss include:


Previously, these updates trickled out as interviews on Superuser following the Design Summit, but with the advent of the Project Teams Gathering and the shifting of the release cycle, the PTLs and core contributors met at the Summit Forum to discuss features and goals.


The post Check out the OpenStack project updates appeared first on OpenStack Superuser.

by Superuser at May 16, 2017 11:07 AM

May 15, 2017

OpenStack Superuser

OpenStack Boston Summit recap: 50+ things you need to know

BOSTON—From the moment the keynote doors opened to the StackCity party at Fenway Park to the close of the last breakout session, the OpenStack Summit Boston covered all the bases. With a focus on new delivery models like Private Cloud-as-a-Service making OpenStack easier to consume, executing on multi-cloud strategy and collaboration across open infrastructure communities, we packed a lot of content and announcements into one week.

If you didn’t attend—or even if you did and want a replay—Superuser collected the announcements, users and Forum discussions you may have missed. You can also catch all of the videos for the Summit sessions on the OpenStack website.

Jump to new case studies
Jump to live demos
Jump to roadmap & technical decisions
Jump to news from the OpenStack ecosystem
Jump to what’s next

Let’s start with the OpenStack Foundation announcements:

1) Remotely Managed Private Cloud opened shop in the OpenStack Marketplace. This new category, under private-cloud-as-a-service, delivers immediate value, cost savings and public cloud experience on-premises. It’s currently offered by nine vendors in the OpenStack ecosystem—Aptira, Canonical, Cisco, EasyStack, IBM, Mirantis, Platform9, Rackspace and UnitedStack. In addition to GE Healthcare, other users taking advantage of this consumption model are the Chemical Abstract Service and Tele2.  

2) The OpenStack board of directors approved China Unicom and FiberHome Telecommunication Technologies as new Gold Members for the Foundation, joining 21 other Gold Members. Both new Gold Members demonstrate OpenStack’s strategic value for networking and large-scale service providers.

3) A new version of the OpenStack project navigator launched. It now features all user-facing services, in addition to sample configurations and new project mascot logos. The project navigator enables OpenStack users to dig deeper into individual project details, so they can make informed decisions on how to consume the software. 

4) The Foundation has heard from users across telecom, retail and manufacturing who are interested in pursuing Edge Computing architectures on OpenStack, and in Boston announced new initiatives to enable collaboration across users and other open source technologies to speed up Edge deployments. The ‘OpenStack on the Edge’ BoF was standing room only, and the community will continue discussions at a collaborative event called OpenDev scheduled for September 7-8 in San Francisco.

5) Surprising Boston Summit attendees and gluing them to their keynote seats, Mark Collier did a live interview with Edward Snowden, who provided a provocative point of view whether or not you support his actions. From a remote location, Snowden fielded questions that OpenStack community submitted via Twitter, including why open source infrastructure matters in regards to data privacy and the ethical implications of working in open source. Catch a transcript of their conversation here.

6) The new Forum concept debuted, bringing developers and operators to the same table for strategic planning discussions. Across four days, over 60 sessions were held to hack, discuss and plan for future releases.

Now, a word from some OpenStack users in production:

7) Speaking for the first time, the U.S. Army Cyber School captivated the keynote audience with their story of educating soldiers with a curriculum powered by OpenStack using Heat templates.

8) Amit Tank and Sunil Jethwani announced that the new AT&T Entertainment Group offering, DirecTV Now is powered by OpenStack. OpenStack enables the platform’s viewers to flip between live TV and on-demand shows and even allows viewers to stream live events like this year’s Super Bowl. Looking to the future, Tank says that their team will be prioritizing containers on bare metal, functions-as-a-service and more usage of Kubernetes to make hybrid clouds more seamless.

9) Gap Inc., which deployed its first OpenStack cloud in 2013, pulled back the curtain for the first time last week in Boston. Gap powers its e-commerce and in-store systems with OpenStack, so when you buy from Banana Republic, Gap, Old Navy or Athleta, you’re using OpenStack. ComputerworldUK shared more insights from lead architect Eli Elliott and you can catch the full session here.

10) Driven by need for compliance, speed and flexibility, Patrick Weeks from GE Digital (Healthcare) spoke about how quickly he was able to migrate sensitive healthcare applications to the cloud using Rackspace’s remotely managed private cloud offering.

11) eBay shared details about their massive OpenStack deployment with 167,000 virtual machines, 68,000 bare metal machines and 13PB storage, which runs 4,000 applications and 95 percent of their business.  Suneet Nandwani announced his team will  open-source TessMaster, a management framework to run Kubernetes across multiple clouds and wants other OpenStack users to collaborate.

12) China Mobile passed the torch for the first ever Superuser Awards tie—Paddy Power Betfair and UKCloud won the sixth round of the awards after community rating and judging by the Superuser editorial advisors.

13) Verizon showcased a cloud-in-a-box system using OpenStack for edge computing that’s about the size of a home router. Beth Cohen, distinguished member of technical staff at Verizon, used data to show why they are looking to the edge: They have a 4LTE network available to 98 percent of the US population, a global IP network covering six continents, IP and data services in over 150 countries and seven security operations centers on four continents.

14) The UK Civil Service, working with Red Hat hosted by UKCloud, spoke for the first time in Boston about their cloud strategy and its effect on the process for UK companies to obtain import and export licenses, including a projected 50 percent cost savings compared to previous IaaS solutions.

15) SkyTV, the leading television provider in New Zealand,  brought their experience building a hybrid cloud using OpenStack and AWS. They shared how their move to a hybrid OpenStack model enabled their provisioning process to be reduced from six weeks to three minutes. More from Superuser here.

16) The Oklahoma Medical Research Foundation (OMRF) shared how they use OpenStack Swift in a stand-alone deployment, integrating it with Docker and Hashicorp. With Swift, OMRF was able to solve their space problems and move to an active archive, which is durable and fast enough to be the primary storage for the scientific data sets and analysis on it.

17) At the end of the week, more than 30 technical leaders distributed across 13 countries from the Latin America, including the América Móvil and Claro teams, met in Boston with members of the OpenStack Foundation to discuss plans, challenges and collaboration opportunities to achieve their open infrastructure innovation goals.

18) Amadeus IT Group, a leading technology partner for the global travel industry, announced deployment of an enterprise private cloud based on VMware Integrated OpenStack (VIO) to support digital transformation initiatives. A supporter of and contributor to open source projects, Amadeus selected OpenStack because seeking a platform conducive to agile development.

19) The Massachusetts Open Cloud (MOC), an initiative dedicated to the creation of publicly available cloud-computing resources for running big data workloads, has standardized on Red Hat OpenStack Platform and Red Hat Ceph Storage for its open, scalable and flexible cloud infrastructure. MOC was formed by a consortium of private and public institutions, including Boston University, Harvard University, Northeastern University, MIT and the University of Massachusetts.

20) The University of Messina and Fujitsu worked together to further develop the Smart City of Messina. The University of Messina developed Stack4Things, a framework to collect and send information from devices to an OpenStack Cloud. Monasca, the OpenStack monitoring service, analyzes and presents the measured data. Monasca monitors the health of the devices, too.

21) GARR, Italy’s leading research and education network consortium, created a countrywide federated OpenStack cloud using Ubuntu OpenStack and Canonical’s model driven operations tools, MAAS and Juju. GARR manages a fiber-optic network of 15,000 kilometers responsible for high-performance connectivity and advanced services to around 4 million users and over 1,200 research institutes, universities, research hospitals, cultural institutions, libraries, museums and schools.

22) The Oak Ridge National Laboratory (ORNL), the largest U.S. Department of Energy science and energy laboratory, also announced it’s using Red Hat’s OpenStack Platform to deliver cloud services in its Compute And Data Environment for Science (CADES).

For more OpenStack user stories, check out all of the sessions from the OpenStack Summit Boston case studies track.

Oh, and we’re always interested in hearing from the community—contact to share your use cases.

It wouldn’t be an OpenStack Summit without some live demos:

23) 15 global IT leaders deployed the CockroachDB cloud-native application on Kubernetes on top of OpenStack in the latest version of the Interop Challenge. The demo successfully showed that containers can run smoothly on and across multiple OpenStack clouds and that distributed enterprise database systems can be deployed automatically using open source tools such as Ansible and OpenStack Shade. All 15 participants successfully deployed the cloud-native application, including Open Telekom Cloud and Vexxhost running directly on their public clouds in production.

24) Julia Kreger, a developer advocate at IBM bootstrapped a mini-data center on a DellEMC rack, with Kubernetes running on bare metal servers using Ironic and Neutron.

25) John Griffith, principal software engineer at NetApp and Kendall Nelson, OpenStack Foundation, upstream developer advocate, joined Kreger and Foundation COO Mark Collier onstage to deploy Cinder as an independent service using Docker. This makes it easy to consume the block storage service alongside other technologies, such as Kubernetes. Live demo fates aside, check out the demo  here.

26) Jakub Pavlik, director of product engineering at Mirantis demonstrated the power of one platform for bare metal, VMs, and containers with a big data application, utilizing Spark, Kafka, and Hadoop Distributed File System on a common OpenContrail-powered network. He performed real-time social analysis that updated throughout the demo, showing the popular Twitter hashtags in the Boston area, including #OpenStackSummit.

Here are some of the key community activities, technical decisions and roadmap discussions:

27) The community wide goals for the OpenStack Queens release were selected for proposal to the Technical Committee and the community not in attendance at the Forum. These goals include finer RBAC support, managing functional tests and completing API reference documentation.

28) In the log messages session, an implementation of error codes within logs using exception names was discussed.  Logs across services will include unique request IDs in all log messages related to a specific request.

29) At the request of large-scale users like Yahoo! who have said that dnsmasq doesn’t scale to fit their needs, operators and developers discussed the need for support for DHCP servers other than dnsmasq like KEA DHCP.  

30) There was an overwhelming amount of support for etcd v3 as the distributed lock management solution.

31) The community will begin transitioning from supporting multiple database solutions to focusing on improving usage with MySQL for better testing and performance.

32) With a whole Forum session dedicated to making simple use cases with Neutron easier, a plan was formed to document and make consistent the process a virtual machine (or other compute instance) can take to configure its networking interface(s).

33) Back by popular demand, 34 project team leads (PTLs) provided project updates to Summit attendees, sharing the latest features and what can be expected in the Pike release. In the upcoming weeks, the videos will be uploaded to the new project navigator.

34) A group of researchers and technologists from around the world gathered in the first of a series of open cloud congresses at the OpenStack Summit Boston and onsite at MIT.  Under the title of the Open Research Cloud Declaration, this group intends to begin the dialogue on, and develop draft conventions to enable a more effective interlacing of the international scientific community.  The declarations from the congress will be circulated at subsequent events in Australia and Europe, with the objective of encouraging further discussions and contributions towards a set of conventions that all can be signatories to. Stay tuned for more info!

35) The OpenStack Summit invited relevant open source communities to participate in the first Open Source Days, dedicated tracks run by the various communities for education and collaboration across projects. Ansible, Ceph, Cloudily, Cloud Foundry,, Kubernetes, OpenContrail, Open Daylight, Open Switch, Open vSwitch and OPNFV all hosted mini-events in Boston.

36) On the last day of the Summit, 11 community members were recognized by the Community Contributor Awards, with quirky categories like the Duct Tape Award. A special trophy—awarded by the OpenStack User Committee—was handed to Tom Fifield for his dedication in fighting for the users.

Over 100 organizations sponsored the OpenStack Summit Boston. Here’s what some of them had to say:  

37) Animbus® Cloud OS is the OpenStack-centric product developed by 99Cloud with further development and optimization in computing, storage, network, image, authentication, telemetry and other modules. Compatible with heterogeneous virtual software, server, storage and network equipment, it offers a one-stop solution for an enterprise cloud computing platform that’s easy to scale and highly available.

38) China Mobile signed a memorandum of understanding (MoU) with UMCloud (Mirantis in China), ARM, Cavium and Enea, the third batch of companies to partner with the China Mobile Open NFV lab. These partners will carry out a series of tests with multiple NFV typical scenarios and services.

39) City Network announced that its OpenStack-powered public infrastructure service City Cloud, has joined Interxion’s Cloud Connect platform. Enterprise customers can now connect directly with City Cloud through City Connect. This enables customers to extend their network with high speed and secure connections to City Cloud points of presence in Interxion’s Stockholm, London and Frankfurt campuses.

40) EasyStack launched ESContainer, integrating Kubernetes and OpenStack technologies. ESCloud focuses on the data center infrastructure management and supports the operations of traditional business applications while ESContainer focuses on providing a one-stop platform for the transformation of applications in the new enterprise.

41) Fujitsu and Mirantis announced a partnership to collaborate around integrating Mirantis Cloud Platform with Fujitsu’s hardware, software and support capabilities. Building on a wide range of private cloud offerings, Fujitsu will add a new privately managed global OpenStack based cloud on the Mirantis Cloud Platform, launching June 2017 in Japan followed by other regions. 

42) HPE launched HPE Helion OpenStack 5.0, the latest version of their enterprise-grade distribution of OpenStack based on Newton.

43) Inspur released the latest version of In-Cloud OS 5.0, completely based on the OpenStack architecture and offering functionality, availability, security and toolchains to further extend the cloud service directory. It uses a micro-service architecture to support rapid deployment on Docker, providing automatic perception, intelligent management and automated service delivery.

44) NetApp introduced the latest version of its next-generation object storage software, StoargeG. The latest enhancements include StorageGRID Webscale, more software deployment options, including Docker container support and installation on bare-metal servers as well as simpler OpenStack storage deployments with Keystone integration, Swift access control list (ACL) support, and Heat-less deployment.

45) Red Hat OpenStack Platform 11, the latest version of Red Hat’s massively scalable and agile cloud IaaS, was released. Based on the OpenStack Ocata release, Red Hat OpenStack Platform 11 delivers enhanced support for upgrades with composable roles, new networking capabilities and improved integration with Red Hat CloudForms for cloud management.

46) NEC participated in trials with Red Hat to confirm the feasibility of KDDI’s next generation integrated platform, which includes infrastructure and IaaS, and can operate with multiple types of systems. KDDI provides high-quality communication and services in line with the company’s life design strategy. In the future, as 5G and IoT services become more prevalent, it’s expected that development and operation of fast and efficient services will be necessary on a larger number of servers and infrastructure.

47) Rackspace announced a partnership with Dell EMC to deliver OpenStack private clouds with leading compute and storage solutions from Dell EMC. They’ll combine Rackspace’s operational expertise with Dell EMC compute and storage solutions to offer an easy-to-consume, private cloud-as-a-service offering. It’s the first step in an expanded relationship with Dell EMC in which both companies will help lower the barrier to entry for private clouds and deliver rapid private cloud elasticity through a utility-based consumption model.

48) After a lot of optimization efforts in memory and TCP connections management, the Shanghai Engineering Research Center for Broadband Networks and Applications announced the latest version of DCFabric can support 3,000 switches with one instance. The new HA solutions of DCFabric, which has been testified at some practical environments, were also introduced.

49) Storage Made Easy launched storage provider support for Keystone v3 with OpenStack Swift. Some of the top governance features include enhancements focusing on PCI DSS. These include PCI DSS notifications, password requirements API, lockout rules and immediate password rotation.

50) SUSE announced OpenStack Cloud Monitoring, an open source software solution that simplifies monitoring and managing the health and performance of enterprise OpenStack cloud environments and workloads.

51) Verizon has expanded its Virtual Network Services product offering by adding x86-based whitebox options, leveraging OpenStack, to its universal customer premises equipment (uCPE) portfolio. Verizon’s uCPE eliminates the need for enterprises to invest in separate hardware appliances for critical virtual network functions (VNF) such as software-defined WAN (SD-WAN), security, routing, WAN optimization, or any other network function that can be virtualized.

52) VirTool Networks Inc. released VirTool Network Analyzer, a tool designed to simplify the process of fixing network problems on OpenStack. VirTool Network Analyzer delivers complete layer 2 virtual network visualization along with advanced packet capture and traffic tracing capabilities. It allows network engineers to identify the cause of failures in a fraction of the time compared to conventional methods.

53) vScaler announced a partnership with Datera to provide storage-as-a-service on OpenStack. This new partnership between vScaler and Datera will deliver scalable private clouds for a wide variety of workloads, from high-performance databases to archival storage.

What’s next

That’s a wrap for the Boston Summit — but we’re not unpacking our suitcases just yet.  There are 16 OpenStack Days, regional community-organized events, scheduled across Europe, Asia, Australia and Latin America in the next four months, so check out the full schedule.

We also can’t wait to get the global community together again at OpenStack Summit Sydney, November 6-8, 2017. See you there!

The post OpenStack Boston Summit recap: 50+ things you need to know appeared first on OpenStack Superuser.

by Superuser at May 15, 2017 04:35 PM

Julien Danjou

OpenStack Summit Boston 2017 recap

The first OpenStack Summit of 2017 was last week, in Boston, MA, USA. I was able to attend as I've been selected to give 3 talks, to help for a hands-on and to animate an on-boarding session. This made sure I was a bit busy every day, which was good.

This is the first summit to happen since the new Project Team Gathering (PTG) happened last February. I was unable to attend this first PTG back then, as there was no way to justify my presence there. The OpenStack Telemetry team that I lead is pretty small. People don't really need to talk to each other face to face to discuss: therefore we decided to not ask to be present during the last PTG event.

The Telemetry on-boarding session that I organized with my fellow developer Gordon Chung on Tuesday had only 3 people showing up to ask a few questions about Telemetry. The session lasted 15 minutes on 90 planned. We shared that session with CloudKitty, for which nobody showed up for. When you think about it, this was really disappointing but did not come as a surprise.

First, the amount of company engaging developers into OpenStack has shrunk drastically during the last year. Secondly, since there's now another event (the PTG) twice a year, it seems pretty clear that every developer will not be able to attend all the 4 events every year, creating dispersion in the community.

I personally was glad to attend the Summit rather than the PTG, as it is more valuable to meet operators and users than developers to gather feedback. However, meeting everyone at the same time would be great, especially for smaller teams. The PTG scattered some teams to a point that many of developers of those lineups won't go to either the PTG nor the OpenStack. As a consequence, I won't have any meeting point in the future with many of my fellow developers around OpenStack. I warned the Technical Committee last year about this when it was decided to reorganize the events. I'm glad to be right but I'm a bit sad that the Foundation did not listen.

Though all the projects I work on tend to follow the good practice I wrote last year. Therefore I cannot say that it has huge consequences on the projects I work on. It's a loss as it makes it harder to reach users and operators for some of us. It also reduces our occasion for social interaction, which was a great benefit. But it will not prevent us from building great software anyway!

The few other sessions of The Forum (the space dedicated to developers during the Summit) that I attended discussed various technical things, and some sessions were pretty empty. I wonder if it was a lack of interest of people or if people were unable to travel to discuss those items. Anyhow, at this stage I am not sure it would have really mattered: this has been my 9th OpenStack Summit and many of the subjects discussed already have been discussed multiple time with barely any change since. Talk is cheap. Furthermore, most of the discussion were not made by stakeholders of the various projects involved, but by people on the side, or by members of the Technical Committee. There is just unfortunately too much of wishful thinking.

On the talk side, my presentation with Alex Krzos entitled Telemetry and the 10,000 instances went pretty well. We demonstrated what how we tested the performance of the telemetry stack. Same goes for my hands-on with the CloudKitty developers, where we managed to explain how Ceilometer, Gnocchi, and CloudKitty were able to work with each other to create nice billing reports. The last day was concluded with my talk on collectd and Gnocchi with Emma, which was short and to the point. My final talk was about the status and roadmap of the OpenStack Telemetry team where I tried to explain how the Telemetry works and what we might do (or not) in the next cycles. It was pretty short as we barely have a roadmap, the project having 3 developers doing 80% of the work.

I was also able to catch up with Nubeliu about their Gnocchi usage. They presented a nice demo of the cloud monitoring solution they build on top of Gnocchi. They completely understood how to use Gnocchi to store a large number of metrics at scale and how to leverage the API to render what's happening in your infrastructure. It is pretty amazing.

While I missed the energy and the drive that the design session used to have in the first summits, it has been a pretty good summit. I was especially happy to be able to discuss OpenStack Telemetry and Gnocchi. The feedback I gathered was tremendous and terrific and I'm looking forward to the work we'll achieve in the next months!

by Julien Danjou at May 15, 2017 03:00 PM

May 12, 2017

NFVPE @ Red Hat

Gleaming the Kube – Building Kubernetes from source

Time to bust out your kicktail skateboards Christian Slater fans, we’re about to gleam the kube, or… really, fire up our terminals and build Kubernetes from source. Then we’ll manually install what we built and configure it and get it running. Most of the time, you can just yum install or docker run the things you need, but, sometimes… That’s just not enough when you’re going to need some finer grain control. Today we’re going to look at exactly how to build Kubernetes from source, and then how to deploy a working Kubernetes given the pieces therein. I base this tutorial on using the official build instructions from the Kube docs to start. But, the reality is as much as it’s easy to say the full instructions are git clone and make release – that just won’t cut the mustard. We’ll need to do a couple ollies and pop-shove-its to really get it to go. Ready? Let’s roll…

by Doug Smith at May 12, 2017 07:21 PM

OpenStack Superuser

For Latin America, OpenStack’s time has come

BOSTON — When your team is distributed across 19 countries, opportunities to meet face-to-face are not common. The OpenStack Summit Boston provided that chance for América Móvil, Latin America’s largest phone company and one of the largest global mobile providers.

On Thursday morning, more than 30 technical leaders from the Latin America Movil and Claro teams met here in Boston with members of the OpenStack Foundation to discuss plans, challenges and collaboration opportunities to achieve their open infrastructure innovation goals.

América Móvil team members and OpenStack Foundation staffers Jonathan Bryce, Mark Collier, Danny Carreno and Heidi Bretz.


The OpenStack Summit provided more than just an opportunity to meet the community—it gave América Móvil the ability to meet the members within their own company who are working on OpenStack initiatives, a common challenge for geographically distributed teams. In Boston, they put their heads together to discuss their public cloud, big data and network functions virtualization (NFV) initiatives and gain insights into how to successfully navigate their journey to being a cloud provider.  All of these efforts are key for a company running 363.5 million access lines, including 280.6 million wireless subscribers, 33.7 million landlines, 27.0 million broadband accesses and 22.2 million PayTV units.

After two hours, the group achieved tighter integration with the OpenStack ecosystem, especially the telecom communities. The result?  They now have plans to drive community building in Latin America and bring like-minded community members and users together through meetups and OpenStack Days events.


The post For Latin America, OpenStack’s time has come appeared first on OpenStack Superuser.

by Heidi Bretz at May 12, 2017 12:57 PM


Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


Last updated:
May 26, 2017 10:21 PM
All times are UTC.

Powered by: