December 02, 2016

Sean Roberts

What does it take to run a business cloud?

There are consistent truths that cannot be ignored. Speed of light is one. There are others below.  Cloud means distributed systems connected by networks. A vast majority of cloud implementations are really rebranded platform...

by sarob at December 02, 2016 08:21 PM

IBM OpenTech Team

Writing a Horizon Plugin

Why write a Horizon plugin?

There are many reasons you will want to package your code as a plugin. The most important reason is that it ensures the longevity and reliability of your code. Things are constantly changing in Horizon, but as long as your adhere to the plugin architecture, we guarantee that our releases will not break your plugin.

Plugins are a way to extend and add functionalities. Since you own the code, you are essentially in the driver seat. You can release it when it is ready or keep it private for internal use. Writing your code as a plugin also modularizes your code – making it easier to translate and test. Finally, if you have independent teams actively working on the same code, our plugin architecture can potentially alleviate merge conflicts that your teams will encounter.

Building your plugin

Below is a skeleton of what your plugin should look like. Note that for brevity, we will focus on creating a custom AngularJS step in the images workflow. An AngularJS plugin is a collection of JavaScript files and static resources. Because it runs entirely in your browser, we need to place all of our static resources inside the static folder. This ensures that the Django static collector picks it up and distributes it to the browser correctly.

Plugin structure

│   └─
     │ └─myplugin.module.js
This file creates a panel that would allow us to hook into Horizon. Hold on a second… why do I need to create a panel if all I want to do is append a simple step to an existing workflow? Unfortunately Horizon uses the Django plugin mechanism underneath and requires a standard file structure. Even though our panel does not show up in the dashboard, we still have to register it so that our other code gets incorporated.

from django.utils.translation import ugettext_lazy as _

import horizon

class MyPanel(horizon.Panel):
    name = _("My Plugin")
    slug = "myplugin"
The enabled file contains the configuration that registers our plugin with Horizon. The file is prefixed with an alpha-numeric string that determines the load order. Here we are saying, install myplugin and automatically discover all of my static resources.

  PANEL = 'myplugin'
  PANEL_DASHBOARD = 'project'
  ADD_INSTALLED_APPS = ['myplugin']

The code below illustrates how your code can plug into an existing workflow. In this example, we are adding mystep to an existing create-volume workflow. The most important thing to note is the templateUrl which points to our view below.

(function() {
  'use strict';


  myplugin.$inject = [

  function myplugin(basePath, workflow) {
      title: gettext('My custom step'),
      templateUrl: basePath + 'steps/mystep/mystep.html',
      formName: 'myStepForm'

This is our view. In this example, we are looping through the list of items provided by the controller and displaying the name and id. The important thing to note is the reference to our controller using the ng-controller directive.

<div ng-controller="horizon.dashboard.project.mystepController as ctrl">
  <div>Loading data from your controller:</div>
    <li ng-repeat="item in ctrl.items">
      <span class="c1">{$ $}</span>
      <span class="c2">{$ $}</span>

The controller is the glue between the model and the view. In this example, we are going to initialize it with some mocked data. This is the data that gets displayed in the view.

(function() {
  'use strict';


  function mystepController() {
    var ctrl = this;
    ctrl.items = [
      { name: 'abc', id: 123 },
      { name: 'def', id: 456 },
      { name: 'ghi', id: 789 },

Deploying Your Plugin

Now that you have a complete plugin, it is time to install and test it. But before we are able to package our plugin, we need to go cover and
This manifest file tells the packager what files to include. In our manifest, we want to include all of the static content.


recursive-include myplugin/static *
This file contains basic information about you and our plugin. I removed the information in classifiers to keep the code compact, feel free to add your own classifiers. Note that you can choose to use pbr and setup.cfg instead, but we will not be covering that in this article.

from setuptools import setup, find_packages

    name = 'myplugin',
    version = '0.0.1',
    description = 'Workflow Plugin for Horizon',
    author = 'Thai Tran',
    author_email = '',
    classifiers = [],
    include_package_data = True,


1. Run "cd <plugin> & python sdist"
2. Run "cp -rv enabled <horizon>/openstack_dashboard/local/"
3. Run "<horizon>/tools/ pip install dist/<package>.tar.gz"
4. Restart Apache

Visit the Horizon dashboard and launch the Angular images workflow. Hopefully you will see the additional step you added in your plugin. Although the example provided in this article is very basic, it highlighted the level of customization that is now available to Horizon developers, something that was previously very hard to do. At the writing of this article, users are able to extend workflows, actions, and table configurations (things like columns, headers, etc...). In the near future, we should be able to use the same architecture to extend forms, search facets, and tabs. To view a working example available on github, click here. For the full write-up, click here.

The post Writing a Horizon Plugin appeared first on IBM OpenTech.

by Thai Q Tran at December 02, 2016 06:24 PM

Elizabeth K. Joseph

OpenStack book and Infra team at the Ocata Summit

At the end of October I attended the OpenStack Ocata Summit in beautiful Barcelona. My participation in this was a bittersweet one for me. It was the first summit following the release of our Common OpenStack Deployments book and OpenStack Infrastructure tooling was featured in a short keynote on Wednesday morning, making for quite the exciting summit. Unfortunately it also marked my last week with HPE and an uncertain future with regard to my continued full time participation with the OpenStack Infrastructure team. It was also the last OpenStack Summit where the conference and design summit are being hosted together, so the next several months will be worth keeping an eye on community-wise. Still, I largely took the position of assuming I’d continue to be able to work on the team, just with more caution in regards to work I was signing up for.

The first thing that I discovered during this summit was how amazing Barcelona is. The end of October presented us with some amazing weather for walking around and the city doesn’t go to sleep early, so we had plenty of time in the evenings to catch up with each other over drinks and scrumptious tapas. It worked out well since there were fewer sponsored parties in the evenings at this summit and attendance seemed limited at the ones that existed.

The high point for me at the summit was having the OpenStack Infrastructure tooling for handling our fleet of compute instances featured in a keynote! Given my speaking history, I was picked from the team to be up on the big stage with Jonathan Bryce to walk through a demonstration where we removed one of our US cloud providers and added three more in Europe. While the change was landing and tests started queuing up we also took time to talk about how tests are done against OpenStack patch sets across our various cloud providers.

Thanks to Johanna Koester for taking this picture (source)

It wasn’t just me presenting though. Clark Boylan and Jeremy Stanley were sitting in the front row making sure the changes landed and everything went according to plan during the brief window that this demonstration took up during the keynote. I’m thrilled to say that this live demonstration was actually the best run we had of all the testing, seeing all the tests start running on our new providers live on stage in front of such a large audience was pretty exciting. The team has built something really special here, and I’m glad I had the opportunity to help highlight that in the community with a keynote.

Mike Perez and David F. Flanders sitting next to Jeremy and Clark as they monitor demonstration progress. Photo credit for this one goes to Chris Hoge (source)

The full video of the keynote is available here: Demoing the World’s Largest Multi-Cloud CI Application

A couple of conference talks were presented by members of the Infrastructure team as well. On Tuesday Colleen Murphy, Paul Belanger and Ricardo Carrillo Cruz presented on the team’s Infra-Cloud. As I’ve written about before, the team has built a fully open source OpenStack cloud using the community Puppet modules and donated hardware and data center space from Hewlett Packard Enterprise. This talk outlined the architecture of that cloud, some of the challenges they’ve encountered, statistics from how it’s doing now and future plans. Video from their talk is here: InfraCloud, a Community Cloud Managed by the Project Infrastructure Team.

James E. Blair also gave a talk during the conference, this time on Zuul version 3. This version of Zuul has been under development for some time, so this was a good opportunity to update the community on the history of the Zuul project in general and why it exists, status of ongoing efforts with an eye on v3 and problems it’s trying to solve. I’m also in love with his slide deck, it was all text-based (including some “animations”!) and all with an Art Deco theme. Video from his talk is here: Zuul v3: OpenStack and Ansible Native CI/CD.

As usual, the Infrastructure team also had a series of sessions related to ongoing work. As a quick rundown, we have Etherpads for all the sessions (read-only links provided):

Friday concluded with a Contributors Meetup for the Infrastructure team in the afternoon where folks split off into small groups to tackle a series of ongoing projects together. I was also able to spend some time with the Internationalization (i18n) team that Friday afternoon. I dragged along Clark so someone else on the team could pick up where I left off in case I have less time in the future. We talked about the pending upgrade of Zanata and plans for a translations checksite, making progress on both fronts, especially when we realized that there’s a chance we could get away with just running a development version of Horizon itself, with a more stable back end.

With the i18n team!

Finally, the book! It was the first time I was able to see Matt Fischer, my contributing author, since the book came out. Catching up with him and signing a book together was fun. Thanks to my publisher I was also thrilled to donate the signed copies I brought along to the Women of OpenStack Speed Mentoring event on Tuesday morning. I wasn’t able to attend the event, but they were given out on my behalf, thanks to Nithya Ruff for handling the giveaway.

Thanks to Nithya Ruff for taking a picture of me with my book at the Women of OpenStack area of the expo hall (source) and Brent Haley for getting the picture of Lisa-Marie and I (source).

I was also invited to sit down with Lisa-Marie Namphy to chat about the book and changes to the OpenStack Infrastructure team in the Newton cycle. The increase in capacity to over 2000 test instances this past cycle was quite the milestone so I enjoyed talking about that. The full video is up on YouTube: OpenStack® Project Infra: Elizabeth K. Joseph shares how test capacity doubled in Newton

In all, it was an interesting summit with a lot of change happening in the community and with partner companies. The people that make the community are still there though and it’s always enjoyable spending time together. My next OpenStack event is coming up quickly, next week I’ll be speaking at OpenStack Days Mountain West on the The OpenStack Project Continuous Integration System. I’ll also have a pile of books to give away at that event!

by pleia2 at December 02, 2016 02:58 PM

OpenStack Superuser

What do you think of a joint Kubernetes OpenStack environment?

Container schedulers like Kubernetes, Swarm, Mesos, Rancher and others have become a very hot topic in cloud infrastructure discussions. They promise a level of abstraction over infrastructure that reduces operational complexity and improves interoperability by hiding many of the differences between different platforms.

Of these benefits, the idea of integrated mechanisms for upgrades and high availability is especially attractive; however, these features are not free.  They require applications be adapted for the platforms.

With such high potential benefits, it’s no surprise that some in the OpenStack community want to use these platforms to also operate OpenStack. In my opinion, this confusing fusion of the technologies is inevitable because of the rapidly expanding footprint of container schedulers.

<figure class="wp-caption alignright" id="attachment_3774" style="width: 474px">Joint OpenStack Kubernetes Environment<figcaption class="wp-caption-text"> CC BY // CC BY</figcaption> </figure>

For the Barcelona Summit, I put together a pragmatic review of the pros and cons of this approach including an iterative path forward. This review included the architecture diagram in this post.

Many people were glad to see a balanced operations focused presentation.

Several points generated discussion:

First was the distinction between plain old container installs (“POCI”) and this more automated approach. The key element being use of immutable 12-factor designs.

Next were the OpenStack services that can be more easily adapted to this design and how to evaluate services that can easily survive IP movement and mid-transaction termination.

We also discussed broader internal refactoring that would be required. Happily, these items would make OpenStack more durable and upgradable in all operational scenarios.

Not everyone agrees with my iterative and cautious recommendation. For example, engineers from Mirantis have been very vocal in the community promoting this approach — dubbing it  “Evolve or Die”  — with support from CoreOS and Intel (see also their  “Lessons Learned” session from Barcelona and the earlier “Stakanetes” intro).

They have been able to run all services in containers including the Nova Compute, Libvirt and agents by pinning containers to specific hosts. This approach does provide benefit of using Kubernetes to manage the version of containers running OpenStack components. However, much work remains to ensure that it’s operationally pragmatic.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

Overall, I believe that declarations of broad success are overstated due to marketing pressures to show progress. More importantly, I believe that the rationale for adding this complexity into the environment not well developed. It is important to note that the Kubernetes community itself is still working on critical operational details like HA, security and upgrades.

That does not mean that I’m giving up on the effort! In fact, I’m very excited to collaborate on taking steps towards a more integrated solution. The RackN team has been pulling together many of the control elements that are required to build the integration (check out our Kubernetes demo).

Right now, I’m refining my “Will it Blend” talk based on feedback and will be taking it on the road to the Mountain OpenStack Days Salt Lake City, the first week of December.

I’d love to hear your thoughts about the presentation, the Kubernetes underlay approach and making OpenStack operationally robust in general.

Rob Hirschfeld has been active in the OpenStack community since 2012. He’s served as a Board member and is a tireless champion of interoperability. You can also find him on Twitter or on his blog.



Superuser is always interested in community content, get in touch at

The post What do you think of a joint Kubernetes OpenStack environment? appeared first on OpenStack Superuser.

by Rob Hirschfeld at December 02, 2016 12:46 PM


hastexo Academy: Now with OpenStack Newton & Juju 2.0!

Every month, our HX201 Cloud Fundamentals for OpenStack course gets a refresh. Usually, those are small, incremental changes, but this December there's a bigger update.

OpenStack Newton

Firstly, we've updated the course from OpenStack Mitaka to OpenStack Newton, the latest OpenStack release that dropped just a few weeks back. This means that you now get to deploy an OpenStack cluster, live and fully interactively as before — except you are now deploying the very latest OpenStack release.

Ubuntu-branded OpenStack Dashboard (Horizon) running on OpenStack  Newton

Ubuntu-branded OpenStack Dashboard (Horizon) running on OpenStack Newton (screenshot taken from HX201 course)

Juju 2.0

As always, the course comes in two flavors: one with OpenStack managed with OpenStack-Ansible, and another with Juju. And as far as Juju is concerned, here's the other big change: the entire course now runs based on Juju 2.0.

This also means you get to deploy your OpenStack services into LXD containers, have a new and improved Juju GUI at your disposal, and are able to take advantage of the much more intuitive CLI.

Juju 2.0 GUI

Juju 2.0 GUI with fully deployed OpenStack (screenshot taken from HX201 course)

Ubuntu 16.04 LTS

And finally, of course, this refresh means that we've also updated to the latest Ubuntu LTS release (16.04 Xenial Xerus). This goes for both the Ansible and the Juju 2.0 course flavor.

Get it while it's hot!

Want to get some new knowledge into your system before the year is over? Now is your chance. HX201 is available for purchase from our web site, and you can get cracking any time.

by florian at December 02, 2016 12:00 AM

December 01, 2016

Assaf Muller

Upstream Contribution – Give Up or Double Down?

Ever since I’ve been involved with OpenStack people have been complaining that upstream is hard. The number one complaint is that it takes forever to get your patches merged. I thought I’d take a look at some data and attempt to visualize it. I wrote some code that accepts an OpenStack project and a list of contributors and spits out a bunch of graphs. For example:

  • How long does it take to merge patches in a given project over time? Looking back, did governance changes affect anything?
  • Is there a correlation between the size of a patch and the length of time it takes to merge it? (Spoiler: The answer is… Kind of)
  • Looking at an author: Does the time to merge patches trend down over time?
  • Looking at the average length of time it takes to merge patches per author, how does it look like when we graph it as a function of the author’s number of patches? Reviews? Emails? Bug reports? Blueprints implemented?

The data suggestes answers for many of those questions and more.

Here’s the code for you to play with:

And some conclusions in the slides embedded below:

<iframe frameborder="0" height="569" marginheight="0" marginwidth="0" src=";loop=false&amp;delayms=10000" width="696"></iframe>

Here’s a few resources about effective upstream contribution. It’s all content written by and for the Neutron community but it’s applicable to any OpenStack project.

by assafmuller at December 01, 2016 11:26 PM


The Next Big Thing for AppFormix is... Juniper

Today, we have some big news at AppFormix. Juniper Networks announced today its intent to acquire AppFormix. This is great news for our team, our customers and our technology.

by Sumeet Singh ( at December 01, 2016 09:15 PM

David Moreau Simard

ARA 0.10, the biggest release yet, is out !

19 commits, 59 changed files, 2,404 additions and 588 deletions… and more than a month’s on and off work.

0.10 is out !

Where to get it ? Get started easily by installing and configuring ARA.

I’m excited to tell you about this new release ! Here’s a few highlights !

An improved web application browsing experience

A lot of work has gone into the browsing experience: less clicks, more information, faster.

I am by no means a professional frontend developer and I would definitely love help on this front but I think things are definitely starting to look good ! I’ve added updated previews to the documentation:

It’s awesome to see how much the interface has improved over time.

Curious ? Look at the Alpha and Beta video previews to see how far the project has come !

Two new Ansible modules

This new release features two new Ansible modules, ara_record and ara_read.

These allow you to respectively write and read persistent data that is made available throughout your playbook run and in the web interface.

For example:

- name: Test playbook
  hosts: localhost
    - name: Get git version of playbooks
      command: git rev-parse HEAD
      register: git_version

    - name: Record git version
        key: "git_version"
        value: "{{ git_version.stdout }}"

That would make the git_version key/value available in the web interface and on the CLI.

And you can record different kind of data, too, see:

- ara_record:
    key: "{{ item.key }}"
    value: "{{ item.value }}"
    type: "{{ item.type }}"
    - { key: "log", value: "error", type: "text" }
    - { key: "website", value: "http://domain.tld", type: "url" }
    - { key: "data", value: '{ "key": "value" }', type: "json" }

Setting a type will make it so it’s displayed accordingly in the interface. More types will eventually be made available !

Improved unit and integration testing coverage

In order to make sure ARA works well — and keeps working well — we need to have a good amount of unit and integration testing.

This coverage was improved since the previous release and additionnally, we now test against different versions of Ansible on both CentOS and Ubuntu.

Database schema finally declared stable

Was ARA unstable before ? Not really. However, at the rapid pace of development we were having with the project, we decided to avoid managing SQL migrations to avoid unnecessary overhead as the database schema was moving a lot.

I feel the database is fairly fleshed out at this point at any new modifications will be handled automatically when running ARA.

Any version of ARA >= 0.9.0 is considered a stable and managed database schema.

This is great, let’s make it better !

The feedback around ARA has been awesome, keep it coming ! A lot of the improvements that are part of this release is directly from ideas and comments provided by users.

If you want to come chat, learn or discuss about ARA, you’ll find us on #ara on IRC in the freenode server !

by dmsimard at December 01, 2016 06:47 PM

OpenStack Superuser

China puts money where the cloud is, ramps up OpenStack adoption

When it comes to OpenStack, companies in China have taken a seat at the table.

In the November board meeting, China Telecom, Inspur and ZTE were approved as the newest Gold Members increasing the proportion of companies from China to one third of the Gold Member representatives. They join 21 other global organizaitons, including 99cloud, China Mobile, Deutsche Telekom and City Network who were recently added as Gold Members at the OpenStack Summit Barcelona board meeting.

“China is a huge market where we have the second highest number of OpenStack developers,” said Anni Lai, head of global business development at Huawei and Huawei’s Gold Member representative on the Foundation’s board of directors. “However, due to the location, time zone, and cultural differences, Chinese developers and users face tremendous challenges being integrated into the core OpenStack community and Foundation.” 

With the increased representation from Chinese companies on the board, Lai says she hopes the impact of the OpenStack community in China, which is the second largest in the world, will be better understood.

This increased investment in OpenStack is not limited to supporting the Foundation; users from industries ranging from telecom to automotive to IT to energy are investing in the OpenStack software to power their businesses.

In his Barcelona Summit keyenote, OpenStack Foundation executive director Jonathan Bryce discussed one of the world’s largest telecoms, China Telecom and the scale of its environment. 

China Telecom brought online two new data centers and the combined space of those data centers is 2 million square meters, 497 acres. Their plan is to get to the point that OpenStack is controlling the infrastructure as they bring these data centers online.  

<script async="async" charset="utf-8" src=""></script>

“China Telecom has a large number and large scale data centers with various infrastructures,” said Yufeng Zhang, China Telecom System Integration deputy general manager. “By joining the [OpenStack] Foundation as a Gold Member, we can accelerate the communication with the Foundation and other Foundation members about telecom related features and promote the process of building China Telecom’s technology architecture.” 

Other OpenStack users from China include:

  • State Grid Corporation of China is the world’s largest electric utility company with 1.5 million employees, supplying electrical power to 27 provinces. It has rolled out OpenStack to 10 data centers to date, around one third of China’s provinces with plans to reach the remaining data centers in the next three years.
  • China Mobile is the most recent Superuser Awards winner and its telecom network has more than 800 million subscribers and 3 million base stations. Its OpenStack footprint is already 10,000 nodes with 100,000 cores and growing.  
  • Lenovo has customers in over 160 countries and is one of the world’s largest PC vendors. Lenovo relies on OpenStack to produce the agile, flexible and scalable enterprise cloud platform required to support new business cases and client uses, meeting needs for business innovation, IT management processes, and leaning more toward internet-based and e-commerce functions, something its legacy IT architecture could not do.
  • Dongfeng Motor Corporation was among the Chinese users Lai presented in her keynote at the OpenStack Summit Barcelona and is one of the big three car makers in China. Last year, Dongfeng sold close to 4 million cars. This year, it announced its 5 year digital transformation plan based on OpenStack. “OpenStack is the key to the success of Dongfeng’s digital transformation,” says Lai.


<script async="async" charset="utf-8" src=""></script>

In addition to , Lai also serves on the Foundation board of directors and is actively involved in planning the first OpenStack Days China, the most recent event which drew 2,400 attendees, another testament to the strength and growth of the OpenStack community in China.  

“I believe having more representation from China in the Foundation will help the OpenStack community and Foundation understand Chinese developer and user community and share the value the Chinese OpenStack community brings,” she says. 

The post China puts money where the cloud is, ramps up OpenStack adoption appeared first on OpenStack Superuser.

by Allison Price at December 01, 2016 04:42 PM

November 30, 2016

OpenStack Superuser

Manage resources on AWS with OpenStack Omni

When it comes to cloud computing, finding a standard that can manage hybrid and multi-cloud environments can be challenging.

In fact, nearly all of the respondents in the April 2016 OpenStack User Survey listed “standardize on the same open platform and APIs that power a global network of public and private clouds” as their top priority.

To that end, Platform9 Systems, a private cloud company that provides OpenStack and Kubernetes as managed SaaS (software-as-a-service), leveraged that need for standardization and built OpenStack Omni, a set of OpenStack drivers that allows OpenStack to manage and control resources on AWS.

After showcasing Omni on the keynote stage of the Barcelona Summit, Madhura Maskasky, co-founder or Platform9 Systems, sat down with Superuser TV to talk about what else her company has been up to since the Tokyo Summit and how the Omni project came about.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

The post Manage resources on AWS with OpenStack Omni appeared first on OpenStack Superuser.

by Superuser at November 30, 2016 06:47 PM

Elizabeth K. Joseph

Ohio LinuxFest 2016

Last month I had the pleasure of finally attending an Ohio LinuxFest. The conference has been on my radar for years, but every year I seemed to have some kind of conflict. When my Tour of OpenStack Deployment Scenarios was accepted I was thrilled to finally be able to attend. My employer at the time also pitched in to the conference as a Bronze sponsor and by sending along a banner that showcased my talk, and my OpenStack book!

The event kicked off on Friday and the first talk I attended was by Jeff Gehlbach on What’s Happening with OpenNMS. I’ve been to several OpenNMS talks over the years and played with it some, so I knew the background of the project. This talk covered several of the latest improvements. Of particular note were some of their UI improvements, including both a website refresh and some stunning improvements to the WebUI. It was also interesting to learn about Newts, the time-series data store they’ve been developing to replace RRDtool, which they struggled to scale with their tooling. Newts is decoupled from the visualization tooling so you can hook in your own, like if you wanted to use Grafana instead.

I then went to Rob Kinyon’s Devs are from Mars, Ops are from Venus. He had some great points about communication between ops, dev and QA, starting with being aware and understanding of the fact that you all have different goals, which sometimes conflict. Pausing to make sure you know why different teams behave the way they do and knowing that they aren’t just doing it to make your life difficult, or because they’re incompetent, makes all the difference. He implored the audience to assume that we’re all smart, hard-working people trying to get our jobs done. He also touched upon improvements to communication, making sure you repeat requests in your own words so misunderstandings don’t occur due to differing vocabularies. Finally, he suggested that some cross-training happen between roles. A developer may never be able to take over full time for an operator, or vice versa, but walking a mile in someone else’s shoes helps build the awareness and understanding that he stresses is important.

The afternoon keynote was given by Catherine Devlin on Hacking Bureaucracy with 18F. She works for the government in the 18F digital services agency. Their mandate is to work with other federal agencies to improve their digital content, from websites to data delivery. Modeled after a startup, she explained that they try not to over-plan, like many government organizations do and can lead to failure, they want to fail fast and keep iterating. She also said their team has a focus on hiring good people and understanding the needs of the people they serve, rather than focusing on raw technical talent and the tools. Their practices center around an open by default philosophy (see: 18F: Open source policy), so much of their work is open source and can be adopted by other agencies. They also make sure they understand the culture of organizations they work with so that the tools they develop together will actually be used, as well as respecting the domain knowledge of teams they’re working with. Slides from her talk here, and include lots of great links to agency tooling they’ve worked on:

Catherine Devlin on 18F

That evening folks gathered in the expo hall to meet and eat! That’s where I caught up with my friends from Computer Reach. This is the non-profit I went to Ghana with back in 2012 to deploy Ubuntu-based desktops. I spent a couple weeks there with Dave, Beth Lynn and Nancy (alas, unable to come to OLF) so it was great to see them again. I learned more about the work they’re continuing to do, having switched to using mostly Xubuntu on new installs which was written about here. On a personal level it was a lot of fun connecting with them too, we really bonded during our adventures over there.

Tyler Lamb, Dave Sevick, Elizabeth K. Joseph, Beth Lynn Eicher

Saturday morning began with a keynote from Ethan Galstad on Becoming the Next Tech Entrepreneur. Ethan is the founder of Nagios, and in his talk he traced some of the history of his work on getting Nagios off the ground as a proper project and company and his belief in why technologists make good founders. In his work he drew from his industry and market expertise from being a technologist and was able to play to the niche he was focused on. He also suggested that folks look to what other founders have done that has been successful, and recommended some books (notably Founders at Work and Work the System). Finaly, he walked through some of what can be done to get started, including the stages of idea development, basic business plan (don’t go crazy), a rough 1.0 release that you can have some early customers test and get feedback from, and then into marketing, documenting and focused product development. He concluded by stressing that open source project leaders are already entrepreneurs and the free users of your software are your initial market.

Next up was Robert Foreman’s Mixed Metaphors: Using Hiera with Foreman where he sketched out the work they’ve done that preserves usage of Hiera’s key-value store system but leverages Foreman for the actual orchestration. The mixing of provisioning and orchestration technologies is becoming more common, but I hadn’t seen this particular mashup.

My talk was A Tour of OpenStack Deployment Scenarios. This is the same talk I gave at FOSSCON back in August, walking the audience through a series of ways that OpenStack could be configured to provide compute instances, metering and two types of storage. For each I gave a live demo using DevStack. I also talked about several other popular components that could be added to a deployment. Slides from my talk are here (PDF), which also link to a text document with instructions for how to run the DevStack demos yourself.

Thanks to Vitaliy Matiyash for taking a picture during my talk! (source)

At lunch I met up with my Ubuntu friends to catch up. We later met at the booth where they had a few Ubuntu phones and tablets that gained a bunch of attention throughout the event. This event was also my first opportunity to meet Unit193 and Svetlana Belkin in person, both of whom I’ve worked with on Ubuntu for years.

Unit193, Svetlana Belkin, José Antonio Rey, Elizabeth K. Joseph and Nathan Handler

After lunch I went over to see David Griggs of Dell give us “A Look Under the Hood of Ohio Supercomputer Center’s Newest Linux Cluster.” Supercomputers are cool and it was interesting to learn about the system it was replacing, the planning that went into the replacement and workload cut-over and see in-progress photos of the installation. From there I saw Ryan Saunders speak on Automating Monitoring with Puppet and Shinken. I wasn’t super familiar with the Shinken monitoring framework, so this talk was an interesting and very applicable demonstration of the benefits.

The last talk I went to before the closing keynotes was from my Computer Reach friends Dave Sevick and Tyler Lamb. They presented their “Island Server” imaging server that’s now being used to image all of the machines that they re-purpose and deploy around the world. With this new imaging server they’re able to image both Mac and Linux PCs from one Macbook Pro rather than having a different imaging server for each. They were also able to do a live demo of a Mac and Linux PC being imaged from the same Island Server at once.

Tyler and Dave with the Island Server in action

The event concluded with a closing keynote by a father and daughter duo, Joe and Lily Born, on The Democratization of Invention. Joe Born first found fame in the 90s when he invented the SkipDoctor CD repair device, and is now the CEO of Aiwa which produces highly rated Bluetooth speakers. Lily Born invented the tip-proof Kangaroo Cup. The pair reflected on their work and how the idea to product in the hands of customers has changed in the past twenty years. While the path to selling SkipDoctor had a very high barrier to entry, globalization, crowd-funding, 3D printers and internet-driven word of mouth and greater access to the press all played a part in the success of Lily’s Kangaroo cup and the new Aiwa Bluetooth speakers. While I have no plans to invent anything any time soon (so much else to do!) it was inspiring to hear how the barriers have been lowered and inventors today have a lot more options. Also, I just bought an Aiwa Exos-9 Bluetooth Speaker, it’s pretty sweet.

My conference adventures concluded with a dinner with my friends José, Nathan and David, all three of whom I also spent time with at FOSSCON in Philadelphia the month before. It was fun getting together again, and we wandered around downtown Columbus until we found a nice little pizzeria. Good times.

More photos from the Ohio LinuxFest here:

by pleia2 at November 30, 2016 06:29 PM


Automate bare metal server provisioning using Ironic (bifrost) and the ansible deploy driver

The post Automate bare metal server provisioning using Ironic (bifrost) and the ansible deploy driver appeared first on Mirantis | The Pure Play OpenStack Company.

On our team, we mostly conduct various research in OpenStack, so we use bare metal machines extensively. To make our lives somewhat easier, we’ve developed a set of simple scripts that enables us to backup and restore the current state of the file system on the server. It also enables us to switch between different backups very easily. The set of scripts is called multi-root (

Unfortunately, we had a problem; in order to use this tool, we had to have our servers configured in a particular way, and we faced different issues with manual provisioning:

  • It is not possible to set up more than one bare metal server at a time using a Java-based IPMI application
  • The Java-based IPMI application does not properly handle disconnection from the remote host due to connectivity problems (you have to start installation from the very beginning)
  • The bare metal server provisioning procedure was really time consuming
  • For our particular case, in order to use multi-root functionality we needed to create software RAID and make required LVM configurations prior to operating system installation

To solve these problems, we decided to automate bare metal node setup, and since we are part of the OpenStack community, we decided to use bifrost instead of other provisioning tools. Bifrost was a good choice for us as it does not require other OpenStack components.

Lab structure

This is how we manage disk partitions and how we use software RAID on our machines:


As you can see here, we have the example of a bare metal server, which includes two physical disks.  Those disks are combined using RAID1, then partitioned by the operating system.  The LVM partition then gets further partitioned, with each copy of an operating system image assigned to its own partition.

This is our network diagram:


In this case we have one network to which our bare metal nodes are attached. Also attached to that network is the IRONIC server. A DHCP server assigns IP addresses for the various instances as they’re provisioned on the bare metal nodes, or prior to the deployment procedure (so that we can bootstrap the destination server).

Now let’s look at how to make this work.

How to set up bifrost with ironic-ansible-driver

So let’s get started.

  1. First, add the following line to the /root/.bashrc file:
    # export LC_ALL="en_US.UTF-8"
  2. Ensure the operating system is up to date:
    # apt-get -y update && apt-get -y upgrade
  3. To avoid issues related to MySQL, we decided to ins tall it prior to bifrost and set the MySQL password to “secret”:
    # apt-get install git python-setuptools mysql-server -y
  4. Using the following guideline, install and configure bifrost:
    # mkdir -p /opt/stack
    # cd /opt/stack
    # git clone
    # cd bifrost
  5. We need to configure a few parameters related to localhost prior to the bifrost installation. Below, you can find an example of an /opt/stack/bifrost/playbooks/inventory/group_vars/localhost file:
    echo "---
    ironic_url: "http://localhost:6385/"
    network_interface: "p1p1"
    ironic_db_password: aSecretPassword473z
    mysql_username: root
    mysql_password: secret
    ssh_public_key_path: "/root/.ssh/"
    deploy_image_filename: "user_image.qcow2"
    create_image_via_dib: false
    transform_boot_image: false
    create_ipa_image: false
    dhcp_lease_time: 12h
    dhcp_static_mask:" > /opt/stack/bifrost/playbooks/inventory/group_vars/localhost

    As you can see, we’re telling Ansible where to find Ironic and how to access it, as well as the authentication information for the database so state information can be retrieved and saved. We’re specifying the image to use, and the networking information.

    Notice that there’s no default gateway for DHCP in the configuration above, so I’m going to fix it manually after the install.yaml playbook execution.

  6. Install ansible and all of bifrost’s dependencies:
    # bash ./scripts/
    # source /opt/stack/bifrost/env-vars
    # source /opt/stack/ansible/hacking/env-setup
    # cd playbooks
  7. After that, let’s install all packages that we need for bifrost (Ironic, MySQL, rabbitmq, and so on) …
    # ansible-playbook -v -i inventory/localhost install.yaml
  8. … and the Ironic staging drivers with already merged patches for enabling Ironic ansible driver functionality:
    # cd /opt/stack/
    # git clone git://
    # cd ironic-staging-drivers/
  9. Now you’re ready to do the actual installation.
    # pip install -e .
    # pip install "ansible>=2.1.0"

    You should see typical “installation” output.

  10. In the /etc/ironic/ironic.conf configuration file, add the “pxe_ipmitool_ansible” value to the list of enabled drivers. In our case, it’s the only driver we need, so let’s remove the other drivers:
    # sed -i '/enabled_drivers =*/c\enabled_drivers = pxe_ipmitool_ansible' /etc/ironic/ironic.conf 
  11. If you want to enable cleaning and disable disk shredding during the cleaning procedure, add these options to /etc/ironic/ironic.conf:
    automated_clean = true
    erase_devices_priority = 0
  12. Finally, restart the Ironic conductor service:
    # service ironic-conductor restart
  13. To check that everything was installed properly, execute the following command:
    # ironic driver-list | grep ansible
    | pxe_ipmitool_ansible | test |

    You should see the pxe_ipmitool_ansible driver in the output.

  14. Finally, add the default gateway to /etc/dnsmasq.conf (be sure to use the IP address for your own gateway).
    # sed -i '/dhcp-option=3,*/c\dhcp-option=3,' /etc/dnsmasq.conf

Now that everything’s set up, let’s look at actually doing the provisioning.

How to use ironic-ansible-driver to provision bare-metal servers with custom configurations

Now let’s look at actually provisioning the servers. Normally, we’d use a custom ansible deployment role that satisfies Ansible’s requirements regarding idempotency to prevent issues that can arise if a role is executed more than once, but because this is essentially a spike solution for us to use in the lab, we’ve relaxed that requirement.  (We’ve also hard-coded a number of values that you certainly wouldn’t in production.)  Still, by walking through the process you can see how it works.

  1. Download the custom ansible deployment role:
    curl -Lk | tar xz -C /opt/stack/ironic-staging-drivers/ironic_staging_drivers/ansible/playbooks/ --strip-components 1
  2. Next, create an inventory file for the bare metal server(s) that need to be provisioned:
    # echo "---
        ipa_kernel_url: ""
        ipa_ramdisk_url: ""
        uuid: 00000000-0000-0000-0000-000000000001
            ipmi_username: IPMI_USERNAME
            ipmi_address: IPMI_IP_ADDRESS
            ipmi_password: IPMI_PASSWORD
            ansible_deploy_playbook: deploy_custom.yaml
            mac: 00:25:90:a6:13:ea
        driver: pxe_ipmitool_ansible
          cpu_arch: x86_64
          ram: 16000
          disk_size: 60
          cpus: 8
        name: server1
          image_source: """ > /opt/stack/bifrost/playbooks/inventory/baremetal.yml
    # export BIFROST_INVENTORY_SOURCE=/opt/stack/bifrost/playbooks/inventory/baremetal.yml

    As you can see the above we have added all required information for bare-metal node provisioning using IPMI. If needed you can add information about various number of bare-metal servers here and all of them will be enrolled and deployed later.

  3. Finally, you’ll need to build a ramdisk for the Ironic ansible deploy driver and create a deploy image using DIB (disk image builder). Start by creating an RSA key that will be used for connectivity from the Ironic ansible driver to the provisioning bare metal host:
    # su - ironic
    # ssh-keygen
    # exit
  4. Next set environment variables for DIB:
    # export ELEMENTS_PATH=/opt/stack/ironic-staging-drivers/imagebuild
    # export DIB_DEV_USER_USERNAME=ansible
    # export DIB_DEV_USER_AUTHORIZED_KEYS=/home/ironic/.ssh/
    # export DIB_DEV_USER_PASSWORD=secret
    # export DIB_DEV_USER_PWDLESS_SUDO=yes
  5. Install DIB:
    # cd /opt/stack/diskimage-builder/
    # pip install .
  6. Create the bootstrap and deployment images using DIB, and move them to the web folder:
    # disk-image-create -a amd64 -t qcow2 ubuntu baremetal grub2 ironic-ansible -o ansible_ubuntu
    # mv ansible_ubuntu.vmlinuz ansible_ubuntu.initramfs /httpboot/
    # disk-image-create -a amd64 -t qcow2 ubuntu baremetal grub2 devuser cloud-init-nocloud -o user_image
    # mv user_image.qcow2 /httpboot/
  7. Fix file permissions:
    # cd /httpboot/
    # chown ironic:ironic *
  8. Now we can enroll anddeploy our bare metal node using ansible:
    # cd /opt/stack/bifrost/playbooks/
    # ansible-playbook -vvvv -i inventory/ enroll-dynamic.yaml

    Wait for the provisioning state to read “available”, as a bare metal server needs to cycle through a few states and could be cleared, if needed. During the enrollment procedure, the node can be cleared by the shred command. This process does take a significant amount of time time, so you can disable or fine tune it in the Ironic configuration (as you saw above where we enabled it).

  9. Now we can start the actual deployment procedure:
    # ansible-playbook -vvvv -i inventory/ deploy-dynamic.yaml

    If deployment completes properly, you will see the provisioning state for your server as “active” in the Ironic node-list.

    | UUID                                                    | Name  | Instance UUID | Power State | Provisioning State | Maintenance |
    | 00000000-0000-0000-0000-000000000001   | server1| None          | power on      | active                     | False            |

Now you can log in to the deployed server via ssh using the login and password that we defined above during image creation (ansible/secret) and then, because the infrastructure to use it has now been created, clone the multi-root tool from Github.


As you can see, bare metal server provisioning isn’t such a complicated procedure. Using the Ironic standalone server (bifrost) with the Ironic ansible driver, you can easily develop a custom ansible role for your specific deployment case and simultaneously deploy any number of bare metal servers in automation mode.

I want to say thank you to Pavlo Shchelokovskyy and Ihor Pukha for your help and support throughout the entire process. I am very grateful to you guys.

The post Automate bare metal server provisioning using Ironic (bifrost) and the ansible deploy driver appeared first on Mirantis | The Pure Play OpenStack Company.

by Guest Post at November 30, 2016 03:49 PM

SUSE Conversations

Watch your PaaS (and IaaS) – SUSE acquires OpenStack and Cloud Foundry technology from HPE

If you follow SUSE, you know we recently made our first technology acquisition—openATTIC—and announced it at our annual customer conference, SUSECON. At the time, I said there would likely be similar acquisitions when it makes sense for our growth strategy, as well as our open source development and business model. Well, that time has come. …

+read more

The post Watch your PaaS (and IaaS) – SUSE acquires OpenStack and Cloud Foundry technology from HPE appeared first on SUSE Blog. Michael Miller

by Michael Miller at November 30, 2016 10:21 AM

David Moreau Simard

Recorded presentations from OpenStack Canada Day

OpenStack Days are sort of local mini one-day OpenStack summits.

Not everyone can travel to the OpenStack summits for a number of reasons so OpenStack days are great for that since there are many across a wide range of countries.

The first one in Canada was in Montreal, at an awesome venue: the Notman house. There’s a lot of history about the place, definitely look it up - it’s very interesting !

Lots of great speakers

There was a lot of great speakers at the event! Have a look at the packed schedule attendees were presented with:


Unfortunately, the talks were not recorded at the event, unlike at the OpenStack summits.

Thankfully, the organizers granted me the permission to record some talks myself so I went ahead and recorded my talk as well as those of my colleagues from Red Hat that were also speaking.

I hope you can enjoy these, you can find the talks below !

Test your application deployments with Ansible on OpenStack for free

  • Speaker: David Moreau Simard (me!)
  • Slides are available here
  • Video is available on YouTube.
<iframe allowfullscreen="allowfullscreen" frameborder="0" height="480" src="" width="853"></iframe>

A community cloud managed by the project infrastructure team

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="480" src="" width="853"></iframe>

A container stack for OpenStack

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="480" src="" width="853"></iframe>

by dmsimard at November 30, 2016 01:00 AM

November 29, 2016

Shannon McFarland

Have Fun with OpenStack and IPv6 Prefix Delegation

In this post I will talk about IPv6 Prefix Delegation (PD) in OpenStack.  IPv6 is a super important feature, especially in OpenStack. Unlike traditional IPv4 deployments with OpenStack where the cloud admin does not have to worry much about overlapping IPs and colliding address scopes, this is a real issue with IPv6 in OpenStack.  Other … Continue reading Have Fun with OpenStack and IPv6 Prefix Delegation

by eyepv6(at)gmail(dot)com at November 29, 2016 11:50 PM

Deploying IPv6-Only Tenants with OpenStack

In this post, I will discuss the deployment of IPv6-only Neutron networks for tenant use.  A few things to note about IPv6-only instances: IPv6-only works out-of-the-box for basic IP connectivity with OpenStack but you are hosed on the metadata service Many people deploy dual stack in order to overcome the metadata issue The metadata service … Continue reading Deploying IPv6-Only Tenants with OpenStack

by eyepv6(at)gmail(dot)com at November 29, 2016 10:29 PM

Deploying IPv6 With OpenStack Provider Networks

Last year I posted on “Tenant IPv6 Deployment in OpenStack Kilo Release” and I also posted on “Using OpenStack Heat to Deploy an IPv6-Enabled Instance”.  Both of those posts were using the Neutron L3 Agent (Tenant routers) for my examples.  In this post I will show the deployment of IPv6 using Provider Networks. There is no … Continue reading Deploying IPv6 With OpenStack Provider Networks

by eyepv6(at)gmail(dot)com at November 29, 2016 09:47 PM

Kenneth Hui

Forget About Your OpenStack Upgrades


Upgrades. Few IT activities are as important yet heartburn-inducing as upgrading a major system.

Often upgrades mean service disruption, both expected and sometimes unexpected, which can negatively affect businesses. Often the cause is unreliable software and/or lack of operational expertise.

I’ve worked with and for enterprise customers who had policies of not upgrading until a particular piece of software or driver had been out for at least two years. I know customers with policies of not ever upgrading software, unless they had no choice. Whatever the reasons, these enterprises have often adopted a risk avoidance stance on upgrades, e.g. “if it isn’t broke, don’t fix it.”

OpenStack has by no means been immune to the upgrade conundrum. The community has prided itself on being able to develop software quickly, as evidenced by the project’s cadence of two major releases per year. But upgrading each time has often been painful with many earlier releases requiring a complete reinstall as the “upgrade” process. While there have been significant improvements, upgrading OpenStack is still not a task to take on lightly, especially without significant operational expertise.

Rackspace has always been committed to making operational IT tasks, like upgrades, as painless as possible for customers. Our goal is that upgrade events should be something customers can forget about because Rackspace has them covered.

In particular, we’ve worked with Red Hat to combine the advances Red Hat OpenStack platform has made around software reliability and non-disruptive upgrades with Rackspace’s operational expertise. The end result is that Rackspace Private Cloud powered by Red Hat, aka RPC-R, is an innovative and yet reliable OpenStack platform for enterprises to consume.

To read more how Rackspace and  Red Hat is making OpenStack upgrades forgettable for customers, please click here to go to my article on the Rackspace blog site.

Filed under: Cloud, Cloud Computing, OpenStack, Private Cloud Tagged: Cloud, Cloud computing, OpenStack, Private Cloud, Rackspace, Red Hat, TripleO, Upgrades

by kenhui at November 29, 2016 06:28 PM

OpenStack Superuser

Fast and furious: Inside China’s startup scene

While Silicon Valley may be home to many large tech companies and thousands of startups, innovation reaches far beyond its borders. Here’s more on how these startups are using OpenStack.


A rapidly growing startup founded in 2012 and one of the OpenStack Foundation’s newest Gold Members, Kai says 99Cloud currently has five core members in different OpenStack projects, underlining its dedication to contributions.Kai says there are three overarching challenges in China: fewer startup buyouts, building differentiation and finding and growing talent. “From day one we knew we needed a long-term survival strategy; this year we’re at the break-even point,” Kai says. That translated to a focus on building differentiation in solutions and service models. For example, they have homed in on plug-in development to fill the gaps between community distro and clients needs, “key pain points during Open-Stack adoption,” Kai says. They also focused on injecting more “vertical industry DNA into OpenStack” for customers with solutions for China Central Bank, China UnionPay and JinZheng.  The company also delivers a cloud to State Grid that will offer electricity to 90 percent of China by the end of 2017.

The advantages, Kai says, are also many.

“China is still one of the most Open-Stack-friendly markets for startups,” says Li Kai, co-founder and vice president, adding that there are three main types of clients. The largest is government cloud or state-run companies who prefer private clouds over public clouds due to information security concerns and customization needs. Then there are traditional companies—banks, for example such as its client China UnionPay—who look to open source technologies for lower costs, better scalability and higher performance. The third are internet companies building community clouds such as, China’s top crowd-sourcing platform, where 99Cloud supports billions of transactions.

Kai outlined 99Cloud’s strategy for success in four points:

  • Talk is cheap, show clients more code and demos.
  • Don’t push the cloud product—be a “cloud coach” and design the solution with clients.
  • Avoiding vendor lock-in is not just a slogan; show clients it’s doable.
  • Focus on the last kilometer issue during adoption over distros. Community releases are not perfect, tell clients the truth. Let them know what’s mature and immature and provide provide alternatives if necessary.


EasyStack is a leading OpenStack service provider in Asia that counts over 100 enterprise customers, contributions to the Mitaka release and OpenStack Foundation Gold Member status among its accomplishments.

“How to translate all of this into a solid global branding and enable company global market extension is more challenging, however, than for our Silicon Valley peers,” says Robert Wen, marketing director. Wen also cites the large scale and technical challenges in OpenStack deployments in China as well as the time difference and language barriers that make it difficult for EasyStack’s engineers to collaborate and contribute code.

On the plus side, Wen says EasyStack’s large enterprise customers implementing OpenStack in production includes many Fortune 500 companies like State Grid, China Mobile, China Telecom, Postal Savings Bank of China and Lenovo. He summarizes its “simple but powerful” strategy in three points:

  • Deliver value by understanding large enterprise businesses and enabling their innovation, rather than focus-ing on your product just to survive.
  • Position EasyStack as a trusted partner rather than a vendor selling a license or service.
  • Work with the ecosystem and community—never fight alone.

This article first appeared in the print edition of Superuser magazine, distributed at the Barcelona Summit. If you’d like to contribute to the next one, get in touch:

Cover Photo // CC BY NC

The post Fast and furious: Inside China’s startup scene appeared first on OpenStack Superuser.

by Nicole Martinelli at November 29, 2016 06:09 PM


How to Install and Run Tempest

Tempest is a set of integration tests to run against an OpenStack cluster. In this blog I'm going to show you, how to install tempest from git repository, how to install all requirements and run tests against an OpenStack cluster.

I'm going to use a fresh installation of Centos7 and OpenStack cluster provided by packstack. If you've done that, follow the instructions below.

Tempest Installation

You have two options how to install tempest. You can install it through RPM or you can clone tempest from GitHub repository. If you choose installation through RPM, follow this link.

Installation from GitHub repository

Now you can clone upstream tempest or you can clone RedHat's fork of upstream tempest. The RedHat's fork provides, which is a configuration tool. It will generate tempest.conf for you, what can be handy.

[1.] Install dependencies:

$ sudo yum install -y gcc python-devel libffi-devel openssl-devel

[2.] Clone tempest:

$ git clone

Or (RedHat's fork):

$ git clone

[3.] Install pip, for example:

$ curl "" -o ""
$ sudo python

[4.] Install tempest globally in the system. If you don't want to do that, skip this step and continue reading.

$ sudo pip install tempest/

Install tempest in a virtual environment

Sometimes you don't want to install things globally in the system. For this reason you may want to use a virtual environment. I'm going to explain installation through virtualenv and tox.

Setting up Tempest using virtualenv

[1.] Install virtualenv:

$ easy_install virtualenv

Or through pip:

$ pip install virtualenv

[2.] Enter tempest directory you've cloned before:

$ cd tempest/

[3.] Create a virtual environment and let's name it .venv:

$ virtualenv .venv
$ source ./venv/bin/activate

[4.] Install requirements:

(.venv) $ pip install -r requirements.txt
(.venv) $ pip install -r test-requirements.txt

NOTE: If problems occur during requirements installation, it may be due to an old version of pip, upgrade may help:

(.venv) $ pip install pip --upgrade

[5.] After dependencies are installed, run following commands, which install tempest within the virtual environment:

(.venv) $ cd ../
(.venv) $ pip install tempest/

Or this command does the same without using pip:

$ python install If you need to trigger installation to developer mode run:

(.venv) $ python develop ` develop` comes from limitations on [pbr]( If you are interested, [here is]( an explanation of difference between `install` and `develop`.

Setting up Tempest using TOX

[1.] Install tox:

$ easy_install tox

Or if you want to use pip:

$ pip install tox

[2.] Install tempest:

$ tox -epy27 --notest
$ source .tox/py27/bin/activate This will create a virtual environment named `.tox`, install all dependencies (*requirements.txt* and *test-requirements.txt*) and tempest within it. If you check `tox.ini` file, you'll see tox actually run tempest installation in develop mode you could run manually as it was explained above.


[3.] If you want to expose system-site packages, tox will do it for you. Deactivate environment, you are currently in (if you followed the step before) and create another environment:

(py27) $ deactivate
$ tox -eall-plugin --notest
$ source .tox/all-plugin/bin/activate

[4.] Then if you want to install plugins test packages based on the OpenStack Components installed, let this script to do it:

(all-plugin) $ sudo python tools/
(all-plugin) $ python develop

Generate tempest.conf

About tempest.conf and what it is used for you can read in this documentation. If you want to create tempest.conf let do it for you. The tool is part of RPM tempest (check this documentation) or if you don’t want to install tempest globally, you can clone RedHat's tempest fork and install it within a virtual environment as it was explained above.

RedHat's tempest fork

Create a virtual environment as I already mentioned and source credentials (if you installed OpenStack cluster by packstack credentials are saved in /root/):

(.venv) $ source /root/keystone_admin

And run config tool:

(.venv) $ python tools/ --debug identity.uri $OS_AUTH_URL \
         identity.admin_password  $OS_PASSWORD --create  After this, `./etc/tempest.conf` is generated.

NOTE: If you running OSP, it’s needed to add a new argument to config_tempest tool:

(.venv) $ ./tools/ object-storage.operator_role swiftoperator

It's because OSP uses lowercase operator for the swift operator_role, however, tempest default value is "SwiftOperator". To override the default value run config_tool like this:

$ python tools/ --debug identity.uri $OS_AUTH_URL \
  identity.admin_password  $OS_PASSWORD \
  object-storage.operator_role swiftoperator --create

Running tests

If you've installed tempest and have tempest.conf, you cant start testing. To run tests you can use testr or ostestr. If you want to run tempest unit tests, check this out.

Note: following commands run withing the virtual environment you've created before. To run specific tests run for example:

$ python -m tempest.api.volume.v2.test_volumes_list OR

$ ostestr --regex tempest.api.volume.v2.test_volumes_list

Alternatively you can use tox, for example:

$ tox -efull Run only tests tagged as smoke:

$ tox -esmoke

by mkopec at November 29, 2016 03:42 PM

Bernard Cafarelli

Service Function Chaining demo with devstack

After a first high-level post, it is time to actually show networking-sfc in action! Based on a documentation example, we will create a simple demo, where we route some HTTP traffic through some VMs, and check the packets on them with tcpdump:

SFC demo diagram

This will be hosted on a single node devstack installation, and all VMs will use the small footprint CirrOS image, so this should run on “small” setups.

Installing the devstack environment

On your demo system (I used Centos 7), check out devstack on the Mitaka branch (remember to run devstack as a sudo-capable user, not root):

[stack@demo ~]$ git clone -b stable/mitaka

Grab my local configuration file that enables the networking-sfc plugin, rename it to local.conf in your devstack/ directory.
If you prefer to adapt your current configuration file, just make sure your devstack checkout is on the mitaka branch, and add the SFC parts:
enable_plugin networking-sfc

Then run the usual “./” command, and go grab a coffee.

Deploy the demo instances

To speed this step up, I regrouped all the following items in a script. You can check it out (at a tested revision for this demo):
[stack@demo ~]$ git clone -b sfc_mitaka_demo

The script will:

  • Configure security (disable port security, set a few things in security groups, create a SSH key pair)
  • Create source, destination systems (with a basic web server)
  • Create service VMs, configuring the network interfaces and static IP routing to forward the packets
  • Create the SFC items (port pair, port pair  group, flow classifier, port chain)

I highly recommend to read it, it is mostly straightforward and commented, and where most of the interesting commands are hidden. So have a look, before running it:
[stack@demo ~]$ ./openstack-scripts/
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
Updated network: private
Created a new port:

route: SIOCADDRT: File exists
WARN: failed: route add -net "" gw ""
You can safely ignore the route errors at the end of the script (they are caused by duplicate default route on the service VMs).

Remember, from now on, to source the credentials file in your current shell before running CLI commands:
[stack@demo ~]$ source ~/devstack/openrc demo demo

We first get the IP addresses for our source and destination demo VMs:[vagrant@defiant-devstack ~]$ openstack server show source_vm -f value -c addresses; openstack server show dest_vm -f value -c addresses

private=, fd73:381c:4fa2:0:f816:3eff:fe65:12fd

Now, we look for the tap devices associated to our service VMs:[stack@demo ~]$ neutron port-list -f table -c id -c name

| name           | id                                   |
| p1in           | 897df85a-26c3-4491-888e-8cc58f19cea1 |
| p1out          | fa838294-317d-46df-b10e-b1734dd62faf |
| p2in           | c86dafc7-bda6-4537-b806-be2282f7e11e |
| p2out          | 12e58ea8-a9ab-4d0b-9fd7-707dc6e99f20 |
| p3in           | ee14f406-e9d6-4047-812b-aa04514f50dd |
| p3out          | 2d86403b-4639-40a0-897e-68fa0c759f01 |

These devices names follow the tap<port ID first 10 digits> pattern, so for example tap897df85a-26 is the associated  for the p1in port here

See SFC in action

In this example we run a request loop from client_vm to dest_vm (remember to use the IP addresses found in the previous section):
[stack@demo ~]$ ssh cirros@
$ while true; do curl; sleep 1; done
Welcome to dest-vm
Welcome to dest-vm
Welcome to dest-vm

So we do have access to the web server! But does the packets really go through the service VMs? To confirm that, in another shell, run tcpdump on the tap interfaces:

# On the outgoing interface of VM 3
$ sudo tcpdump port 80 -i tap2d86403b-46
tcpdump: WARNING: tap2d86403b-46: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap2d86403b-46, link-type EN10MB (Ethernet), capture size 65535 bytes
11:43:20.806571 IP > Flags [S], seq 2951844356, win 14100, options [mss 1410,sackOK,TS val 5010056 ecr 0,nop,wscale 2], length 0
11:43:20.809472 IP > Flags [.], ack 3583226889, win 3525, options [nop,nop,TS val 5010057 ecr 5008744], length 0
11:43:20.809788 IP > Flags [P.], seq 0:136, ack 1, win 3525, options [nop,nop,TS val 5010057 ecr 5008744], length 136
11:43:20.812226 IP > Flags [.], ack 39, win 3525, options [nop,nop,TS val 5010057 ecr 5008744], length 0
11:43:20.817599 IP > Flags [F.], seq 136, ack 40, win 3525, options [nop,nop,TS val 5010059 ecr 5008746], length 0

Here are some other examples (skipping the tcpdump output for clarity):
# You can check other tap devices, confirming both VM 1 and VM2 get traffic
$ sudo tcpdump port 80 -i tapfa838294-31
$ sudo tcpdump port 80 -i tap12e58ea8-a9

# Now we remove the flow classifier, and check the tcpdump output
$ neutron port-chain-update --no-flow-classifier PC1
$ sudo tcpdump port 80 -i tap2d86403b-46 # Quiet time

# We restore the classifier, but remove the group for VM3, so tcpdump will only show traffic on other VMs
$ neutron port-chain-update --flow-classifier FC_demo --port-pair-group PG1 PC1
$ sudo tcpdump port 80 -i tap2d86403b-46 # No traffic
$ sudo tcpdump port 80 -i tapfa838294-31 # Packets!

# Now we remove VM1 from the first group
$ neutron port-pair-group-update PG1 --port-pair PP2
$ sudo tcpdump port 80 -i tapfa838294-31 # No more traffic
$ sudo tcpdump port 80 -i tap12e58ea8-a9 # Here it is

# Restore the chain to its initial demo status
$ neutron port-pair-group-update PG1 --port-pair PP1 --port-pair PP2
$ neutron port-chain-update --flow-classifier FC_demo --port-pair-group PG1 --port-pair-group PG2 PC1

Where to go from here

Between these examples, the commands used in the demo script, and the documentation, you should have enough material to try your own commands! So have fun experimenting with these VMs.

Note that in the meantime we released the Newton version (3.0.0), which also includes the initial OpenStackClient (OSC) interface, so I will probably update this to run on Newton and with some shiny “openstack sfc xxx” commands. I also hope to make a nicer-than-tcpdumping-around demo later on, when time permits.

by Bernard Cafarelli at November 29, 2016 02:09 PM

4 OpenStack guides to help you build your open source cloud

Every month, brings you the latest in OpenStack tips, tricks, guides, and tutorials created by the open source community.

by Jason Baker at November 29, 2016 08:00 AM


Introducing Decapod, an easier way to manage Ceph

The post Introducing Decapod, an easier way to manage Ceph appeared first on Mirantis | The Pure Play OpenStack Company.

Ceph is a de-facto standard in building robust distributed storage systems. It enables users to get a reliable, highly available, and easily scalable storage cluster using commodity hardware. Also, Ceph is becoming a storage basis for production OpenStack clusters.

There are several ways of managing Ceph clusters, including:

  • Using the ceph-deploy tool
  • Using custom in-house or open source manifests for configuration management software such as Puppet or Ansible
  • Using standalone solutions such as VSM or Fuel

Another solution in that third bucket is Decapod, a standalone solution that simplifies deployment of clusters and management of their lifecycles.

In this article, we’ll compare the different means for deploying Ceph.

Deployment using ceph-deploy

The ceph-deploy tool is available with Ceph itself. According to the official documentation:

The ceph-deploy tool is a way to deploy Ceph relying only upon SSH access to the servers, sudo, and some Python. It runs on your workstation, and does not require servers, databases, or any other tools. If you set up and tear down Ceph clusters a lot, and want minimal extra bureaucracy, ceph-deploy is an ideal tool. The ceph-deploy tool is not a generic deployment system. It was designed exclusively for Ceph users who want to get Ceph up and running quickly with sensible initial configuration settings without the overhead of installing Chef, Puppet or Juju. Users who want fine-control over security settings, partitions or directory locations should use a tool such as Juju, Puppet, Chef or Crowbar.

As described, ceph-deploy is mostly limited to some quick cluster deployment. This is perfectly applicable for deploying a test environment, but production deployment still requires a lot of thorough configuration using external tools.

Deployment using manifests for configuration management tools

Configuration management tools enable you to deploy Ceph clusters as while maintaining great possibilities to tune the cluster. It is also possible to scale or shrink these clusters using the same code base.

The only problem here is high learning curve of such solutions: you need to know, in detail, every configuration option, and you need to read the source code of manifests/playbooks/formulas to understand in detail how they works.

Also, in most cases these manifests focus on a single use case: cluster deployment. They do not provide enough possibilities to manage the cluster after it is up and running. When you operate the cluster, if you need to extend it with new machines, disable existing machines to do maintenance, reconfigure hosts to add new storage pools or hardware, and so on, you will need to create and debug new manifests by yourself.

Standalone solutions

Decapod and VSM are examples of standalone configuration tools. They provide you with a unified view of the whole storage system, eliminating the need to understand low level details of cluster management. They integrate with a monitoring system, and they simplify operations on the cluster. They both have a low learning curve, providing best management practices with a simple interface.

Unfortunately, VSM has some flaws, including the following:

  • It has tightly coupled business and automation logic, which makes it hard to extend the tool, or even customize some deployment steps
  • By design, it is limited in scale. It works great for small clusters, but at a bigger scale the software itself becomes a bottleneck
  • It lacks community support
  • It has an overcomplicated design

Decapod takes a slightly different approach: it separates provisioning and management logic from the start, using an official community project, ceph-ansible. Decapod uses Ansible to do all remote management work, and uses its proven ability to create scalable deployments.

The Decapod architecture

Since Decapod uses Ansible to manage remote nodes, it does not need a complex architecture. Moreover, we’ve been trying to keep it as simple as possible. The architecture looks like this:


As you can see, Decapod has two main services: API and controller.

The API service is responsible for management entities and the handling of HTTP requests. If you request execution of an action on a Ceph node, the API service creates the task in the database for the controller. Each request for that task returns its status.

The Controller listens for new tasks in the database, prepares Ansible for execution (generates Ansible inventory, injects variables for playbooks) and tracks the progress of execution. Every step of the execution is trackable in the UI. You can also download the whole log afterwards.

Decapod performs every management action using a plugin, including cluster deployment and purging object storage daemons from hosts. Basically, a plugin is a playbook to execute, and a Python class used to generate the correct variables and dynamic inventory for Ansible based on the incoming class. Installation is dynamically extendable, so there is no need to redeploy Decapod with another set of plugins. Also, each plugin provides a set of sensible settings for your current setup, but if you want, you may modify every aspect and each setting.

Decapod usage

Decapod has rich CLI and UI interfaces, which enable you to manage clusters. We gave a lot of attention to the UI because we believe that a good interface can help users to accomplish  their goals without paying a lot of attention to low level details. If you want to do some operation work on a cluster, Decapod will try to help you with the most sensible settings possible.

Also, another important feature of Decapod is its ability to audit changes. Every action or operation on the cluster is trackable, and it is always possible to check the history of modifications for every entity, from its history of execution on a certain cluster to changes in the name of a user.

The Decapod workflow is rather simple, and involves a traditional user/role permission based model of access. To deploy a cluster, you  needs to create it, providing a name for the deployed cluster. After that, you select the management action you wants to do, and select the required servers, and Decapod will generate sensible defaults for that action. If you’re is satisfied, you can execute this action in a single click. If not, you can tune these defaults.

You may find more information about using Decapod in our demo:

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="" width="560"></iframe>

So what do you think? What are you using to deploy Ceph now, and how do you think Decapod will affect your workflow? Leave us a comment and let us know.

The post Introducing Decapod, an easier way to manage Ceph appeared first on Mirantis | The Pure Play OpenStack Company.

by Nick Chase at November 29, 2016 03:33 AM

November 28, 2016

Flavio Percoco

On communities: Empower humans to be amazing

When it comes to communities, a system is the set of processes you put in place to allow for humans to be amazing. It's the means to empower these humans to contribute to your community, learn from it and grow with it.

These systems are essential for your community to exist. Your community is a system in itself and it functions through the processes that have been created along the way. These processes exist even if you're not aware they do. The processes that help your community function are not some kind of magic tunnel through which things happen automatically. These are processes created and tailored for your community. There are many things that can be shared across different communities but there are others that are simply specific to yours.

The way you merge code, the means through which you communicate in your community, the standards you put in place. These are all processes that allow your community to cope with growth and chaos. These processes are all meant to be created, evolve and sometimes die. If you want your community to survive change, you must change your community therefore you must change your processes.

There's no secret recipe for managing these processes, though. One thing to always keep in mind is that the humans that interact with these processes are more important than the processes themselves. If the way you review code is too complex for most of the humans that are doing reviews, change it. If the way you define new processes doesn't allow other humans to actively participate, change it. If the way you allow for contributions to be submitted ends up frustrating your contributors, change it. Your community is made by humans and the sooner you acknowledge that, the sooner you'll adapt your processes to better empower these humans. Remember that humans react to emotions (you can read more here) but they act based on their cultures.

Culture has been defined in many ways. Some definitions involve long explanations about society, evolution and human interaction. When it comes to communities a, perhaps oversimplified, way to define culture is that it's the way humans in the community actually do things. I heard this definition at Zingtrain in June 2016 and it stuck with me. Not only it makes sense on paper but it's also true in reality.

We spend a huge deal of time defining new processes in OpenStack that would help the community evolve and adapt itself and we see over and over how many of these processes change as soon as other humans start interacting with them. Sometimes this interaction ends up in processes being "officially" changed and many times they are left as they are so people can use them the way it works best.

In other words, the processes in your community will be bent by the cultures in your community more often than not and this is fine. You want this to happen. You want your community to adapt itself to the cultures that it embraces. You want your community to embrace more cultures and to allow for new cultures to be created within the community itself. Different cultures have different ways to solve problems and there's lots your community can learn from this.

Your community must be flexible for it to be able to adapt itself and tolerate the bending of its processes caused by the interaction with other cultures. If the processes in your community can't be bent a bit, then some cultures won't be able to interact with them and this will, of course, affect your community.

For humans from different cultures (even the same culture, really) to be able to interact with each other, they must be tolerant to variance. Putting the tolerance to variance at the bases of your community will set the principle that humans in your community will interact based on. There must be balance, though. Being either too tolerant or too intolerant won't help your community. You can't make everyone happy but you definitely must make enough humans happy.

Eventually it will come down to how good your community is at allowing humans to interact with each other and I believe a good summary of this post could be:

You must tolerate variance in your community, to empower humans, hopefully from any culture, to be amazing.

If you liked this post, you may also like:

If you liked this post, you may be interested in the keynote I gave at Pycon South Africa. Keeping up with the pace of a fast growing community without dying

by Flavio Percoco at November 28, 2016 11:00 PM

Kenneth Hui

Enabling Software Defined Storage as a Service with OpenStack and Ceph


Data storage growth is both a reality and a burden for today’s enterprises.

With the move to digital, big data and, data intensive workloads, enterprises are storing an increasing amount of data, growing 30-40 percent per year, according to Vanson Bourne LTD. Much of this growth comes from applications running on cloud platforms such as OpenStack, and these applications have unique requirements that are difficult to meet using traditional storage solutions.


<figure class="wp-caption aligncenter" id="attachment_41933"> <figcaption class="wp-caption-text">Source: Red Hat</figcaption> </figure>

In fact, the Vanson Bourne survey indicates that 70 percent of IT decision makers don’t believe their current storage solutions can handle these next generation workloads. The storage requirements for these new cloud-native applications often include:

Which brings us to the Ceph open source storage project. Ceph was developed by a community of engineers and researchers who needed a distributed object store and file system. Over time, block storage capabilities were added to Ceph which has helped it to become a primary storage option for OpenStack and a valuable software defined storage solution for enterprises.

To read more about Ceph 2 and how Rackspace is using it to enabled software defined storage as a service, please click here to go to my article on the Rackspace blog site.

Filed under: Cloud, Cloud Computing, OpenStack, Private Cloud, Software Defined Storage, Storage Tagged: Ceph, Cloud, OpenStack, Private Cloud, Rackspace, Red Hat, Software Defined Storage, Storage

by kenhui at November 28, 2016 06:59 PM

OpenStack Blog

OpenStack Developer Mailing List Digest November 18-25th


  • Nova placement/resource provider work [4]
  • New release-announce list and other changes to openstack-announce [5]
  • Formal Discussion of Documenting Upgrades[6]
  • Stewardship Working Group description/update [7]
  • OpenStack Liberty has reached EOL [8]
  • Switching test jobs from Ubuntu Trusty to Xenial on the gate is happening on December 6th [9]

A Continuously Changing Environment:

  • We have core developers who’ve been around for a long while stepping down and giving the opportunity to the “next generation” to take on the responsibility of leadership
  • Thank you for your presence, for teaching and for showing other contributors a good example by embracing open source and OpenStack
    • Andrew Laski (Nova): “As I’ve told people many times when they ask me what it’s like to work on an open source project like this: working on proprietary software exposes you to smart people but you’re limited to the small set of people within an organization, working on a project like this exposed me to smart people from many companies and many parts of the world. I have learned a lot working with you all. Thanks.”
    • Carl Baldwin (Neutron): “This is a great community and I’ve had a great time participating and learning with you all.”
    • Marek Denis (Keystone): “It’s been a great journey, I surely learned a lot and improved both my technical and soft skills.”
  • Thank you for all your hard work!

Community goals for Ocata:

  • Starting with the Newton, our community commits to release goals in order to provide the minimum level of consistency and user experience and to improve certain areas OpenStack-wide [1]
  • The goal is to remove all remaining incubated Oslo code in Ocata[2][3]

Unit Test Setup Changes [10]:

  • Attempt to remove DB dependency from the unit test jobs
    • Special DB jobs still exist to provide workaround where needed along with a script in ‘tools/’
  • Long term goal is for projects to not use the -db jobs anymore, new changes for them should not be accepted.

Project Info in README Files [11]

  • Increase visibility of fundamental project information that is already available on the governance web site [12]
  • Badges are automatically generated as part of the governance CI [13]
  • Every project is strongly recommended to use this new system to provide information about
    • The project’s state (in Big Tent or not, etc.)
    • Project tags
    • Project capabilities














by Kendall Nelson at November 28, 2016 06:58 PM


How are you using RDO? (Survey results)

Over the last few weeks, we've been conducting a survey of RDO users, to get an idea of who is using RDO, how, and for what.

While the sample size here is small, it's a start at understanding the makeup of our user community. And we'll conduct the survey again around the next release, so if you missed this one, stay tuned.

The Numbers

Let's start with the numbers.

First of all, there were only 39 responses to the survey. Hopefully next time we can do a better job of getting responses, but this is a good start for the first time.

Most of our users (ie, more than half) are running the Mitaka release, witht he next-largest number (46%) having already moved over to the Newton release.


Nearly half of our users are running RDO as their production cloud.


38% of users are deploying with Packstack, with just 20% using TripleO.


There's no clear leader in terms of what industry our users are in, however, Research, Service Providers and Telecom are the three at the top.


Finally, in the distribution of cloud size, over half of respondants were in the 1-10 nodes range, with the rest spread everywhere from there to more than 7500 nodes.


There were some additional questions that will be summarized to the rdo-list over the coming days, regarding how people want to get more involved in the project, and what things they feel are missing.

Next Time

Doing surveys is hard, and invaribly as soon as you send a survey out into the wild, you realize some things that you wish you'd done differently. This time, one thing we did was make the "industry" and "size" fields free-form entry, rather than providing options. This made it a lot more work to tally the results.

Beyond that, if there's some things that you feel that we should have done differently in the survey, please speak up.

by Rich Bowen at November 28, 2016 05:30 PM

OpenStack Superuser

Your guide to the Certified OpenStack Administrator exam

If you’re itching to demonstrate your OpenStack skills, now you can really put them to the test.

The Certified OpenStack Administrator (COA) exam is the first professional certification offered by the OpenStack Foundation. It is designed to help companies identify top talent in the industry and help job seekers demonstrate their skills. So far, more than 500 people have already taken it with 500 more lined up.

At the OpenStack Summit Barcelona, Alan Clark, chairman of the OpenStack Foundation’s Board of Directors, and Heidi Bretz, director of business development at the Foundation, discuss why the Foundation felt the need to create this certification and the benefits it can bring to both OpenStack experts and beginners. If you’d like to brush up on your skills before taking the test, Bretz advises checking out the OpenStack training marketplace.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

Learn more about the COA:

The post Your guide to the Certified OpenStack Administrator exam appeared first on OpenStack Superuser.

by Superuser at November 28, 2016 05:15 PM

Thierry Carrez

OpenStack TC mythbusting

On several occasions over the last months, I heard people exposing truths about the OpenStack Technical Committee. However, those positions were often inaccurate or incomplete. Arguably we are not communicating enough: governance changes and resolutions that are brought to the TC are just approved or rejected. That binary answer is generally not the whole story, and with only the headline, it is easy to read too much in a past decision. We should do a better job at communicating beyond that simple answer when the topic is more complex, and at continuously explaining the role and limits of the TC. Hopefully this blogpost will help, by busting some of those myths.

Myth #1: "the TC doesn't want Go in OpenStack"

This one comes from the recent rejection of a resolution proposing to add golang to the list of approved languages, to support merging the hummingbird feature branch into Swift's master branch. A more accurate way to present that decision would be to say that a (short) majority of the TC members was not supporting the addition of golang at that time and under the proposed conditions. In summary it was more of a "not now, not this way" than a "never".

There were a number of reasons for this decision, but the crux was: adding a language creates fragmentation and a support cost for all of the OpenStack community, so we need to make sure "OpenStack" as a whole is successful with it, beyond any specific project. That means for example having a clear plan for integrating Go within all our standing processes, while trying to prevent duplication of effort or gratuitous rewrites. The discussion was actually recently restarted by Flavio in this change. We'll need resources to make this a success, so if you care, please jump in to help.

Myth #2: "the TC doesn't like competition with existing projects"

I'm not exactly sure where this one comes from. I can't think of any specific TC decision that would explain it. Historically, back when we had programs, a given team would own a given problem space and could essentially lock alternatives out. But the Big Tent project structure reform changed that, allowing competition to happen within our community.

Yes, we still want to encourage cooperation wherever it's possible, so we still have a requirement which says "where it makes sense, the project cooperates with existing projects rather than gratuitously competing or reinventing the wheel". But as long as the new project is different or presents a different value proposal, it should be fair game. For example we accepted the Monasca team in the same problem space as the Telemetry team. And we have several teams working on various deployment recipes.

That said, having competitive solutions in OpenStack on key problem spaces (like base compute or networking services) creates interesting second-order questions in terms of what is "core", trademark usage and interoperability. Those are arguably more downstream concerns than upstream concerns, but that explains why the deeper you go, the more difficult the community discussion is likely to be.

Myth #3: "the TC does not set any direction"

There are multiple variants on that one, from "OpenStack needs a benevolent dictator" to "a camel is a horse designed by a committee". The idea behind it is that the TC needs to have a very opinionated plan for OpenStack and somehow force everyone in our community to follow it. Part of this myth is trying to apply single-vendor software development theory to an open collaboration, and misunderstanding how other large open source projects (like the Linux kernel) work.

While the TC members are all well-respected in our community, we can't unilaterally decide everything for 2500+ developers from 300+ different organizations, and expect them all to execute The Plan. What the TC can do, however, is to define the mission, provide an environment, set principles, enforce common practices, and arbitrate conflicts. In painting terms, the TC provides the frame, the subject of the painting, a color palette and the techniques that can be used. But it doesn't paint. Painting is done at each project team level.

In this cycle, the TC started to drive some simple cross-community goals. The idea is to collectively make visible progress on a given topic over the course of a release cycle, to pay back technical debt or to implement a simple feature across all of OpenStack projects. But this is done as a goal the community agrees to work on, rather than a top-down mandate.

Myth #4: "the TC, due to the Big Tent, prevents proper focus"

This one is interesting, and I think its roots lie in some misunderstanding of open source community dynamics. If you consider a finite set of resources and a zero-sum-game community, then of course adding more projects results in less resources being dedicated to "important projects". But an open community like OpenStack is not a finite set of resources. The people and the organizations in the OpenStack community work and cooperate on a number of projects. Before the big tent, some of those would not be considered part of the OpenStack projects, despite helping with the OpenStack mission and following the OpenStack development principles, and therefore essentially being developed by the OpenStack community.

Considering more (or less) projects as being part of our community doesn't decrease (or increase) focus on "important projects". It's up to each organization in OpenStack to focus on the set of projects it considers important. For more on that, go read Ed Leafe's brilliant blogpost, he expressed it better than I can. Of course there are some efforts (like packaging) where adding more projects results in diluting focus. But with every added project comes new blood in our community (rather than artificially keeping it out), and some of that new blood ends up helping on those efforts. It's not a zero-sum game, and the big tent makes sure we are open to new ways of achieving the OpenStack mission and have an inclusive and welcoming community.

by Thierry Carrez at November 28, 2016 09:38 AM

Hugh Blemings



Welcome to Last week on OpenStack Dev (“Lwood”) for the week just past. For more background on Lwood, please refer here.

Basic Stats for the week 21 to 27 November for openstack-dev:

  • ~367 Messages (down about 16% relative to last week)
  • ~125 Unique threads (down about 24% percent relative to last week)

Not surprisingly Message and Thread countsdropped back down again last week as many in the OpenStack community observed the Thanksgiving Holidays in the US. In a spare moment I goofed about a bit and looked at the overall trends since July 2015 when I started Lwood and the overall trend for Messages is quite noticeably down. The trend for Threads is down too but not as markedly – I conject that the overall variety/level of discourse is staying fairly constant, but the average thread is getting shorter – a sign perhaps of consensus being reached more rapidly and/or more involved discussions shifting to commentary in blueprints, reviews etc.

Notable Discussions – openstack-dev

What exactly is the Steward Working Group anyhow ?

Is the quasi-rhetorical question posed by Colette Alexander in her email. In her post you’ll find a very succinct explanation of the SWG activities – I dare say of interest to all involved in OpenStack and worth a read.

Liberty released tagged End of Life

Tony Breeds notes that in keeping with usual OpenStack process, the Liberty release is now tagged EoL – if you’re still using it, now might be a good to time to plan not to :)

OpenStack Summit Barcelona summary of summaries

Just a reminder that there’s a summary of summaries from the summit in Barcelona here.

Notable Discussions – other OpenStack lists

Things were pretty quiet, at least from an Lwood standpoint on the various other OpenStack lists.

People and Projects

Core nominations & changes


Further reading

Don’t forget these excellent sources of OpenStack news – most recent ones linked in each case


This edition of Lwood brought to you by Mr. Mister (Welcome to the Real World), Glenn Hughes (Resonate) and a couple of other tunes. On the off chance you like rock/heavy rock, please do yourself a favour and check out Resonate – it’s goosebumps inducing good – best thing I’ve heard all year and I fully expect will end up in my long term top ten album list – it’s excellent.

Last but by no means least, thanks, as always, to Rackspace :)

by hugh at November 28, 2016 07:40 AM


The Dos and Don'ts for Ceph for OpenStack

Ceph and OpenStack are an extremely useful and highly popular combination. Still, new Ceph/OpenStack deployments frequently come with easily avoided shortcomings — we'll help you fix them!

Do use show_image_direct_url and the Glance v2 API

With Ceph RBD (RADOS Block Device), you have the ability to create clones. You can think of clones as the writable siblings of snapshots (which are read-only). A clone creates RADOS objects only for those parts of your block device which have been modified relative to its parent snapshot, and this means two things:

  1. You save space. That's a no-brainer, but in and of itself it's not a very compelling argument as storage space is one of the cheapest things in a distributed system.

  2. What's not been modified in the clone can be served from the original volume. This is important because, of course, it means you are effectively hitting the same RADOS objects — and thus, the same OSDs — no matter which clone you're talking to. And that, in turn, means, those objects are likely to be served from the respective OSD's page caches, in other words, from RAM. RAM is way faster to access than any persistent storage device, so being able to serve lots of reads from the page cache is good. That, in turn, means, that serving data from a clone will be faster than serving the same data from a full copy of a volume.

Both Cinder (when creating a volume from an image) and Nova (when serving ephemeral disks from Ceph) will make use of cloning RBD images in the Ceph backend, and will do so automatically. But they will do so only if show_image_direct_url=true is set in glance‑api.conf, and they are configured to connect to Glance using the Glance v2 API. So do both.

Do set libvirt/images_type = rbd on Nova compute nodes

In Nova (using the libvirt compute driver with KVM), you have several options of storing ephemeral disk images, that is, storage for any VM that is not booted from a Cinder volume. You do so by setting the images_type option in the [libvirt] section in nova‑compute.conf:

images_type = <type>

The default type is disk, which means that when you fire up a new VM, the following events occur:

  • nova‑compute on your hypervisor node connects to the Glance API, looks up the desired image, and downloads the image to your compute node (into the /var/lib/nova/instances/_base directory by default).
  • It then creates a new qcow2 file which uses the downloaded image as its backing file.

This process uses up a fair amount of space on your compute nodes, and can quite seriously delay spawning a new VM if it has been scheduled to a host that hasn't downloaded the desired image before. It also makes it impossible for such a VM to be live-migrated to another host without downtime.

Flipping images_type to rbd means the disk lives in the RBD backend, as an RBD clone of the original image, and can be created instantaneously. No delay on boot, no wasting space, all the benefits of clones. Use it.

Do enable RBD caching on Nova compute nodes

librbd, the library that underpins the Qemu/KVM RBD storage driver, can enable a disk cache that uses the hypervisor host's RAM for caching purposes. You should use this.

Yes, it's a cache that is safe to use. On the one hand, the combination of virtio-blk with the Qemu RBD storage driver will properly honor disk flushes. That is to say, when an application inside your VM says "I want this data on disk now," then virtio‑blk, Qemu, and Ceph will all work together to only report the write as complete when it has been

  • written to the primary OSD,
  • replicated to the available replica OSDs,
  • acknowledged to have hit at least the persistent journal on all OSDs.

In addition, Ceph RBD has an intelligent safeguard in place: even if it is configured to cache in write-back mode, it will refuse to do so (meaning, it will operate in write-through mode) until it has received the first flush request from its user. Thus, if you run a VM that just never does that — because it has been misconfigured or its guest OS is just ages old — then RBD will stubbornly refuse to cache any writes. The corresponding RBD option is called rbd cache writethrough until flush, it defaults to true and you should never disable it.

You can enable writeback caching for Ceph by setting the following nova-compute configuration option:

images_type = rbd

And you just should.

Do use separate pools for images, volumes, and ephemeral disks

Now that you have enabled show_image_direct_url=true in Glance, configured Cinder and nova-compute to talk to Glance using the v2 API, and configured nova-compute with libvirt/images_type=rbd, all your VMs and volumes will be using RBD clones. Clones can span multiple RADOS pools, meaning you can have an RBD image (and its snapshots) in one pool, and its clones in another.

You should do exactly that, for several reasons:

  1. Separate pools means you can lock down access to those pools separately. This is just a standard threat mitigation approach: if your nova-compute node gets compromised and the attacker can corrupt or delete ephemeral disks, then that's bad — but it would be worse if they could also corrupt your Glance images.
  2. Separate pools also means that you can have different pool settings, such as the settings for size or pg_num.
  3. Most importantly, separate pools can use separate crush_ruleset settings. We'll get back to this in a second, it'll come in handy shortly.

It's common to have three different pools: one for your Glance images (usually named glance or images), one for your Cinder volumes (cinder or volumes), and one for your VMs (nova-compute or vms).

Don't necessarily use SSDs for your Ceph OSD journals

Of the recommendations in this article, this one will probably be the one that surprises the most people. Of course, conventional wisdom holds that you should always put your OSD journals on fast OSDs, and that you should deploy SSDs and spinners in a 1:4 to 1:6 ratio, right?

Let's take a look. Suppose you're following the 1:6 approach, and your SATA spinners are capable of writing at 100 MB/s. 6 spinners make 6 OSDs, and each OSD uses a journal device that's on a partition on an enterprise SSD. Suppose further that the SSD is capable of writing at 500 MB/s.

Congratulations, in that scenario you've just made your SSD your bottleneck. While you would be able to hit your OSDs at 600 MB/s on aggregate, your SSD limits you to about 83% of that.

In that scenario you would actually be fine with a 1:4 ratio, but make your spindles just a little faster and the SSD advantage goes out the window again.

Now, of course, do consider the alternative: if you're putting your journals on the same drive as your OSD filestores, then you effectively get only half the nominal bandwidth of your drive, on average, because you write everything twice, to the same device. So that means that without SSDs, your effective spinner bandwidth is only about 50 MB/s, so the total bandwidth you get out of 6 drives that way is more like 300 MB/s, against which 500 MB/s is still a substantial improvement.

So you will need to plug your own numbers into this, and make your own evaluation for price and performance. Just don't assume that journal SSD will be a panacea, or that it's always a good idea to use them.

Do create all-flash OSDs

One thing your journal SSDs don't help with are reads. So, what can you do to take advantage of SSDs on reads, too?

Make them OSDs. That is, not OSD journals, but actual OSDs with a filestore and journal. What this will create are OSDs that don't just write fast, but read fast, too.

Do put your all-flash OSDs into a separate CRUSH root

Assuming you don't run on all-flash hardware, but operate a cost-effective mixed cluster where some OSDs are spinners and others are SSDs (or NVMe devices or whatever), you obviously want to treat those OSDs separately. The simplest and easiest way to do that is to create a separate CRUSH root in addition to the normally configured default root.

For example, you could set up your CRUSH hierarchy as follows:

-1 4.85994 root default
-2 1.61998     host elk
 0 0.53999         osd.0          up  1.00000          1.00000 
 1 0.53999         osd.1          up  1.00000          1.00000 
 2 0.53999         osd.2          up  1.00000          1.00000 
-3 1.61998     host moose
 3 0.53999         osd.3          up  1.00000          1.00000 
 4 0.53999         osd.4          up  1.00000          1.00000 
 5 0.53999         osd.5          up  1.00000          1.00000 
-4 1.61998     host reindeer
 6 0.53999         osd.6          up  1.00000          1.00000 
 7 0.53999         osd.7          up  1.00000          1.00000 
 8 0.53999         osd.8          up  1.00000          1.00000
-5 4.85994 root highperf
-6 1.61998     host elk-ssd
 9 0.53999         osd.9          up  1.00000          1.00000 
10 0.53999         osd.10         up  1.00000          1.00000 
11 0.53999         osd.11         up  1.00000          1.00000 
-7 1.61998     host moose-ssd
12 0.53999         osd.12         up  1.00000          1.00000 
13 0.53999         osd.13         up  1.00000          1.00000 
14 0.53999         osd.14         up  1.00000          1.00000 
-8 1.61998     host reindeer-ssd
15 0.53999         osd.15         up  1.00000          1.00000 
16 0.53999         osd.16         up  1.00000          1.00000 
17 0.53999         osd.17         up  1.00000          1.00000

In the example above, OSDs 0-8 are assigned to the default root, whereas OSDs 9-17 (our SSDs) belong to the root highperf. We can now create two separate CRUSH rulesets:

rule replicated_ruleset {
    ruleset 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit

rule highperf_ruleset {
    ruleset 1
    type replicated
    min_size 1
    max_size 10
    step take highperf
    step chooseleaf firstn 0 type host
    step emit

The default ruleset, replicated_ruleset, picks OSDs from the default root, whereas step take highperf in highperf_ruleset means it covers only OSDs in the highperf root.

Do assign individual pools to your all-flash ruleset

Assigning individual pools to a new CRUSH ruleset (and hence, to a whole different set of OSDs) is a matter of issuing a single command:

ceph osd pool set <name> crush_ruleset <number>

... where <name> name of your pool and <number> is the numerical ID of your ruleset as per your CRUSH map. You can do this while the pool is online, and while clients are accessing its data — although of course, there will be a lot of remapping and backfilling so your overall performance may be affected somewhat.

Now, the assumption is that you will have more spinner storage than SSD storage. Thus, you will want to select individual pools for your all-flash OSDs. Here are a handful of pools that might come in handy as first candidates to migrate to all-flash. You can interpret the list below as a priority list: as you add more SSD capacity to your cluster, you can move pools over to all-flash storage one by one.

  1. Nova ephemeral RBD pools (vms, nova-compute)
  2. radosgw bucket indexes (.rgw.buckets.index and friends) — if you're using radosgw as your drop-in OpenStack Swift replacement
  3. Cinder volume pools (cinder, volumes)
  4. radosgw data pools (.rgw.buckets and friends) — if you need low-latency reads and writes on Swift storage
  5. Glance image pools (glance, images)
  6. Cinder backup pools (cinder-backup) — usually the last pool to convert to all-flash OSDs.

Do designate some non-Ceph compute hosts with low-latency local storage

Now, there will undoubtedly be some applications where Ceph does not produce the latency you desire. Or, for that matter, any network-based storage. That's just a direct consequence of recent developments in storage and network technology.

Just a few years ago, the average latency of a single-sector uncached write to a block device was on the order of a millisecond, or 1,000 microseconds (µs). In contrast, the latency incurred on a TCP packet carrying a 512-byte (1-sector) payload was about 50 µs, which makes for a 100-µs round trip. All in all, the additional latency incurred from writing to a device over the network, as opposed to locally, was approximately 10%.

In the interim, a single-sector write for a device of the same price is itself about 100 µs, tops, with some reasonably-priced devices down to about 40 µs. Network latency, in contrast, hasn't changed all that much — going down about 20% from Gigabit Ethernet to 10 GbE.

So even going to a single, un-replicated SSD device over the network will now be 40 + 80 = 120 µs latency, vs. just 40 µs locally. That's not a 10% overhead anymore, that's a whopping factor of 3.

With Ceph, that gets worse. Ceph writes data multiple times, first to the primary OSD, then (in parallel) to all replicas. So in contrast to a single-sector write at 40 µs, we now incur a latency of at least two writes, plus two network round-trips, to that's 40 x 2 + 80 x 2 = 240 µs, six times the local write latency.

The good news is, most applications don't care about this sort of latency overhead, because they're not latency-critical at all. The bad news is, some will.

So, should you ditch Ceph because of that? Nope. But do consider adding a handful of compute nodes that are not configured with libvirt/images_type=rbd, but that use local disk images instead. Roll those hosts into a host aggregate, and map them to a specific flavor. Recommend to your users that they use that flavor for low-latency applications.

by hastexo at November 28, 2016 12:00 AM

Carlos Camacho

Testing composable upgrades

This is a brief recipe about how I’m testing composable upgrades N->O.

Based on the original shardy’s notes from this link.

The following steps are followed to upgrade your Overcloud from Mitaka to latest master (Ocata).

  • Deploy latest master TripleO following this post.

  • Remove the current Overcloud deployment.

  source stackrc
  heat stack-delete overcloud
  • Remove the Overcloud images and create new ones (for the Newton Overcloud).
  openstack image list
  openstack image delete <image_ID> #Delete all the Overcloud images overcloud-full* 
  rm -rf /home/stack/overcloud-full.*

  export STABLE_RELEASE=newton
  export DELOREAN_REPO_FILE="delorean.repo"
  /home/stack/tripleo-ci/scripts/ --overcloud-images

  # Or reuse images
  # wget
  # tar -xvf overcloud-full.tar
  # openstack overcloud image upload --update-existing
  • Download Newton tripleo-heat-templates.
  git clone -b stable/newton tht-newton
  • Configure the DNS (needed when upgrading the Overcloud).
  neutron subnet-update `neutron subnet-list -f value | awk '{print $1}'` --dns-nameserver
  • Deploy a Newton Overcloud.
  openstack overcloud deploy \
  --libvirt-type qemu \
  --templates /home/stack/tht-newton/ \
  -e /home/stack/tht-newton/overcloud-resource-registry-puppet.yaml \
  -e /home/stack/tht-newton/environments/puppet-pacemaker.yaml
  • Install prerequisites in nodes (if no DNS configured this will fail).
cat > upgrade_repos.yaml << EOF
  UpgradeInitCommand: |
    set -e
    export HOME=/root
    cd /root/
    if [ ! -d tripleo-ci ]; then
      git clone
      pushd tripleo-ci
      git checkout master
      git pull

    if [ ! -d tripleo-heat-templates ]; then
      git clone
      pushd tripleo-heat-templates
      git checkout master
      git pull

    ./tripleo-ci/scripts/ --repo-setup
    sed -i "s/includepkgs=/includepkgs=python-heat-agent*,/" /etc/yum.repos.d/delorean-current.repo
    yum -y install python-heat-agent-ansible
  • Download master tripleo-heat-templates, composable upgrades patch and rebase (part of this step will be removed when the composable upgrades with Ansible patch lands).
  git clone tht-master
  cd tht-master #Remove after landing this feature
  git review -d 403441 #Remove after landing this feature
  git rebase -i master #Remove after landing this feature
  • Upgrade Overcloud to master
  openstack overcloud deploy \
  --templates /home/stack/tht-master/ \
  -e /home/stack/tht-master/overcloud-resource-registry-puppet.yaml \
  -e /home/stack/tht-master/environments/puppet-pacemaker.yaml \
  -e /home/stack/tht-master/environments/major-upgrade-composable-steps.yaml \
  -e upgrade_repos.yaml

If the last steps manage to finish successfully (WIP), you just have upgraded your Overcloud from Newton to Ocata (latest master).

For more resources related to TripleO deployments, check out the TripleO YouTube channel.

Updated 2016/11/28: Not working properly due to an error in the galera cluster (probably related to my env).

TASK [setup] *******************************************************************
ok: [localhost]
TASK [Stop cinder_api service (running under httpd)] ***************************
changed: [localhost]
TASK [Stop cinder_scheduler service] *******************************************
changed: [localhost]\n\nTASK [Stop cinder_volume service] **********************************************
changed: [localhost]
TASK [Stop keystone service (running under httpd)] *****************************\nok: [localhost]
TASK [Stop glance_api service] *************************************************\nchanged: [localhost]
TASK [Stop glance_registry service] ********************************************\nchanged: [localhost]
TASK [Stop heat_api service] ***************************************************\nchanged: [localhost]
TASK [Stop heat_api_cfn service] ***********************************************\nchanged: [localhost]
TASK [Stop heat_api_cloudwatch service] ****************************************\nchanged: [localhost]
TASK [Stop heat_engine service] ************************************************\nchanged: [localhost]
TASK [Stop service] ************************************************************\nok: [localhost]
TASK [Stop neutron_dhcp service] ***********************************************\nchanged: [localhost]
TASK [Stop neutron_l3 service] *************************************************\nchanged: [localhost]
TASK [Stop neutron_metadata service] *******************************************\nchanged: [localhost]
TASK [Stop neutron_api service] ************************************************\nchanged: [localhost]
TASK [Stop neutron_ovs_agent service] ******************************************\nchanged: [localhost]
TASK [Stop rabbitmq service] ***************************************************\nok: [localhost]
TASK [Stop nova_conductor service] *********************************************\nchanged: [localhost]
TASK [Stop nova_api service (running under httpd)] *****************************\nok: [localhost]
TASK [Stop nova_scheduler service] *********************************************\nchanged: [localhost]
TASK [Stop nova_consoleauth service] *******************************************\nchanged: [localhost]
TASK [Stop nova_vnc_proxy service] *********************************************\nok: [localhost]\
TASK [Stop openstack-swift-proxy service] **************************************\nchanged: [localhost]

TASK [Stop swift object services] **********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: need more than 1 value to unpack
fatal: [localhost]: FAILED! => {\"changed\": false, \"failed\": true, \"module_stderr\": \"Traceback (most recent call last):\
  File \\\"/tmp/ansible_9YZa1c/\\\", line 1515, in <module>\
  File \\\"/tmp/ansible_9YZa1c/\\\", line 1471, in main\\
  File \\\"/tmp/ansible_9YZa1c/\\\", line 561, in get_service_status\
    return self.get_systemd_service_status()\\
  File \\\"/tmp/ansible_9YZa1c/\\\", line 542, in get_systemd_service_status\
    d = self.get_systemd_status_dict()\
  File \\\"/tmp/ansible_9YZa1c/\\\", line 518, in get_systemd_status_dict\
    key, value = line.split('=', 1)\
ValueError: need more than 1 value to unpack\
\", \"module_stdout\": \"\", \"msg\": \"MODULE FAILURE\", \"parsed\": false}
NO MORE HOSTS LEFT *************************************************************
\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/7585c039-7f38-412a-8392-5ecd9e12fd39_playbook.retry
PLAY RECAP *********************************************************************
localhost                  : ok=24   changed=18   unreachable=0    failed=1   
    "deploy_stderr": "", 

TASK [Stop swift object services] **********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: need more than 1 value to unpack
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_ueI7a6/\", line 1515, in <module>\n    main()\n  File \"/tmp/ansible_ueI7a6/\", line 1471, in main\n    service.get_service_status()\n  File \"/tmp/ansible_ueI7a6/\", line 561, in get_service_status\n    return self.get_systemd_service_status()\n  File \"/tmp/ansible_ueI7a6/\", line 542, in get_systemd_service_status\n    d = self.get_systemd_status_dict()\n  File \"/tmp/ansible_ueI7a6/\", line 518, in get_systemd_status_dict\n    key, value = line.split('=', 1)\nValueError: need more than 1 value to unpack\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}


New error:

2016-11-26 23:47:05Z [AllNodesUpgradeSteps.ControllerUpgrade_Step4.0]: SIGNAL_IN_PROGRESS  Signal: deployment 4b3dfa0c-a1d3-444c-a98b-cc61f5db49f0 failed (2)
2016-11-26 23:47:05Z [AllNodesUpgradeSteps.ControllerUpgrade_Step4.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2016-11-26 23:47:05Z [AllNodesUpgradeSteps.ControllerUpgrade_Step4]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2016-11-26 23:47:06Z [AllNodesUpgradeSteps.ControllerUpgrade_Step4]: CREATE_FAILED  Error: resources.ControllerUpgrade_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2016-11-26 23:47:06Z [AllNodesUpgradeSteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerUpgrade_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2016-11-26 23:47:07Z [AllNodesUpgradeSteps]: CREATE_FAILED  Error: resources.AllNodesUpgradeSteps.resources.ControllerUpgrade_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2016-11-26 23:47:07Z [overcloud]: UPDATE_FAILED  Error: resources.AllNodesUpgradeSteps.resources.ControllerUpgrade_Step4.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2016-11-26 23:47:10Z [overcloud-ControllerAllNodesDeployment-zdhtbbghswuq.0]: SIGNAL_COMPLETE  Unknown
2016-11-26 23:47:16Z [overcloud-Controller-hmkrkckqdnez-0-zxwqd4xqp3br.ControllerDeployment]: SIGNAL_COMPLETE  Unknown
2016-11-26 23:47:18Z [overcloud-ControllerHostsDeployment-3mscoxbvr3lu.0]: SIGNAL_COMPLETE  Unknown

"deploy_stdout": "
PLAY [localhost] ***************************************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [Start service] ***********************************************************
fatal: [localhost]: FAILED! => {\"changed\": false, \"failed\": true, \"msg\": \"Job for mariadb.service failed because the control process exited with error code. See \\\"systemctl status mariadb.service\\\" and \\\"journalctl -xe\\\" for details.\
NO MORE HOSTS LEFT *************************************************************
\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/45fee150-1bf7-4580-bb3f-a3c7ce60c199_playbook.retry
PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   

become: True for all services restarts

by Carlos Camacho at November 28, 2016 12:00 AM

November 26, 2016

Carlos Camacho

TripleO cheatsheet

This is a cheatsheet some of my regularly used commands to test, develop or debug TripleO deployments.


  • swift download overcloud

    Download the overcloud swift container files in the current folder (With the rendered j2 templates).

  • heat resource-list --nested-depth 5 overcloud | grep FAILED

    Show resources, filtering to get those who have failed.

  • heat deployment-show <deployment_ID>

    Get the deployment details for <deployment_ID>.

  • openstack image list

    List images.

  • openstack image delete <image_ID>

    Delete <image_ID>.

  • wget<release>/delorean/overcloud-full.tar

    Download <release> overcloud images tar file [liberty|mitaka|newton|...]

  • openstack overcloud image upload --update-existing

    Once downloaded the images, this command will upload them to glance.

Debugging CI


    Check submissions CI status.

  • wget -e robots=off -r --no-parent <patch_ID>

    Download all logs from <patch_ID>.

  • console.html & logs/postci.txt.gz

    Relevant log files when debuging a TripleO Gerrit job.

If you think there are more useful commands to add to the list just add a comment!

Happy TripleOing!

by Carlos Camacho at November 26, 2016 12:00 AM

November 25, 2016

Stackmasters team

Stackmasters partners with tripsta to build their OpenStack cloud

Stackmasters, the European OpenStack experts and Managed Cloud services provider, today announced its ongoing partnership with one of the biggest online travel agencies in Europe, tripsta, to develop an OpenStack based private cloud and related processes.

The OpenStack based private cloud is going to host existing applications as well as to support the company’s software development teams to their cloud-native modern software architecture by leveraging open technology and efficient process models.

“The most significant step in order to adopt the cloud with success is made by companies where business and IT leadership is aware that their digital transformation is driven by more than the technology solution,” said Thanassis Parathyras, CTO of Stackmasters. “Tripsta, along with other companies we support in EU, digested the open technology and cloud initiatives to achieve agility and efficiency in their business objectives, and we ‘re excited to participate in the project for building the tripsta cloud.”

“Tripsta has long before embraced open-source software and understands its important role in cloud technology which allows both the operators team and the development teams to utilize company’s resources in order to develop and enrich our leading on line booking services,” said Yiannis Polizois, Systems Administration Manager at tripsta. “Making things happen, we first had to build a cloud infrastructure and setup the respective processes. With Stackmasters support we are able to iterate over several cycles resulting in defining the right strategy and select suitable technologies towards bringing improved performance for our business.”

About Stackmasters

Stackmasters is a Cloud Specialist company founded in 2013, uniquely combining years of technical and business experience in IT with the enthusiasm of a cutting- edge tech startup. Its founders are specialists in Cloud Computing and have been following the emerging paradigm since its beginning from all its aspects, wearing the hat of the provider, the consumer, the engineer and consultant, and, last but not least, that of the academic researcher. Stackmasters carries the legacy and technical expertise of Virtual Trip, the established Greek IT services company and technology incubator, and focuses on grid computing, distributed systems and architecture, virtualisation and open-source software implementations. View more

About tripsta

Tripsta Travelplanet
In 2005 two restless entrepreneurs founded a startup initially called travelplanet24 in the heart of one of the world’s travel hot spots, Athens Greece. Today our online travel agency, tripsta, is one of the largest and most recognized leaders in the industry. Our trusted brands, tripsta, airtickets®, and travelplanet24 are preferred by more than a million loyal travelers throughout the 45 global markets that we presently operate in. With offices in Athens, Bucharest and Istanbul our committed team of 280 passionate travel geniuses ensure that you reach your destination with guaranteed low prices and dedicated support.

The post Stackmasters partners with tripsta to build their OpenStack cloud appeared first on Stackmasters.

by Stackmasters at November 25, 2016 10:11 AM

November 24, 2016


Monitoring an Openstack deployment with Prometheus and Grafana

Following our previous blog post, we are still looking at tools for collecting metrics from an Openstack deployment in order to understand its resource utilization. Although Monasca has a comprehensive set of metrics and alarm definitions, the complex installation process combined with a lack of documentation makes it a frustrating experience to get it up and running. Further, although it is complex, with many moving parts, it was difficult to configure it to obtain the analysis we wanted from the raw data, viz how many of our servers are overloaded over different timescales in different respects (cpu, memory, disk io, network io). For these reasons we decided to try Prometheus with Grafana which turned out to be much easier to install and configure (taking less than an hour to set up!). This blog post covers the installation process and configuration of Prometheus and Grafana in a Docker container and how to install and configure Canonical’s Prometheus Openstack exporter to collect a small set of metrics related to an Openstack deployment.

Note that minor changes to this HOWTO are required to install theses services in a VM or in a host machine when using containers is not an option. As preparation, take note of your Openstack deployment’s locations for Keystone and the Docker host. Remember that all downloads should be verified by signature comparison for production use.

Installing and configuring Prometheus

First of all pull the Ubuntu image into you docker machine. Let’s call it docker-host.

Note that in this blog post we describe Prometheus installation process step-by-step – we chose to install it from scratch to get a better understanding of the system, but using the pre-canned Docker Hub image is also possible.

docker pull ubuntu:14.04

Then create the docker container opening the port 9090 which will be used to get/push metrics into Prometheus.

docker run -it -p 9090:9090 --name prometheus ubuntu:14.04

Inside the container download the latest version of Prometheus and uncompress it (version 1.3.1 is used in this HOWTO; the download size is ca. 16 MB).

tar xvf prometheus-1.3.1.linux-amd64.tar.gz
cd prometheus-1.3.1.linux-amd64

Configure prometheus.yml adding the targets from which prometheus should scrape metrics. See the example below for the Openstack exporter (assuming it is installed in the same docker-host):

  - job_name: 'openstack-deployment-1'
    scrape_interval: 5m
      - targets: ['docker-host:9183']

Start the Prometheus service:

./prometheus -config.file=prometheus.yml

Similarly, install and configure the Prometheus Openstack exporter in another container. Note that this container needs to be set up manually as there are configuration files to be changed and Openstack libraries to be installed.

docker run -it -p 9183:9183 --name prometheus-openstack-exporter ubuntu:14.04
sudo apt-get install python-neutronclient python-novaclient python-keystoneclient python-netaddr unzip wget python-pip python-dev python-yaml
pip install prometheus_client
cd prometheus-openstack-exporter-master/

Next, configure prometheus-openstack-exporter.yaml create the /var/cache/prometheus-openstack-exporter/ directory and the novarc file containing credentials for Nova user.

mkdir /var/cache/prometheus-openstack-exporter/
echo  export OS_USERNAME=nova-username \
      export OS_PASSWORD=nova-password \
      export OS_AUTH_URL=http://keystone-url:5000/v2.0 \
      export OS_REGION_NAME=RegionOne \
      export OS_TENANT_NAME=services > novarc
source novarc
./prometheus-openstack-exporter prometheus-openstack-exporter.yaml

Then you’ve got a fully functional Prometheus system with some Openstack metrics on it! Visit http://docker-host:9090 to graph and see which metrics are available.

Here is the list of the 18 metrics currently collected by Prometheus Openstack exporter:

neutron_public_ip_usage hypervisor_memory_mbs_total
neutron_net_size hypervisor_running_vms
hypervisor_memory_mbs_used hypervisor_disk_gbs_total
hypervisor_vcpus_total hypervisor_disk_gbs_used
openstack_allocation_ratio hypervisor_vcpus_used
nova_instances nova_resources_ram_mbs
nova_resources_disk_gbs swift_replication_duration_seconds
openstack_exporter_cache_age_seconds swift_disk_usage_bytes
swift_replication_stats swift_quarantined_objects

Alternatively you could use Prometheus’s Node exporter for more detailed metrics on node usage – this needs to be installed in the controller/compute nodes and the prometheus.yml configuration file also needs to be changed. A docker container is also available at Docker Hub.

Although Prometheus provides some rudimentary graph support, combining it with a more powerful graphing solution makes it much easier to see what’s going on in your system. For this reason, we set up Grafana.

Installing Grafana

The latest version of Grafana (currently 4.0.0-beta2) had a lot of improvements in its user interface, it also supports now alerting and notifications for every panel available – refer to the documentation for more information. Its integration with Prometheus is very straightforward, as described below.

First of all, pull the grafana image into your docker-host and create the docker container opening the port 3000 used to access it.

docker pull grafana/grafana
docker run -d -p 3000:3000 grafana/grafana:4.0.0-beta2

Visit http://docker-host:3000 and use the credentials admin/admin to log into the dashboard. In the Data Sources tab, add a new corresponding data source.


Create a new dashboard and add panels containing graphs using the Prometheus datasource.


Play around with metrics available and create your own dashboard! See a simple example below.



Although not many metrics are available yet to monitor an Openstack deployment the combination of Prometheus and Grafana is quite powerful for visualising data; also it was much easier to set up in comparison with Monasca. Further, from a cursory glance, Prometheus seems to be more flexible than Monasca and for these reasons it appears more promising. That said, we are still looking into Prometheus and how it can be used to properly understand resource consumption in an Openstack context, but that will come in another blog post!

by Bruno Grazioli at November 24, 2016 09:34 AM


Installing OpenStack on OpenStack with External Network

Installing OpenStack on OpenStack and enabling external network on the upper layer of OpenStack.

For testing I need to install an All-in-One OpenStack in a VM on our bare-metal OpenStack. Installing an All-in-One OpenStack is easy, but it requires that the VMs on the All-in-One OpenStack be accessed from other VMs on the bare-metal OpenStack, which is a challenge. I’ll show you how installing OpenStack on OpenStack works in this blog.

Firstly, here is the diagram showing what I will set up.


The goals are to let VM1 and VM2 talk to each other, and let VM2 have Internet access.

I am going to install an All-in-One OpenStack on a bare-metal OpenStack. Firstly, I need to create a tenant network in the bare-metal OpenStack for a VM running All-in-One OpenStack. When doing that, I reserve some IPs and use them in the All-in-One OpenStack as floating IPs ( The bare-metal OpenStack is set up properly and there is already an external network in the bare-metal OpenStack so that VM1 can access the Internet via a virtual router, thus I just need to create a tenant network. To do that, use

neutron net-create mynet
neutron subnet-create --name mysubnet --dns-nameserver --allocation-pool start=,end= mynet

In this case, it will create a VxLAN tenant network. But other network types should work as well. Also remember to create a router for this tenant network to access the Internet.
Now I can start a VM for the All-in-One OpenStack. The VM needs at least 8GB ram and 2 CPUs to run everything and get a reasonable performance for testing. Once the VM is up, you can test if the VM has Internet access before installing OpenStack.

There are many ways to set up an All-in-One OpenStack. The easiest way is probably using PackStack, or you can use Puppet modules. Normally when launching a VM in OpenStack, the VM has one network interface called eth0. I am going to use that as the external network interface for the All-in-One OpenStack.

PackStack can do that automatically; I just need to add some arguments to the packstack command, as in

packstack --allinone --provision-demo=n --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --os-neutron-ml2-type-drivers=vxlan,flat

In this example, it creates a new br-ex bridge and connects eth0 to it. It also maps the logical name “extnet” to br-ex so I can reference it in my external provider network in Neutron. “flat” is added to the type driver because the external provider network is a flat network.

If using Puppet modules, or another method that doesn’t do this, I need to create br-ex manually.
Edit /etc/sysconfig/network-scripts/ifcfg-eth0 so it becomes a port of br-ex.


Edit /etc/sysconfig/network-scripts/ifcfg-br-ex

OVS_EXTRA="set bridge br-ex other-config:hwaddr=fa:16:3e:06:95:92"

Here br-ex is created as an OVS bridge. It uses DHCP to get an IP address from the bare-metal OpenStack’s tenant network ( via eth0. Therefore we need an extra option to set its MAC address to be the same as eth0 (as below) for DHCP to work. Neutron only assigns IP addresses to known MAC addresses and the mapping is put into Dnsmasq’s config. If we don’t specify the MAC address here, br-ex won’t be able to get any IP address from the DHCP server. After making all these changes, restart network service.

2: eth0: mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000
link/ether fa:16:3e:06:95:92 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe06:9592/64 scope link
valid_lft forever preferred_lft forever

Double check the config of br-ex and eth0 no matter which method you use to install All-in-One OpenStack, and make sure the network is working.

Now I can create networks in the All-in-One OpenStack. Note that the physical network name “extnet” is mapped to br-ex in Neutron’s config. The IP range of the external network is reserved previously and not overlapped with the tenant network in the bare-metal OpenStack. Also the external network has DHCP disabled and it is using br-ex’s IP address as the gateway.

neutron net-create public --router:external True --provider:physical_network extnet --provider:network_type flat
neutron subnet-create --allocation-pool start=,end= --dns-nameserver --name public_subnet --gateway public --disable-dhcp
neutron net-create private_network
neutron subnet-create --name private_subnet private_network
neutron router-create router1
neutron router-gateway-set router1 public
neutron router-interface-add router1 private_subnet

Next I will create an instance, VM2, with a floating IP.

Until now I can access VM2 from its host, the VM running All-in-One OpenStack but I can’t access to it from another VM in the same tenant network in bare-metal OpenStack, such as VM1.

In order to allow the instances in All-in-One Openstack to talk to the VMs in bare-metal OpenStack, such as VM1 in the diagram, there is one more thing to do on the bare-metal OpenStack. By default, Neutron creates an iptables rule to only allow traffic matching the VM’s IP/MAC address pair to get into the VM; other traffic will be filtered out, such as the following example.

Chain neutron-openvswi-s27a6fc06-7 (1 references)
target prot opt source destination
RETURN all -- MAC FA:16:3E:86:6E:78 /* Allow traffic from defined IP/MAC pairs. */
DROP all -- /* Drop traffic without an IP/MAC allow rule. */

However, since I am running an All-in-One OpenStack on the VM, I need the VM to see all traffic that targets the floating IPs that are associated with All-in-One OpenStack’s VMs.

There are two ways that you can let the VM see traffic targeting other IPs. One is to add extra IP/MAC pairs to the port of this VM in bare-metal OpenStack.

neutron port-update [Port-UUID] --allowed-address-pairs type=dict list=true mac_address=XXX,ip_address=YYY

The MAC address and IP address has to match the ones of the instance in All-in-One OpenStack. Since we are using DHCP for the tenant network in All-in-One OpenStack, the MAC/IP pair is unknown beforehand. Therefore I will use the second method which is to disable IP/MAC filtering so that this VM can see all traffic in the network.

To do that, I need to enable the port_security plugin in Neutron of the bare-metal OpenStack. On controller node, edit /etc/neutron/plugins/ml2/ml2_conf.ini

extension_drivers = port_security

Then restart neutron server.

systemctl restart neutron-server

Now I can disable port security on the port of the All-in-One OpenStack VM. Firstly, find the UUID of the port. Then use this command to disable security group first.

neutron port-update --no-security-groups [Port-UUID]

This means you need to use iptables in the VM to protect it. Next you can use the following command to disable port security:

neutron port-update [Port-UUID] --port-security-enabled=False

Now the VMs in All-in-One OpenStack are accessible from other VMs in the bare-metal OpenStack using floating IPs. On VM1, I can ssh to VM2 using its floating IP. In VM2, I can access the Internet.


The post Installing OpenStack on OpenStack with External Network appeared first on Aptira Cloud Solutions.

by Shunde Zhang at November 24, 2016 09:06 AM

Adam Young

JSON Home Tests and Keystone API changes

If you change the public signature of an API, or add a new API in Keystone, there is a good chance the Tests that confirm JSON home layout will break.  And that test is fairly unfriendly:  It compares a JSON doc with another JSON doc, and spews out the entirety of both JSON docs, without telling you which section breaks.  Here is how I deal with it:

First, run the test in a manner that you can do some qyerying. I did this:


tox -e py27 -- keystone.tests.unit.test_versions.VersionTestCase.test_json_home_v3 2>&1 | less

That lets me search through the output dynamically.  It runs only the one test.


Here is the change I made to the keystone/assignment/ file:


If I run the test and search through the output for the value url_pattern I see that this section is new:

 u'': {u'href-template': u'/url_patterns/{url_pattern_id}',
 u'href-vars': {u'url_pattern_id': u''}},
 u'': {u'href': u'/url_patterns'},

Sorry for the formatting:  it is really long lines in the output.

To start, I modify the file to add something that I think will be comparable:

 json_home.build_v3_resource_relation('url_pattern'): {
 'href-template': '/url_pattern/{url_pattern_id}',
 'href-vars': {
 'url_pattern_id': json_home.Parameters.URL_PATTERN_ID},

Rerunning the test, I now see that it matches earlier:  this is in the expected output:

 '': {'hints': {'status': 'experimental'},
 'href-template': '/url_patterns/{url_pattern_id}',
 'href-vars': {'url_pattern_id': ''}},

Which looks good, but I need the second line, with url_patterns.  So I add:

 json_home.build_v3_resource_relation('url_patterns'): {
 'href': '/url_patterns'},

Note that it is href, and not href-template.



by Adam Young at November 24, 2016 03:31 AM

November 23, 2016

OpenStack Superuser

Tests demonstrate OpenStack NFV interoperability

One of the big questions surrounding OpenStack is whether distributions can play well together.

When it comes to NFV, the European Advanced Networking Test Center (EANTC), helped put that question to rest. The EANTC is an objective and independent testing center based in Berlin, Germany. The organization provides vendor-neutral performance testing laboratories for telecommunications solutions. They tested NFV on various distributions of OpenStack.

At the recent OpenStack Summit in Barcelona, EANTC’s managing director Carsten Rossenhovel showcased recent NFV testing that included an interoperability test and performance tests run in two different scenarios. According to Rossenhovel, “without OpenStack, none of this would be possible.”

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

Check out the video below for a live demo of their interoperability tests.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

The post Tests demonstrate OpenStack NFV interoperability appeared first on OpenStack Superuser.

by Superuser at November 23, 2016 01:00 PM


Tristan Goode interviewed by WIN News Canberra

Last week, we held the first ever OpenStack Government Day. Held in Canberra, this event attracted a range of industry leaders from around the country to our nations capital to share insights into the future of technology within the public sector.

Aptira’s CEO Tristan Goode and Vault Systems CEO Rupert Taylor-Price were interviewed by WIN News Canberra to discuss the growth of cloud, and the huge demand for employment within the industry.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="" width="560"></iframe>

The post Tristan Goode interviewed by WIN News Canberra appeared first on Aptira Cloud Solutions.

by Jessica Field at November 23, 2016 09:05 AM

November 22, 2016

Kenneth Hui

OpenStack Summit Day 2 Recap: The Future is Multi-Cloud


Jonathan Bryce, Executive Director of the OpenStack Foundation, kicked off day two of the Summit by tying it back to the first day’s keynote about the challenge of meeting the scalability requirements of projects such as the Square Kilometer Array radio telescope.

The SKA project is expecting to require multi-exaFLOPS — that’s one quintillion (10^18) floating point operations per second — once it’s fully online. With those requirements, Bryce said, we need to abandon the old paradigm of picking winners and losers in the cloud, or seeing the cloud space as zero sum game. Instead, he proposed that we embrace a “positive sum cloud world.”

To do that, Bryce believes the OpenStack community needs to focus on three key principles: keep innovating, collaborate with other technologies and replicate success.

To read more about day 2 of the Barcelona OpenStack Summit, please click here to go to my article on the Rackspace blog site.

Filed under: Cloud, Cloud Computing, OpenStack, Private Cloud Tagged: Cloud, Cloud computing, OpenStack, OpenStack Summit, Private Cloud, Rackspace

by kenhui at November 22, 2016 06:19 PM


Blog posts in the past week

Here's what RDO enthusiasts have been blogging about over the last week or so:

Chasing the trunk, but not too fast by amoralej

As explained in a previous post, in RDO Trunk repositories we try to provide packages for new commits in OpenStack projects as soon as possible after they are merged upstream. This has a number of advantages:


Enabling nested KVM support for a instack-virt-setup deployment. by Carlos Camacho

The following bash snippet will enable nested KVM support in the host when deploying TripleO using instack-virt-setup.


Ocata OpenStack summit 2016 - Barcelona by Carlos Camacho

A few weeks ago I had the opportunity to attend to the Barcelona OpenStack summit ‘Ocata design session’ and this post is related to collect some overall information about it. In order to achieve this, I’m crawling into my paper notes to highlight the aspects IMHO are relevant.


On communities: Emotions matter by Flavio Percoco

Technology is social before it's technical - Gilles Deleuze


New TLS algorithm priority config for libvirt with gnutls on Fedora >= 25 by Daniel Berrange

Libvirt has long supported use of TLS for its remote API service, using the gnutls library as its backend. When negotiating a TLS session, there are a huge number of possible algorithms that could be used and the client & server need to decide on the best one, where “best” is commonly some notion of “most secure”. The preference for negotiation is expressed by simply having an list of possible algorithms, sorted best to worst, and the client & server choose the first matching entry in their respective lists. Historically libvirt has not expressed any interest in the handshake priority configuration, simply delegating the decision to the gnutls library on that basis that its developers knew better than libvirt developers which are best. In gnutls terminology, this means that libvirt has historically used the “DEFAULT” priority string.


New libvirt website design by Daniel Berrange

The current previous libvirt website design dated from circa 2008 just a few years after the libvirt project started. We have grown alot of content since that time, but the overall styling and layout of the libvirt website has not substantially changed. Compared to websites for more recently launched projects, libvirt was starting to look rather outdated. So I spent a little time to come up with a new design for the libvirt website to bring it into the modern era. There were two core aspects to the new design, simplify the layout and navigation, and implement new branding.


Quick Guide: How to Plan Your Red Hat Virtualization 4.0 Deployment by Eric D. Schabell

On August 24th of this year Red Hat announced the newest release of Red Hat Virtualization (RHV) 4.0.


Visualizing Kolla’s Ansible playbooks with ARA by dmsimard

Kolla is an OpenStack deployment tool that’s growing in popularity right now.


Recapping OpenStack Summit Barcelona by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform

More than 5,200 OpenStack professionals and enthusiasts gathered in Barcelona, Spain to attend the 2016 OpenStack Summit. From the keynotes to the break-out sessions to the marketplace to the evening events and the project work sessions on Friday, there was plenty to keep attendees busy throughout the week. In fact, if you were one of the lucky ones who attended OpenStack Summit, there was probably many sessions and activities you wanted to make it to but couldn’t.


by Rich Bowen at November 22, 2016 05:39 PM

Chasing the trunk, but not too fast

As explained in a previous post, in RDO Trunk repositories we try to provide packages for new commits in OpenStack projects as soon as possible after they are merged upstream. This has a number of advantages:

  • It allows packagers to identify packaging issues just after introduced.
  • For project developers, their changes are tested in non-devstack environments, so test coverage is extended.
  • For deployment tools projects, they can use these repos to identify problems with new versions of packages and to start integrating any enhancement added to projects as soon as it's merged.
  • For operators, they can use these packages as hot-fixes to install in their RDO clouds before patches are included in official packages.

This means that, for every merged commit a new package is created and a yum repository is published in RDO trunk server. This repo includes the just built package and the latest builds for the rest of packages in the same release.

Initially, we applied this approach to every package included in RDO. However, while testing these repos during the Newton cycle we observed that jobs failed with errors that didn't affect OpenStack upstream gates. The reason behind this is that commits in OpenStack gate jobs are tested with versions of libraries and clients defined in upper-constraints.txt files in requirements project for the branch where the change is proposed. Typically these are the last tag-released versions. As RDO was testing using libraries from last commit instead of last release, we were effectively ahead of upstream tests, running too fast.

While this provided some interesting information and we could identify issues very early, it made very difficult to get stable repositories that could be promoted and used. After some discussions in RDO weekly meeting, it was decided to apply some changes in the way libraries are managed to leverage the work done in upstream gates but try to keep catching issues as soon as possible:

  • For master branch, it was decided to pin libraries and clients to the versions included in upper-constraints for repositories served in This repositories are used by RDO CI promotion jobs and marked as current-passed-ci when all tests succeed.
  • Additionally, a new builder was created that chases master in all packages, including libraries, clients, etc… This builder is able to catch issues based on unit tests executed when packages are created. The produced repos are available in but promotion jobs are not executed using them.

The differences between master and master-head are shown in following diagram:

RDO master pins

  • For releases in maintenance phase we pin libraries to what's found in upper-constraints.txt file in the corresponding branch.


In order to manage the libraries versions properly, RDO is using a peer-reviewed workflow of gerrit reviews proposed to the rdoinfo project in You can see an example here.

A job is executed periodically that automatically creates gerrit reviews when versions are updated in upper-constraints files. Manual approval is needed to get the changes merged and the new versions built by DLRN builders.

by amoralej at November 22, 2016 03:29 PM

OpenStack Superuser

OpenStack gains momentum at Supercomputing 2016

Last week at Supercomputing 2016 there was a lot of activity around OpenStack. Working Group co-chair Blair Bethwaite put together a pick of the schedule’s OpenStack highlights, and used the event to launch the book edition of my OpenStack / high-performance computing (HPC) studies, featuring user stories and case studies from many prominent working group members.


In testament to the momentum building behind using OpenStack for research computing, the Scientific OpenStack birds of a feather (BoF) filled the room, and was even bigger than the scientific OpenStack BoF at the OpenStack Summit in Barcelona. Topics discussed included the trade-offs between virtualization and containers for scientific use cases and different strategies for finding performance on scientific OpenStack clouds.


One interesting observation came from a show of hands on who in the room was already using OpenStack – it turned out to be no more than a significant minority. OpenStack already feels like it has arrived with a bang in HPC, but there’s substantially more interest out there, and the clear implication is that even greater things are yet to come.


Earlier in the day, the Indiana University booth hosted a series of lightning talks, including from myself and several Scientific Working Group members. Bethwaite gave an excellent and informative talk on the recent studies he has done at Monash University on the overhead of virtualization.

OpenStack’s presence at the Supercomputing conference was wrapped up with a panel session featuring a great set of luminaries from the melting pot of OpenStack and research computing:

I took the chair as moderator. With such a great panel it’s no surprise that many interesting points came up from the discussion, including OpenStack’s true capabilities for HPC, and the level of investment in (but the ultimate value of) self-supported OpenStack. There were also some insightful comments on a wish list for future OpenStack development.

Finally, we ended the conference with a working group dinner, confirming the trend that when it comes to socials, the Scientific Working Group is the team to beat!

The OpenStack Foundation also made a splash last week with the launch of the Scientific OpenStack landing page, which highlights the contributions of the Scientific Working Group (including a free digital download of the OpenStack/HPC book). The Scientific OpenStack landing page is also being promoted as the masthead banner graphic for the OpenStack home page. What an accolade!

This post first appeared on the StackHPC blog. Superuser is always interested in community content, email:

The post OpenStack gains momentum at Supercomputing 2016 appeared first on OpenStack Superuser.

by Stig Telfer at November 22, 2016 02:00 PM

Mark McLoughlin

Sustainable Investment in Open Source

This is the prose version of a talk I gave today at OpenStack Day, France. The slides are available here.

Today, I want to speak about a somewhat subtle, meta-topic. I hope to share some insight on the question of how a company should think about its investment in any given open-source project. The reason I think the subject is important right now is that, as OpenStack follows its inevitable path through the “hype curve”, we are seeing some major players in the OpenStack ecosystem have recently re-shaped their investment in the project. I think there is plenty of opportunity for companies to take a much more realistic view on this subject, and my hope is that this will lead to a much more sustainable level of investment in the project.

My Story

My views on this question are so heavily influenced by an early personal experience, that I’m tempted to talk at length about my own story. But time is short. Very quickly, though, as I was finishing university, I realized I wanted to focus my career on open-source, and I was faced with the obvious question of who was going to pay me to work on open-source? What value would be open-source work bring to my employer? I spent a lot of time thinking about this, even after I found myself employed to work full time on open-source.

My Employer

An interesting thing happened. Because I had spent a good deal of time thinking about the value of my own open-source work, I was well-placed to help my employer think about how, where, and why to invest in open-source projects. It quickly became apparent to me then - and is still apparent to me now - that this fundamental question is fraught with difficulty, and most people struggle greatly to answer it.

Investment vs Business Needs

Over the years, I’ve boiled my answer to this simple-sounding question to something equally simple - your investment in a project should be directly related to what the business needs from the project. I think it’s important to frame it this way, because if the business doesn’t have a good have a good understanding in the value of an investment, the business is going to be quick to discontinue that investment when it is looking at how to make the best use of its resources.


What do I mean about an investment that isn’t directly related to the needs of the business? Let me dig into that a little bit with some anti-patterns.

Focus on Features

First, features. Or, as Thierry Carrez puts it, “tactical contributions”. Companies often look at the “what do we need from this project” question through the lens of feature analysis - what are my requirements? What is missing?

The good thing about contributing features is that it is directly related to business needs. The thing about your primary focus being on new feature development is it misses the bigger picture. A community of developers that is focused only on their own new features is not going to get anywhere. Who is thinking about the long-term future of the project? Who is looking at feedback from users? Who is merging the code for these new features?

All of these types of activities are the basic necessities of any software project. They are the basis through which new features get added. You must invest in ensuring that this is happening in this project whose future you care about.

The Donation

This anti-pattern is about our choice of language, and how it affects our thinking. People often talk about “donating” code to a project. Calling it a donation suggests you have a feeling that the return on your investment is quite intangible. How long will you continue with these “donations”?

For Recognition

Related, it’s often quite clear that a major motivation for companies investing in open-source is the recognition they receive in doing so. Certainly, in OpenStack, with Stackalytics we have spawned an impressively pointless competition between companies for ranking in contributor statistics. If you give your employees the goal of achieving a particular ranking in the contributor statistics, you may achieve this, but what does that really achieve?

The type of business value that it delivers is essentially short-term marketing value. We don’t want investment in open-source projects to be made out marketing budgets, since that’s possibly the least reliable source of funding!

"100% Upstream"

Not so long ago, when companies were falling over themselves to demonstrate their commitment to OpenStack, we saw an interesting phenomenon - the “100% dedicated upstream resource”. These were individuals and teams at various companies who were employed to contribute to the project but - as far as I can tell - were self-directed and deliberately kept isolated from any “downstream” work.

This is an incredibly alluring opportunity for those involved! The company is saying that the project is critically important to the business and whatever you or your team does to help the project be successful, that’s valuable to the company! Unfortunately, we can see how these things go in cycles and, at some point, the company will take a harder look at what this person or team is doing. When that happens, the likelihood is that the company has very little visibility into - or understanding of - the work being done and, worse, the person or team has no experience articulating how the value of the work is meaningful to the business!

There are some exceptions to this, even within Red Hat. But as a systematic way of investing in a project … it’s unlikely to be sustainable.

Non-Profit Staff

Finally, the culmination of a number of these anti-patterns - the idea that companies should "donate" to a non-profit organization like the OpenStack Foundation, have that organization use the donations to employ staff who are "100% dedicated upstream resources", and the value to companies ...? The recognition it receives for being such generous benefactors?

Having some technical staff employed by the non-profit is fine. Some of my favorite people work for the OpenStack Foundation! My preference would be for Foundation staff to be in facilitation roles. But certainly, you would do well to avoid this pattern for the majority of the contributors on a project.

One example of this is the Linux Foundation's Core Infrastructure Initiative. In the wake of the OpenSSL Heartbleed vulnerability, the Linux Foundation brought together funding from a number of companies to invest resources for key projects like OpenSSL. It's fascinating because here is a project like OpenSSL that many, many businesses depended on, but few invested in. What the Linux Foundation has done is positive, but I can't help feel we have failed as a community if this is the only way to sustain these projects. You'll notice that Red Hat hasn't joined this initiative, despite us being supportive of it - we have always invested in these projects directly, and think that is actually a healthier model for the future.

Think Strategically

Ok, so that's a bunch of anti-patterns ... what you generally shouldn't do. What should you do? Well, you need to think strategically. You are choosing to have your business depend on something you don't control, but you should not allow yourself to feel powerless about how successful that choice will be.

The Future

Think about the future, the long-term. Given the business choice you are making, for how long will your business success depend on the success of the project? Indefinitely? What are you doing to ensure that success?

One way to think about that would be a worst-case scenario. The unimaginable happens, and the rest of our community disappears. You are left almost completely alone maintaining the project. What you focus on? Obviously, you wouldn't choose to depend on a project where there is a possibility of that happening, but the thought exercise does help you think about your priorities.


If you are determined to ensure the success of the project, you'll have your own view of what that success looks like. Are you happy to hope the community will find a direction that fits your needs, without any input from community leaders that understand the needs of your business?

At Red Hat, we talk about wearing "two hats". Given a particular problem, a particular question of direction, you should have the ability to understand what's good for the project overall and what's good for Red Hat. And crucially, you should have a way to reconcile those two views. This is not a zero sum game. Almost always, you can find a solution that is good for both the project and Red Hat. Why would Red Hat want a solution that is harmful to the project?


In order to be successful with any technology, you need to have access to expertise that understands the technology. There is no better expertise than the authors or maintainers of the project. These are the people who understand not just how the technology works now, but how it evolved there, and how things are likely to change in the future. They understand the pitfalls, the know the cool new tricks.

You can have access to that expertise by having those experts join your team. They stop being experts pretty quickly if they stop having time to work on the project, so their upstream responsibilities should continue to be a significant proportion of their time. But you'd be amazed at how their presence on the team can help the whole team be successful.

Red Hat's Model

As somewhat of an aside, since this presentation is not a sales pitch, think about Red Hat's business model, and our proposition to our customers. Certainly part of the model is that, through our product subscriptions, the customer is investing in the projects involved and, by proxy, is safeguarding the future of the project, gaining a level of influence in the project, and has access to expertise relating to the project.

A Measurable Goal

Recently, on Red Hat's OpenStack team, and thanks to the influence of Alexis Monville, we've been using the "Objectives and Key Results" framework for using well-defined, measurable goals to guide our teams. We started brainstorming on these OKRs almost a year ago, and from those early discussions, I wondered “how do we frame an measurable goal around our investment in upstream?”. What indicator could we use to know whether we were on the right track, and to ensure we’d stay on the right track?

Our Vision

Our first thoughts on this was to look at data like our position in the contributor statistics, or the number of core contributors, PTLs, or TC members on our team. None of this sat well with us, though, because we have seen that these type of goals don’t drive the right behaviors. When we started drafting a vision statement for each goal, I wrote:

“Teams of engineers with the required motivation, an understanding of Red Hat's vision, and empowered with the necessary time, encouragement, and recognition, working upstream to drive and influence many diverse parts of OpenStack.”

And there it sat for a while, us all lacking ideas on how to measure this. Quite recently, Russell Bryant and Doug Hellmann hit on a promising solution. Survey the team with a small set of questions and use that sentiment of our measure of success with this goal.

The Questions

The questions that Russell and Doug developed are:

  1. Does your team have effective input into the features accepted and design decisions made upstream?
  2. Does your team have effective input into bug fixes and backports made upstream?
  3. Does your team have effective input into discussions related to processes, schedules, requirements, infrastructure management, and other decisions made by the upstream OpenStack community in general?
  4. What level of investment does your team make in upstream work?

The allowed answers are “not enough”, “just right”, “too much”, and “don’t know”. We also had a free-form comments section which is helping us gain insight into the results.

Notice one important aspect of this - we are asking individuals about the effectiveness of the investment their team is making upstream. That reflects the vision above.

The Results

We only recently started running this survey, and we will do one every release cycle. So far, we’ve had over 70 responses, so that’s a pretty good start.

Level of Invesment

The really interesting judgment call we need to make is what percentage of “just right” answers do we want to aim for? It would seem misguided to aim for 100% - do we really think that it’s important that every individual feels we’re striking the right balance? We’ve arbitrarily chosen 80% as our target, which means we feel pretty good about where we’re at, but there’s still opportunities for improvement.


One thing I’m loving about my job these days is the direct exposure I have to Red Hat customers who are choosing to invest in OpenStack. Deciding to build a significant cloud for production use is a huge decision, and no-one takes it lightly. It’s exciting, and companies doing this can look forward to a true transformation in how they think about IT resources. The truly daunting part is the sense of responsibility that comes with it.

These are long-term choices being made, and they’re being made because people believe in the future of this project. Red Hat takes the responsibility seriously by making choices that we hope will ensure the project has a long, sustainable future.

by markmc at November 22, 2016 11:00 AM

November 21, 2016

OpenStack Blog

OpenStack Developer Mailing List Digest November 5-18

SuccessBot Says

  • mriedem: We’re now running neutron by default in Ocata CI jobs [1].
  • stevemar: fernet token format is now the default format in keystone! thanks lbragstad samueldmq and dolphm for making this happen!
  • Ajaegar: is now hosted by OpenStack infra.
  • Tonyb: OpenStack requirements on pypi [2] is now a thing!
  • All

Registration Open For the Project Teams Gathering

  • The first OpenStack Project Teams Gathering event geared toward existing upstream team members, providing a venue for those project teams to meet, discuss and organize the development work for the Pike release.
  • Where: Atlanta, GA
  • When: The week of February 20, 2017
  • Register and get more info [3]
  • Read the FAQ for any questions. If you still have questions, contact Thierry (ttx) over IRC on free node, or email foundation staff at
  • Full thread

Follow up on Barcelona Review Cadence Discussions

  • Summary of concerns were Nova is a complex beast. Very few people know even most of it well.
  • There are areas in Nova where mistakes are costly and hard to rectify later.
  • Large amount of code does not merge quickly.
  • Barrier of entry for Nova core is very high.
  • Subsystem maintainer model has been pitched [4].
  • Some believe this is still worth giving a try again in attempt to merge good code quickly.
  • Nova today uses a list of experts [5] to sign off on various changes today.
  • Nova PTL Matt Riedemann’s take:
    • Dislikes the constant comparison of Nova and the Linux kernel. Lets instead say all of OpenStack is the Linux Kernel, and the subsystems are Nova, Cinder, Glance, etc.
    • The bar for Nova core isn’t as high as some people make it out to be:
      • Involvement
      • Maintenance
      • Willingness to own and fix problems.
      • Helpful code reviews.
    • Good code is subjective. A worthwhile and useful change might actually break some other part of the system.
  • Nova core Jay Pipes is supportive of the proposal of subsystems, but with a commitment to gathering data about total review load, merge velocity, and some kind of metric to assess code quality impact.
  • Full thread

Embracing New Languages in OpenStack

  • Technical Committee member Flavio Percoco proposes a list of what the community should know/do before accepting a new language:
    • Define a way to share code/libraries for projects using the language
      • A very important piece is feature parity on the operator.
      • Oslo.config for example, our config files shouldn’t change because of a different implementation language.
      • Keystone auth to drive more service-service interactions through the catalog to reduce the number of things an operator needs to configure directly.
      • oslo.log so the logging is routed to the same places and same format as other things.
      • oslo.messaging and oslo.db as well
    • Work on a basic set of libraries for OpenStack base services
    • Define how the deliverables are distributed
    • Define how stable maintenance will work
    • Setup the CI pipelines for the new language
      • Requirements management and caching/mirroring for the gate.
    • Longer version of this [6].
  • Previous notes when the Golang discussion was started to work out questions [7].
  • TC member Thierry Carrez says the most important in introducing the Go should not another way for some of our community to be different, but another way for our community to be one.
  • TC member Flavio Percoco sees part of the community wide concerns that were raised originated from the lack of an actual process of this evaluation to be done and the lack of up front work, which is something trying to be addressed in this thread.
  • TC member Doug Hellmann request has been to demonstrate not just that Swift needs Go, but that Swift is willing to help the rest of the community in the adoption.
    • Signs of that is happening, for example discussion about how oslo.config can be used in the current version of Swift.
  • Flavio has started a patch that documents his post and the feedback from the thread [8]
  • Full thread

API Working Group News

  • Guidelines that have been recently merged:
    • Clarify why CRUD is not a great descriptor [9]
    • Add guidelines for complex queries [10]
    • Specify time intervals based filtering queries [11]
  • Guidelines currently under review:
    • Define pagination guidelines [12]
    • WIP add API capabilities discovery guideline [13]
    • Add the operator for “not in” to the filter guideline [14]
  • Full thread

OakTree – A Friendly End-user Oriented API Layer

  • The OpenStack summit results of the Interop Challenge shown on stage was awesome. 17 different people from 17 different clouds ran the same workload!
  • One of the reasons it worked is because they all used the Ansible modules we wrote based on the Shade library.
    • Shade contains business logic needed to hide vendor difference in clouds.
    • This means that there is a fantastic OpenStack interoperability story – but only if you program in Python.
  • OakTree is a gRPC-based APO service for OpenStack that is based on the Shade library.
  • Basing OakTree on Shade gets not only the business logic, Shade understands:
    • Multi-cloud world
    • Caching
    • Batching
    • Thundering herd protection sorted to handle very high loads efficiently.
  • The barrier to deployers adding it to their clouds needs to be as low as humanly possible.
  • Exists in two repositories:
    • openstack/oaktree [15]
    • openstack/oaktreemodel [16]
  • OakTree model contains the Protobuf definitions and build scripts to produce Python, C++ and Go code from them.
  • OakTree itself depends on python OakTree model and Shade.
    • It can currently list and search for flavors, images, and floating ips.
    • A few major things that need good community design listed in the todo.rst [17]
  • Full thread


by Mike Perez at November 21, 2016 08:10 PM


Best practices for running RabbitMQ in OpenStack

The post Best practices for running RabbitMQ in OpenStack appeared first on Mirantis | The Pure Play OpenStack Company.

RabbitMQ in OpenStackOpenStack is dependent on message queues, so it’s crucial that you have the best possible setup. Most deployments include RabbitMQ, so let’s take a few minutes to look at best practices for making certain it runs as efficiently as possible.

Deploy RabbitMQ on dedicated nodes

With dedicated nodes, RabbitMQ is isolated from other CPU-hungry processes, and hence can sustain more stress.

This isolation option is available in Mirantis OpenStack starting from version 8.0. For more information, do a search for ‘Detach RabbitMQ’ on the validated plugins page.

Run RabbitMQ with HiPE

HiPE stands for High Performance Erlang. When HiPE is enabled, the Erlang application is pre-compiled into machine code before being executed. Our benchmark showed that this gives RabbitMQ a performance boost up to 30%. (If you’re into that sort of thing, you can find the benchmark details here and the results are here.)

The drawback with doing things this way is that application initial start time increases considerably while the Erlang application is compiled. With HiPE, the first RabbitMQ start takes around 2 minutes.

Another subtle drawback we have discovered is that if HiPE is enabled, debugging RabbitMQ might be hard as HiPE can spoil error tracebacks, rendering them unreadable.

HiPE is enabled in Mirantis OpenStack starting with version 9.0.

Do not use queue mirroring for RPC queues

Our research shows that enabling queue mirroring on a 3-node cluster makes message throughput drop twice. You can see this effect in publicly available data produced by Mirantis Scale team – test reports.

On the other side, RPC messages become obsolete pretty quickly (1 minute) and if messages are lost, it leads only to failure of current operations in progress, so overall RPC queues without mirroring seem to be a good tradeoff.

At Mirantis, you generally enable queue mirroring only for Ceilometer queues, where messages must be preserved. You can see how we define such a RabbitMQ policy here.

The option to turn off queue mirroring is available in MOS starting in Mirantis OpenStack 8.0 and is enabled by default for RPC queues starting in version 9.0.

Use a separate RabbitMQ cluster for Ceilometer

In general, Ceilometer doesn’t send many messages through RabbitMQ. But if Ceilometer gets stuck, its queues overflow. That leads to RabbitMQ crashing, which in turn causes outages for other OpenStack services.

The ability to use a separate RabbitMQ cluster for notifications is available starting with OpenStack Mitaka (MOS 9.0) and is not supported in MOS out of the box. The feature is not documented yet, but you can find the implementation here.

Reduce Ceilometer metrics volume

Another best practice when it comes to running RabbitMQ beneath OpenStack is to reduce the number of metrics sent and/or their frequency. Obviously that reduces stress put on RabbitMQ, Ceilometer and MongoDB, but it also reduces the chance of messages piling up in RabbitMQ if Ceilometer/MongoDB can’t cope with their volume. In turn, messages piling up in a queue reduce overall RabbitMQ performance.

You can also mitigate the effect of messages piling up by using RabbitMQ’s lazy queues feature (available starting with RabbitMQ 3.6.0), but as of this writing, MOS does not make use of lazy queues..

(Carefully) consider disabling queue mirroring for Ceilometer queues

In the Mirantis OpenStack architecture, queue mirroring is the only ‘persistence’ measure used. We do not use durable queues, so do not disable queue mirroring if losing Ceilometer notifications will hurt you. For example, if notification data is used for billing, you can’t afford to lose those notifications.

The ability to disable mirroring for Ceilometer queues is available in Mirantis OpenStack starting with version 8.0, but it is disabled by default.

So what do you think?  Did we leave out any of your favorite tips? Let us know in the comments!

The post Best practices for running RabbitMQ in OpenStack appeared first on Mirantis | The Pure Play OpenStack Company.

by Dmitry Mescheryakov at November 21, 2016 08:09 PM

Kenneth Hui

OpenStack Summit Day 1 Recap: OpenStack for Work That Matters


Like the city in which we are meeting, the Barcelona OpenStack Summit feels both old and new at the same time.

Barcelona is a new venue for the bi-annual Summit and as usual, there are an impressive number of first time attendees. But there is also a very familiar feel to the Summit this time — after all, this is the 14th such Summit, the fifth international Summit and the fourth outside North America. The Summits are now mainstream tech conferences — which makes sense, since OpenStack is a mainstream enterprise technology.

This was reflected in the themes of the first day’s keynotes.

Mark Collier, chief operating officer for the OpenStack Foundation, kicked the Summit off by reviewing findings from 451 Research’s latest study: that OpenStack has become a versatile and reliable platform used for “work that matters” across varying industries. Fully 80 percent of the companies running OpenStack are not technology companies, as demonstrated by morning keynotes highlighting use cases from banking, media, telecom and scientific research.

Each use case reinforced that OpenStack is being used to build software that solves big problems.

To read more about day 1 of the Barcelona OpenStack Summit, please click here to go to my article on the Rackspace blog site.

Filed under: Cloud, Cloud Computing, OpenStack, Private Cloud Tagged: Cloud, Cloud computing, OpenStack, OpenStack Summit, Private Cloud, Rackspace

by kenhui at November 21, 2016 06:38 PM

OSIC - The OpenStack Innovation Center

PLUMgrid Proves SDN Scalability with OSIC

PLUMgrid Proves SDN Scalability with OSIC


By Travis Broughton


One of the more gratifying aspects of working with OSIC is seeing firsthand how the cluster's capacity is being used to advance the scalability of OpenStack.  A recent example from PLUMgrid demonstrates that the concept of scalability means more than just the number of nodes.  Other dimensions of scalability include multitenancy, security policies, and addressability, and those were all aspects that PLUMgrid chose to cover with their time in the cluster.  


Over two 3-week sessions, PLUMgrid performed testing on Liberty- and Mitaka-based distributions using two different versions of their Open Networking Suite and CloudApex SDN monitoring software.  Their testing spanned basic functionality and performance benchmarking, along with more extensive tests to validate scalability in terms of number of projects, number of Security Groups, and using NAT and Floating IP instances concurrently running within the system.  


The PLUMgrid testing uncovered some new issues within the OpenStack environment, some of which were fixed with configuration and tuning changes, and some requiring upstream patches.  These changes were documented and contributed back to the community, and testing on both Liberty and Mitaka demonstrated improvements in many areas.


One of the OSIC tenets is upstream sharing of all of the work done in the cluster.  I was pleased to be able to participate in some of this sharing at the recent OpenStack Summit in Barcelona, where PLUMgrid led a session detailing their test results.  A companion scale-out report is also available.


On December 7, 2016,  we will give a webinar with a more in-depth look at the OSIC cluster, along with PLUMgrid's case study of an SDN overlay network connecting 131 nodes running over 1000 VMs.  Join this session to find out more about OSIC and the SDN scale-out test configuration, scenarios, and results.

by OSIC Team at November 21, 2016 04:35 PM

IBM OpenTech Team

OpenStack in containers

If you have installed OpenStack manually, you would have gone through a long sequence of installing and configuring various services and drivers. In every case, these services and drivers run as processes on hosts designated as controllers, compute nodes, etc. When containers became a topic of interest to the community about two years ago, it quickly became apparent that these processes can easily be wrapped in containers. There are several benefits for containerizing the OpenStack services:

1. Filesystem isolation: required packages can be installed freely without modifying the base host and potentially impacting other services. This provides a clean way to add and remove services.

2. Simplified management: common way to start, suspend, stop containers.

3. Easier upgrading and packaging.

Fast forward to the present, there are now several projects that deploy OpenStack cloud where all its services run in containers, and development is continuing to push OpenStack further onto the container platform. In this blog, we will take a quick look at some key aspects of two projects: OpenStack Ansible (OSA) and Kolla. I will take the perspective of a user installing OpenStack, so this is not intended to be an exhaustive comparison of the two projects, although there may be useful information if you are trying to choose between the two. Please refer to the documents of these projects for the actual instructions on using OSA or Kolla.

First, what are the common themes between OSA and Kolla? All OpenStack services and drivers run in containers. But the containers themselves still need to be deployed onto the hosts by some mechanism. Both OSA and Kolla use Ansible to deploy and manage the containers. To give users a quick way to evaluate the tool, both OSA and Kolla provide an all-in-one installation: the full OpenStack is installed onto one server, similar to DevStack. Both assume the hosts to be prepared with certain prerequisites for the containers to work. The prerequisites are documented and there are some utilities to help prepare the hosts, but I found it does take some experimentation to sort out the details and get the environment to the correct starting state. For instance, OSA assumes that a set of network bridges with specific names exists for the containers to connect to.


What are the differences? The key difference is in the container technology being used, and this in turn leads to other differences. OSA deploys LXC containers, while Kolla uses Docker containers. Compared to LXC, Docker provides a rich set of utilities to manage its containers, a reason behind its popularity. Kolla leverages these advantages by packaging particular versions of the services into Docker images. Official release versions are available on Docker Hub, or you can build your own.


OSA, on the other hand, creates the containers but treats them more or less as a VM: a private repo is created to serve the installation packages of a particular version of OpenStack, then the LXC containers are created and the software are installed onto the containers.

Since Kolla images are fully pre-built, they are faster to deploy. For an environment of around 100 nodes, Kolla has been reported to install OpenStack in less than 30 minutes, while OSA would take about a day since it would go through the actual installation of the software.

The layering in Docker images also allows for Kolla to support more distros than OSA: Kolla can install onto Ubuntu, CentOS, RHEL and OracleLinux while OSA only supports Ubuntu.

OSA, on the other hand, does derive some benefits for being fully based on Ansible. If the dev-op team is fluent in Ansible, then OSA can be very appealing. When a user needs to troubleshoot a problem, understand the steps being done, or make some customization, the playbooks are readily available on the system. In this respect, OSA does offer more flexibility in configuring and managing the cloud.

What’s next for OpenStack in containers? Currently Ansible is used to deploy and manage the containers. However, container platform such as Kubernetes offers a rich set of functionality to manage containers, so the natural question is whether it makes sense to let Kubernetes manages the OpenStack containers themselves. Although this may seem convoluted, there are benefits to this approach. Kubernetes can ensure high availability of the containers, and can provide rolling upgrade of OpenStack services. However, one gap to be filled is that Kubernetes’ descriptor for container is not quite an orchestration language since it does not describe dependencies to express a sequence of actions. This functionality will need to be provided elsewhere.

Kolla is embarking on this development in the Ocata release: running OpenStack within Kubernetes. Check back later for an update when this new tool is available.

The post OpenStack in containers appeared first on IBM OpenTech.

by KQ7R_Ton_Ngo at November 21, 2016 04:12 PM

OpenStack Superuser

The Interoperability Challenge demonstrates app portability

There has been significant progress in the quest for application portability across a diversity of OpenStack public and private clouds. This progress was demonstrated on day two of the OpenStack Summit in Barcelona, when 16 global IT leaders in the OpenStack ecosystem participated in the live OpenStack Interoperability Challenge.

The challenge took a holistic approach to prove interoperability by asking all participants to run the same workload and automated deployment tools across their OpenStack distributions and public clouds. Caleb Boylan and Stefano Maffulli describe what the challenge entailed in more detail and describe their experiences being on stage as participants of the event.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="" width="640"></iframe>

Check out the Interop Challenge that wowed the crowd.

The post The Interoperability Challenge demonstrates app portability appeared first on OpenStack Superuser.

by Superuser at November 21, 2016 02:00 PM


Installing Monasca – a happy ending after much sadness and woe

In one of our projects we are making contributions to an Openstack project called Watcher, this project focuses on optimizing resource utilization of a cloud according to a given strategy. As part of this work it is important to understand the resource utilization of the cloud beforehand in order to make a meaningful contribution. This requires collection of metrics from the system and processing them to understand how the system is performing. The Ceilometer project was our default choice for collecting metrics in an Openstack deployment but as work has evolved we are also exploring alternatives – specifically Monasca. In this blog post I will cover my personal experience installing Monasca (which was more challenging than expected) and how we hacked the monasca/demo docker image to connect it to our Openstack deployment.

In our earlier series of blog posts we explained how to install, configure and test Monasca step-by-step using git, Maven and pip – a good option for learning the moving parts of the giant beast that is Monasca and how it is all linked together. This content was posted over a year ago and hence it is now a little bit stale; since that time work has been done on deployment using automation tools such as  Ansible or Puppet. Generally, when working on this, we encountered significant issues with lack of documentation, unspecified versioning issues, unclear dependencies etc.

Our first attempt to install Monasca was based on different Ansible repositories available on Github – pointed to by the official page of Monasca. Although most of the repositories are Ansible roles, some try to put everything in a single place. Interestingly, many of those assume that Openstack is deployed with Devstack (which was not true in our case – we had a small  Packstack deployment) and this caused many steps to fail during the installation as files were not where they were expected to be (in devstack many files are into /opt/stack/). Even after solving these issues Monasca itself did not seem to work and we always ended up with a system where only few monasca services were up and running. We’re not certain where the problems were, but there were multiple issues with Kafka and in some cases it was not possible to get monasca-agent or monasca-notifications working.

A tip if you want to go this path: Most of the repos use Ansible 1.9.2, which is not obvious. We had many issues using Ansible >= 2.0 and some roles using version 1.9.2 which required modification to the source code as there were changes in how loops are defined.

After the frustrating experience with Ansible we then decided to get back to basics, there is a functional out-of-the-box docker image in Docker Hub used for testing Monasca. It contains pretty much everything Monasca needs for running – even a local Keystone, Nova and Glance. We then decided to modify Monasca components in this image in order to connect it to our to Openstack cluster. Note that only Openstack credentials were modified inside the container – InfluxDB, MySQL and other services credentials also provided in the container were not modified. After relatively little tinkering, we managed to get this working as described below.

First of all, pull the image into your docker machine – let’s call it docker-host. Note that docker-host must be able to access your Openstack deployment (via the public APIs) and vice-versa

docker pull monasca/demo

Next, create the docker container but it is necessary to modify its entrypoint, so it does not start setting up monasca automatically and we can change the Monasca configuration files; also, note that the since it will not be running Keystone, we will remove the Keystone port:

docker run -it -p {port_to_horizon}:80 -p 8080:8080 --name monasca --entrypoint=/bin/bash monasca/demo

Change the value of port_to_horizon in case port 80 is already used in docker-host.

Inside the docker container you will be in the ‘/setup’ directory, in this directory you will see the  ansible files used to set up this container. The main file in this directory is called, this file will start all the services required by Monasca but before running this script it is necessary to modify the following Monasca Openstack configuration details.

In /etc/monasca/api-config.yml add your credentials in the middleware section (note that you will need to install a text editor in the container):

##  /etc/monasca/api-config.yml
# region usually defaults to RegionOne
region: “{SERVICE_REGION}”
defaultAuthorizedRoles: [user, domainuser, domainadmin, monasca-user, admin]
agentAuthorizedRoles: [monasca-agent]
adminAuthMethod: password
adminUser: "monasca"
adminPassword: "MONASCA_PASSWORD"
adminProjectName: "monasca"

Similarly in /usr/local/bin/monasca-reconfigure and /setup/alarms.yml:

##  /usr/local/bin/monasca-reconfigure
'/opt/monasca/bin/monasca-setup' \
     -u 'monasca-agent' \
      -s 'monitoring'  \
     --keystone_url 'KEYSTONE_URL' \
     --project_name 'MONASCA_AGENT_PROJECT' \
       --monasca_url 'http://localhost:8080/v2.0'  \
      --check_frequency '5'  \

## /setup/alarms.yml
- name: Setup default alarms
   hosts: localhost
       keystone_url: KEYSTONE_URL
       keystone_user: monasca
       keystone_password: MONASCA_PASSWORD
       keystone_project: monasca
       - {role: monasca-default-alarms, tags: [alarms]}

In /setup/ there is an error regarding the location of monasca-notification init script, modify the ‘/usr/local/bin/monasca-notification &’ (line 44) to ‘/opt/monasca/bin/monasca-notification &’. Also remove the playbook which installs the Keystone service (line 11):

## /setup/

# remove this line 
# ansible-playbook -i /setup/hosts /setup/keystone.yml -c local
/opt/monasca/bin/monasca-notification &

Finally remove services which are not necessary anymore, eg the local Keystone, from, leave apache2 and change the variable OPENSTACK_HOST in /etc/openstack-dashboard/ to the IP where Keystone is installed:

## /setup/
/etc/init.d/apache2 start

## /etc/openstack-dashboard/

Before running the main script to set everything up make sure you have created the Keystone users, projects and roles which are assumed in the way monasca is configured:

 keystone tenant-create --name monasca --description "Monasca tenant"
 keystone user-create --name monasca-agent --pass password --tenant [monasca-tenant-id]
 keystone user-create --name monasca --pass password --tenant [monasca-tenant-id]
 keystone role-create --name monasca-agent
 keystone role-create --name monasca-user
 keystone user-role-add --user [monasca-agent-id] --role [monasca-agent-role-id] --tenant [monasca-tenant-id]
 keystone user-role-add --user [monasca-id] --role [monasca-user-role-id] --tenant [monasca-tenant-id]
 keystone service-create --type monitoring --name monasca --description "Monasca monitoring service"
 keystone endpoint-create --service [service-id] --publicurl http://docker-host:8080/v2.0 --internalurl http://docker-host:8080/v2.0 --adminurl http://docker-host:8080/v2.0

Once all of this is done, you can now run /setup/ to set everything up – it may take a couple of minutes. When it has finished, you can visit your monasca-dashboard page at http://docker-host:{port_to_horizon}. Log in as the user Monasca and visit the monitoring tab in the dashboard.

In this tab you will be able to create alarms definitions and notifications. You will also be able to see an overview of the current state of the system in the Grafana dashboard. To collect more data regarding your Openstack cluster you will need to install and configure monasca-agent service in each of the controller/compute nodes. It is possible to do it in a virtualenv so it does not modify any library in the system.

# on compute/controller nodes
virtualenv monasca_agent_env
source monasca_agent_env/bin/activate
pip install --upgrade monasca-agent

monasca-setup -u monasca-agent -p password --project_name monasca -s monitoring --keystone_url {KEYSTONE_URL} --monasca_url http://docker-host:8080/v2.0 --config_dir /etc/monasca/agent --log_dir /var/log/monasca/agent --overwrite

Then you’ve got a fully functional Monasca system which works with your Openstack cluster – it’s a good solution for experimenting with Monasca to understand how powerful it really is. This solution is probably not the best solution for a production deployment, however – it would probably make more sense to decouple the services and put the data in a persistent data store, but that’s another day’s work!


If you are getting “requests.exceptions.ConnectionError: (‘Connection aborted.’, gaierror(-2, ‘Name or service not known’))” error when running script that means the container is not able reach the endpoint defined in Openstack for the monitoring service. Make sure that the container is able to connect to the external interface in your Openstack cluster and the Keystone endpoint for the monitoring service is configured correctly.

The is the main script in this container, in case you find any issues you can execute each of the commands in this script sequentially to understand more clearly what is happenning.  

“Exception in thread “main” java.lang.RuntimeException: Topology with name `thresh-cluster` already exists on cluster”. This error is known to happen often as the thresh-cluster service tries to configure itself for a second time, you can ignore it.

by Bruno Grazioli at November 21, 2016 01:12 PM

Carlos Camacho

Enabling nested KVM support for a instack-virt-setup deployment.

The following bash snippet will enable nested KVM support in the host when deploying TripleO using instack-virt-setup.

This will work in AMD or Intel architectures.

echo "Checking if nested KVM is enabled in the host."
ARCH=$(lscpu | grep Architecture | head -1 | awk '{print $2}')
if [[ $ARCH == 'x86_64' ]]; then
if [[ -f $KVM_STATUS_FILE ]]; then
    if [[ "${KVM_CURRENT_STATUS^^}" -ne "${ENABLE_NESTED_KVM^^}" ]]; then
        echo "This host does not have nested KVM enabled, enabling."
        sudo rmmod kvm-$ARCH_BRAND
        sudo sh -c "echo 'options kvm-$ARCH_BRAND nested=$ENABLE_NESTED_KVM' >> /etc/modprobe.d/dist.conf"
        sudo modprobe kvm-$ARCH_BRAND
        echo "Nested KVM support is already enabled."
    echo "$KVM_STATUS_FILE does not exist."

By default nested virtualization with KVM is disabled in the host, so in order to run the overcloud-pingtest correctly we have two options. Either run the previous snippet on the host, or, when deploying the Compute node in a virtual machine add --libvirt-type qemu to the deployment command. Otherwise launching instances on the deployed overcloud will fail.

Here you have an example of the deployment command, fixing libvirt to qemu.

openstack overcloud deploy \
--libvirt-type qemu \
--ntp-server \
--templates /home/stack/tripleo-heat-templates \
-e /home/stack/tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
-e /home/stack/tripleo-heat-templates/environments/puppet-pacemaker.yaml

Have a happy TripleO deployment!

by Carlos Camacho at November 21, 2016 12:30 PM

Hugh Blemings



Welcome to Last week on OpenStack Dev (“Lwood”) for the week just past. For more background on Lwood, please refer here.

Basic Stats for the week 14 to 20 November for openstack-dev:

  • ~436 Messages (up about 14% relative to last week)
  • ~164 Unique threads (up about two percent relative to last week)

Traffic levels slowly coming back up post-Barcelona.  I suspect the Thanksgiving Holidays in the US may be slowing things a little too with folk in that neck of the woods traveling towards the end of the week.

Notable Discussions – openstack-dev

OpenStack Summit Barcelona summary of summaries

A handful more of summaries from the summit in Barcelona – the complete list aggregated from the last few Lwoods is here.

Oaktree – A friendly end-user oriented API layer

Monty Taylor shared some information about Oaktree – a work in progress that will provide a consistent API that works across clouds from multiple vendors.  As Monty puts it “oaktree is a gRPC-based API porcelain service for OpenStack that is based on the shade library and I’d love some help in writing it.”

Early days but looks like a pretty cool endeavour and has been well received by the openstack-dev community thus far.  Check it out :)

Tricircle now officially an OpenStack “Big Tent” project

Joe Huang wrote to say that Tricircle, a project that aims to provide networking automation for Neutron in multi-region OpenStack clouds, has officially been accepted as a big-tent project.  Congrats to all involved!

New Release Announcement list up and running

Thierry Carrez posted to advise the new mailing list for automatically generated (and hence sometimes quite noisy) release announcements.  More details in Thierry’s email or you can subscribe here.

Acknowledging Community goals for Ocata

Doug Hellmann reminded the community that quite a few teams are yet to acknowledge the community communication goals for Ocata.  Anne Gentle followed up to say she and Doug had done a short video that explains this new process.

Notable Discussions – other OpenStack lists

Over on OpenStack Operators, Blair Bethwaite pointed out a live blog he was curating that details OpenStack related goings on at SuperComputer16. Melvin Hillsman posted a followup about the poll to decide the location for the Ops Midcycle on March 15 & 16 next year – off to Milano it is! :)

People and Projects

Core nominations & changes


Further reading

Don’t forget these excellent sources of OpenStack news – most recent ones linked in each case


No tunes, but did enjoy playing a little virtual Hammond B3 during typing breaks this evening (I didn’t record this evenings noodling, but if you’re curious I’m playing the same virtual instrument here, albeit turned up to eleven…)  Aforementioned CD didn’t arrive – there are downsides to being in the antipodes sometimes :/

Last but by no means least, thanks, as always, to Rackspace :)


by hugh at November 21, 2016 10:48 AM

OpenStack Summit Barcelona 2016 Summary of Summaries


The last few editions of Lwood have had a list of links to the mailing list posts where the writers had provided a summary of particular Design Summit Project sessions.

Like the one from earlier in the year for Austin, the list below is the combination of these in one (hopefully!) easily searchable list :)

Project Summaries

Summaries posted to the OpenStack-Dev Mailing list

Apologies for any I’ve missed, corrections welcome :)



by hugh at November 21, 2016 09:28 AM

Mark McLoughlin

November 17th OpenStack Foundation Board Meeting

I had previously posted summaries of most board meetings, but had stopped doing this when I was appointed as Red Hat's representative on the board. Lately, I've sensed folks at Red Hat becoming more interested in the workings of the Foundation, so I figured it might be useful to start doing this again.

The OpenStack Foundation Board of Directors met for a two hour conference call last week. The usual disclaimer applies - this my informal recollection of the meeting. It’s not an official record.

Gold Member Applications

The first half of the meeting was given over to an Executive Session, after which the board voted to approve China Telecom, Inspur, and ZTE as Gold Members of the Foundation.

This completes the handling of the 7 Gold Member applications that were presented to the board at the October 24 meeting in Barcelona. 99Cloud, China Mobile, City Networks, and Deutsche Telecom were approved at that meeting.

For the first time, the Foundation now has reached the maximum of 24 Gold Members. We will only be able to consider applications for new Gold Members if an existing member departs, or if we go through a difficult process to change the limit in our bylaws.

User Committee Proposal

Next up, Edgar Magana updated the board on some proposed changes to the User Committee that was first discussed at the October meeting.

Currently the bylaws describe the User Committee is an "advisory committee" of three members appointed by the Board and TC, which prepares regular reports for the Board and TC. The idea with this proposal is to make the User Committee a parallel entity to the TC, with its members chosen through an election of Active User Contributors (AUC).

The bylaws changes outline how the User Committee has at least five members, that elections of Active User Contributors (AUCs) are held every six months, an affiliation limit for members of the UC, how AUC status is determined, and more.

The hope is that feedback will be collected as comments in the proposal document and that the Board will vote on the proposal during our December 6 meeting. A vote of the Board is sufficient to enact the changes.

One point of discussion was whether bylaws changes are necessary at all. The hope when the bylaws were originally drafted was that the User Committee would have plenty of lattitude to evolve via changes to the UC charter. Edgar made the point that the key change is for the UC to become a parallel entity to the TC rather than an advisory committee to the Board and TC.

"Futures" Discussion

Next, Toby Ford gave a presentation on a topic he had proposed along with Imad Sousou, Allison Randal, and Mark Baker. Toby introduced the topic by saying that as any project evolves and matures, it is important to reflect on competitive threats and the need to evolve the project.

Toby talked about a wide set of competitive threats to OpenStack from the container ecosystem (including Kubernetes and Mesos), to other storage implementations, to various projects in the Telco world (OpenVIM, E2, 3GPPP), and the obvious challenge of AWS, Azure, and Google with AWS revenue over $10B.

Toby moved on to talk about "the good and the bad of the Big Tent", describing the need for a balance between diversity and innovation versus consolidation and refactoring. Toby made the case that we're seeing a lot of expansion in the Big Tent, but not consolidation and refactoring. He expressed particular concern about how and whether core projects will be able to evolve, especially if the Big Tent does not allow for competitors to existing projects.

Toby then put on his "AT&T hat" and talked about the challenges for OpenStack from an AT&T perspective - the need to get to a control plane that scales to 10k servers, the problem of managing over 100 different production sites, the challenge of more and more happening "at the edge" with 5G, innovation happening in the networking space and how it relates to OpenStack, and the unsolved problem of keeping a production environment current with latest OpenStack releases.

To wrap up his presentation, Toby listed a few "possible recommendations" the Board could make to the technical community - (1) allow core projects to have competition within the Big Tent, (2) a mechanism to incubate new approaches, and (3) a mechanism to reationalize, clean up, or refactor.

What followed was a pretty lively discussion that covered much ground, over and beyond the concerns raised by Toby. While the success of OpenStack in bringing together such a diverse set of interests was acknowledged, there was a definite frustration that some form of necessary change is failing to emerge naturally, and whether it falls on the Board to try to at least try to clearly articulate the problem at a strategic level.

Jonathan tried to focus the conversation by giving his perspective that the Big Tent concerns had tied down until "recent comments in the media", and that is probably a distraction from the concerns people are expressing about core projects like Nova and Neutron. He was at pains to say that this isn't about project teams doing bad work - we continue to make huge progress in each release, and valuable work is being done. However, we're definitely seeing frustration from some quarters that it is difficult to influence the community with their ideas.

Jonathan warned that talking too much in the abstract creates the risk that the discussion will go around in circles. Despite this, the discussion never did go into any particular detail on where we've witnessed the problems we were discussing. From my own perspective, it was clear that much of frustration was as a result of how the Ciao and Gluon projects have been received, but we're failing to learn anything significant from those experiences by not talking about them in detail.

By the end of the discussion, we had agree to collaborate on a concrete description of specific problem areas, the questions they raise, and some possible solutions. The hope is that we would complete this before our December meeting, and we may then plan a longer meeting (possibly face-to-face, possibly with the TC) to dig into this.

by markmc at November 21, 2016 09:00 AM

Upstream training, scientific research, and more OpenStack news

Are you interested in keeping track of what is happening in the open source cloud? is your source for news in OpenStack, the open source cloud infrastructure project.

OpenStack around the web

There is a lot of interesting stuff being written about OpenStack. Here's a sampling from some of our favorites:

by Jason Baker at November 21, 2016 06:00 AM

Carlos Camacho

Ocata OpenStack summit 2016 - Barcelona

A few weeks ago I had the opportunity to attend to the Barcelona OpenStack summit ‘Ocata design session’ and this post is related to collect some overall information about it. In order to achieve this, I’m crawling into my paper notes to highlight the aspects IMHO are relevant.

Sessions list by date.

Tuesday - Oct. 25th

  • RDO Booth: Carlos Camacho TripleO composable roles demo (12:15pm-12:55pm)
  • What the Heck is OoO: Owls All the Way Down (5:55pm – 6:35)

Wednesday - Oct. 26th

  • Anomaly Detection in Contrail Networking (1:15pm-1:29pm)
  • Freezer: Plugin Architecture and Deduplication (3:05pm-3:45pm)
  • TripleO: Containers - Current Status and Roadmap (3:55pm-4:35pm)
  • TripleO: Work Session - Growing the team (5:05pm-5:45pm)
  • TripleO: Work Session - CI - current status and roadmap (5:55pm-6:35pm)

Thursday - Oct. 27th

  • Zuul v3: OpenStack and Ansible Native CI/CD (11:00am-11:40am)
  • The Latest in the Container World and the Role of Container in OpenStack (11:50am-12:30pm)
  • TripleO: Upgrades - current status and roadmap (1:50pm-2:30pm)
  • Mistral: Mistral and StackStorm (3:30pm-4:10pm)
  • Nokia: TOSCA & Mistral: Orchestrating End-to-End Telco Grade NFV (5:30pm-6:10pm)

Friday - Oct. 28th

  • TripleO: Work Session - Composable Undercloud deployment with Heat (9:00am-9:20am)
  • TripleO: Work Session - GUI, CLI, Validations current status, roadmap, requirements (9:20am-9:40am)
  • TripleO: Work Session - Multiple topics - Blueprints, specs, tools and Ocata summary. (9:50am-10:30am)

Beyond the analysis of “What I did there in a week” I want to state a few relevenat facts for me.

Why is important to attend to a design session (My case: Upstream TripleO developer)?

I think when working remotely in OpenStack projects, for example, in such a complex project as TripleO is really hard to know what other people are doing. So forth, design sessions force engineers to realize about your peers work on different OpenStack projects or even in the same project ;)

This will give you some ideas for future features, new services to integrate, issues that you might have in the future among many others. Also for TripleO specific case if you are interested in working in a specific service, you can get into those services sessions to know more about them.

Where is the value for companies when sending engineers to design sessions?

There might be several answers for this question, but I believe the overall answer will be that sending engineers to the design sessions allow engineers to be aligned based on companies goals, mostly for cases when several companies are related to the same project. Also allow team members to know each others, maybe this can be a soft benefit, for me as important as being aligned for future features or architectural agreements.

Is it really mandatory to send people to design sessions?

I think this is not mandatory at all, but at the a relevant factor might be that all knowledge/value generate in those sessions can be delivered and processed by the rest of the team members.

Do attendees gain value when attending to these design sessions?

Off course they will gain lot of value:

  • You might have a wrong impression about people on IRC, meeting them in person can change this dramatically ‘or not’.
  • Know better and engage with your team members and other peers.
  • Better alignment with your project goals.
  • Discuss blueprints and have a better understanding about the features life-cycle and roadmap.
  • Know what other people are doing.
  • Improve you overall knowledge about other projects (This time is for doing that “Not in your free time if you like it like me”).

If we are in a design session, are we in “working mode” or just in “learning mode”?

Hardest question by far, I had the remorse of feeling that I was not working enough. There were a lot of distractions if you wanted to actually make some time for coding or reviewing submissions. But that’s the thing, I believe is good time to align and after the summit you will always have time for coding :)

What about some business alignment?

I believe this is also an important factor to know about how OpenStack evolves with the time, release announcements and how summits are actually evolving.

by Carlos Camacho at November 21, 2016 12:00 AM

November 18, 2016

Dougal Matthews

Mistral Workflow Engine

This year I have been getting to know Mistral, the OpenStack workflow service and recently joined the Mistral core team. This post is going to be a bit of an introduction to the project and share some of my experiences with it.

But, first, what is a workflow service anyway?

A workflow management system (WFMS) provides an infrastructure for the set-up, performance and monitoring of a defined sequence of tasks, arranged as a workflow.

That is the one-line description from Wikipedia. Put simply, a workflow is a set of connected tasks that will be executed. However, the real advantage with a service like Mistral is that it manages and stores the state of the workflow. This means that it is able to restart or retry from any point.

The project is developed in the OpenStack community, but can be used outside of OpenStack. We’ll come back to that later.

The Hello Workflow

Okay, so, let’s start with a simple workflow and see what it does and how it works. This should give you a taste of what the Mistral Workflow language looks like but this example is poor - I plan to demonstrate more interesting examples in future posts.

version: '2.0'

            action: std.echo output="Hello Workflow!”
                output: <% task(say_hello).result %>

This is almost the simplest workflow possible, but it reveals a number of features in the Workflow language. We’ll step through it.

First we need to specify the version we are working with, at the moment this needs to be version: ‘2.0’. It is the only supported version.

Next we have the name of our workflow, hello_world and this workflow is made up of a set of tasks, in this case we only have one task, called say_hello. The workflow calls the std.echo action and passes a string as input. Once the task is finished, it will publish the result of the task and the workflow ends.

If we save the above workflow in a file called hello_world.yaml we can then register it with the Mistral API.

$ mistral workflow-create hello_world.yaml;

Now that this workflow is stored in Mistral we can easily trigger it at any point. Workflows are started by creating executions.

$ mistral execution-create hello_world;

After this workflow has been started it can be monitored easily with the following commands. The hello_world workflow should finish almost instantly, but workflows can run for an extended period of time.

$ mistral execution-list;
$ mistral execution-get $UUID;
$ mistral execution-get-output $UUID;

What are actions?

Actions are really where all the power comes from, without them the workflow wouldn't do very much, workflows just link together and connect a series of actions.

It is often helpful to isolate and run Mistral actions directly as you design the workflow. With the following action call we can replicate the functionality of the workflow above (since it only called one action).

$ mistral run-action std.echo '{"output": "Hello Workflow!"}'
{"result": "Hello Workflow!"}

Mistral only provides a small set of system actions (Mistrals standard library), but they are all useful. Mistral also ships OpenStack actions for all the OpenStack services - these are extremely useful.

Most Mistral users will want to write custom actions eventually, the Mistral documentation has a brief primer for writing custom actions. I hope to write more about this later and also make some improvements to the official documentation.

Without OpenStack?

OpenStack is a great project, and it is huge. So unless you are using it already it is unlikely you will install OpenStack to use Mistral. Thankfully Mistral isn't tied to OpenStack at all, but it can integrate well. Essentially, if you want to use Mistral with OpenStack you would setup KeyStone authentication (Keystone is the OpenStack identity service). Otherwise you can use keycloak or run Mistral without authentication (but behind a proxy which should add authentication).

The documentation for this again is poor and something I want to write about and improve. So watch this space.

by Dougal Matthews at November 18, 2016 02:35 PM

OpenStack Superuser

How scores big with OpenStack Swift

Neymar da Silva Santos Jr. has been called the present and future of world soccer. Just 24, the Brazil native wears jersey number 11 for FC Barcelona as he racks up international awards for top scorer.

The infrastructure holding up his official website,, has to perform as flawlessly as the sure-footed forward, handling up to 6,000 simultaneous network connections and a vast media library serving over 70,000 photos.

Neymar’s website, on, runs on OpenStack Swift. The move to Swift marked a major transformation for, the largest media portal in Latin America and one of the top 10 most trafficked websites in Brazil., the front door to media colossus Grupo Globo, offers up a public instance serving around 300 million requests a day to worldwide audiences distributed over more than 70 projects in production.

<script async="async" charset="utf-8" src=""></script>

The journey began in 2013, with what dev ops analyst Caio Brentano dos Passos calls a “shift in mindset” that led the team to create an open source project for platform-as-a-service (PaaS) called Tsuru. The team designed the PaaS, whose name means “crane” in Japanese, to depend on object storage in its deploy pipeline and looked into Swift. They were sold on the highly available, distributed, consistent object store that would allow them to store lots of data efficiently, safely and cheaply—important factors as the 60-year old media conglomerate on-boards older archive materials to the web and launches internet-native projects.

As a next step, they built an OpenStack Swift cluster with a few servers and Tempauth authentication. “It worked perfectly,” Brentano says, adding that the results enticed a number of internal clients. To keep operations running smoothly with more volume, they soon adopted OpenStack Keystone for authentication. “Since then, our cluster more than doubled its storage capacity and throughput,” Brentano says.

Today, Swift acts as the wind beneath the wings for many major projects at Globo and its partners, including a pioneering broadcast effort serving and storing videos in 4k high resolution called GloboPlay. The platform also warehouses over 14 million objects for millions of users who create uniforms, flags and crests for their teams in fantasy soccer game CartolaFC. Tsuru uses Swift to store Docker images, allowing hundreds of deploys per day.

Brentano says cost is the first, most obvious benefit of adopting open-source software. “We’re gradually reducing the dependency of expensive enterprise storage solutions. In addition, projects that were using AWS S3 can easily migrate to Openstack Swift, with no loss of quality for the clients.” Another benefit was helping their homegrown PaaS become the main platform to develop and deploy new products.

There are still a few areas to smooth over, he concedes, like monitoring. The team currently monitors dozens of metrics, but needs to improve them to have a better understanding about how each project is performing and help clients optimize Swift use. They are also taking their first steps with billing Swift usage, so they’re studying up and testing OpenStack Ceilometer.

Still, their now-nimble adoption is a long way from where they started. “When we started the object storage project at, the concepts related to cloud were new to us,” says Brentano. “We also had that cultural challenge you face when you need to convince people to change the way they do things. But as we showed teams how Openstack Swift works they noticed all the benefits and started testing and adopting it.”

<script async="async" charset="utf-8" src=""></script>

The team got another big boost from the “active and helpful” members of the Swift community, which he calls the most valuable asset of OpenStack. “I am pretty sure that their help is much better than the major paid support services available in the market,” says Brentano.

John Dickinson, Swift’s project team lead (PTL) who has been involved in Swift development since 2009, says that the Globo team’s results underline the project’s versatility.

“One of the highlights of working in OpenStack is seeing development and deployment in every corner of the planet—from NTT in Asia, OVH in Europe, SoftLayer in North America and Globo in South America,” Dickinson says. “I love seeing the passion the Globo team has in providing world-class infrastructure and it’s a pleasure to work with them to make Swift the best object storage system in the world.”

Object storage is one game where everyone comes out a winner.

This article first appeared in the print edition of Superuser magazine, distributed at the Barcelona Summit. If you’d like to contribute to the next one, get in touch:

Cover Photo // CC BY NC

The post How scores big with OpenStack Swift appeared first on OpenStack Superuser.

by Nicole Martinelli at November 18, 2016 12:02 PM


Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


Last updated:
December 03, 2016 12:13 AM
All times are UTC.

Powered by: