March 17, 2018


Open Networking Summit: Aptira Solutionauts head to LA

Open Networking Summit: Solutionauts in Los Angeles

Open Networking Summit North America

The Open Networking Summit (ONS) is taking place at the Intercontinental Los Angeles Downtown from the 26th to 29th of March, and some of our Solutionauts will be there. This is the industry’s premier Open Networking event, gathering enterprises, service providers and cloud providers across the ecosystem to share learnings, highlight innovation and discuss the future of Open Source Networking, including SDN, NFV, orchestration and the automation of Cloud, network, & IoT services.

We’ve been quite busy in the open networking space lately, recently joining the Linux Foundation Networking Project as a founding gold sponsor, as well as working on some very exciting ONAP projects. We’ve also launched Australia’s first Open Networking User Group (AONUG!) with two meetups taking place in Sydney and Melbourne last month. This is where local open network enthusiasts can meet and learn more about Open Networking, covering topics such as ONAP, OPNFV, OpenDaylight and lots more. This meetup is currently looking for speakers, so if you’re working on something open in networking we’d love to hear from you. We’ll also be adding an Open Networking theme to the upcoming OpenStack Day which will be held in Sydney in July. Stay tuned for more details regarding speaking opportunities, sponsorship and to reserve your place.

If you’re attending ONS, please let us know. Our Solutionauts would love to catch up with you to learn more about any Open Networking projects you are currently working on, or to discuss anything and everything open in networking. You can also also catch them at some of the event activities, such as the 5k fun run or the Women in networking lunch. Please note our Solutionauts love to run, so if you’d prefer you can meet us at the finish for a much needed beverage.

You can register for the Open Networking Summit here. Hope to see you there!

The post Open Networking Summit: Aptira Solutionauts head to LA appeared first on Aptira.

by Jessica Field at March 17, 2018 07:39 AM

March 16, 2018

Doug Hellmann

openstack-summit-counter 0.2.0

openstack-summit-counter is a plugin for python-openstackclient, the command line tool for interacting with OpenStack. This plugin helps you answer the summit registration question about how many summits you have attended in the past. What’s new in 0.2.0? Add support for counting PTGs (contributed by Colleen Murphy)

by doug at March 16, 2018 10:05 PM

openstack-summit-counter 0.1.0

openstack-summit-counter is a plugin for python-openstackclient, the command line tool for interacting with OpenStack. This plugin helps you answer the summit registration question about how many summits you have attended in the past. This is the first public release.

by doug at March 16, 2018 09:13 PM

Adam Young

Generating a list of URL patterns for OpenStack services.

Last year at the Boston OpenStack summit, I presented on an Idea of using URL patterns to enforce RBAC. While this idea is on hold for the time being, a related approach is moving forward building on top of application credentials. In this approach, the set of acceptable URLs is added to the role, so it is an additional check. This is a lower barrier to entry approach.

One thing I requested on the specification was to use the same mechanism as I had put forth on the RBAC in Middleware spec: the URL pattern. The set of acceptable URL patterns will be specified by an operator.

The user selects the URL pattern they want to add as a “white-list” to their application credential. A user could further specify a dictionary to fill in the segments of that URL pattern, to get a delegation down to an individual resource.

I wanted to see how easy it would be to generate a list of URL patterns. It turns out that, for the projects that are using the oslo-policy-in-code approach, it is pretty easy;

cd /opt/stack/nova
 . .tox/py35/bin/activate
(py35) [ayoung@ayoung541 nova]$ oslopolicy-sample-generator  --namespace nova | egrep "POST|GET|DELETE|PUT" | sed 's!#!!'
 POST  /servers/{server_id}/action (os-resetState)
 POST  /servers/{server_id}/action (injectNetworkInfo)
 POST  /servers/{server_id}/action (resetNetwork)
 POST  /servers/{server_id}/action (changePassword)
 GET  /os-agents
 POST  /os-agents
 PUT  /os-agents/{agent_build_id}
 DELETE  /os-agents/{agent_build_id}

Similar for Keystone

$ oslopolicy-sample-generator  --namespace keystone  | egrep "POST|GET|DELETE|PUT" | sed 's!# !!' | head -10
GET  /v3/users/{user_id}/application_credentials/{application_credential_id}
GET  /v3/users/{user_id}/application_credentials
POST  /v3/users/{user_id}/application_credentials
DELETE  /v3/users/{user_id}/application_credentials/{application_credential_id}
PUT  /v3/OS-OAUTH1/authorize/{request_token_id}
GET  /v3/users/{user_id}/OS-OAUTH1/access_tokens/{access_token_id}
GET  /v3/users/{user_id}/OS-OAUTH1/access_tokens/{access_token_id}/roles/{role_id}
GET  /v3/users/{user_id}/OS-OAUTH1/access_tokens
GET  /v3/users/{user_id}/OS-OAUTH1/access_tokens/{access_token_id}/roles
DELETE  /v3/users/{user_id}/OS-OAUTH1/access_tokens/{access_token_id}

The output of the tool is a little sub-optimal, as the oslo policy enforcement used to be done using only JSON, and JSON does not allow comments, so I had to scrape the comments out of the YAML format. Ideally, we could tweak the tool to output the URL patterns and the policy rules that enforce them in a clean format.

What roles are used? Turns out, we can figure that out, too:

$ oslopolicy-sample-generator  --namespace keystone  |  grep \"role:
#"admin_required": "role:admin or is_admin:1"
#"service_role": "role:service"

So only admin or service are actually used. On Nova:

$ oslopolicy-sample-generator  --namespace nova  |  grep \"role:
#"context_is_admin": "role:admin"

Only admin.

How about matching the URL pattern to the policy rule?
If I run

oslopolicy-sample-generator  --namespace nova  |  less

In the middle I can see an example like this (# marsk removed for syntax):

# Create, list, update, and delete guest agent builds

# This is XenAPI driver specific.
# It is used to force the upgrade of the XenAPI guest agent on
# instance boot.
 GET  /os-agents
 POST  /os-agents
 PUT  /os-agents/{agent_build_id}
 DELETE  /os-agents/{agent_build_id}
"os_compute_api:os-agents": "rule:admin_api"

This is not 100% deterministic, though, as some services, Nova in particular, enforce policy based on the payload.

For example, these operations can be done by the resource owner:

# Restore a soft deleted server or force delete a server before
# deferred cleanup
 POST  /servers/{server_id}/action (restore)
 POST  /servers/{server_id}/action (forceDelete)
"os_compute_api:os-deferred-delete": "rule:admin_or_owner"

Where as these operations must be done by an admin operator:

# Evacuate a server from a failed host to a new host
 POST  /servers/{server_id}/action (evacuate)
"os_compute_api:os-evacuate": "rule:admin_api"

Both map to the same URL pattern. We tripped over this when working on RBAC in Middleware, and it is going to be an issue with the Whitelist as well.

Looking at the API docs, we can see that difference in the bodies of the operations. The Evacuate call has a body like this:

    "evacuate": {
        "host": "b419863b7d814906a68fb31703c0dbd6",
        "adminPass": "MySecretPass",
        "onSharedStorage": "False"

Whereas the forceDelete call has a body like this:

    "forceDelete": null

From these, it is pretty straight forward to figure out what policy to apply, but as of yet, there is no programmatic way to access that.

It would take a little more scripting to try and identity the set of rules that mean a user should be able to perform those actions with a project scoped token versus the set of APIs that are reserved for cloud operations. However, just looking at the admin_or_owner rule for most is sufficient to indicate that it should be performed using a scoped token. Thus, an end user should be able to determine the set of operations that she can include in a white-list.

by Adam Young at March 16, 2018 05:35 PM

OpenStack Superuser

Network Function Virtualization test results: Management and Orchestration solutions

In the first post of this series, I shared a short introduction to NFV and its importance, as well as a summary about ETSI’s second NFV Plugtests. In the second post I shared the overall results of the tests and some details from the VIM perspective.

In this third and final post, I’ll share a summary of the results related to the Management and Orchestration (MANO) solutions, which as you may already know are the software pieces (VNF Manager and NFV Orchestrator) that define how virtualized network services are built from the resources provided by the infrastructure through the VIM (VMs, containers, networks, etc), their life cycle, and of course, their behavior.

As you can see from the official report and the table below, 10 different MANO commercial solutions were tested, including six proprietary and four open-source based.

It’s important to note that all MANO platforms included a Generic VNF Manager. This means that even though there is room for Specific VNF Managers at the standards, where a VNF brings its own manager (and there were tests for this kind of set-up), in most cases, a generic VNF Manager can cover all life cycle functions on most VNFs.

Another interesting observation is that at the VIM layer, 90 percent + of the vendors presented a single open-source based solution: OpenStack, while at the MANO layer, only 40 percent of the platforms were open-source based.

Still, as opposed to the first Plugtests edition, where open-source projects participated directly, this time there were two commercial (branded) MANO solutions based on open-source software and both of those are based on Open Source MANO (OSM)

OSM is an ETSI-hosted project to develop an open source NFV management and orchestration (MANO) software stack aligned with ETSI NFV. From my point of view, from the open-source MANO solutions available today (where OSM, ONAP and Open Baton stand out), OSM has reached the highest level of maturity and production-readiness across the industry. It also has a lean approach and a large, growing and diverse community that keeps enhancing the project, which is about to launch its fourth release. That’s why Whitestack  selected OSM as the foundation for our WhiteNFV product. We believe that open source brings a brighter future for our industry and we are committed to launching ETSI-standard NFV deployments in the near future.

Back to the tests:  as we did in the previous post with the VIM, let’s explore the main features expected from the MANO layer during the Plugtests:

From those features, the ones that seem to have reached maturity are related to basic life cycle management and Multi-VIM integration (especially with OpenStack, including orchestration of advanced features like EPA, multi-site, etc.), however, performance management, fault management and scaling features seem to be the next set of features that need to reach that level of maturity.

Let’s take a look a the following table from the report:

As you can see, basic VNF lifecycle testing had 300 tests with 99 percent success, followed by basic performance management (where metrics were grabbed from the VIM), with 176 tests executed and 72 percent success. We also see tests with less amount of participation, support and/or interoperability, with features related to performance management (where metrics were obtained directly from the VNFs), KPI-triggered scaling, and fault management.

In our particular case, with WhiteNFV/OSM, we took the chance to enhance our product by increasing our contributions to the open source project at those features required for this Plugtests event. In particular, we are strongly contributing to the existing OSM’s Monitoring module and with new modules for achieving VNF metrics collection and auto-scaling, as well as other important features that should be starting to appear in Release 4 (April 2018). Plugtests gave us the unique opportunity to be side-by-side with engineers from the VNF vendors, so we started developing some of the features onsite during the sessions, taking advantage of the VNF vendor’s feedback. Thanks to that opportunity we were able to successfully test VIM/VNF-based performance management and auto-scaling.

In summary, as basic life cycle features are working consistently across solutions, we can say that at the MANO level, NFV is ready for prime time, however, we should carefully select the right solution depending on the project’s specific demands beyond basic VNF life cycle, especially when related to performance/fault management and scaling, or other more advanced ones like network slicing and service function chaining.

To close this series, I would like to thank the ETSI Plugtests team for organizing such an important event, and of course, the OSM Community and the OpenStack Foundation, for working so hard on these strong open source projects that enable us to prepare a world-class, end-to-end NFV solution.

If  you’re interested in any of the topics in the series, please leave a comment. Thanks for reading!

About the author

Gianpietro Lavado is a network solutions architect interested in the latest software technologies to achieve efficient and innovative network operations. He currently works at Whitestack, a company whose mission is to promote SDN, NFV, cloud and related deployments all around the world, including a new OpenStack distribution.

This post first appeared on LinkedIn. Superuser is always interested in community content, get in touch at:

The post Network Function Virtualization test results: Management and Orchestration solutions appeared first on Superuser.

by Gianpietro Lavado at March 16, 2018 02:11 PM

James Page

Winning with OpenStack Upgrades?

On the Monday of the project teams gathering in Dublin a now somewhat familiar gathering of developers and operators got together to discuss upgrades – specifically fast forward upgrades but discussion over the day drifted into rolling upgrades and how to minimize downtime in supporting components as well. This discussion has been a regular feature over the last 18 months at PTG’s, Forums and Ops Meetups.

Fast Forward Upgrades?

So what is a fast forward upgrade? A fast forward upgrade takes an OpenStack deployment through multiple OpenStack releases without the requirement to run agents/daemons at each upgrade step; it does not allow you to skip an OpenStack release – the process allows you to just not run a release as you pass through it. This enables operators using older OpenStack releases to catch up with the latest OpenStack release in as short an amount of time as possible, accepting the compromise that the cloud control plane is down during the upgrade process.

This is somewhat adjunct to a rolling upgrade, where access to the control plane of the cloud is maintained during the upgrade process by upgrading units of a specific service individually, and leveraging database migration approaches such as expand/migrate/contract (EMC) to provide as seamless an upgrade process as possible for an OpenStack cloud. In common with fast forward upgrades, releases cannot be skipped.

Both upgrade approaches specifically aim to not disrupt the data plane of the cloud – instances, networking and storage – however this may be unavoidable if components such as Open vSwitch and the Linux kernel need to be upgraded as part of the upgrade process.

Deployment Project Updates

The TripleO team have been working towards fast forward upgrades during the Queens cycle and have a ‘pretty well defined model’ for what they’re aiming for with their upgrade process. They still have some challenges around ordering to minimize downtime specifically around Linux and OVS upgrades.

The OpenStack Ansible team gave an update – they have a concept of ‘leap upgrades’ which is similar to fast-forward upgrades – this work appears to lag behind the main upgrade path for OSA, which is a rolling upgrade approach which aims to be 100% online.

The OpenStack Charms team still continue to have a primary upgrade focus on rolling upgrades, minimizing downtime as much as possible for both the control and data plane of the Cloud. The primary focus for this team right now is supporting upgrades of the underlying Ubuntu OS between LTS releases with the imminent release of 18.04 on the horizon in April 2018, so no immediate work is planned on adopting fast-forward upgrades.

The Kolla team also have a primary focus on rolling upgrades, for which support starts at OpenStack Queens or later. There was some general discussion around automated configuration generation using Oslo to ease migration between OpenStack releases.

No one was present to represent the OpenStack Helm team.

Keeping Networking Alive

Challenges around keeping the Neutron data-plane alive during an upgrade where discussed – this included:

  • Minimising Open vSwitch downtime by saving and restoring flows.
  • Use of the ‘neutron-ha-tool’ from AT&T to manage routers across network nodes during an OpenStack cloud upgrade – there was also a bit of bike shedding on approaches to Neutron router HA in larger clouds. Plan are afoot to endeavor to make this part of the neutron code base.

Ceph Upgrades

We had a specific slot to discuss upgrade Ceph as part of an OpenStack Cloud upgrade; some deployment projects upgrade Ceph first (Charms), some last (TripleO) but there was general agreement that Ceph upgrades are pretty much always a rolling upgrade – i.e. no disruption to the storage services being provided. Generally there seems to be less pain in this area so it was not a long session.

Operator Feedback

A number of operators shared experiences of walking their OpenStack deployments through fast forward upgrades including some of the gotchas and trip hazards encountered.

Oath provided a lot of feedback on their experience of fast-forward upgrading their cloud from Juno to Ocata which included some increased complexity due to the move to using cells internally for Ocata. Ensuring compatibility between OpenStack and supporting projects was one challenge encountered – for example, snapshots worked fine with Juno and Libvirt 1.5.3, however on upgrade live snapshots where broken until Libvirt was upgraded to 2.9.0. Not all test combinations are covered in the gate!

Some of these have been shared on the OpenStack Wiki.

Upgrade SIG

Upgrade discussion has become a regular fixture at PTG’s, Forums, Summits and Meetups over the last few years; getting it right is tricky and the general feeling in the session was that this is something that we should talk about more between events.

The formation of an Upgrade SIG was proposed and supported by key participants in the session. The objective of the SIG is to improve the overall upgrade process for OpenStack Clouds, covering both offline ‘fast-forward’ and online ‘rolling’ upgrades by providing a forum for cross-project collaboration between operators and developers to document and codify best practice for upgrading OpenStack.

The SIG will initially be led by Lujin Luo (Fujitsu), Lee Yarwood (Redhat) and myself (Canonical) – we’ll be sorting out the schedule for bi-weekly IRC meetings in the next week or so – OpenStack operators and developers from across all projects are invited to participate in the SIG and help move OpenStack life cycle management forward!

by JavaCruft at March 16, 2018 10:00 AM


New Solutionaut Troops: Welcome Prashant, Shane and Adam to the Aptira Army.

The Aptira Army of Solutionauts continues to expand with 3 new Solutionaut troops joining the Aptira Army, and more on the way!

Prashant Abkari - Aptira staff headshot

Prashant Abkari


Prashant joins Aptira after working mainly on building telco products starting from 2G roaming solutions to the 4G VoLTE. He has also been involved in virtualization of some of the LTE core products in Nokia. Outside of work, Prashant enjoys to eat and drink, go to the movies, and play cricket. Without them life would be bland.

Shane Howearth - Aptira staff headshot

Shane Howearth


Shane has a broad background that stems from a love of computer science and networking. He also has interests in Machine Learning, AI (graph search), writing simple device drivers, mechanical keyboards, and LOVES Linux! Outside of work, Shane walks – a LOT! Last year it was ~100 Km/week, but now it’s ~55Km/week. Back in New Zealand he was a volunteer for Forest and Bird, helping with pest management (trapping and poisoning pests).

Adam Russell - Aptira staff headshot

Adam Russell

Mission Controllernaut

Adam has been working in the IT business for 40 years, after turning his back on a career as a zoologist and started learning to code in the mid 70’s. Starting as a developer in the nascent days of the PC industry, Adam has spent the last 35 years working on complex systems integration projects both in pre and post Sales roles. Since first taking the role of a Project Manager in the early 80’s, Adam has worked on many large-scale enterprise integration projects, mostly involving custom software, vendor product integration and a global supply chain. For much of the last 25 years these projects have been in the Telco domain and Adam has worked with some of the largest Telco’s in the world to help them integrate solutions in both BSS and OSS domains. Check out Adam’s views on project management at

Our troops are scattered all over the globe, but be sure to catch some of them at one of our upcoming Open Networking meetups, an OpenStack meetup, or the Open Infrastructure Day.

The post New Solutionaut Troops: Welcome Prashant, Shane and Adam to the Aptira Army. appeared first on Aptira.

by Jessica Field at March 16, 2018 06:49 AM

March 15, 2018

Graham Hayes

Dublin PTG Summary

Not getting on a plane was a nice change for an OpenStack event :) - especially as it looks like I would not have made it home for a few days.

Cross Project Days (Monday / Tuesday)

These days are where I think the major value of the PTG is. The cross project days feel like the Summit of old, with more time for topics, and less running around a conference centre trying to cram 2 weeks worth of talks / developer sessions, and other meetings into a few days.

Unified Limits / olso.limits / keystone stored limit data

First up for me was the keystone based limits API for unifying quota data in keystone. It was decided to create oslo.limts (olso.limits repo, olso.limits spec & oslo.limits etherpad). The keystone team already created a keystone-limits-api that is currently experimental, and the feeling in the room was that we should try and implement it using a new oslo library to find where changes need to be made.

The migration procedure was discussed, and how we (the services) would need to run multiple quota systems for quite a few cycles, due to partial upgrades that happen in OpenStack. [1] Possible implementations were discussed, and the oslo.limits etherpad has a good overview of them.


This is an idea that I have been very interested in since it was discussed in Sydney. We actually had 3 sessions on this in Dublin, across 3 different cross project rooms - API-SIG, Oslo and Self Healing SIG.

Overall, most people were receptive - the commentary is that the spec is too wordy, and contains my favorite description:

It feels like OpenStack, but not really in a good way.

After listening to feedback, and talking offline to a few people I think I have a handle on where the issues are, and I think I have a rough spec I can work flesh out over the next few days. I think I will just start writing code at that point as well - I think with a more concrete example it could help clear up issues for people.

Edge Computing

I stopped by the Edge room on the Tuesday to listen in on the discussion. Personally, I really think this group needs to stop bikesheding on, well, everything, and actually go and implement a POC and see what breaks.

The push still seems to be "make OpenStack work on the edge" instead of (what I think is the quickest / most productive way forward) "write extra tooling to orchestrate OpenStack on the edge."

There was some interesting items brought up, like Glance, and image / data residency. I think that actually engaging with the Glance team might have been helpful, as they were completely unaware that the discussion was being held, but the concepts raised sounded interesting.

I lasted about an hour or so, before I gave up. From my limited exposure, it sounded exactly like the discussions I have heard on the Edge Calls, which were the same as the ones I heard in Sydney.

Designate Sessions

The Designate room was a quiet enough affair, but it marks the first time since the PTG's started that getting a dedicated room was justified. We did some onboarding with new folks, and laid out a plan for the cycle.

The plan so far looks like this:


    • A starting point, using signing keys in the Designate database, which we can use as a jumping point to storing keys in a HSM / Barbican
    • People are currently looking at PowerDNS's inline signing as a short term solution.
  • Docs (it will never not be a priority :) )

  • Shared Zones

  • Improve the UI

    • This really relies on us either rm -rf openstack/designate-dashboard or finding people who understand Angular.

TC / QA / Tempest / Trademark Programs

If you follow the mailing list, or openstack/governance reviews, you may have seen a long running discussion over where tempest tests used for Trademark Programs should go. From memory I seem to remember this being raised in Boston, but it could have been Barcelona. There was tension between QA, me, the InterOp Work Group, and others about the location. Chris Dent covered this pretty well in his TC updates over the last while, so I am not going to rehash it, but it does look like we finally have some sort of agreement on the location, after what was 2 other times I thought we had agreement :).

Board of Directors Meeting

The OpenStack Board of Directors met on the first day of the PTG. This was an issue in its own right, which was highlighted in a thread on the foundation list. Thankfully, I have been told that this situation will not happen again (I can't seem to find any record of the decision, so it may have been an informal board discussion, but if anyone from the board is reading, replying to the foundation list would be great.).

As it met on day one, I didn't get to see much - I arrived to wait for the Add On Trademark program approvals, and happened to catch a very interesting presentation by Dims, Chris and Melvin. I then got to see the board approve DNS as a Trademark add on, which is great for the project, and people who want a constant DNSaaS experience across multiple clouds.

Johnathon Price's Board Recap is also a good overview, with links to things that were presented at the meeting.

The Board, Community, and how we interact

One topic that was highlighted by the TC / QA / Tempest / Trademark Programs discussion was that the QA team is very under resourced. This, combined with the board discussing future of the PTGs due to cost makes me very worried.

The foundation has (in my mind) two main focuses.

  1. Promote the use of the software or IP produced by people working on projects under the foundation, and protect its reputation.
  2. Empower the developers working on said software or IP to do so.

In my eyes, Trademark programs are very much part of #1, and the board should either:

  1. Fund / find resources for the QA team, to ensure they have enough bandwidth to maintain all trademark programs, the associated tests, and tooling.
  2. Fund / find a team that does it separately, but removes the entire burden from the QA team.

The PTG falls firmly under #2. I was initially a PTG skeptic, but I really think it works as an event, and adds much more value than the old mid-cycles did. I understand it has problems, but without it, teams will go back to the mid cycles, which may have looked cheaper at first glance, but for some people either meant multiple trips, or missing discussions.

One very disappointing thing to see was the list of Travel Support Program donors - there was some very generous individuals in the community that stood up and donated, but none of the corporate foundation members contributed. This, with members being added to the foundation that seem to stop at paying the membership fee (see tencent who were added at the Sydney board meeting), makes me wonder about the value placed on the community by the board.

I know the OpenStack Foundation is diversifying its portfolio of projects beyond just the OpenStack Project (this is going to get confusing :/), but we should still be supporting the community that currently exists.

Other great write ups

This turned into a bit of a PTG + 2 weeks after update, so here are some other write ups I have read over the last week or so, and prompted me to remember things that I would have otherwise forgotten.

The Hotel

And, I saved the best until last. The Croke Park Hotel was absolutely amazing during the conference. When we needed to leave the main venue on Thursday, they managed the transition of a few hundred developers into all the public spaces we could find extremely well. They kept us fed, watered and happy the entire time we were in the hotel. The fact they managed to do this, while not leaving the hotel to go home and sleep themselves! I cannot say enough good things about them, and encourage anyone who is looking for a hotel in Dublin to stay there, or is running an event to use Croke Park and the hotel.

[1]I have heard of companies running Newton / Ocata Designate (and other projects) on clouds as old as Liberty.

by Graham Hayes at March 15, 2018 07:23 PM

OpenStack Superuser

What’s new at the Vancouver Summit

With 85,000 community members and over 6 million compute cores in production, OpenStack is already one of the largest and most active open source communities in history. The OpenStack Foundation (OSF) and events are the home for people who are build and operate infrastructure.

Superuser has been talking to these extraordinary users for four years and recognizing them with the coveted Superuser Award for the past three. Whether they’re unraveling the mysteries of space or running high-speed train networks, what can be done with open infrastructure is impressive.

We’ve also been hearing how infrastructure use cases are evolving — users are combining virtual machines, containers and bare metal in new architectures — and the walls of the data centers themselves are breaking down as computing power moves to the edge.

As use cases for infrastructure have evolved, so has the OSF. With a focus on integration and cross-project collaboration, the OpenStack community started to add new top level projects under the OSF including Kata Containers and the Summit also reflects these changes.

What’s new at the Vancouver Summit

  • The Summit schedule, published today, features over 100 sessions organized by use cases: including AI and machine learning; HPC; edge computing; NFV; container infrastructure; and public, private and multi-cloud strategies. The Vancouver Summit will gather the open infrastructure operators and developers to drive these use cases forward by focusing on the integration and operations of these technologies.
  • In addition to OpenStack, there will be featured sessions on the newest project at the Foundation — Kata Containers — and a strong focus on other open source technologies relevant to infrastructure operators, including Kubernetes, Docker, Ansible, Ceph, Istio, Envoy, Spinnaker, OpenContrail, ONAP, OPNFV and many more.
  • The second edition of OpenDev will be co-located at the Vancouver Summit and focus on CI/CD, featuring collaborative working sessions, architecture show-and-tells and demos. Sessions are backed by a programming committee including Google, Netflix, Spinnaker, Mesosphere, TravisCI, Zuul, Ericsson, GoDaddy and the University of Cambridge.
  • More than 30 new and featured speakers presenting their use cases including JPMorgan Chase & Co., Wells Fargo, Progressive Insurance, Target, Gap Inc., Ohio State University, Italy’s University of Messina, Oath Inc. and AT&T. Stay tuned on Superuser as we preview these user stories.

Want to see it in person? Ticket prices for the Summit increase April 4 so register now to join the developers and operators driving the future of open infrastructure. If you need help covering costs, apply for the Travel Support Program by March 22.

Get involved

Community is central to OSF. In addition to the above events, check out the contributor portal for how to get involved with code and documentation, user groups, users and sponsorship.

During the Summit, you can also attend the Forum. With three parallel tracks over four days, OpenStack users and developers gather to brainstorm requirements for the next release, give feedback on the past version and have strategic discussions that go beyond just one release cycle. Development teams, working groups and interested individuals talk about release requirements, connect with community members about cross-project collaboration and exchange feedback between design and operations members.

The post What’s new at the Vancouver Summit appeared first on Superuser.

by Superuser at March 15, 2018 03:50 PM


March 14 Community Blog Round-up

Hardware burn-in in the CERN datacenter by Tim Bell

During the Ironic sessions at the recent OpenStack Dublin PTG in Spring 2018, there were some discussions on adding a further burn in step to the OpenStack Bare Metal project (Ironic) state machine. The notes summarising the sessions were reported to the openstack-dev list. This blog covers the CERN burn in process for the systems delivered to the data centers as one example of how OpenStack Ironic users could benefit from a set of open source tools to burn in newly delivered servers as a stage within the Ironic workflow.


Using Docker macvlan networks by Lars Kellogg-Stedman

A question that crops up regularly on #docker is “How do I attach a container directly to my local network?” One possible answer to that question is the macvlan network type, which lets you create “clones” of a physical interface on your host and use that to attach containers directly to your local network. For the most part it works great, but it does come with some minor caveats and limitations. I would like to explore those here.


A New Fencing Mechanism (TBD) by Andrew Beekhof

Protecting Database Centric Applications. In the same way that some application require the ability to persist records to disk, for some applications the loss of access to the database means game over – more so than disconnection from the storage.


Generating a Callgraph for Keystone by Adam Young

Once I know a starting point for a call, I want to track the other functions that it calls. pycallgraph will generate an image that shows me that.


Inspecting Keystone Routes by Adam Young

What Policy is enforced when you call a Keystone API? Right now, there is no definitive way to say. However, with some programmatic help, we might be able to figure it out from the source code. Lets start by getting a complete list of the Keystone routes.


SnowpenStack by rbowen

I’m heading home from SnowpenStack and it was quite a ride. As Theirry said in our interview at the end of Friday (coming soon to a YouTube channel near you), rather than spoiling things, the freak storm and subsequent closure of the event venue served to create a shared experience and camaraderie that made it even better.


Expiry of VMs in the CERN cloud by Jose Castro Leon

The CERN cloud resources are used for a variety of purposes from running compute intensive workloads to long running services. The cloud also provides personal projects for each user who is registered to the service. This allows a small quota (5 VMs, 10 cores) where the user can have resources dedicated for their use such as boxes for testing. A typical case would be for the CERN IT Tools training where personal projects are used as sandboxes for trying out tools such as Puppet.


My 2nd birthday as a Red Hatter by Carlos Camacho

This post will be about to speak about my experience working in TripleO as a Red Hatter for the last 2 years. In my 2nd birthday as a Red Hatter, I have learned about many technologies, really a lot… But the most intriguing thing is that here you never stop learning. Not just because you just don’t want to learn new things, instead, is because of the project’s nature, this project… TripleO…


by Mary Thengvall at March 15, 2018 02:27 AM

March 14, 2018

OpenStack Superuser

Kata Containers and OpenStack support open container standards

To continue the strategy of cross-community collaboration and supporting standard technologies, today the OpenStack Foundation joined the Open Container Initiative (OCI). OCI was established in 2015 by Docker, CoreOS and other industry leaders to create open industry standards around container formats and runtimes.

The move for the OpenStack Foundation to join OCI was driven by the new Kata Containers project, launched in December 2017, with a mission to combine the speed and performance of containers with the isolation and security of virtual machines.

The Kata Containers project delivers extremely lightweight virtual machine hosted containers that confirm to the OCI runtime specification. The team plans to get involved with OCI-related tooling, compatibility and interoperability testing. The Kata Containers community is currently working on their 1.0 release scheduled for late Q2 and you can learn more about their mission or contribute to their efforts at .

“OCI is an important place for the container ecosystem to come together and drive common formats across tools and deployments. We’re excited to support OCI specs in the Kata Containers project and look forward to working more closely with the OCI community to support the future of container image standardization” says James Kulina, COO of Hyper and a member of the Kata Containers Working Committee.

Under the OpenStack project, there’s also a team packaging the cloud services as OCI compliant images and industry standard tooling, which makes it easier to deploy and manage the lifecycle of OpenStack clouds using services like OpenStack-Helm and Kolla.

In the future, the OCI community plans to launch a certification program for projects, which will be another opportunity for both Kata Containers and OpenStack to participate.

Get involved

To get involved with OCI efforts, you can join the community mailing list.

To follow the OpenStack community efforts, check out the contributor portal for how to get involved with code and documentation, user groups, users and sponsorship.

If you’re new to Kata, here are some ways to connect with the community.

Code and documentation:
GitHub / How to contribute
Overview deck
Overview One Pager

Mailing list
Freenode IRC: #kata-dev and #kata-general
Slack channel or invite





Cover Photo // CC BY NC

The post Kata Containers and OpenStack support open container standards appeared first on Superuser.

by Lauren Sell at March 14, 2018 05:14 PM

Ed Leafe

Dublin PTG Recap

We recently held the OpenStack PTG for the Rocky cycle. The PTG ran from Monday to Friday, February 26 – March 2, in Dublin, Ireland. So of course the big stuff to write about would be the interesting meetings between the teams, and the discussions about future development, right? Wrong! The big news from the … Continue reading "Dublin PTG Recap"

by ed at March 14, 2018 04:48 PM

March 13, 2018

OpenStack Superuser

Check out these OpenStack project updates

If you’re interested in getting up to speed on what’s next for OpenStack software in the current development cycle,  project updates from the recent Project Teams Gathering (PTG) are a good place to start.

RDO’s Rich Bowen sat down with attendees in Dublin to talk about what features have just landed with the Queens release and what’s coming for Rocky. In his video interviews you’ll hear from the project team leaders (PTLs) and core contributors about what they’ve accomplished, where they’re heading for future releases plus how you can get involved and influence the roadmap.

Here are some excerpts:

Compute (Nova)

Matt Riedemann and Melanie Witt talk about what’s new to this release including multi-cells support for Cells V2, support for vGPUs and QEMU native LUKS decryption and volume multi-attach. At the PTG, the developers were starting conversations with users including Go Daddy and CERN about implementing  the new features. “It’s really cool that people are actually upgrading and migrating to this stuff that we’ve been working on for a couple of years now,”  Riedemann says. “They’re starting to use it and they’re finding problems but that’s good — they’re coming back to us and we’re working together.”

Block storage (Cinder)

In addition to the much-talked about volume multi-attach, PTL Jay Bryant says there are a couple of other updates that the end user will appreciate in this release. These include default policies in code, the ability to create a volume from a backup and better quota handling. This last item, which Byrant says is still in a work in progress, aims to improve the way quotas are handled for each of the different settings within Cinder. “It was possible to actually use more storage on your backend than you intended, so now we’ve added checks  0to make sure that creating a snapshot isn’t going to go over the available storage quota you set.” Bryant marks this as an area for further improvement but at least it will avoid the head-scratching situation “where Cinder thinks that 200 gig is getting used but the back end thinks I’ve got 350.” The future focus for Cinder is on usability and improvements and that users can expect “fewer major enhancements with more focus on improving functionality.”

DNS-as-a-service (Designate)

“The last cycle was fairly quiet, we were recovering from a period of low activity so we built up some debt… we spent this cycle paying it off,” says PTL Graham Hayes. “We started the groundwork for improving our upgrades, working towards rolling upgrades – meaning you can upgrade some parts of Designate and not others, so you have zero down time.” The foundation for that was laid for that during Queens and merged right after the release, he adds. The team is also working on improving the dashboard and continuous live integration – keeping an eye on ensuring that drivers are working properly in conjunction with the interoperability team. At the PTG, the board approved the DNS trademark program, “so if you have an OpenStack product or a public cloud you can have a certified of the stock DNS program as well too offer a standard it’s an interoperable DNS experience for your users.”


Because the OpenStack TripleO (that’s OpenStack on OpenStack) project focuses on deploying, upgrading and operating OpenStack clouds, it has to interact with pretty much all of the other OpenStack projects.

“During the Queens cycle, I focused pretty extensively on Ansible integration and moving more towards an Ansible-driven architecture,” says developer James Slagle. Responding to the rapid growth of the project recently, Slagle says “We’ve integrated with a lot of the OpenStack services, we have a lot of contributions from all the different projects for configuring deployment of their services and so we’ve been converting a lot of that to Ansible as well as using Ansible for a lot of upgrade tasks. As for what’s next, Slagle says “During rocky we want to switch to this Ansible deployment mode by default as well as move all of our CI over to use that new model going forward, we’re going to deprecate the old model. We’ll also be looking at more Kubernetes integration to also be deployed with real ansible and also integrating with Ceph Ansible more, to use that as a deployment mechanism for Ceph as well.”

For more on the other OpenStack project updates from the PTG, here’s a roundup from the developer’s mailing list:

Get involved!

If you’re new to OpenStack,  check out the contributor portal for how to get involved with code and documentation, user groups, users and sponsorship. You can also get up and running with the project navigator or join the mailing lists or IRC-channels.

The post Check out these OpenStack project updates appeared first on Superuser.

by Superuser at March 13, 2018 04:13 PM

Chris Dent

TC Report 18-11

Much of the activity of the TC in the past week has been devoted to discussing and sometimes arguing about two pending resolutions: Location of Interop Tests and Extended Maintenance (links below). While there has been a lot of IRC chat, and back and forth on the gerrit reviews, it has resulted in things moving forward.

Since I like to do this: a theme I would identify from this week's discussions is continued exploration of what the TC feels it can and should assert when making resolutions. This is especially apparent in the discussions surrounding Interop Tests. The various options run the gamut from describing and enforcing many details, through providing a limited (but relatively clear) set of options, to letting someone else decide.

I've always wanted the TC to provide enabling but not overly limiting guidance that actively acknowledges concerns.

Location of Interop Tests

There are three reviews related to the location of the Interop Tests (aka Trademark Tests):

It's looking like the middle one has the most support now, but that is after a lot of discussion. On Wednesday I introduced the middle of the road version to make sure the previous discussion was represented in a relatively clear way. Then on Thursday, a version which effectively moves responsibility to the InteropWG.

Throughout this process there have been hidden goals whereby this minor(?) crisis in policy is being used to attempt to address shortcomings in the bigger picture. It's great to be working on the bigger picture, but hidden doesn't sound like the right approach.

Tracking TC Goals

One of the outcomes from the PTG was an awareness that some granular and/or middle-distance TC goals tend to get lost. The TC is going to try to use StoryBoard to track these sort of things. The hope is that this will result in more active and visible progress.

Extended Maintenance

A proposal to leave branches open for patches for longer has received at least as much attention as the Interop discussions.

Some talk starts on Thursday afternoon and then carries on intermittently for the rest of time^w^w^w^w^wthrough today. The review has a great deal of interaction as well.

There's general agreement on the principal ("let's not limit people from being able to patch branches for longer") but reaching consensus on the details has been more challenging. Different people have different goals.

What's a SIG for?

Discussion about renaming the recently named PowerStackers group eventually migrated into talking about what SIGs are for or mean. There seems to be a few different interpretations, with some overlap:

None of these are really complete. I think of SIGs as a way to break down boundaries, provide forums for discussion and make progress without worrying too much about bureaucracy. We probably don't need to define them to death.

by Chris Dent at March 13, 2018 03:10 PM


Upgrade Newton on Heat : what’s new for orchestration service ?

Dear Users,

Orchestration service is now in Newton version: let’s deep dive into new features and Heat resources improvements.

New Features

Convergence feature allows to improve performance of your create, update, delete actions. Most effective when stacks are large. Enabled by default when service upgraded, you will take benefit from it without any action from your side.

Observe and Update feature is also enabled, it checks resources state before doing any update. For example, taking a stack with a volume and an instance, you aim to update, adding a volume attachment to this instance. If instance or/and volume do not exist anymore, the update without Observe and Update feature will fail. Reversely, with it, the update ends with success.

Heat resources improvements

Volume resources

If you plan to update one of your stack, we recommend you strongly to explicit the metadata property filling “{}” in its template.

New resources available


A resource for managing Neutron security group rules. Rules to use in security group resource.

Example :

type: OS::Neutron::SecurityGroupRule
  description: String
  direction: String
  ethertype: String
  port_range_max: Integer
  port_range_min: Integer
  protocol: String
  remote_group: String
  remote_ip_prefix: String
  security_group: String


A resource which exposes its value property as an attribute.

This is useful for exposing a value that is a simple manipulation of other template parameters and/or other resources.

Example :

type: OS::Heat::Value
  type: String
  value: Any


This resource allows to create multiple resources with same properties.

Example :

  type: OS::Heat::ResourceChain            
      concurrent: True                   
      resource_properties: {size: 5, volume_type: "performant"}              
      resources: ["OS::Cinder::Volume","OS::Cinder::Volume", "OS::Cinder::Volume"]

Here above, 3 performant volumes of 5G have been created.

Property “concurrent” defines if resource creation can be done in parallel.

Property “resource_properties” defines properties to apply to resources

Property “resources” defines resources to create.

This resource allows mainly to factorize yaml templates.


It allows to disable easily some resources through “resource_registry” on environment file.

This resource works with all properties and return any attribute.

Example :

  OS::Cinder::Volume: OS::Heat::None


This a resource which store the value from the chain.

Some control actions are available, such as ‘action_wait_secs’, ‘wait_secs’, ‘attr_wait_secs’. This allow to adjust time for stack creation or for waiting period to obtain resource output.

Example ::

    type: OS::Heat::TestResource
            create: 60
            update: 60
        value: "test"
        value: {get_attr: [alarm_cache_wait, output]}


This resource associates a group of servers with some configuration.

Replaces the resource OS::Heat::SoftwareDeployments

Example :

type: OS::Heat::SoftwareDeploymentGroup
  actions: [Value, Value, ...]
  config: String
  input_values: {...}
  name: String
  servers: {...}
  signal_transport: String


This resource associates a group of servers with some configuration.

This resource works similar as OS::Heat::SoftwareDeploymentGroup, but for structured resources.

Replaces the resource OS::Heat::StructuredDeployments

Example :

type: OS::Heat::StructuredDeploymentGroup
  actions: [Value, Value, ...]
  config: String
  input_key: String
  input_values: {...}
  input_values_validate: String
  name: String
  servers: {...}
  signal_transport: String

New attributes

For resource OS::Heat::AutoScalingGroup

  • refs : A list of resource IDs for the resources in the group
  • refs_map : A map of resource names to IDs for the resources in the group

For resource OS::Heat::ResourceGroup

  • refs_map : A map of resource names to IDs for the resources in the group
  • removed_rsrc_list : A list of removed resource names

New properties

For resource OS::Nova::Server

For block_device_mapping_v2:

  • image : The ID or name of the image to create a volume from. Updated by image property.

For OS::Glance::Image

  • architecture
  • os_distro ( In order to make images more easily searchable in different openstack installations )
  • kernel_id ( The ID of image stored in Glance that should be used as the kernel when booting an AMI-style image,)
  • ramdisk_id ( The ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image )
  • extra_properties
  • tags
  • owner

New update policies

For resource OS::Heat::SoftwareDeploymentGroup and OS::Heat::StructuredDeploymentGroup (resources added in mitaka)

  • Batch_create
    • Map value expected :
    • max_batch_size : The maximum number of resources to create at once
    • pause_time
  • Rolling_update
    • Map value expected :
    • max_batch_size : The maximum number of deployments to replace at once
    • Pause_time

Example :

type: OS::Heat::SoftwareDeploymentGroup
      max_batch_size: 2
      pause_time: 1
    config: {get_resource: config}
      foo: {get_param: input}
     '0': dummy0

Deprecated resources

For resource OS::Nova::Server and For block_device_mapping_v2

The image_id has to be replaced by image

End of support for the following resources


You can contact our support service

by Horia Merchi at March 13, 2018 12:00 AM

March 12, 2018

OpenStack Superuser

Getting started with multi-cloud

There are many reasons companies consider going multi-cloud: to lift and shift their data center into the cloud, extend their data center into the cloud or to move applications to a different environment. Or maybe it’s because a single cloud service provider can’t meet all of their complex compute and storage needs.

Start with the right questions

Before you start multiplying your clouds though, it makes sense to take a step back.

“Far too many times IT managers and directors assume they can cram their entire infrastructure into one platform and that might not be the best case scenario for every project,” says Morgan Martensen, business development at IT hosting company SingleHop on the company’s tech chat, SingleHoppy Hour. “Our number one rule is: application first, infrastructure second.” That means asking questions about your applications before starting to plan what will power them.

Too many people don’t consider what’s driving the move. “It’s like walking into a car dealership and asking: ‘Do you sell cars?’ You have to have to understand what that IT director is trying to accomplish because there are a lot of delivery models,” says TJ Waldorf, VP of inside sales and marketing at SingleHop.

Asking questions is key, some to start with include:

  • What workloads do they run
  • What apps are they running and supporting
  • What are the compliance requirements
  • What are the geographical requirements

Keep an eye on the bills

Cost is often a key driver for going multi-cloud, but it may not be the simplest route. Instead of hashing out cost optimization on one platform, you’ve got five. “It’s definitely challenging,” says Waldorf. “If you’re using multiple vendors you’ve got multiple bills. AWS in particular…if you’ve ever seen an invoice from AWS it’s probably 47 pages with 10,000 line items that you’re not entirely sure what they are and when costs go up go or down it’s really hard to pinpoint why.” There are plenty of tools to help with cost optimization, but you still have to have the expertise to know how to run those tools and understand how to do it right, he adds.
Waldorf cites the case of one customer who moved from their own dedicated servers to the cloud looking for greater scalability, but realized that the unexpected bandwidth costs were much higher than anticipated. In a lot of cases, it makes sense to leave a few applications and workloads on dedicated servers or a private cloud. “It boils down to figuring out where those things need to go and optimizing for performance but also cost. There’s a lot of trade-offs with public cloud in any cloud platform,” he says.

Catch the whole 19-minute conversation on YouTube

The post Getting started with multi-cloud appeared first on Superuser.

by Superuser at March 12, 2018 03:36 PM

OpenStack in Production

Hardware burn-in in the CERN datacenter

During the Ironic sessions at the recent OpenStack Dublin PTG in Spring 2018, there were some discussions on adding a further burn in step to the OpenStack Bare Metal project (Ironic) state machine. The notes summarising the sessions were reported to the openstack-dev list. This blog covers the CERN burn in process for the systems delivered to the data centers as one example of how OpenStack Ironic users could benefit from a set of open source tools to burn in newly delivered servers as a stage within the Ironic workflow.

CERN hardware procurement follows a formal process compliant with public procurements. Following a market survey to identify potential companies in CERN's member states, a tender specification is sent to the companies asking for offers based on technical requirements.

Server burn in goals

Following the public procurement processes at CERN, large hardware deliveries occur once or twice a year and smaller deliveries multiple times per year. The overall resource management at CERN was covered in a previous blog. Part of the steps before production involves burn in of new servers. The goals are
  • Ensure that the hardware delivered complies with CERN Technical Specifications
  • Find systematic issues with all machines in a delivery such as bad firmware
  • Identify failed components in single machines
  • Provoke early failure in failing components due to high load during stress testing
Depending on the hardware configuration, the burn-in tests take on average around two weeks but do vary significantly (e.g. for systems with large memory amounts, the memory tests alone can take up to two weeks). This has been found to be a reasonable balance between achieving the goals above compared to delaying the production use of the machines with further testing which may not find more errors.

Successful execution of the CERN burn in processes is required in the tender documents prior to completion of the invoicing.


The CERN hardware follows a lifecycle from procurement to retirement as outlined below. The parts marked in red are the ones currently being implemented as part of the CERN Bare Metal deployment.

As part of the evaluation, test systems are requested from the vendor and these are used to validate compliance with the specifications. The results are also retained to ensure that the bulk equipment deliveries correspond to the initial test system configurations and performance.

Preliminary Checks

CERN requires that the Purchase Order ID  and an unique System Serial Number are set in the NVRAM of the Baseboard Management Controller (BMC), in the Field Replaceable Unit (FRU) fields Product Asset Tag (PAT) and Product Serial (PS) respectively:

# ipmitool fru print 0 | tail -2
 Product Serial        : 245410-1
 Product Asset Tag     : CD5792984

The Product Asset Tag is set to the CERN delivery number and the Product Serial is set to the unique serial number for the system unit.

Likewise, certain BIOS fields have to be set correctly such as booting from network before disk to ensure the systems can be easily commissioned.

Once these basic checks have been done, the burn in process can start. A configuration file, containing the burn-in tests to be run, is created according on the information stored in the PAT and PS FRU fields. Based on the content of the configuration file, the enabled tests will automatically start.

Burn in

The burn in process itself is highlighted in red in the workflow above, consisting of the following steps
  • Memory
  • CPU
  • Storage
  • Benchmarking
  • Network


The memtest stress tester is used for validation of the RAM in the system. Details of the tool are available at


Testing the CPU is performed using a set of burn tools, burnK7 or burnP6, and burn MMX. These tools not only test the CPU itself but are also useful to find cooling issues such as broken fans since the power load is significant with the processors running these tests.


Disk burn ins are intended to create the conditions for early drive failure. The bathtub curve aims to cause the early failure drives to fail prior to production.

With this aim, we use the badblocks code to repeatedly read/write the disks. SMART counters are then checked to see if there are significant numbers of relocated bad blocks and the CERN tenders require disk replacement if the error rate is high.

We still use this process although the primary disk storage for the operating system has now changed to SSD. There may be a case for minimising the writing on an SSD to maximise the life cycle of the units.


Many of the CERN hardware procurements are based on price for total compute capacity needed. With the nature of most of the physics processing, the total throughput of the compute farm is more important than the individual processor performance. Thus, it may be that the most total performance can be achieved by choosing processors which are slightly slower but less expensive.

CERN currently measures the CPU performance using a set of benchmarks based on a subset of the SPEC 2006 suite. The subset, called HEPSpec06, is run in parallel on each of the cores in the server to determine the total throughput from the system. Details are available at the HEPiX Benchmarking Working Group web site.

Since the offers include the expected benchmark performance, the results of the benchmarking process are used to validate the technical questionnaire submitted by the vendors. All machines in the same delivery would be expected to produce similar results so variations between different machines in the same batch are investigated.

CPU benchmarking can also be used to find problems where there is significant difference across a batch, such as incorrect BIOS settings on a particular system.

Disk performance is checked using a reference fio access suite. A minimum performance level in I/O is also required in the tender documents.


Networking interfaces are difficult to burn in compared to disks or CPU. To do a reasonable validation,  at lest two machines are needed. With batches of 100s of servers, a simple test against a single end point will produce unpredictable results.

Using a network broadcast, the test finds other machines running the stress test, they pair up and run a number of tests.

  • iperf3 is used for bandwidth, reversed bandwidth, udp and reversed udp
  • iperf for full duplex testing (currently missing from iperf3)
  • ping is used for congestion testing

Looking forward

CERN is currently deploying Ironic into production for bare metal management of machines. Integrating the burn in and retirement stages into the bare metal management states would bring easy visibility of the current state as the deliveries are processed.

The retirement stage is also of interest to ensure that there is no CERN configuration in the servers (such as Ironic BMC credentials or IP addresses).  CERN has often donated retired servers to other high energy physics sites such as SESAME in Jordan and Morocco which requires a full server factory reset before dismounting. This retirement step would be a more extreme cleaning followed by complete removal from the cloud.

Discussing with other scientific laboratories such as SKA through the OpenStack Scientific special interest group has shown interest in extending Ironic to automate the server on-boarding and retirement processes as described in the session at the OpenStack Sydney summit. We'll be following up on these discussions at Vancouver.


  • CERN IT department -
  • CERN Ironic and Rework Contributors 
    • Alexandru Grigore
    • Daniel Abad
    • Mateusz Kowalski


by Tim Bell ( at March 12, 2018 11:29 AM


OpenStack embraces the future with GPU, edge computing support

It wasn't that long ago that OpenStack was the hot new kid on the infrastructure block. That just might change with the latest release of OpenStack, code-named Queens.

by Nick Chase at March 12, 2018 07:38 AM

March 09, 2018

Adam Spiers

Report from the OpenStack PTG in Dublin

Last week I attended OpenStack’s PTG (Project Teams Gathering) in Dublin. This event happens every 6 months in a different city, and is a fantastic opportunity for OpenStack developers and upstream contributors to get together and turbo-charge the next phase of collaboration.

I wrote a private report for my SUSE colleagues summarising my experience, but then Colleen posted her report publicly, which made me realise that it would be far more in keeping with OpenStack’s Four Opens to publish mine online. So here it is!

General observations

The PTG was well-attended with just over 400 people from all over the world. As with previous PTGs this felt like just the right size: enough to have most projects / organizations well represented, but not too big to prevent serendipitous meetings happening many times throughout the week, where I would just be wandering around and bump into exactly the right person I needed to talk to 🙂

All the etherpads are publicly available in case you want to dive into the gory details.

IMHO the ptgbot continued to work really well for adapting to the dynamic and fast-changing agenda. I submitted a few patches to polish it further.

The “extreme” (for Ireland, anyway) weather, which was reportedly the heaviest snow in 40 years, caused a lot of disruption, not just for travel, but also that our conference venue in Croke Park Stadium was forced to shut on Thursday afternoon, rather than Friday evening as originally planned. Luckily our amazing hotel, The Croke Park, stepped up to the plate and allowed us to continue the conference there. It was quite a squash fitting everyone in, but somehow they managed it, and the ensuing organized mayhem had the unexpected side effect of building really strong comradery between everyone.

There was a piano-bar style sing-along night (with OpenStack’s Executive Director Jonathan Bryce doing a great job on the piano), a board games night … I think it was my colleague Keith Berger who shrewdly observed that it was like being on a cruise ship 🙂 Having the best Guinness in the world on tap probably helped too. Check out #SnowpenStack on Twitter for some of the hilarious antics.

We managed a couple of SUSE team dinners and it was great to spend more time with teammates, especially the ones who are normally on a different continent 🙂

Self-healing SIG

My main reason for attending the PTG was bootstrapping this new SIG which I founded a few months ago. We had a really good session filling Tuesday afternoon, with about 30 people attending.

The subset of self-healing attendees brave enough to venture out into the cold

There were discussions about various use cases, and some people are already keen to start work on use cases immediately. We also had representatives from the API SIG, including Graham Hayes a.k.a. mugsie who is leading the new health-check API initiative, and from OPNFV, since we are trying to promote better sharing of work on automated testing of self-healing / HA functionality. (If you’re interested, please sign up here!)

We agreed some concrete next steps, some of which I already started working on:

You can follow all activity via the StoryBoard project.

The only disappointment with this session was that no operators attended. However I managed to coordinate with some people involved with the Tokyo Ops meetup this week, in order to gather feedback from ops, and we got a session slot booked in that agenda which happened yesterday. Additionally I am speaking on self-healing and leading a discussion session on it at the London OpenStack Meetup next Monday, so hopefully we’ll get some good feedback there too.

First Contact SIG

I joined the contributor guide part of this session briefly to give some feedback and also organise collaboration on writing a quickstart guide for people who are used to GitHub and don’t know Gerrit, since I promised to write that after creating a comparison matrix in Denver, which provides a base for the guide.


The Self-healing SIG previously agreed to use StoryBoard for tracking work, not just to aid collaboration within the SIG, but also to help dogfood StoryBoard and provide feedback. So I spent quite a bit of time with the StoryBoard folks giving feedback and brainstorming ideas for improving usability. I also started work on a “Rosetta stone” matrix which aims to show the key differences between StoryBoard and other issue trackers people are used to, such as JIRA, Trello, GitHub, Bugzilla, Redmine etc. Similarly to the approach with the GitHub / Gerrit matrix above, it could serve as the basis for a document which helps “port” people to StoryBoard from other issue trackers.

Fast-forward upgrades (FFU)

This was a really full session, with some good discussions. Main points:

  • It was agreed to form an Upgrade SIG, and this has now happened.
  • AFAICS everyone is not yet on the same page about what FFU means exactly, especially regarding online migrations. There is perhaps room for improvement in the governance tags provided for tracking upgradability (assert:supports-upgrade, assert:supports-accessible-upgrade), which would help clarify this.
  • IIUC Kolla seem to be going in a slightly different direction based on online rolling upgrades, although I don’t think they necessarily have any strong objections to supporting offline upgrades so maybe things will converge in the long term.
  • Noone from openstack-helm present 🙁
  • Lots of gotchas, and consequent debates on how best to capture and share these.


I only had time to pop into the oslo room briefly, to discuss an interesting new initiative in oslo.config to automate migration of config between releases (a bit like DB migrations, but for oslo.config instead of DB schema). This was also mentioned in the FFU session.

I posed some questions about how this would handle config which requires extra per-project logic to migrate, and also how it would handle config which has been explicitly set to the default value. We had some interesting discussions on that but I think ultimately the answer was “let’s cross that bridge when we come to it”.

I also had a quick chat with Ken Giusti about a long-standing oslo.service bug we’ve worked on, but we ran out of time and agreed to take it online.

OpenStack Ansible

I demo’d ansible-viz, the Ansible visualization tool of which I recently became the maintainer, to the OpenStack Ansible folks. They seemed really enthusiastic, and are looking into using it for embedding diagrams in their docs. Hopefully this means they will start contributing to make it better!

My colleague Dirk Müller and I also talked to them in general about other areas for potential collaboration. We were both impressed with how open they are to help and collaborate!


As usual with the PTGs, I got huge value from random unplanned conversations in hallways, at lunch tables, in the bar etc.

I spent a lot of time talking to Emmet Hickory (a.k.a. persia) and he gave me some great ideas about new use cases for the git workflow tools I have been building in hackweeks and spare time. I already knew that my git-deps tool allows fairly accurate predictions of how difficult it will be to backport a given commit to a stable branch, but a new idea emerged that this could be added as a non-voting job to Zuul‘s checks pipeline, to facilitate a more proactive / preemptive approach to backporting.

I talked with Tony Breeds (PTL of stable branch maintenance) about this who told me that Neutron is already extremely proactive when it comes to backporting, so they might be interested in this. So I met Miguel Lavalle (Neutron PTL) and he was really interested and requested that I put it on the agenda for a future Neutron IRC meeting. This has now happened and we will discuss during next Tuesday’s meeting.

Emmet helped me realise that my git-explode tool could also help with backporting work, so I need to finish packaging that and publish it. There is a London Git meetup coming up soon, and I might present the tools at that.

I’m getting off-topic now so I’ll finish here 🙂 If you found this report useful please leave a comment below, so that I know it’s worth bothering to do next time 🙂


The post Report from the OpenStack PTG in Dublin appeared first on Structured Procrastination.

by Adam at March 09, 2018 08:27 PM

OpenStack Superuser

How to develop a new Horizon dashboard without DevStack

Everybody knows that the easiest way to build a new Horizon dashboard is to use DevStack. But, as you also already know, DevStack requires a fair amount of resources (16GB of RAM for a workable DevStack, in my experience).

So what can you do if you only have a laptop with 4-8GB RAM? Easy peasy, just follow these steps:

1. Install Keystone (the minimum requirement for Horizon)

Follow these official instructions:

Here are some common issues you may run into and fixes for them:

2. Install Horizon

Follow these official instructions on how to install Horizon on your computer:

3. Build your own dashboard

Using this tutorial:

Note: you may have to install additional services if your dashboard needs to communicate with those services (e.g: Tacker, Nova…)

4. Change the world


Trinh Nguyen is a hacker and ed-tech enthusiast with a startup called EdLab.XYZ. This post first appeared on his blog.

Superuser is always interested in community content, especially tutorials. Get in touch at

Cover Photo // CC BY NC

The post How to develop a new Horizon dashboard without DevStack appeared first on Superuser.

by Trinh Nguyen at March 09, 2018 05:11 PM

OpenStack in Production

Expiry of VMs in the CERN cloud

The CERN cloud resources are used for a variety of purposes from running compute intensive workloads to long running services. The cloud also provides personal projects for each user who is registered to the service. This allows a small quota (5 VMs, 10 cores) where the user can have resources dedicated for their use such as boxes for testing. A typical case would be for the CERN IT Tools training where personal projects are used as sandboxes for trying out tools such as Puppet.

Personal projects have a number of differences compares to other projects in the cloud
  • No non-standard flavors
  • No additional quota can be requested
  • Should not be used for production services
  • VMs are deleted automatically when the person stops being a CERN user
With the number of cloud users increasing to over 3,000, there is a corresponding growth in the number of cores used by personal projects, growing by 1,200 cores in the past year. For cases like training users, there is often the case that the VMs are created and the user then does not remember to delete the resources so they consume cores which could be used for compute capacity to analyse the data from the LHC.

One possible approach would be to reduce the quota further. However, tests such as setting up a Kubernetes cluster with OpenStack Magnum often need several VMs to perform the different roles so this would limit the usefulness of personal projects. The usage of the full quota is also rare.

VM Expiration

Based on a previous service which offered resources on demand (called CVI based on Microsoft SCVMM), the approach was taken to expire personal virtual machines.
  • Users can create virtual machines up to the limit of their quota
  • Personal VMs are marked with an expiry date
  • Prior to their expiry, the user is sent several mails to inform them their VM will expire soon and how to extend it if it is still useful.
  • On expiry, the virtual machine is locked and shutdown. This helps to catch cases where people have forgotten to prolong their VMs.
  • One week later, the virtual machine is deleted, freeing up the resources.


We use Mistral to automate several OpenStack tasks in the cloud (such as regular snapshots and project creation/deletion). This has the benefit of a clean audit log to show what steps worked/failed along with clear input/output states supporting retries and an authenticated cloud cron for scheduling.

Our OpenStack projects have some properties set when they are created. This is used to indicate additional information like the accounting codes to be charged for the usage. There are properties for indicating if the type of project such as personal and if the expiration workflow should apply. Mistral YAQL code can then select resources where expiration applies.

task(retrieve_all_projects) => $.id, name => $.name, enabled => $.enabled, type => $.get('type','none'),expire => $.get('expire','off'))).where($.type='personal').where($.enabled).where($.expire='on')

The expire_at parameter is stored as a VM property. This makes it visible for automation such as CLIs through the openstack client show server CLI.

There are several parts to the process
  • A cron trigger'd workflow which
    • Machines in error state or currently building are ignored
    • A newly created machine which does not have an expiry date set has the expiration date set according to the grace period
    • Sees if any machines are entering close to their expiry time and sends a mail to the owner
    • Checks for invalid settings of the expire_at property (such as people setting it a long way in the future or deleting the property) and restores a reasonable value if this is detected
    • If a machine has reached it's expiry date, it's locked and shutdown
    • If a machine has past it's date by the grace period, it's deleted
    • A workflow, launched by Horizon or from the CLI
      • Retrieves the expire_at value and extends it by the prolongation period
    The user notification is done using a set of mail templates and a dedicated workflow ( This allows templates such as instance reminders to have details about the resources included, such as the example from the mail template.

    The Virtual Machine {instance} from the project {project_name} in the Cloud Infrastructure Service will expire on {expire_date}.

    A couple of changes to Mistral will be submitted upstream
    • Support for HTML mail bodies which allows us to have a nicer looking e-mail for notification with links included
    • Support for BCC/CC on the mail so that the OpenStack cloud administrator e-mail can also be kept on copy when there are notifications
    A few minor changes to Horizon were also done (currently local patches)
    • Display expire_at value on the instance details page
    • Add a 'prolong' action so that instances can be prolonged via the web by using the properties editor to set the date of the expiry (defaulting to the current date with the expiry time). This launches the workflow for prolonging the instance.


    Jose Castro Leon from the CERN cloud team


    by Jose Castro Leon ( at March 09, 2018 03:11 PM

    Julie Pichon

    OpenStack PTG Dublin - Rocky

    I was so excited when it was first hinted in Denver that the next OpenStack PTG would be in Dublin. In my town! Zero jet lag! Commuting from home! Showing people around! Alas, it was not to be. Thanks, Beast from the East. Now everybody hates Ireland forever.

    The weather definitely had some impact on sessions and productivity. People were jokingly then worryingly checking on the news, dropping in and out of rooms as they tried to rebook their cancelled flights. Still we did what we could and had snow-related activities too - good for building team spirit, if nothing else!

    I mostly dropped in and out of rooms, here are some of my notes and sometimes highlights.

    OpenStack Client

    Like before, the first two days of the PTG were focused on cross-projects concerns. The OpenStack Client didn't have a room this time, which seems fair as it was sparsely attended the last couple of times - I would have thought there'd have been one helproom session at least but if there was I missed it.

    I regret missing the API Working Group morning sessions on API discovery and micro-versions, which I think were relevant. The afternoon API sessions were more focused on services and less applicable for me. I need to be smarter about it next time.

    First Contact SIG

    Instead, that morning I attended the First Contact Special Interest Group sessions, which aim to make OpenStack more accessible to newcomers. It was well attended, with even a few new and want-to-be contributors who were first-time PTG attendes - I think having the PTG in Europe really helped with that. The session focused on making sure everyone in the room/SIG is aware of the resources that are out there, to be able to help people looking to get started.

    The SIG is also looking for points of contact for every project, so that newcomers have someone to ask questions to directly (even better if there's a backup person too, but difficult enough to find one as it is!).

    Some of the questions that came up from people in the room related to being able to map projects to IRC channel (e.g. devstack questions go to #openstack-qa).

    Also, the OpenStack community has a ton of mentoring programs, both formal and informal and just going through the list to explain them took a while. Outreachy, Google Summer of Code, Upstream Institute, Women of OpenStack, First Contact Liaisons (see above). Didn't realise there were so many!

    I remember when a lot of the initiatives discussed were started. It was interesting to hear the perspectives from people who arrived later, especially the discussions around the ones that have become irrelevant.

    Packaging RPMs

    On Tuesday I dropped by the packaging RPMs Working Group session. A small group made up of very focused RDO/Red Hat/SUSE people. The discussions were intense, with Python 2 going End Of Life in under 2 years now.

    The current consensus seems to be to create a RPM-based Python 3 gate based on 3.6. There's no supported distro that offers this at the moment, so we will create our own Fedora-based distro with only what we need at the versions we need it. Once RDO is ready with this, it could be moved upstream.

    There were some concerns about 3.5 vs 3.6 as the current gating is done on 3.5. Debian also appears to prefer 3.6. In general it was agreed there should not be major differences and generally ok.

    The clients must still support Python 2.

    There was a little bit of discussion about the stable policy and how it doesn't apply to the specs or the rpm-packaging project (I think the example was with Monasca and the default backend not working (?), so a spec change to modify the backend was backported - which could be considered a feature backport, but since the project isn't under the stable policy remit it could be done).

    There was a brief chat at the end about whether there is still interest in packaging services, as opposed to just shipping them as containers. There certainly still seems to be at this point.

    Release management

    A much more complete summary has already been posted on the list, and I had to leave the session halfway to attend something else.

    There seems to be an agreement that it is getting easier to upgrade (although some people still don't want to do it, perhaps an education effort is needed to help with this). People do use the stable point release tags.

    The "pressure to upgrade": would Long-Term Support release actually help? Probably it would make it worse. The pressure to upgrade will still be there except there won't be a need to work on it for another year, and it'll make life worse for operators/etc submitting back fixes because it'll take over a year for a patch to make it into their system.

    Fast-Forward Upgrade (which is not skip-level upgrades) may help with that pressure... Or not, maybe different problems will come up because of things like not restarting services in between upgrades. It batches things and helps to upgrade faster, but changes nothing.

    The conversation moved to one year release cycles just before I left. It seemed to be all concerns and I don't recall active support for the idea. Some of the concerns:

    • Concerns about backports - so many changes
    • Concerns about marketing - it's already hard to keep up with all that's going on, and it's good to show the community is active and stuff is happening more than once a year. It's not that closely tied to releases though, announces could still go out more regularly.
    • Planning when something will land may become even harder as so much can happen in a year
    • It's painful for both people who keep up and people who don't, because there is so much new stuff happening at once.


    The sessions began with a retrospective on Wednesday. I was really excited to hear that tripleo-common was going to get unit tests for workflows. I still love the idea of workflows but I found them becoming more and more difficult to work with as they get larger, and difficult to review. Boilerplate gets copy-pasted, can't work without a few changes that are easy to miss unless manually tested and these get missed in reviews all the time.

    The next session was about CI. The focus during Queens was on reliability, which worked well although promotions suffered as a result. There were some questions as to whether we should try to prevent people from merging anything when the promotion pipeline is broken but no consensus was really reached.

    The Workflows session was really interesting, there's been a lot of Lessons Learnt from our initial attempt with Mistral this last couple of years and it looks like we're setting up for a v2 overhaul that'll get rid of many of the issues we found. Exciting! There was a brief moment of talk about ripping Mistral out and reimplementing everything in Ansible, conclusions unclear.

    I didn't take good notes during the other sessions and once the venue closed down (snow!) it became a bit difficult to find people in the hotel and then actually hear them. Most etherpads with the notes are linked from the main TripleO etherpad.

    by jpichon at March 09, 2018 11:57 AM

    March 08, 2018

    OpenStack Superuser

    When CI/CD becomes a battleground

    You can almost hear the roar vibrating from the cubicles. It’s the epic tussle between tiger teams — those elite IT professionals — as they decide the future of operations. The decisions about how applications are deployed and who gets to decide that process are part of an ongoing fight spurred by the move to continuous integration and development.

    Kong Yang spoke about the human side that often gets overlooked when it comes to adopting CI/CD and dev-ops on a recent edition of L8ist Sh9y podcast (that’s “Latest Shiny,” for those who appreciate vowels.)

    Yang, a Dell alum whose current title is actually “head geek” at SolarWinds, spoke of a scenario where two teams of engineers battle over the CI/CD pipeline, with each team given six months to see whose is better.  This kind of showdown rarely works long term: typically, he says, both will walk away and leave it to somebody else to do the maintenance of a system that’s designed to fail fast. And, in the meantime, one of those battle-scarred distinguished engineers probably has to go looking for greener pastures. Or the company loses both — because they can always walk away. Everyone loses.

    “That’s the piece that leaders on the business side don’t get… What they want to see is more revenue, they want more profits. They want to see more stuff that they can offer to customers,” Yang says. “That human aspect is one thing that a lot of organizations don’t factor in because guess what: You have the option to go out and hire new talent. Well, the talent that you have in-house has the option to leave as well.”

    Host of the podcast Rob Hirschfeld, CEO of RackN, agreed, adding that he’s dubbed it the SRE half life, where people are brought into an organization to usher in change. When the company resists those changes, the site reliability engineer or architect leaves for a more conducive habitat. “Companies have to figure out how to accept and embrace these changes, it’s a very big deal.”

    The pair covered a lot of other interesting ground, including:

    • Why we’re in the golden age of cloud 2:34
    • The disrupting force of mobile apps and the CI/CD pipeline 15:38
    • “Living data points” vs. monolith apps 17:15
    • The challenge (and the opportunities) for IT ops professionals now 23:20
    • More on the political debate around building CI/CD pipelines 27:11
    • From AWS to bare metal 30:18

    You can catch the whole 37-minute podcast on SoundCloud.

    Cover Photo // CC BY NC

    The post When CI/CD becomes a battleground appeared first on Superuser.

    by Superuser at March 08, 2018 05:29 PM

    Mark T. Voelker

    Dublin PTG 2018: The Edge Sessions

    I recently got snowed in at a hotel in Dublin with a couple hundred people I consider friends, a well-stocked bar/restaurant, a fantastic event staff, and a lot of energy. I know…poor me, right? =) Before Storm Emma and the Beast from the East settled in on the OpenStack Project Team Gathering and forced the Croke Park Stadium to close (thereby giving birth to the hashtag “snowpenstack”), we got a solid start to the week.

    March 08, 2018 12:00 AM

    March 07, 2018

    Major Hayden

    Reaching the fork in the road

    Mountain trail

    Walt Disney said it best:

    We keep moving forward, opening new doors, and doing new things, because we’re curious and curiosity keeps leading us down new paths.

    The world of technology is all about change. We tear down the old things that get in our way and we build new technology that takes us to new heights. Tearing down these old things can often be difficult and that forces us to make difficult choices.

    Rackspace has been a great home for me for over 11 years. I’ve made the incredibly difficult choice to leave Rackspace on March 9th to pursue new challenges.

    Humble beginnings

    I came to Rackspace as an entry-level Linux administrator and was amazed by the culture generated by Rackers. The dedication to customers, technology, and quality was palpable from the first few minutes I spent with my team. Although I didn’t know it at the time, I had landed at the epicenter of a sink-or-swim technology learning experience. My team had some very demanding customers with complex infrastructures and it forced me to take plenty of notes (and hard knocks). My manager and teammates supported me through it all.

    From there, I served in several different roles. I was a manager of technicians on a support team and had the opportunity to learn how to mentor. One of my favorite leaders said that “good managers know when to put their arm around to people and when to put a boot in their rear.” I reluctantly learned how to do both and I watched my people grow into senior engineers and great leaders.

    Closing the Datapoint office

    Datapoint office closing in 2011

    I was pulled to Mosso, Rackspace’s first cloud offering, shortly after that and discovered an entirely new world. Rackers force-fed me “Why’s (Poignant) Guide to Ruby” and I started building scripts and web front-ends for various services. Rackspace acquired Slicehost after that and I jumped at the chance to work as an operations engineer on the new infrastructure. That led to a lot of late nights diagnosing problems with Xen hypervisors and rails applications. I met some amazing people and began to realize that St. Louis has some pretty good barbecue (but Texas still has them beat).

    Slicehost humor in 2009

    Slicehost humor in 2009

    Not long after that, I found myself managing an operations team that cared for Slicehost’s infrastructure and Rackspace’s growing Cloud Servers infrastructure. OpenStack appeared later and I jumped at the chance to do operations there. It was an extremely rough experience in the Diablo release, but it taught me a lot. My start with OpenStack involved fixing lots of broken Keystone tests that didn’t run on Python 2.6.

    Working on OpenStack in 2012

    Working on OpenStack in 2012

    If you’ve attended some of my talks on impostor syndrome, you may know what came next. We had a security issue and I sent some direct feedback to our CSO about how it was handled. I expected to be told to “pack a box” after that, but I was actually asked to lead a security architecture team in the corporate security group. It was definitely a surprise. I accepted and joined the team as Chief Security Architect. My coworkers called it “joining the dark side”, but I did my best to build bridges between security teams and the rest of the company.

    Talking at Rackspace::Solve in 2015

    Talking at Rackspace::Solve in 2015

    This role really challenged me. I had never operated at the Director level before and our team had a ton of work to do. I found myself stumbling (and floundering) fairly often and I leaned on other leaders in the business for advice. This led me to take some courses on critical thinking, accounting, finance, and tough conversations. I’ve never had a role as difficult as this one.

    Our cloud team came calling and asked me to come back and help with some critical projects in the public cloud. We worked on some awesome skunkworks projects that could really change the business. Although they didn’t get deployed in one piece, we found ways to take chunks of the work and optimize different areas of our work. An opportunity came up to bring public cloud experience to the private cloud team and I jumped on that one. I discovered the awesome OpenStack-Ansible project and a strong set of Rackers who were dedicated to bringing high-touch service to customers who wanted OpenStack in their own datacenter.

    Impostor syndrome talk at the Boston OpenStack Summit in 2017

    Impostor syndrome talk at the Boston OpenStack Summit in 2017

    During this time, I had the opportunity to deliver several conference talks about OpenStack, Fedora, security, and Ansible. My favorite topic was impostor syndrome and I set out on a mission to help people understand it. My first big talk was at the Fedora Flock conference in Rochester in 2015. This led to deep conversations with technical people in conference hallways, evening events, and even airport terminals about how impostor syndrome affects them. I took those conversations and refined my message several times over.

    Talking about impostor syndrome at Fedora Flock 2015

    Talking about impostor syndrome at Fedora Flock 2015 (Photo credit: Kushal Das)


    I couldn’t even begin to name a list of Rackers who have helped me along the way. I wouldn’t be where I am now without the help of hundreds of Rackers. They’ve taught me how to build technology, how to navigate a business, and how to be a better human. They have made me who I am today and I’m eternally grateful. I’ve had an incredible amount of hugs this week at the office and I’ve tried my best not to get a face full of tears in the process.

    I’d also like to thank all of the people who have allowed me to mentor them and teach them something along the way. One of the best ways to understand something is to teach it to someone else. I relish any opportunity to help someone avoid a mistake I made, or at least be able to throw something soft under them to catch their fall. These people put up with my thick Texas accent, my erratic whiteboard diagrams, and worse of all, my dad jokes.

    Another big “thank you” goes out to all of the members of the open source communities who have mentored me and dealt with my patches.

    The first big community I joined was the Fedora Linux community. I’ve been fortunate to serve on the board and participate in different working groups. Everyone has been helpful and accommodating, even when I pushed broken package builds. I plan to keep working in the community as long as they will have me!

    The OpenStack community has been like family. Everyone — from developers to foundation leaders — has truly been a treat to work with over several years. My work on Rackspace’s public and private clouds has pushed me into various projects within the OpenStack ecosystem and I’ve found everyone to be responsive. OpenStack events are truly inspiring and it is incredible to see so many people from so many places who dedicate themselves to the software and the people that make cloud infrastructure work.

    The next adventure

    I plan to talk more on this later, but I will be working from home on some projects that are entirely different from what I’m working on now. That adventure starts on March 19th after a week of “funemployment.” I’m incredibly excited about the new opportunity and I’ll share more details when I can.

    Top photo credit: Wikipedia

    The post Reaching the fork in the road appeared first on

    by Major Hayden at March 07, 2018 04:18 PM

    OpenStack Superuser

    Takeaways from the first Helm Summit

    Snow and freezing temperatures in Portland didn’t stop the inaugural Helm Summit from a well-attended two-day conference devoted to the Kubernetes packaging system.

    The first day was devoted to user stories, the use cases and techniques who have adopted Helm to manage packages for their Kubernetes infrastructure. Following on experiences from the first day, the second day got down to unconference work sessions devoted to shaping the upcoming design and development of Helm 3.  About 200 people braved the weather to attend.

    Adam Reese, one of the original Helm developers, kicked off the summit with a keynote that covered the history and philosophy of Helm. “Let’s solve problems instead of create more software” was the main takeaway that also doubles as kind of a mission statement for the project.

    The rest of the day included user stories from Ubisoft on how they use Helm to deliver core services to their games,  Josh Dolitsky of Codefresh on launching the open-source project ChartMuseum to be a Helm chart repository and Greg Taylor about how Helm backs the development-staging-production application life cycle at Reddit. Security and scalability were common themes of the talks.

    Day two was more focused on getting developers and users together to scope out requirements for Helm 3.
    Some highlights from the second day keynote:

    • Maintaining backwards compatibility with the wealth of existing Helm 2 charts
    • Cleaning up the code base by removing deprecated code
    • Refactoring code to make it cleaner, more readable, and less redundant
    • Introducing more role based access controls (RBAC) and trying to design with “sensible security” in mind
    • Making a commitment to excellent documentation to benefit the new and seasoned user alike

    From there, attendees gave a series of lightning talks, many focused on proposing new features for Helm 3 based on current needs and use cases. Building a strong community was also a topic, with a panel of Kubernetes Community leaders emphasizing that kindness was one of the most powerful tools a project can use to grow both talented individual developers and a strong community.

    As the day moved into afternoon work sessions, one theme that became clear was that within Helm there will always be a tension between the slower pace of Helm packaging and the faster pace of Kubernetes development. Backwards compatibility between not only Helm releases but for Kubernetes releases is important to K8s users whose organizations may be slower to adopt more recent releases of Kubernetes. The strength of open source projects that deliver applications depends on the strength of package managers, and Helm has already demonstrated itself as the essential Kubernetes package manager. It was exciting to participate in this event and I’m looking forward to seeing what changes are in store for Helm 3.

    For a deeper write-up of the event from one of the Helm core maintainers, head over to Taylor Thomas’s post at the Microsoft + Open Source blog.

    You can also catch all the talks on the Helm YouTube Channel.

    Get involved

    In May, I’ll be attending the first Kubernetes-SIG-Docs Summit in Portland, representing SIG-OpenStack and helping out with provider documentation. If you’re at the Write the Docs conference and want to get involved with Kubernetes documentation, be sure to stop by! Details on the Kubernetes #sig-docs channel or here.


    Chris Hoge is the OpenStack Foundation’s senior strategic program manager.


    Cover Photo // CC BY NC

    The post Takeaways from the first Helm Summit appeared first on Superuser.

    by Chris Hoge at March 07, 2018 03:54 PM

    Chris Dent

    TC Report 18-10

    This is a TC Report, but since everything that happened in its window of observation is preparing for the PTG, being at the PTG, trying to get home from the PTG, and recovering from the PTG, perhaps think of this as "What the TC talked about [at] the PTG". As it is impossible to be everywhere at once (especially when the board meeting overlaps with other responsibilities) this will miss a lot of important stuff. I hope there are other summaries.

    As you may be aware, it snowed in Dublin causing plenty of disruption to the PTG but everyone (foundation staff, venue staff, hotel staff, attendees, uisce beatha) worked together to make a good week.

    Talking about the PTG at the PTG

    At the board meeting, the future of the PTG was a big topic. As currently constituted it presents some challenges:

    • It is difficult for some people to attend because of visa and other travel related issues.
    • It is expensive to run and not everyone is convinced of the return on investment.
    • Some people don't like it (they either miss the old way of doing the design summit, or midcycles, or $OTHER).
    • Plenty of other reasons that I'm probably not aware of.

    This same topic was reviewed at yesterday's office hours.

    For now, the next 2018 PTG is going to happen (destination unknown) but plans for 2019 are still being discussed.

    If you have opinions about the PTG, there will be an opportunity to express them in a forthcoming survey. Beyond that, however, it is important that management at contributing companies hear from more people (notably their employees) than the foundation about the value of the PTG.

    My own position is that of the three different styles of in-person events for technical contributors to OpenStack that I've experienced (design summit, mid-cycles, PTG), the PTG is the best yet. It minimizes distractions from other obligations (customer meetings, presentations, marketing requirements) while maximizing cross-project interaction.

    One idea, discussed yesterday and earlier today was to have the PTG be open to technical participants of any sort, not just so-called "OpenStack developers". Make it more of a place for people who hack on and with OpenStack to hack and talk. Leave the summit (without a forum) for presentations, marketing, pre-sales, etc.

    An issue raised with conflating the PTG and the Forum is that it would remove the inward/outward focus concept that is supposed to distinguish the two events.

    I guess it depends on how we define "we" but I've always assumed that both events were for outward focus and that for any inward focussing effort we ought to be able use asynchronous tools more.

    Foundation and OCI

    Thierry mentioned yesterday that it is likely that the OpenStack Foundation will join the Open Container Initiative because of Kata and LOCI.

    This segued into some brief concerns about the attentions and intentions of the Foundation, aggravated by the board meeting schedule conflict (there's agreement that will never ever happen again), and the rumor milling about the PTG.

    Friday at the PTG with the TC

    The TC had scheduled a half day of discussion for Friday at the PTG. A big agenda, a fun filled week, and the snow meant we went nearly all day (and since there's no place to go, let's talk, let's talk, let's talk) with some reasonable progress. Some highlights:

    • There was some discussion on trying to move forward with constellations concept, but I don't recall specific outcomes from that discussion.

    • The team diversity tags need to be updated to reflect adjustments in the very high bars we set earlier in the history of OpenStack. We agreed to not remove projects from the tc-approved tag, as that could be taken the wrong way. Instead we'll create a new tag for projects that are in the trademark program.

    • Rather than having Long Term Support, which implies too much, a better thing to do is enable extended maintenance for those parties who want to do it.

    • Heat was approved to be a part of the trademark program, but then there were issues with where to put their tests and the tooling used to manage them. By the power of getting the right people in the room at the same time, we reached some consensus which is being finalized on a proposed resolution.

    • We need to make an official timeline for the deprecation (and eventual removal) of support for Python 2, meaning we also need to accelerate the adoption of Python 3 as the primary environment.

    • In a discussion about the availability of etcd it was decided that tooz needs to be finished.

    See the etherpad for additional details.

    by Chris Dent at March 07, 2018 12:10 PM

    March 06, 2018

    Adam Young

    Generating a Callgraph for Keystone

    Once I know a starting point for a call, I want to track the other functions that it calls. pycallgraph will generate an image that shows me that.

    All this is done inside the virtual env set up by tox at keystone/.tox/py35

    I need a stub of a script file in order to run it. I’ll put this in tmp:

    from keystone.identity import controllers
    from keystone.server import wsgi
    from keystone.common import request
    def main():
        d = dict()
        r  = request.Request(d)
        c = controllers.UserV3()
    if __name__ == '__main__':

    To install pycallgraph:

    pip install pycallgraph

    And to run it:

     pycallgraph  --max-depth 6  graphviz /tmp/ 

    It errors out do to auth issues (it is actually rtunning the code, so don’t do this on a production server)

    Here is what it generated.

    Click to enlarge. Not great, but it is a start.

    by Adam Young at March 06, 2018 09:53 PM

    Inspecting Keystone Routes

    What Policy is enforced when you call a Keystone API? Right now, there is no definitive way to say. However, with some programmatic help, we might be able to figure it out from the source code. Lets start by getting a complete list of the Keystone routes.

    In the WSGI framework that Keystone uses, a Route is the object that used to match the URL. For example, when I try to look at the user with UserId abcd1234, I submit a GET request to the URL https://hostname:port/v3/users/abcd1234. The route path is the pattern /users/{user_id}. The WSGI framework handles the parts of the URL prior to that, and eventually needs to pull out a Python function to execute for the route. Here is how we can generate a list of the route paths in Keystone

    from keystone.server import wsgi
    app = wsgi.initialize_admin_application()
    composing = app['/v3'].application.application.application.application.application.application._app.application.application.application.application
    for route in composing._router.mapper.matchlist:

    I’ll put the output at the end of this post.

    That long chain of .application properties is due to the way that the pipeline is built using the paste file. In keystone/etc/keystone-paste.ini we see:

    # The last item in this pipeline must be service_v3 or an equivalent
    # application. It cannot be a filter.
    pipeline = healthcheck cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3

    Each of those pipeline elements are python classes specified earlier in the file, that honor the middleware contract. Most of these can be traced back to the keystone.common.wsgi.Middleware base class, which implements this as __call__ method.

        def __call__(self, request):
            response = self.process_request(request)
            if response:
                return response
            response = request.get_response(self.application)
            return self.process_response(request, response)

    The odd middleware out is AuthContextMiddleware which extends from from keystonemiddleware.auth_token.BaseAuthProtocol. See if you can spot the difference:

        def __call__(self, req):
            """Handle incoming request."""
            response = self.process_request(req)
            if response:c
                return response
            response = req.get_response(self._app)
            return self.process_response(response

    Yep: self._app.

    Here is the output from the above code, executed in the python interpreter. This does not have the Verbs in it yet, but a little more poking should show where they are stored:

    >>> for route in composing._router.mapper.matchlist:
    ...     print(route.routepath)

    by Adam Young at March 06, 2018 05:17 PM

    OpenStack Superuser

    Pushing embedded devices closer to the edge

    Lattice Semiconductor is a Silicon Valley maker of programmable logic devices founded in 1983 (yep, that’s the “Halt and Catch Fire” era), that sees the future in edge computing.

    “AI and the cloud is old news, it’s been implemented over the last few years,” says Deepak Boppana, senior director, product and segment marketing at Lattice, said at the recent edition of Embedded World held in Nuremberg, Germany. “The move of AI is towards the edge: that’s the future.”

    To show how they’re pushing closer to that future, Boppana has been on the conference circuit with the company’s low-power, wireless field-programmable gate array (FPGA) devices.  And when they say low power, they mean low power: some of the solutions are in the range of milliwatts — as in your average laser pointer.

    At the 2018 edition of the Consumer Electronics Show (CES), he showcased this demo of vision applications for collision avoidance — with the toy robot gracefully moving around cones without hitting any.

    The autonomous robot glides without incident across the factory floor using under one watt of power, Boppana says, adding “that’s really key for battery powered applications at the edge.” A wireless HD link uploads the video of the robot to its human overlords. Some key specs:


    Other solutions Lattice is working on include AI face-based tracking (it works for objects, too) that could be implemented with drones or surveillance cameras and facial recognition solution.  That low power iCE40 UltraPlus FPGA performs edge computing applications locally on a device, sending metadata to the cloud but does that despite consuming than a milliwatt of power. Lattice offerings also feature a host of city applications — cameras that could be used for traffic flow monitoring, driving violations, smart parking and toll collection.

    Check out the interview from Embedded World 2018 (and take a look the robots!) below. And you can catch up on these IRL at the upcoming Embedded Vision Conference in May 2018.

    Via Electronics Weekly

    The post Pushing embedded devices closer to the edge appeared first on Superuser.

    by Superuser at March 06, 2018 04:01 PM

    March 05, 2018

    OpenStack Superuser

    OpenStack Community Contributor Awards now open for nominations

    So many folks work tirelessly behind the scenes to make OpenStack great, whether they are fixing bugs, contributing code, helping newbies on IRC or just making everyone laugh at the right moment.

    You can help them get recognized (with a very shiny medal!) by nominating them for the next Contributor Awards given out at the upcoming OpenStack Summit Vancouver. These are informal, quirky awards — winners in previous editions included the “Duct Tape” award and the “Don’t Stop Believin’ Cup” — that shine a light on the extremely valuable work that makes OpenStack excel.

    There’s so many different areas worthy of celebration, but there are a few kinds of community members who deserve a little extra love:

    • They are undervalued
    • They don’t know they are appreciated
    • They bind the community together
    • They keep it fun
    • They challenge the norm
      Other: (write-in)

    As before, rather than starting with a defined set of awards, the community is asked to submit  names in those broad categories. The OSF community team then has a little bit of fun on the back end, massaging the award titles to devise something worthy of their underappreciated efforts.


    “There are A LOT of people out there who could use a pat on the back and the affirmation that they do good work for the community, ” says OSF’s Kendall Nelson. Please nominate anyone you think is deserving of an award! Deadline is May 14. Vancouver

    Awards will be presented during the feedback session at the Summit.

    Cover Photo // CC BY NC

    The post OpenStack Community Contributor Awards now open for nominations appeared first on Superuser.

    by Superuser at March 05, 2018 12:15 PM

    March 04, 2018

    Rich Bowen


    I’m heading home from SnowpenStack and it was quite a ride. As Theirry said in our interview at the end of Friday (coming soon to a YouTube channel near you), rather than spoiling things, the freak storm and subsequent closure of the event venue served to create a shared experience and camaraderie that made it even better.

    In the end I believe I got 29 interviews, and I’ll hopefully be supplementing this with a dozen online interviews in the coming weeks.

    If you missed your interview, or weren’t at the PTG, please contact me and we’ll set something up. And I’ll be in touch with all of the PTLs who were not already represented in one of my interviews.

    A huge thank you to everyone that made time to do an interview, and to Erin and Kendall for making everything onsite go so smoothly.

    by rbowen at March 04, 2018 07:52 AM

    March 02, 2018

    OpenStack Superuser

    How a large multinational seeds cloud native

    By any measure, the scale of Huawei is impressive. To keep things humming for its 180,000 employees worldwide, the internal IT department of the Shenzen-based  multinational telco giant runs eight data centers, powering over 800 applications and 100,000 virtual machines. Cost and agility were key factors in deciding the future of that system.

    Huawei’s Kubernetes journey began two years ago with one developer, who started contributing code to the open-source project and quickly became an internal evangelist.  It seemed like a good solution for the internal IT department, which needs to launch tens of thousands of containers. Because those containerspower tasks across thousands of global nodes, managing them in a consistent way is always a challenge, says Peixin Hou, Huawei’s chief software architect and community director for open source, in a recent case study.  About 30 percent of them are now cloud native.

    It’s hardly the first time Huawei embraced open source.  A gold member of the OpenStack Foundation, Huawei offers  customers a carrier-grade OpenStack distribution supporting NFV deployments and Huawei FusionSphere to deploy private, public and hybrid clouds.  They’re also a top contributor to Hadoop, OPNFV, a founding member of the Cloud Native Computing Foundation (CCNF) and employ Spark. In addition to an online lab and courses, the company’s website also offers an open source release center where anyone can download or request open source code found in its phones, computers, wearables and smart home devices.

    Although took a year to adopt that engineer’s suggestion – the process “is not overnight,” says Hou – but once in use, he says: “Kubernetes basically solved most of our problems. Before, the time of deployment took about a week, now it only takes minutes. The developers are happy. That department is also quite happy.”

    The results were impressive enough that the company also decided to become a vendor in 2016 with FusionStage, a platform-as-a-service offering. Huawei’s customers, from very big telecommunications operators to banks, love the idea of cloud native, Hou says.

    After overseeing the initial move to Kubernetes at Huawei, Hou has some advice for other companies considering the technology: “When you start to design the architecture of your application, think about cloud native, think about microservice architecture from the beginning,” he says.

    For companies dealing with legacy apps, Hou advises to dip a toe in with the microservice-friendly part of those applications first, parts that are relatively easy to be decomposed into simpler pieces and are relatively lightweight.

    More from the full case study here.


    Cover image: Huawei Shenzen R&D center, courtesy Huawei.

    The post How a large multinational seeds cloud native appeared first on Superuser.

    by Superuser at March 02, 2018 04:52 PM


    March 1 Blogpost Roundup

    It’s been a busy few weeks of blogging! Thanks as always to those of you who continue to write great content.

    OpenStack Role Assignment Inheritance for CloudForms by Adam Young

    Operators expect to use CloudForms to perform administrative tasks. For this reason, the documentation for OpenStack states that the Keystone user must have an ‘admin’ role. We found at least one case, however, where this was not sufficient. Fortunately, we have a better approach, and one that can lead to success in a wider array of deployments.


    Listening for connections on all ports/any port by Lars Kellogg-Stedman

    On IRC — and other online communities — it is common to use a “pastebin” service to share snippets of code, logs, and other material, rather than pasting them directly into a conversation. These services will typically return a URL that you can share with others so that they can see the content in their browser.


    Grouping aggregation queries in Gnocchi 4.0.x by Lars Kellogg-Stedman

    In this article, we’re going to ask Gnocchi (the OpenStack telemetry storage service) how much memory was used, on average, over the course of each day by each project in an OpenStack environment.


    TripleO deep dive session #12 (config-download) by Carlos Camacho

    This is the 12th release of the TripleO “Deep Dive” sessions. In this session we will have an update for the TripleO ansible integration called config-download. It’s about applying all the software configuration with Ansible instead of doing it with the Heat agents.


    Maximizing resource utilization with Preemptible Instances by Theodoros Tsioutsias

    The CERN cloud consists of around 8,500 hypervisors providing over 36,000 virtual machines. These provide the compute resources for both the laboratory’s physics program but also for the organisation’s administrative operations such as paying bills and reserving rooms at the hostel.


    Testing TripleO on own OpenStack deployment by mrunge

    For some use cases, it’s quite useful to test TripleO deployments on a OpenStack powered cloud, rather than using a baremetal system. The following article will show you how to do it.


    A New Thing by Andrew Beekhof

    If you’re interested in Kubernetes and/or managing replicated applications, such as Galera, then you might also be interested in an operator that allows this class of applications to be managed natively by Kubernetes.


    Two Nodes – The Devil is in the Details by Andrew Beekhof

    tl;dr – Many people love 2-node clusters because they seem conceptually simpler and 33% cheaper, but while it’s possible to construct good ones, most will have subtle failure modes.


    by Mary Thengvall at March 02, 2018 01:08 AM

    March 01, 2018

    Corey Bryant

    OpenStack Queens for Ubuntu 16.04 LTS

    Hi All,

    The Ubuntu OpenStack team at Canonical is pleased to announce the general availability of OpenStack Queens on Ubuntu 16.04 LTS via the Ubuntu Cloud Archive. Details of the Queens release can be found at:

    To get access to the Ubuntu Queens packages:

    Ubuntu 16.04 LTS

    You can enable the Ubuntu Cloud Archive pocket for OpenStack Queens on Ubuntu 16.04 installations by running the following commands:

    sudo add-apt-repository cloud-archive:queens
    sudo apt update

    The Ubuntu Cloud Archive for Queens includes updates for:

    aodh, barbican, ceilometer, ceph (12.2.2), cinder, congress, designate, designate-dashboard, dpdk (17.11), glance, glusterfs (3.13.2), gnocchi, heat, heat-dashboard, horizon, ironic, keystone, libvirt (4.0.0), magnum, manila, manila-ui, mistral, murano, murano-dashboard, networking-bagpipe, networking-bgpvpn, networking-hyperv, networking-l2gw, networking-odl, networking-ovn, networking-sfc, neutron, neutron-dynamic-routing, neutron-fwaas, neutron-lbaas, neutron-lbaas-dashboard, neutron-taas, neutron-vpnaas, nova, nova-lxd, openstack-trove, openvswitch (2.9.0), panko, qemu (2.11), rabbitmq-server (3.6.10), sahara, sahara-dashboard, senlin, swift, trove-dashboard, vmware-nsx, watcher, and zaqar.

    For a full list of packages and versions, please refer to [0].

    Branch Package Builds

    If you would like to try out the latest updates to branches, we deliver continuously integrated packages on each upstream commit via the following PPA’s:

    sudo add-apt-repository ppa:openstack-ubuntu-testing/mitaka
    sudo add-apt-repository ppa:openstack-ubuntu-testing/newton
    sudo add-apt-repository ppa:openstack-ubuntu-testing/ocata
    sudo add-apt-repository ppa:openstack-ubuntu-testing/pike
    sudo add-apt-repository ppa:openstack-ubuntu-testing/queens

    Reporting bugs

    If you have any issues please report bugs using the ‘ubuntu-bug’ tool to ensure that bugs get logged in the right place in Launchpad:

    sudo ubuntu-bug nova-conductor

    Thanks to everyone who has contributed to OpenStack Queens, both upstream and downstream!

    Have fun and see you in Rocky!

    (on behalf of the Ubuntu OpenStack team)


    by coreycb at March 01, 2018 09:31 PM

    OpenStack Superuser

    Standards tests show that network functions virtualization has VIM

    In the first post of this series, I shared a summary about the week I spent in France as part of the European Telecommunications Standards Institute‘s (ETSI) second edition of NFV Plugtests.

    Now that the official results are out, let’s dive in!

    A quick summary: 95 percent of the 189 interoperability test sessions succeeded. (A  ‘session’ means a combination of different vendor’s VNF, VIM+NFVI and MANO solutions.) Overall, only about 30 percent of the features were not yet supported by one of the involved participants, compared to 61 percent at the first Plugtest session. This speaks very well to NFV readiness, considering that more than 50 percent of additional features were considered for this second edition, including multi-vendor network services, fault and performance management, enhanced platform awareness and multi-site deployments.

    Note: all graphics and tables are as published in the second NFV Plugtestsreport

    Here, I’ll focus on the results related to the NFVI and virtualized infrastructure manager (VIM, a.k.a. telco cloud), which as you may already know, are the physical compute, storage and network resources (NFVI), as well as their manager, that provide  a cloud environment for virtualized network functions.

    Twelve different NFVI+VIM combinations were tested, includin 11 with different OpenStack distributions (from seven VIM vendors) and one with VMWare’s vCloud Director.

    Since NFVI is expected to be based on off-the-shelf generic servers with homogeneous features, most challenges were presented to the VIM software. The following tables show some examples of the kind of tests that were included.
    There were basic tests like these, requesting basic lifecycle management:

    … and more advanced ones, like these, requesting autoscaling based on VIM metrics:

    As you can see, test descriptions cover the NFV system as whole, but, how do these tests relate to specific features that VIM platforms had to support?

    VIMs were expected to support:

      • Standard APIs: to provide a way for MANO platforms to connect and orchestrate VIM functions.
      • Standard hypervisor features: to be ready to support different VM image formats that VNFs may require.
      • Complete VM lifecycle: to be able to start, pause, stop and destroy VMs, along with their compute, networking and storage resources.
      • Networking: to provide standard networking features for VMs to interconnect in a number of ways: private and public networks, NAT, security features, etc.
      • Performance management: to collect and store performance metrics for all VMs, while being able to set thresholds and alarms for any of them, so that MANO platforms can monitor KPIs and trigger actions.
      • Fault Management: to generate events and alarms and be able to propagate them to higher layers.
      • Enhanced Platform Awareness (EPA): to be able to set EPA-related configurations on NFVI resources (SR-IOV, CPU pinning, NUMA topology awareness, Huge pages, etc.)

    Cloud platforms are currently mature enough so most of the features are completely supported by both OpenStack distributions and VMWare vCD. Maybe the only one that today still presents some challenges at the VIM layer is EPA management, since this is a recent concept created during the NFV evolution to optimize the NFVI performance using techniques that should work with generic servers.

    During these Plugtests, EPA testing was included in the optional set of sessions and the actual feature to test was not specified, so participants could choose to test simple control-plane related features like Huge Pages or CPU Pinning, or choose more complex, data-plane related features like SR-IOV, OVS-DPDK.

    As evidenced in the following table, for Multi-VNF EPA there were only eight sessions, which included 26 tests. Only 12 could actually be run, while 14 asked for features that were not supported by participants. Out of these 12 tests, nine succeeded and three exposed interoperability issues. In short, only about a third of the EPA tests were successful.

    We participated with Whitestack’s VIM solution, WhiteCloud, a multi-site telco cloud environment based on OpenStack Pike over generic servers and switches. I’m happy to report that it passed most tests successfully when integrating with different MANO solutions and VNFs. We didn’t focus our testing efforts for data-plane related EPA features this time, but we expect to disclose full (and successful) results in the third edition of these tests, expected to take place in May 2018.

    In conclusion, because most features and interoperability worked fine for participants, we can say that at the NFVI/VIM level, NFV is ready for prime time! Of course, new features will keep being developed at both standards and by vendors at different speeds, which increases the importance for continuing this event.

    For production environments, additional attention must be paid when evaluating telco cloud solutions that require high throughput from servers. Data-plane acceleration features of the NFVI and VIM must be carefully assessed and tested, including the most popular alternatives: OVS-DPDK and SR-IOV.

    Stay tuned for the final post in this series, where I’ll share the results from the MANO perspective. Thanks for reading!

    About the author

    Gianpietro Lavado is a network solutions architect interested in the latest software technologies to achieve efficient and innovative network operations. He currently works at Whitestack, a company whose mission is to promote SDN, NFV, cloud and related deployments all around the world, including a new OpenStack distribution.

    This post first appeared on LinkedIn. Superuser is always interested in community content, get in touch at:

    Cover Photo // CC BY NC

    The post Standards tests show that network functions virtualization has VIM appeared first on Superuser.

    by Gianpietro Lavado at March 01, 2018 04:19 PM


    RDO Queens Released

    The RDO community is pleased to announce the general availability of the RDO build for OpenStack Queens for RPM-based distributions, CentOS Linux 7 and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Queens is the 17th release from the OpenStack project, which is the work of more than 1600 contributors from around the world (source – ).

    RDO team doing the release at the PTG

    ]2 RDO team doing the release at the PTG

    The release is making its way out to the CentOS mirror network, and should be on your favorite mirror site momentarily.

    The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds.

    All work on RDO, and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first.

    New and Improved

    Interesting things in the Queens release include:

    • Ironic now supports Neutron routed networks with flat networking and introduces support for Nova traits when scheduling
    • RDO now includes rsdclient, an OpenStack client plugin for Rack Scale Design architecture
    • Support for octaviaclient and Octavia Horizon plugin has been added to improve Octavia service deployments.
    • Tap-as-a-Service (TaaS) network extension to the OpenStack network service (Neutron) has been included.
    • Multi-vendor Modular Layer 2 (ML2) driver networking-generic-switch si now available of operators deploying RDO Queens.

    Other improvements include:

    • Most of the bundled intree tempest plugins have been moved to their own repository during Queens cycle. RDO has adapted plugin packages for these new model.
    • In an effort to improve the quality and reduce the delivery time for our users, RDO keeps refining and automating all required processes needed to build, test and publish the packages included in RDO distribution.

    Note that packages for OpenStack projects with cycle-trailing release models[] will be created after a release is delivered according to the OpenStack Queens schedule. []


    During the Queens cycle, we saw the following new contributors:

    • Aditya Ramteke
    • Jatan Malde
    • Ade Lee
    • James Slagle
    • Alex Schultz
    • Artom Lifshitz
    • Mathieu Bultel
    • Petr Viktorin
    • Radomir Dopieralski
    • Mark Hamzy
    • Sagar Ippalpalli
    • Martin Kopec
    • Victoria Martinez de la Cruz
    • Harald Jensas
    • Kashyap Chamarthy
    • dparalen
    • Thiago da Silva
    • chenxing
    • Johan Guldmyr
    • David J Peacock
    • Sagi Shnaidman
    • Jose Luis Franco Arza

    Welcome to all of you, and thank you so much for participating!

    But, we wouldn’t want to overlook anyone. Thank you to all 76 contributors who participated in producing this release. This list includes commits to rdo-packages and rdo-infra repositories, and is provided in no particular order:

    • Yatin Karel
    • Aditya Ramteke
    • Javier Pena
    • Alfredo Moralejo
    • Christopher Brown
    • Jon Schlueter
    • Chandan Kumar
    • Haikel Guemar
    • Emilien Macchi
    • Jatan Malde
    • Pradeep Kilambi
    • Luigi Toscano
    • Alan Pevec
    • Eric Harney
    • Ben Nemec
    • Matthias Runge
    • Ade Lee
    • Jakub Libosvar
    • Thierry Vignaud
    • Alex Schultz
    • Juan Antonio Osorio Robles
    • Mohammed Naser
    • James Slagle
    • Jason Joyce
    • Artom Lifshitz
    • Lon Hohberger
    • rabi
    • Dmitry Tantsur
    • Oliver Walsh
    • Mathieu Bultel
    • Steve Baker
    • Daniel Mellado
    • Terry Wilson
    • Tom Barron
    • Jiri Stransky
    • Ricardo Noriega
    • Petr Viktorin
    • Juan Antonio Osorio Robles
    • Eduardo Gonzalez
    • Radomir Dopieralski
    • Mark Hamzy
    • Sagar Ippalpalli
    • Martin Kopec
    • Ihar Hrachyshka
    • Tristan Cacqueray
    • Victoria Martinez de la Cruz
    • Bernard Cafarelli
    • Harald Jensas
    • Assaf Muller
    • Kashyap Chamarthy
    • Jeremy Liu
    • Daniel Alvarez
    • Mehdi Abaakouk
    • dparalen
    • Thiago da Silva
    • Brad P. Crochet
    • chenxing
    • Johan Guldmyr
    • Antoni Segura Puimedon
    • David J Peacock
    • Sagi Shnaidman
    • Jose Luis Franco Arza
    • Julie Pichon
    • David Moreau-Simard
    • Wes Hayutin
    • Attila Darazs
    • Gabriele Cerami
    • John Trowbridge
    • Gonéri Le Bouder
    • Ronelle Landy
    • Matt Young
    • Arx Cruz
    • Joe H. Rahme
    • marios
    • Sofer Athlan-Guyot
    • Paul Belanger

    Getting Started

    There are two ways to get started with RDO.

    To spin up a proof of concept cloud, quickly, and on limited hardware, try an All-In-One Packstack installation. You can run RDO on a single node to get a feel for how it works. For a production deployment of RDO, use the TripleO Quickstart and you’ll be running a production cloud in short order.

    Getting Help

    The RDO Project participates in a Q&A service at We also have our for RDO-specific users and operrators. For more developer-oriented content we recommend joining the mailing list. Remember to post a brief introduction about yourself and your RDO story. The mailng lists archives are all available at You can also find extensive documentation on the RDO docs site.

    The #rdo channel on Freenode IRC is also an excellent place to find help and give help.

    We also welcome comments and requests on the CentOS mailing lists and the CentOS and TripleO IRC channels (#centos, #centos-devel, and #tripleo on, however we have a more focused audience in the RDO venues.

    Getting Involved

    To get involved in the OpenStack RPM packaging effort, see the RDO community pages and the CentOS Cloud SIG page. See also the RDO packaging documentation.

    Join us in #rdo on the Freenode IRC network, and follow us at @RDOCommunity on Twitter. If you prefer Facebook, we’re there too, and also Google+.

    by Rich Bowen at March 01, 2018 10:05 AM

    SWITCH Cloud Blog

    Deploy Kubernetes on the SWITCHengines Openstack cloud

    Increasing demand for container orchestration tools is coming from our users. Kubernetes has currently a lot of hype, and often it comes the question if we are providing a Kubernetes cluster at SWITCH.

    At the moment we suggest that our users deploy their own Kubernetes cluster on top of SWITCHengines. To make sure our Openstack deployment works with this solution we tried ourself.

    After deploying manually with kubeadm to learn the tool, I found a well written ansible playbook from Francois Deppierraz. I extended the playbook to make Kubernetes aware that SWITCHengines implements the LBaaSv2, and the patch is now merged in the original version.

    The first problem I discovered deploying Kubernetes is the total lack of support for IPv6. Because instances in SWITCHengines get IPv6 addresses by default, I run into problems running the playbook and nothing was working. The first thing you should do is to create your own tenant network with a router, with IPv4 only connectivity. This is already explained in detail in our standard documentation.

    Now we are ready to clone the ansible playbook:

    git clone

    Because the ansible playbook creates instances through the Openstack API, you will have to source your Openstack configuration file. We extend a little bit the usual configuration file with more variables that are specific to this ansible playbook. Lets see a template:

    export OS_USERNAME=username
    export OS_PASSWORD=mypassword
    export OS_PROJECT_NAME=myproject
    export OS_PROJECT_ID=myproject_uuid
    export OS_AUTH_URL=
    export OS_REGION_NAME=ZH
    export KEY=keyname
    export IMAGE="Ubuntu Xenial 16.04 (SWITCHengines)"
    export NETWORK=k8s
    export SUBNET_UUID=subnet_uuid
    export FLOATING_IP_NETWORK_UUID=network_uuid

    Lets review what changes. It is important to add also the variable OS_PROJECT_ID because the Kubernetes code that creates Load Balancers requires this value, and it is not able to extract it from the project name. To find the uuid just use the Openstack cli:

    openstack project show myprojectname -f value -c id

    The KEY is the name of an existing keypair that will be used to start the instances. The IMAGE is also self explicative, at the moment only Xenial is tested by me. The variable NETWORK is the name of the tenant network you created earlier. When you created a network you created also a subnet, and you need to set the uuid into SUBNET_UUID. The last variable is FLOATING_IP_NETWORK_UUID that tells kubernetes the network where to get floating IPs. In SWITCHengines this network is always called public, so you can extract the uuid like this:

    openstack network show public -f value -c id

    You can customize your configuration even more, reading the README file in the git repository you will find more options like the flavors to use or the cluster size. When your configuration file is ready you can run the playbook:

    source /path/to/config_file
    cd k8s-on-openstack
    ansible-playbook site.yaml

    It will take a few minutes to go through all the tasks. When everything is done you can ssh into the kubernetes master instance and check that everything is running as expected:

    ubuntu@k8s-master:~$ kubectl get nodes
    NAME         STATUS    AGE       VERSION
    k8s-1        Ready     2d        v1.6.2
    k8s-2        Ready     2d        v1.6.2
    k8s-3        Ready     2d        v1.6.2
    k8s-master   Ready     2d        v1.6.2

    I found very useful adding bash completion for kubectl:

    source <(kubectl completion bash)

    Lets deploy an instance of nginx to test if everything works:

    kubectl run my-nginx --image=nginx --replicas=2 --port=80

    This will create two containers with nginx. You can monitor the progress with the commands:

    kubectl get pods
    kubectl get events

    At this stage you have your containers running but the service is still not accessible from the outside. One option is to use the Openstack LBaaS to expose it, you can do it with this command:

    kubectl expose deployment my-nginx --port=80 --type=LoadBalancer

    The expose command will create the Openstack Load Balancer and will configure it. To know the public floating ip address you can use this command to describe the service:

    ubuntu@k8s-master:~$ kubectl describe service my-nginx
    Name:			my-nginx
    Namespace:		default
    Labels:			run=my-nginx
    Selector:		run=my-nginx
    Type:			LoadBalancer
    LoadBalancer Ingress:,
    Port:				80/TCP
    NodePort:			30620/TCP
    Session Affinity:	None
      FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
      ---------	--------	-----	----			-------------	--------	------			-------
      1m		1m		1	service-controller			Normal		CreatingLoadBalancer	Creating load balancer
      10s		10s		1	service-controller			Normal		CreatedLoadBalancer	Created load balancer


    Following this blog post you should be able to deploy Kubernetes on Openstack to understand how things work. For a real deployment you might want to make some customisations, we encourage you to share any patch to the ansible playbook with github pull requests.
    Please note that Kubernetes is not bug free. When you will delete your deployment you might find this bug where Kubernetes is not able to delete correctly the load balancer. Hopefully this is fixed by the time you read this blog post.

    by Saverio Proto at March 01, 2018 12:23 AM

    Carlos Camacho

    My 2nd birthday as a Red Hatter

    This post will be about to speak about my experience working in TripleO as a Red Hatter for the last 2 years. In my 2nd birthday as a Red Hatter, I have learned about many technologies, really a lot… But the most intriguing thing is that here you never stop learning. Not just because you just don’t want to learn new things, instead, is because of the project’s nature, this project… TripleO…

    TripleO (Openstack On Openstack) is a software aimed to deploy OpenStack services using the same OpenStack ecosystem, this means that we will deploy a minimal OpenStack instance (Undercloud) and from there, deploy our production environment (Overcloud)… Yikes! What a mouthful, huh? Put simply, TripleO is an installer which should make integrators/operators/developers lives easier, but the reality sometimes is far away from the expectation.

    TripleO is capable of doing wonderful things, with a little of patience, love, and dedication, your hands can be the right hands to deploy complex environments at easy.

    One of the cool things being one of the programmers who write TripleO, from now on TripleOers, is that many of us also use the software regularly. We are writing code not just because we are told to do it, but because we want to improve it for our own purposes.

    Part of the programmers’ motivation momentum have to do with TripleO’s open‐source nature, so if you code in TripleO you are part of a community.

    Congratulations! As a TripleO user or a TripleOer, you are a part of our community and it means that you’re joining a diverse group that spans all age ranges, ethnicities, professional backgrounds, and parts of the globe. We are a passionate bunch of crazy people, proud of this “little” monster and more than willing to help others enjoy using it as much as we do.

    Getting to know the interface (the templates, Mistral, Heat, Ansible, Docker, Puppet, Jinja, …) and how all components are tight together, probably is one of the most daunting aspects of TripleO for newcomers (and not newcomers). This for sure will raise the blood pressure of some of you who tried using TripleO in the past, but failed miserably and gave up in frustration when it did not behave as expected. Yeah.. sometimes that “$h1t” happens…

    Although learning TripleO isn’t that easy, the architecture updates, the decoupling of the role services “compostable roles”, the backup and restore strategies, the integration of Ansible among many others have made great strides toward alleviating that frustration, and the improvements continue through to today.

    So this is the question…

    Is TripleO meant to be “fast to use” or “fast to learn”?

    There is a significant way of describing software products, but we need to know what our software will be used for… TripleO is designed to work at scale, it might be easier to deploy manually a few controllers and computes, but what about deploying 100 computes, 3 controllers and 50 cinder nodes, all of them configured to be integrated and work as one single “cloud”? Buum!. So there we find the TripleO benefits if we want to make it scale we need to make it fast to use…

    This means that we will find several customizations, hacks, workarounds, to make it work as we need it.

    The upside to this approach is that TripleO evolved to be super-ultra-giga customizable so operators are enabled to produce great environments blazingly fast..

    The downside, Jaja, yes.. there is a downside “or several”. As with most things that are customized, TripleO became somewhat difficult for new people to understand. Also, it’s incredibly hard to test all the possible deployments, and when a user does non-standard or not supported customizations, the upgrades are not as intuitive as they need…

    This trade‐off is what I mean when I say “fast to use versus fast to learn.” You can be extremely productive with TripleO after you understand how it thinks “yes, it thinks”.

    However, your first few deployments and patches might be arduous. Of course, alleviating that potential pain is what our work is about. IMHO the pros are more than the cons and once you find a niche to improve it will be a really nice experience.

    Also, we have the TripleO YouTube channel a place to push video tutorials and deep dive sessions driven by the community for the community.

    For the Spanish community we have a 100% translated TripleO UI, go to and help us to reach as many languages as possible!!! was born on July 5th of 2016 (first GitHub commit), yeah is my way of expressing my gratitude to the community doing some CtrlC + CtrlV recipes to avoid the frustration of working with TripleO and not having something deployed and easy to be used ASAP.

    Anstack does not have much traffic but it reached, the TripleO cheatsheets were on and FOSDEM, so in general, is really nice. When people reference your writings anywhere. Maybe in the future can evolve to be more related to ANsible and openSTACK ;) as TripleO is adding more and more support for Ansible.

    What about Red Hat? Yeahp, I have a long time speaking about the project but haven’t spoken about the company making it all real. Red Hat is the world’s leading provider of open source solutions, using a community-powered approach to provide reliable and high-performing cloud, virtualization, storage, Linux, and middleware technologies.

    There is a strong feeling of belonging in Red Hat, you are part of a team, a culture and you are able to find a perfect balance between your work and life. Also, having all people from all over the globe makes a perfect place for sharing ideas and collaborate. Not all of it is good, i.e. Working mostly remotely in upstream communities can be really hard to manage if you are not 100% sure about the tasks that need to be done.

    Keep rocking and become part of the TripleO community!

    by Carlos Camacho at March 01, 2018 12:00 AM

    February 28, 2018

    Adam Young

    OpenStack Role Assignment Inheritance for CloudForms

    Operators expect to use CloudForms to perform administrative tasks. For this reason, the documentation for OpenStack states that the Keystone user must have an ‘admin’ role. We found at least one case, however, where this was not sufficient. Fortunately, we have a better approach, and one that can lead to success in a wider array of deployments.


    CloudForms uses the role assignments for the give user account to enumerate the set of projects. Internally it creates a representation of these projects to be used to track resources. However, The way that ‘admin’ is defined on OpenStack is tied to a single project. This means that CloudForms really has no way to ask “what projects can this user manage?” Now, while admin anywhere is admin everywhere so you would not think that you need to enumeration projects, but it turns out that some of the more complex operations, such as mounting a volume, has to cross service boundaries, and need the project abstraction to link the sets of operations. CloudForms design did not see this disconnect, and so some of those operations fail.

    Lets assume, for the moment, that a user had to have a role on project in order to perform operations on that project. The current admin-everywhere approach would break. What CloudForms would require is an automated way to give a user a role on a project as soon as that project was created. It turns out that CloudForms is not the only thing that has this requirement.

    Role Assignment Inheritance

    Keystone projects do not have to be organized as a flat collection. They can be nested into a tree form. This is called “Hierarchical Multitenancy.” Added to that, a role can be assigned to a user or group on parent project and that role assignment is inherited down the tree. This is called “Role Assignment Inheritance.”

    This presentation, while old, does a great job of putting the details together.

    You don’t need to do anything different in your project setup to take advantage of this mechanism. Here’s something that is subtle: a Domain IS A project. Every project is already in a domain, and thus has a parent project. Thus, you can assign a user a role on the domain-as-a-project, and they will have that role on every project inside that domain.

    Sample Code

    Here is in command line form.

    openstack role add --user CloudAdmin --user-domain Default --project Default --project-domain Default --inherited admin

    Lets take those arguments step by step.

    --user CloudAdmin  --user-domain Default

    This is the user that CloudForms is using to connect to Keystone and OpenStack. Every user is owned by a domain, and this user is owned by the Default” domain.

    --project Default --project-domain Default

    This is blackest of magic. The Default domain IS-A project. So it owns itself.


    A role assignment is either on a project OR on all its subprojects. So, the user does not actually have a role that is usable against the Default DOMAIN-AS-A-PROJECT, but only on all odf the subordinate projects. This might seem strange, but it was built this way for exactly this reason: being able to distinguish between levels of a hierarchy.


    This is the role name.


    With this role assignment, the CloudForms Management Engine instance can perform all operations on all projects within the default domain. If you add another domain to manage a separate set of projects, you would need to perform this same role assignment on the new domain as well.

    I assume this is going to leave people with a lot of questions. Please leave comments, and I will try to update this with any major concepts that people want made lucid.

    by Adam Young at February 28, 2018 07:04 PM

    OpenStack Superuser

    A close-up on the Queens release

    As OpenStack’s 17th release comes to a cloud near you, Superuser answers some common questions about Queens and the ecosystem. Bonus answer, in case you were wondering: the name comes from Queens Park, a suburb of Sydney, Australia. (More on the release naming process here.)

    What are the main trends in user adoption?

    • People want their clouds to do more than they used to, expanding to new workloads including machine learning and edge.
    • Integration is key. There’s a growing demand for large enterprises to run mission critical and backup as apps in a cloud environment, so the Queens release has a lot of high availability features, there’s a new project called Masakari and multi-attach in Cinder block storage. Multi-attach allows you to have more than one virtual machine attaching to the same volume, so if one goes down you can rely on another VM accessing the same storage.

    What’s the current situation with GPU capabilities in OpenStack and where’s it headed?

    People running GPUs today generally do it two ways:

    • PCI passthrough (the downside — it requires post-provisioning manual setup so it can be time consuming)
    • Using Ironic to manage the entire physical server

    Early users of the underlying hardware include eBay and Commonwealth Bank. eBay is using Ironic on top of hardware layering TensorFlow and Kubernetes on top of the GPUs. Once these components are getting delivered as fully supported technologies inside OpenStack, it’s a pretty good indication the use case will go mainstream across a lot of different environments and organizations — and now it’s much easier to get started.

    New for the Queens release is support for virtual GPUs. Nova now lets cloud administrators define flavors to request vGPU resources and set vGPU resolutions. The feature currently supports Intel GVT-g and NVIDIA GRID vGPU. See the Nova documentation for additional details.

    What enhancements in Queens are the most important for edge computing?

    The three most relevant things for edge are OpenStack-Helm, LOCI and acceleration support, which will be a requirement in a lot of edge environments.

    Helm containerizes OpenStack services and puts them in a Kubernetes pod. (You can see how this works with a live upgrade demoed by AT&T at the Sydney Summit).

    LOCI is a set of lightweight container images, the project takes a thinner approach to the container – offering a smaller, much more portable, lower footprint. Then operators can add configuration to the outside and use Kubernetes to manage the overall service functionality.

    Together they make it easier to deploy a lot of OpenStack environments and to upgrade in an automated way as well as operate in a zero-touch way over the long term and with a smaller footprint than a traditional datacenter OpenStack deployment. Those attributes will all be very important in edge deployments.

    There are a lot of existing projects for OpenStack and containers; where does OpenStack-Helm fit in?

    OpenStack-Helm provides a collection of Helm charts and tools for managing the lifecycle of OpenStack on top of Kubernetes and running OpenStack projects as independent services. It’s a promising project for users who want to put OpenStack services at the edge, or who want to containerize OpenStack services for easier upgrade paths.

    Which end users also contributed code to Queens?

    A total of 158 companies and organizations contributed to this release; Some end users who also contributed code include:

    • AT&T
    • Verizon
    • Walmart
    • Universidade Federal de Campina Grande
    • Boston University
    • China Mobile
    • Deutsche Telekom
    • MIT
    • Johns Hopkins University Applied Physics Laboratory
    • HFBank
    • Workday
    • University of Melbourne

    What’s the main direction for the next release, Rocky?

    This week, the development teams are in Dublin at the PTG to start working on the Rocky release. Although the 18th release of OpenStack is not due until the end of August, there are already a few planned features to get excited about:

    • Fast forward upgrade work will continue, which is a method for letting users jump more than two releases ahead with alternative install paths that help users speed their way through the intermediary releases and get to where they want to be.
    • Minimum bandwidth and bandwidth-based scheduling is a networking feature that may be coming, which will be of particular interest to NFV and cloud service providers for things like ensuring a minimum level of performance for streaming services.
    • One of the community wide goals for Rocky is to enable mutable configuration across services, which will let operators change configuration settings without restarting a service.

    Get involved!

    Learn more about how organizations are using OpenStack at the upcoming Summit, meet the community and start contributing, or visit the Marketplace to find an OpenStack service provider.



    Cover Photo // CC BY NC

    The post A close-up on the Queens release appeared first on Superuser.

    by Nicole Martinelli at February 28, 2018 05:16 PM

    Celebrate the Queens release with 5 new OpenStack tips and guides

    Today marks the release of OpenStack's seventeenth release, Queens. After a 26-week release schedule, Queens brings into the fold new projects and new features, including strong container integration, support for vGPUs, and many advancements around NFV, edge computing, and machine learning applications.

    by Jason Baker at February 28, 2018 09:00 AM

    February 27, 2018

    OpenStack Superuser

    Meet volume multi-attach, a great new feature in OpenStack Queens

    Volume multi-attach is a feature that enables users to attach and access a single block storage volume to/from multiple servers.
    The use cases for this functionality include active-active and hot-standby scenarios, when the data on a volume needs to be accessible by multiple servers in order to recover fast in case of failures or being able to handle increased load on the system. Volume multi-attach requires support from two OpenStack projects, Cinder (block storage) and Nova (compute.)

    The support for it added in the Queens release (February 28, 2018) represents years of collaboration, so it’s with great pride that we introduce you to it here.

    It’s not cloudy anyway

    The question often arises whether or not we should add features to the cloud platform that we don’t consider a good fit. It’s even more common for functionalities that are more challenging to add due to previous architectural decisions. Multi-attach was no different.

    While the desire for cloud native applications is industry-wide, it’s quite a luxury to rewrite everything to fit. In order to support this transformation period, we needed to make the platform suitable for workloads that are not fully adapted yet.

    In the past couple of release cycles, while working to overcome the challenges that prevented us from implementing the volume multi-attach functionality in Cinder and Nova, we had weekly meetings to decide on the design of new APIs and further improvements to the platform.

    So, what do you get?

    As a starting point, we added a new attach/detach API to Cinder to make the flows more flexible. The new API introduces the concept of ‘attachment’ which you can create, update, complete or delete in order to attach or detach a volume.

    You can use the new attach/detach API to attach a volume multiple times in case the Cinder back end you chose reports support for the functionality. In the Queens release, three drivers support multi-attach: LVM, NetApp/SolidFire and Oracle ZFSSA. You can check the Driver Support Matrix  for updates on when additional drivers add support.

    You need to create a multi-attach volume type to create a multi-attach-capable volume to be able to use this functionality. You can also re-type a volume to be non-multi-attach-capable or vice-versa, but only when the volume is in ‘available’ state. Using the ‘multiattach’ flag during the volume create operation is deprecated starting from the Queens release.

    On the compute side, in order to attach a single volume to multiple VM instances you’ll need to run Nova microversion 2.60 or higher. Similarly, to backends in Cinder you’ll find limitations regarding hypervisors in Nova as well. In the Queens release only Libvirt is supporting volume multi-attach and you’ll need either Libvirt >= 3.10 or QEMU < 2.10. For further information please refer to the Feature Support Matrix.

    Currently all the subsequent volume attachments will be attached in read-write (RW) mode including the boot from volume case. You can turn the functionality on or off with two new Cinder policies:


    You can read more about the functionality and its limitations in the admin guide chapters of Cinder and Nova.

    Beyond new APIs and functionality, we’ve also started to cover the testing side not just within the projects, but also Tempest. We’ve added test cases like the basic functionality, booting a server from a multi-attach volume, or resizing servers with a multi-attach volume attached to both.

     Check out this overview video with a demo from Matt Riedemann:

    What’s next ?

    The two teams are looking for feedback and discussing improvements to the functionality. Current improvement ideas to discuss include providing support to specify the attach-mode for the secondary attachments. For the latest information, please take a look into Cinder’s specs list for the current release.

    For further improvements in the area of testing, we are considering adding more robust integration testing such as writing and reading to/from a shared volume between two VMs.

    If you’d like to get involved or have feedback, you can reach the teams on the OpenStack Developer Mailing List, please use the tags ‘[cinder]’ and ‘[nova]’ in the subject line  to ensure it gets the right people’s attention. You can also reach out on IRC, each teams has its own channel: #openstack-cinder and #openstack-nova.

    Special thanks to

    This long-awaited feature wouldn’t have landed without the hard work of the two teams and especially the following contributors: John Griffith, Matt Riedemann, Steve Noyes, John Garbutt, Walter Boring, Balazs Gibizer, Sylvain Bauza, Melanie Witt, Sean McGinnis and Jay Bryant.

    The post Meet volume multi-attach, a great new feature in OpenStack Queens appeared first on Superuser.

    by Ildiko Vancsa at February 27, 2018 04:58 PM

    SWITCH Cloud Blog

    Openstack Keystone authentication for your Kubernetes cluster

    At SWITCH we are looking to provide a container platform as a Service solution. We are working on Kubernetes and Openshift to gauge what is possible and how a service could be structured. It would be really nice to use the existing Openstack username and password to authenticate to Kubernetes. We tested this solution and it works great.

    How does it work ? Lets start from the client side.

    Kubernetes users use the kubectl client to access the cluster. The good news is that since version v1.8.0 of the client, kubectl is able to read the usual openstack env variables, contact keystone to request a token, and forward the request to the kubernetes cluster using the token. This was merged the 7th of August 2017. I could not find anywhere how to correctly configure the client to use this functionality. Finally I wrote some documentation notes HERE.

    How does it work on the Kubernetes master side ?

    The Kubernetes API receives a request with a keystone token. In the Kubernetes language this is a Bearer Token. To verify the keystone token the Kubernetes API server will use a WebHook. What does it means ? That the Kubernetes API will contact yet another Kubernetes component that is able to authenticate the keystone token.

    The k8s-keystone-auth component developed by Dims makes exactly this. I tested his code and I created a Docker container to integrate the k8s-keystone-auth in my kube-system namespace. When you run the k8s-keystone-auth container your pass as an argument the URL of your keystone server.

    If you are deploying your cluster with k8s-on-openstack you find this integration summarized in a single commit.

    Now that everything is setup I can try:

    source ~/openstackcredentials
    kubectl get pods

    I will be correctly authenticated by keystone that will verify my identity, but I will have no authorization to do anything:

    Error from server (Forbidden): pods is forbidden: User "" cannot list pods in the namespace "default"

    This is because we need to set up some authorization for this keystone user. You can find detailed documentation about RBAC but I make here a simple example:

    kubectl create rolebinding saverio-view --clusterrole view --user --namespace default

    Now the my user is able to view anything at the default namespace, and I will be able to do kubectl get pods

    Of course setting up RBAC specific rules for every user is not optimal. You can at least use the keystone projects, that are mapped to kind: Group in Kubernetes. Here an example:

    kind: RoleBinding
     name: read-pods
     namespace: default
    - kind: Group
     name: <openstack_project_uuid>
     kind: Role
     name: pod-reader

    You can then achieve a “soft multitenancy” where every user belonging to a specific keystone project is limited with permissions to a specific namespace. I talk about soft multitenancy because all the pods from all the namespaces, depending on your networking solution, could end up on the same network with a completely open policy.

    I would like to thank Dims and the other people on the Slack channel #sig-openstack for the great help while developing this Kubernetes deployment.

    by Saverio Proto at February 27, 2018 09:59 AM

    Deploy Kubernetes v1.8.3 on Openstack with native Neutron networking

    I wrote in the past how to deploy Kubernetes on SWITCHengines (Openstack) using this ansible playbook. When I wrote that article, I did not care about the networking setup, and I used the proposed weavenet plugin. I went to Sydney at the Openstack Summit and I saw the great presentation from Angus Lees. It was the right time to see the presentation because I recently watched this video where they explain the networking of Kubernetes when running on GCE. Going back to Openstack, Angus mentioned that the Kubernetes master can talk to neutron, to inject routes in the tenant router to provide connectivity without NAT among the pods that live in different instances. This would make easier the troubleshooting, and would leave MTU 1500 between the pods.

    It looked very easy, just use:


    and specify in the cloud config the router uuid.

    Our first tests with version 1.7.0 did not work. First of all I had to fix the Kubernetes documentation, because the syntax to specify the router UUID was wrong. Then I had a problem with Security groups disappearing from the instances. After troubleshooting and asking for help on the Kubernetes slack channel, I found out that I was hitting a gophercloud known bug.

    The bug was already fixed in gophercloud at the time of my finding, but I learned that Kubernetes freezes an older version of this library in the folder “vendor/”. So the only way to get the updated library version was to upgrade to Kubernetes v1.8.0, or any newer version including this commit.

    After a bit of testing every works now. The changes are summarised in this PR, or you can just use the master branch from my git repository.

    After you deploy, the K8s master will assign from network ClusterCIDR (usually a /16 address space) a smaller /24 subnet per each Openstack instance. The Pods will get addresses from the subnet assigned to the instance. The kubernetes master will inject static routes to the neutron router, to be able to route packets to the Pods. It will also configure the neutron ports of the instances with the correct allowed_address_pairs value, so that the traffic is not dropped by the Openstack antispoofing rules.

    This is what a show of the Openstack router looks like:

    $ openstack router show b11216cb-a725-4006-9a55-7853d66e5894 -c routes
    | Field  | Value                                            |
    | routes | destination='', gateway=''  |
    |        | destination='', gateway=''  |
    |        | destination='', gateway='' |
    |        | destination='', gateway='' |

    And this is what the allowed_address_pairs on the port of one instance looks like:

    $ openstack port show 42f2a063-a316-4fe2-808c-cd2d4ed6592f -c allowed_address_pairs
    | Field                 | Value                                                      |
    | allowed_address_pairs | ip_address='', mac_address='fa:16:3e:3e:34:2c' |

    There is of course more work to be done.

    I will improve the ansible playbook to create automatically the Openstack router and network, at the moment these steps are done manually before starting the playbook.

    Working with network-plugin=kubenet is actually deprecated, so I have to understand what is the long term plan for this way of deployment.

    The Kubernetes master is still running on a single VM, the playbook can be extended to have an HA setup.

    I really would like to have feedback from users of Kubernetes on Openstack. If you use this playbook please let me know, and if you improve it, the Pull Requests on github are very welcome! 🙂

    by Saverio Proto at February 27, 2018 09:58 AM

    February 26, 2018

    OpenStack Superuser

    Why cyborgs need free software

    There probably aren’t many questions from patients that leave electrophysiologists speechless. Karen Sandler stumped hers with just one. Sandler, executive director at the Software Freedom Conservancy, wanted to know about the software powering the pacemaker that she needed.

    Sandler has an outsize heart (three times the size of the average ticker), a condition called hypertrophic cardiomyopathy. For her, the condition is without symptoms but she runs an increased chance of sudden death with each year that passes.

    “I was told that everything would be fine, because I could get this defibrillator implanted in my body. If I get into trouble it’ll be like a fairy godmother shocking me,” Sandler recounted in a recent Index Conference keynote. The specialist was showing Sandler and her mother the small, lightweight device that some patients refer to as “internal bling.”

    “I had just one question for him: what does it run on?”

    Sandler, a lawyer by profession, said the doctor didn’t understand the question. Then he called in a representative for the device, who also didn’t know anything about the software inside. After the doctor’s visit, Sandler dogged the company’s help line, a scenario she describes as “phone tree hell” but found that none of the manufacturers had answers for her.

    “I did what any normal person would do, I put it off,” she says. She changed her mind, however, after both her mother and a close friend admitted to fearing Sandler was dead every time they got her voicemail.

    So Sandler got the implant – embracing her “inner cyborg” she says while flashing a slide of “Star Trek: Voyager” character Seven of Nine – and embarked on a research project about the proprietary software inside medical devices.

    The results? “Software has bugs. Yeah, surprise!” she notes, demonstrating exactly zero surprise. “These devices are totally vulnerable,” Sandler concludes. Unsurprisingly, she found that free and open software was less vulnerable than proprietary systems for medical devices. The Four Freedoms — to use, study, share and improve the software – not only lead to fewer bugs but help wipe out the remaining ones. (More on this in a paper from the Software Freedom Law Center titled “Killed with Code.”)

    “That means that if you face a problem, probably other people do too and you can share those changes with others,” Sandler says. Patients don’t have to rely on any particular company, which is really important when it comes to things like your heart.

    “Right now what we have in in the medical device space is the worst of both worlds,” she says. “These devices are broadcasting remote wirelessly all the time, they have radio telemetry on them — but no real security and there’s no basic password protection.”

    Image // CC BY NC

    Sandler had a terrifying experience with the standard issue programming of the device going wrong when she was pregnant – her slightly elevated heart rate caused it to shock her. Because as a pregnant woman she was not the typical for the device, the doctor’s solution was medication to slow down her heart rate. It could’ve been worse:  there were 16,000 deaths associated with medical devices in the FDA database. But with the devices as black boxes, it’s hard to diagnose where the devices went wrong.

    “I would have liked to have seen the source code to my device — even if I got all the pregnant women with defibrillators together, there’s not enough money to get those to get the device manufacturer to consider our case,” she says. It’s a perfect example of a company’s good intentions leading to dangerous situations for some.

    Sandler, a one-time “Mac fan girl” who previously worked at the Gnome Foundation and has been a proponent of FOSS since the 90s, says that now her interest in it has taken on a new dimension.

    “Now I’m someone who thinks that software freedom is essential, not this esoteric issue,” she says. “It’s in fact something that is going to undermine the safety and the very nature of our society.”

    Still, she considers it a privilege to be what she calls a voluntary cyborg.

    “The truth is we will all become cyborgs over time,” she says. “Because I want to incorporate enough technology in my body to take advantage of advances as they go forward.”

    “So I say cyborgs unite!…We’re at the beginning of a road with free and open source software and with it, we have options.”

    Catch the entire 43-minute session below.

    Hat tip: Monty Taylor

    Cover Photo // CC BY NC

    The post Why cyborgs need free software appeared first on OpenStack Superuser.

    by Nicole Martinelli at February 26, 2018 05:37 PM

    Lars Kellogg-Stedman

    Grouping aggregation queries in Gnocchi 4.0.x

    In this article, we're going to ask Gnocchi (the OpenStack telemetry storage service) how much memory was used, on average, over the course of each day by each project in an OpenStack environment.


    I'm working with an OpenStack "Pike" deployment, which means I have Gnocchi 4.0.x. More …

    by Lars Kellogg-Stedman at February 26, 2018 05:00 AM

    February 24, 2018

    OpenStack in Production

    Maximizing resource utilization with Preemptible Instances


    The CERN cloud consists of around 8,500 hypervisors providing over 36,000
    virtual machines. These provide the compute resources for both the laboratory's
    physics program but also for the organisation's administrative operations such
    as paying bills and reserving rooms at the hostel.

    The resources themselves are generally ordered once to twice a year with servers being kept for around 5 years. Within the CERN budget, the resource planning teams looks at:
    • The resources required to run the computing services requirements for the CERN laboratory. These are projected using capacity planning trend data and upcoming projects such as video conferencing.
    With the installation and commissioning of thousands of servers concurrently
    (along with their associated decommissioning 5 years later), there are scenarios
    to exploit underutilised servers. Programs such as LHC@Home are used but we have also been interested to expand the cloud to provide virtual machine instances which can be rapidly terminated in the event of
    • Resources being required for IT services as they scale out for events such as a large scale web cast on a popular topic or to provision instances for a new version of an application.
    • Partially full hypervisors where the last remaining cores are not being requested (the Tetris problem).
    • Compute servers at the end of their lifetime which are used to the full before being removed from the computer centre to make room for new deliveries which are more efficient and in warranty.
    The characteristics of this workload is that it should be possible to stop an
    instance within a short time (a few minutes) compared to a traditional physics job.

    Resource Management In Openstack

    Operators use project quotas for ensuring the fair sharing of their infrastructure. The problem with this, is that quotas pose as hard limits.This
    leads to actually dedicating resources for workloads even if they are not used
    all the time or to situations where resources are not available even though
    there is quota still to use.

    At the same time, the demand for cloud resources is increasing rapidly. Since
    there is no cloud with infinite capabilities, operators need a way to optimize
    the resource utilization before proceeding to the expansion of their infrastructure.

    Resources in idle state can occur, showing lower cloud utilization than the full
    potential of the acquired equipment while the users’ requirements are growing.

    The concept of Preemptible Instances can be the solution to this problem. These
    type of servers can be spawned on top of the project's quota, making use of the
    underutilised  capabilities. When the resources are requested by tasks with
    higher priority (such as approved quota), the preemptible instances are
    terminated to make space for the new VM.

    Preemptible Instances with Openstack

    Supporting preemptible instances, would mirror the AWS Spot Market and the
    Google Preemptible Instances. There are multiple things to be addressed here as
    part of an implementation with OpenStack, but the most important can be reduced to these:
    1. Tagging Servers as Preemptible
    In order to be able to distinguish between preemptible and non-preemptible
    servers, there is the need to tag the instances at creation time. This property
    should be immutable for the lifetime of the servers.
    1. Who gets to use preemptible instances
    There is also the need to limit which user/project is allowed to use preemptible
    instances. An operator should be able to choose which users are allowed to spawn this type of VMs.
    1. Selecting servers to be terminated
    Considering that the preemptible instances can be scattered across the different cells/availability zones/aggregates, there has to be “someone” able to find the existing instances, decide the way to free up the requested resources according to the operator’s needs and, finally, terminate the appropriate VMs.
    1. Quota on top of project’s quota
    In order to avoid possible misuse, there could to be a way to control the amount of preemptible resources that each user/project can use. This means that apart from the quota for the standard resource classes, there could be a way to enforce quotas on the preemptible resources too.

    OPIE : IFCA and Indigo Dataclouds

    In 2014, there were the first investigations into approaches by Alvaro Lopez
    from IFCA (
    As part of the EU Indigo Datacloud project, this led to the development of the
    OpenStack Pre-Emptible Instances package (
    This was written up in a paper to Journal of Physics: Conference Series
    ( and
    presented at the OpenStack summit (

    Prototype Reaper Service

    At the OpenStack Forum during a recent OpenStack summit, a detailed discussion took place on how spot instances could be implemented without significant changes to Nova. The ideas were then followed up with the OpenStack Scientific Special Interest Group.

    Trying to address the different aspects of the problem, we are currently
    prototyping a “Reaper” service. This service acts as an orchestrator for
    preemptible instances. It’s sole purpose is to decide the way to free up the
    preemptible resources when they are requested for another task.

    The reason for implementing this prototype, is mainly to help us identify
    possible changes that are needed in Nova codebase to support Preemptible

    More on this WIP can be found here: 


    The concept of Preemptible Instances gives operators the ability to provide a
    more "elastic" capacity. At the same time, it enables the handling of increased
    demand for resources, with the same infrastructure, by maximizing the cloud

    This type of servers is perfect for tasks/apps that can be terminated at any
    time, enabling the users to take advantage of extra cpu power on demand without the fixed limits that quotas enforce.

    Finally, here in CERN, there is an ongoing effort to provide a prototype
    orchestrator for Preemptible Servers with Openstack, in order to pinpoint the
    changes needed in Nova to support this feature optimally. This could also be
    available in future for other OpenStack clouds in use by CERN such as the
    T-Systems Open Telekom Cloud through the Helix Nebula Open Science Cloud


    • Theodoros Tsioutsias (CERN openlab fellow working on Huawei collaboration)
    • Spyridon Trigazis (CERN)
    • Belmiro Moreira (CERN)


    by Theodoros Tsioutsias ( at February 24, 2018 08:13 AM

    David Moreau Simard

    Awesome things in software engineering: open source

    This is part of a blog series highlighting awesome things in software engineering because not everything has to be depressing, about bugs, vulnerabilities, outages or deadlines. If you’d like to collaborate and write about awesome things in software engineering too, let’s chat: reach out on Twitter or LinkedIn. What’s this blog series about ? Between you and me, software engineering isn’t always fun. You’re not always working on what you like.

    February 24, 2018 12:00 AM

    February 23, 2018

    OpenStack Superuser

    5G and the really smart self-driving car

    On a regular weekday, some time in the future, you’ll be heading to work like everyone else, comfortably chauffeured in a self-driving car. As you approach a school zone, your car and the cars next to you heed the speed limit but stop sharply as a child rushes out into the street.

    That’s what happened during tests at K-City, testing grounds for self-driving cars in Hwaseong, South Korea. The recent experiment from SK telecom and the Korean Transportation Safety Authority (TS) ran two 5G self-driving cars successfully through what they’re calling “cooperative driving,” where smart-self driving cars constantly ping each other with updated traffic information.

    Cooperative driving in action on the K-City track. Courtesy SK telecom.

    And while the cars  in the tests were on a closed track and the “child” running out into the street was a life-size dummy, the experiment proves that the ultra-low-latency of less than 1ms of response time can handle everyday traffic situations. The incessant chatter — the car communicated more than 100 times a second using 5G network with the control center and with another vehicle — means that vehicles reroute for construction sites, gauge the right speed for freeway on-ramps and determine how best to merge traffic after an accident closes a lane.

    An earlier test below shows the car whizzing at over 100 mph to demonstrate potential point-to-point data transmission speeds.


    Putting software-defined networking in the driver’s seat

    SK telecom is the largest wireless provider in South Korea with a 50 percent market share. The company has been at the forefront of developing and commercializing advanced wireless technology — for example, it was the first to commercialize CDMA technology way back in 1990. And while 5G is said to have had its “coming out party” with a number of firms including SK telecom at the Winter Olympics, SKT is once again ahead of the curve.

    Their team has been pioneering the network slicing and virtualization key to make 5G run smoothly. Back in 2017, they released TACO (T-All Container OpenStack) for SKT’s 5G infrastructure. The team has also been developing software-defined networking (SDN) based network virtualization solution (SONA) for TACO. SONA uses VxLAN to provide fully independent virtual networks to 5G services and supports multiple gateways to provide multiple paths to gateways for its performance.

    SONA also supports containers (Docker Swarm and Kubernetes) and allows communication between containers and VMs by controlling their flows using an SDN controller. (This video using Amazon’s Alexa to deploy a container in just a few minutes was a crowd pleaser.)  The connections among containers and VMs are essential to 5G infrastructure where some of VNFs have micro-service architecture utilizing containers to improve its agility.

    For more details on the carrier-grade SDN-based OpenStack networking solution, check out this 30-minute session by SK telecom’s Daniel Park and Sangho Shin at the OpenStack Summit Sydney.

    Cover Photo // CC BY NC

    The post 5G and the really smart self-driving car appeared first on OpenStack Superuser.

    by Superuser at February 23, 2018 05:10 PM

    David Moreau Simard

    Rebranding Ansible Run Analysis to ARA Records Ansible

    So I got an idea recently… Let’s rebrand Ansible Run Analysis to ARA records Ansible. If you’d like to review and comment on the code change, you can do so here: Why ? I watched the last season of Sillicon Valley recently. The series, while exaggerated, provides a humorous look at the world of startups. I don’t have any plans on creating a startup but I love that it makes you think about things like needing a clever name or how you would do a proper “elevator” pitch to get funding.

    February 23, 2018 12:00 AM

    Rebranding Ansible Run Analysis to ARA Records Ansible

    So I got an idea recently… Let’s rebrand Ansible Run Analysis to ARA records Ansible. If you’d like to review and comment on the code change, you can do so here: Why ? I watched the last season of Sillicon Valley recently. The series, while exaggerated, provides a humorous look at the world of startups. I don’t have any plans on creating a startup but I love that it makes you think about things like needing a clever name or how you would do a proper “elevator” pitch to get funding.

    February 23, 2018 12:00 AM

    Carlos Camacho

    TripleO deep dive session #12 (config-download)

    This is the 12th release of the TripleO “Deep Dive” sessions

    Thanks to James Slagle for this new session, in which he will describe and speak about a feature called config-download.

    In this session we will have an update for the TripleO ansible integration called config-download. It’s about aplying all the software configuration with Ansible instead of doing it with the Heat agents.

    So please, check the full session content on the TripleO YouTube channel.

    Please check the sessions index to have access to all available content.

    by Carlos Camacho at February 23, 2018 12:00 AM

    February 22, 2018

    OpenStack Superuser

    Getting Network Function Virtualization ready for prime time: Inside the standards tests

    Network Function Virtualization (NFV) aims to virtualize over generic servers the network functions that typically run in dedicated appliances, so that they acquire all the advantages of cloud applications — almost a must for today’s applications like 5G.

    Since its conception in 2012 by the world’s leading telecom network operators through this white paper and motivated by the increasing costs of building networks with proprietary hardware appliances, NFV has been constantly evolving through European Telecommunications Standards Institute (ETSI) standards. Deployments have been growing slowly but surely, paving the way to a brighter future for the telecom industry.  Since 2017, activities like the ETSI NFV Plugtests have been accelerating deployments by improving both interoperability and technology readiness.

    In this first post in a three-part series, I’ll summarize results from ETSI’s second NFV Plugtests held in January 2018, where I participated representing Whitestack’s NFV solutions. It was an opportunity for vendors and open source communities to get together, assess interoperability and validate solutions against ETSI NFV specifications.

    Before diving into the tests, let’s quickly review ETSI NFV Architecture, where we can see three fundamental divisions:

    • NFVI + VIM: The physical compute, storage and networking resources that will be virtualized (NFV Infrastructure), and the software that manages their lifecycle (Virtual Infrastructure Manager). The VIM belongs to NFV MANO according to the ETSI Architecture, but since OpenStack and other VIM tools already solve VM lifecycle, actual MANO focus is on the higher layers.
    • MANO: The Management & Orchestration software components (NFV Orchestrator and VNF Manager) that take care of the lifecycle of Network Services and Virtual Network Functions.
    • VNFs: The actual network functions, comprised by one or more virtual machines (and containers, if generically speaking), that can be integrated together to build end-to-end (virtualized) Network Services.

    So back to the Plugtests: many experienced VNF vendors and the most relevant NFVI, VIM and MANO providers of the industry met at the ETSI headquarters at Sophia Antipolis, France where we spent a whole week inside a big room going over a number of interoperability tests.

    Testing inside ETSI’s headquarters.

    Of course, to make this big challenge possible (dozens of companies, some of them competitors, working together), ETSI organized things well in advance. They set up weekly group calls around four months beforehand, had a VPN in place, called the NFV Plugtests HIVE, so everyone could connect their solutions in advance (see the image below with the participants spanning the globe) and ensured everyone completed a ‘pre-testing’ process before the onsite week. A special thanks to the ETSI team for achieving a second time and for managing to get more testing done, with more features and in half the time compared to the first edition!

    So, what was tested?

    • Multi-VNF Network Services lifecycle (instantiation, manual scaling and termination) –> Two or more VNFs, from different vendors, in the same service.
    • Multi-site/VIM Network Service deployments –> VNFs from the same Network Services in different data centers.
    • Enhanced Platform Awareness features (SR-IOV, CPU Pinning, Huge Pages, etc.) –> performance boost for VNFs!
    • End-to-end Performance Management –> to be able to grab metrics from and create thresholds on both the VIM and the VNFs.
    • MANO Automatic Scaling (out/in) capabilities based on performance metrics from both VNFs and VIMs –> to add or remove VNFs/VDUs based on VIM/VNF metrics.
    • End-to-end Fault Management –> events and alarms propagation to higher layers
    • An optional API test track provided by ETSI to experiment with compliance testing on some specific interfaces (NFV SOL 002 for the Ve-Vnfm reference point, NFV SOL 003 for the Or-Vnfm reference point)

    Even though our obvious objective was to run tests successfully, ETSI encouraged us to ensure marking results as ‘failed’ when interoperability did not work so that this rich feedback will influence NFV standards in a positive way.

    We brought both our MANO Solution (WhiteNFV, based on Open Source MANO Release 3), and our VIM Solution for Cloud and NFV environments (WhiteCloud, based on OpenStack Pike distribution), so we had to find interoperability with other NFVI/VIM and MANO providers respectively. I’m glad to say that both performed pretty well!

    In parts two and three of this series, we’ll dive into the official Plugtest results which ETSI just made public this week, so stay tuned.

    About the author

    Gianpietro Lavado is a network solutions architect interested in the latest software technologies to achieve efficient and innovative network operations. He currently works at Whitestack, a company whose mission is to promote SDN, NFV, cloud and related deployments all around the world, including a new OpenStack distribution.

    This post first appeared on LinkedIn. Superuser is always interested in community content, get in touch at:

    The post Getting Network Function Virtualization ready for prime time: Inside the standards tests appeared first on OpenStack Superuser.

    by Gianpietro Lavado at February 22, 2018 05:03 PM

    Carlos Camacho

    TripleO deep dive session #12 (config-download)

    This is the 12th release of the TripleO “Deep Dive” sessions

    Thanks to James Slagle for this new session, in which he will describe and speak about a feature called config-download.

    In this session we will have an update for the TripleO ansible integration called config-dpwnload. It’s about aplying all the software configuration with Ansible instead of doing it with the Heat agents.

    So please, check the full session content on the TripleO YouTube channel.

    Please check the sessions index to have access to all available content.

    by Carlos Camacho at February 22, 2018 12:00 AM

    February 21, 2018

    OpenStack Superuser

    Inside CERN’s Open Data portal

    Although the European Organization for Nuclear Research occupies a relatively modest patch of land on the French-Swiss border, the scope of CERN’s research is big — from the Higgs boson (aka “God particle”), anti-matter and dark matter, to extra dimensions – and the amount of data generated is truly vast.

    Since 2014, the CERN Open Data portal, which runs on OpenStack, has been making it available for high schoolers, data scientists and armchair physics buffs. The most recent information made public in late 2017 includes a petabyte of data, including sets related to the discovery of the Higgs boson glimpsed through the Large Hadron Collider.

    Superuser talks to Tibor Simko, technology lead behind the CERN Open Data portal, about the backend as well as the future of the project.

    CERN’s Open Data Portal runs on OpenStack — what can you tell us about the backend?

    The CERN Open Data portal was first released in 2014. The portal ran on about eight virtual machines on the CERN OpenStack cloud infrastructure. The machines were managed by Puppet. The architecture includes the front-end load balancing servers running HAproxy, dispatching user requests to the caching servers running Nginx that either serve the cached content, or, if needed, dispatch the user request further to the CERN Open Data web application itself. The application runs on top of the Invenio digital repository framework that further uses Redis caching service and SQL relational database services.

    The CERN Open Data portal hosts several thousands of records representing datasets, software, configuration files, documentation and related supplementary information released as open data by the LHC experiments. The total amount of released data represents more than 1400 Terabytes. The data assets themselves are stored on the CERN EOS Open Storage system. The CERN Open Data portal relies heavily on the EOS distributed storage system regarding its backend data storage needs.

    What were the resources at your disposal and scope of the portal project when it launched and what are they now?

    On the application side, the CERN Open Data portal recently underwent a major change of the underlying repository framework, from Invenio 2 to Invenio 3. We have upgraded our data model, the user interface and improved the faceted search experience. The new portal was released in December 2017.

    On the infrastructure side, we have been early adopters of container technologies and used Docker for the CERN Open Data portal development since its beginning. We are now using containers also for the portal production deployment itself using the OpenShift platform.

    Besides the changes in the portal technology and deployment, the amount of the open data released by the LHC experiments has grown in an exponential manner. For example, the CMS collaboration released a part of the 2010 datasets of about 30 TB in 2014 initially; the 2011 datasets of about 300 TB were released in 2016, and the 2012 datasets that we just released were about 1 PB in size!

    The CERN Open Data portal’s storage component relies heavily on the scalability of the EOS storage system to host large amounts of released open data. Since the CERN EOS system manages over 250 Petabytes at CERN overall, the amount of the open data pool remains moderate when compared to the regular daily usage by the physicists.

    What are the challenges particular to planning an open-access cloud like this one?

    The challenges were of diverse nature.

    First, we needed to organize the datasets, the software, the configuration, the documentation and the auxiliary information so that it would be understandable by non-specialists. We were working closely with the LHC experiments on the data organization and management.

    Histograms created from the CMS experiment at CERN’s Open Data Portal.

    Second, we had to present the research data to a non-typical audience consisting of data scientists, high-school students and the general public. We’ve integrated tools that permitted to explore the data via event display visualization or basic histogramming; we’ve also provided detailed guides on how to run more advanced physics analyses on the released primary datasets.


    Third, we had to design a scalable system that would be capable of serving possibly thousands of parallel user requests at a time, following the usage peaks coming after widely-covered press releases or social media events.

    What can you tell us about future plans?

    We plan to improve the discoverability of the CERN Open Data material by exposing our datasets using general standards such as JSON-LD with We plan to publish REST API interfaces to enable users to easily write applications against the portal.

    We are looking forward to forthcoming open data releases from LHC experiments. We are excited to host a first non-LHC experiment open data issued by the OPERA collaboration.

    Finally, we plan to facilitate working with the open data by providing richer and more easily runnable analysis examples.

    How did your previous work on the Invenio digital library platform inform this project?

    The Invenio digital library framework has been developed at CERN to run services that were originally oriented towards managing articles, books, preprints, theses, audios, photos, videos. Progressively, the experimental particle physics collaborations have been using the open access publications services to share the supplementary material to publications, such as numerical “data behind plots,” the plotting macro snippets, or the event display files. This brought a natural evolution of Invenio’s focus from targeting the “small data” domain towards the “big data” domain.

    We have been trying to describe and capture the structured information about the whole research process, including the experimental collision and simulated datasets, the virtual machines, the analysis software and the computational workflows used by the physicists to analyze the data that all together produce the original scientific results and publications. Capturing the research analysis process and its computational workflows in order to make the science more easily reproducible and reusable in the future light of new theories is an exciting topic that coherently subscribes to the wider evolution of the open access movement through open data to open science.

    Anything else you’d like to share?

    Before joining CERN, I worked in the field of computational plasma physics, all the while being involved with parallel software programming activities and side projects. I feel therefore particularly enthusiastic and privileged to work at CERN in the field of open science that bridges computing and physics. We’re trying to foster better open and reproducible science practices through offering better open science software tools that would assist the particle physicists researcher in their data analysis computing needs.

    About Tibor Simko
    Tibor Simko holds a PhD in plasma physics from Comenius University Bratislava, Slovakia and from the University of Paris Sud, France. He joined CERN to work as a computing engineer where he founded the Invenio digital repository framework. Simko later worked as a technology director of INSPIRE, the high energy physics information system. He now leads the development of the CERN Analysis Preservation, CERN Open Data and the Reusable Analyses projects. His professional interests include open science and reproducible research, information management and retrieval, software architecture and development, psychology of programming, free software culture and more.

    You can find him on Twitter  and GitHub.

    The post Inside CERN’s Open Data portal appeared first on OpenStack Superuser.

    by Nicole Martinelli at February 21, 2018 05:10 PM

    February 20, 2018

    Chris Dent

    TC Report 18-08

    Most TC activity has either been in preparation for the PTG or stalling to avoid starting something that won't be finished before the PTG. But a few discussions to point at.

    When's the Next PTG?

    Last Tuesday evening had a brief discussion asking when (and where) the next PTG will be, after Dublin. The answer? We don't know yet. It will likely come up in Dublin.

    Base Services and Eventlet

    A question about base services led to discussion about ways to technically and socially avoid the use of eventlet. Notable was the observation that we continue to have new projects that adopt patterns established in Nova that while perfectly workable are no longer considered ideal. There's some work to do to make sure we provide a bit more guidance.

    Naming for S Release

    Rocky is starting, so it is time to be thinking about naming for S. Berlin is the geographic location that will be the source for names beginning with "S".

    Python 3.6

    Most of Friday was devoted to Python 3.6. Many distros are headed that way and OpenStack CI is currently 2.7 and 3.5.

    TC Topics at the PTG

    A reminder that there is an etherpad for TC topics at the PTG.

    Because of the PTG there won't be a TC Report next week, but I will endeavor to write up standalone reports of the discussions started by that etherpad. Those discussion will hopefully grant a bit more vigor, drive, and context to the TC Report, which has wandered a bit of late.

    by Chris Dent at February 20, 2018 06:45 PM

    OpenStack Superuser

    What you need to know about cloud edge computing

    There’s a lot of speculation about whether edge computing will disperse the dominance of the cloud.

    But what exactly is it?

    A group of open infrastructure experts put their heads together for recent white paper (.PDF) to detail the what, how and when of cloud edge computing.

    The OpenStack Foundation (OSF)’s Edge Computing Group counts members from AT&T, Cisco, Ericsson, HPE, Inmarsat, Red Hat, Verizon and Walmart Labs and the report aims to disperse any doubts: At its simplest, cloud edge computing means “offering application developers and service providers cloud computing capabilities, as well as an IT service environment at the edge of a network.” The basic characteristic of edge computing, the authors elaborate, is that the infrastructure is located closer to the end user, that the scale of site distribution is high and edge nodes are connected by wide area network (WAN) network connections.

    Titled “Cloud edge computing: Beyond the data center,” the report details the core benefits (mainly reduced latency and mitigating bandwidth) as well as use cases, scenarios and current challenges involved in cloud edge.

    Allowing that there are probably already dozens of ways cloud edge can be used, the authors outline some common use cases:

    • Data collection and analytics: “Internet of things, where data is often collected from a large network of microsites, benefits from the edge computing model. Sending masses of data over often limited network connections to an analytics engine located in a centralized data center is counterproductive…”
    • Real-time/immersive: “AR/VR, connected cars, telemedicine, tactile internet Industry 4.0 and smart cities, are unable to tolerate more than a few milliseconds of latency and can be extremely sensitive to jitter, or latency variation.” More on how this could change our daily lives here.
    • Self-contained and autonomous site operations “These could include transportation (planes, buses, ships), mining operations (oil rigs, pipelines, mines), power infrastructure (wind farms, solar power plants) and even environments that should typically have good connectivity, like stores.”
    • Network functions virtualization (NFV)
      “NFV is at its heart the quintessential edge computing application because it provides infrastructure functionality. Telecom operators are looking to transform their service delivery models by running virtual network functions as part of, or layered on top of, an edge computing infrastructure.”
    • Network efficiency
    • Security
    • Privacy “Enterprises may have needs for edge computing capacity depending on workloads,
      connectivity limits and privacy. For example, medical applications that need to
      anonymize personal health information (PHI) before sending it to the cloud could
      do this utilizing edge computing infrastructure.”
    • Compliance
    Mobile will become a common environment for cloud edge computing.

    How to get involved

    The 17-page report winds up with a call to action. “We recognize that there is work to be done to achieve our goals of creating the tools to meet these new requirements…and encourage the entire open-source community to join in to define and develop cloud-edge computing.”

    In addition to checking for updates on the OSF Edge Computing page, you can help shape the future by:



    The post What you need to know about cloud edge computing appeared first on OpenStack Superuser.

    by Superuser at February 20, 2018 04:44 PM

    February 17, 2018

    Doug Hellmann

    beagle 0.1.0

    beagle is a command line tool for querying a hound code search service, such as This is the first release of beagle.

    by doug at February 17, 2018 12:20 AM

    February 16, 2018

    OpenStack Superuser

    For a happier open source community, give recognition

    Every open source community is made up of real people with real feelings. Many open source contributors are working in their free time to provide essential software that we use daily. Sometimes praise is lost in the feedback of bugs or missing features. Focusing on too much negative feedback can lead contributors to frustration and burnout.

    However you end up contributing to OpenStack, or any open source project, I believe that what gets people excited about working with a community is some form of recognition.

    My first answer to people coming into the OpenStack community is to join our Project Team Gathering event. Significant changes are discussed here to understand the technical details to carry out the work in the new release. You should seek out people who are owners of these changes and volunteer to work on a portion of the work. Not only are these people interested in your success by having you take on some of the work they have invested in, but you will be doing work that interests the entire team. You’ll finish the improvements and be known as the person in the project with the expertise in that area. You’ll receive some recognition from the team and the community using your software. And just like that, you’re hooked because you know your work is making a difference. Maybe you’ll improve that area of the project more, venture onto other parts of the project, or even expand to other open source projects.

    If you work in the OpenStack community, there’s also another way you can give and get recognition. In OpenStack IRC channels, you can thank members of the community publicly with the following command:

    #thanks <irc_nick> for being a swell person in that heated discussion!

    To be clear,  <irc_nick> is replaced with the person you want to give thanks.

    Where does this information go? Just like the Success Bot in which we can share successes as a community, Thanks Bot will post them to the OpenStack wiki. They will also be featured in the OpenStack Developer Digest.

    In developing this feature, I’vee had help and feedback from various members of the community. You can see my history of thanking people along the way, too.

    At the next OpenStack event, you’re still welcome to buy a tasty beverage for someone to say thanks. But why not give them recognition now too and let them know how much they’re appreciated in the community?

    Mike Perez is the cross-project developer coordinator at the OpenStack Foundation. You can find him as as thingee on IRC and Twitter.


    The post For a happier open source community, give recognition appeared first on OpenStack Superuser.

    by Mike Perez at February 16, 2018 09:53 PM


    Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


    Last updated:
    March 17, 2018 05:52 PM
    All times are UTC.

    Powered by: