January 22, 2018

OpenStack Superuser

Challenges and battle scars from OpenStack deployments

Sometimes,  helping a large enterprise client choose and implement OpenStack is a real conquest.

DXC’s Anupriya Ramraj and Rick Mathot were joined by American Airlines’ Farhad Sayeed at the OpenStack Summit in Sydney to talk about the toughest battles (and a success story) in building OpenStack deployments for enterprise clients.

The first challenge is helping a client choose either a distribution or upstream OpenStack. Ramraj recommends the distribution from multiple vendors. Even then, it’s important to make that infrastructure needs of the company match up with the right distribution. “You might end up with a load balance that’s not supported with a particular version of OpenStack that’s supported by the distribution,” she said. “Just getting that mix right, and delays because of that, is something that we see as a challenge.”

The next trick is dealing with getting OpenStack on older hardware. “It’s got its own unique challenges when you’ve got to go look up those drivers, get everything updated,” Ramraj said. “It can easily add up to months of delays there.”

Even once you’ve gotten OpenStack deployed, there’s bound to be a chaotic incident at some point. “Trouble-shooting OpenStack does require some unique knowledge of OpenStack services and understanding the dependencies between the services,” she said.

Getting help from the distribution provider can be tricky, too, as just reproducing the error or finding the right log files requires OpenStack expertise, which your client may or may not have in-house. “OpenStack skills are tough to hire and they’re tough to retain,” said Ramraj. “The audience out here, you should be proud that you’ve got OpenStack on your resume. It makes you highly valuable, because it is a tough skill to get out there in the market.”

Once all this ground has been covered, you still need to win over the apps group. “It’s always a tug-of-war between apps and infra(structure),” said Ramraj. Apps owners can be reluctant to move to OpenStack and the business case for OpenStack is getting those apps to successfully run. Sometimes, app teams want to use newer app deployment methods, and convincing them that Ansible and containers work on OpenStack is another challenge.

“Is it a Heat API to do the infrastructure automation, is it Kubernetes on OpenStack,” Ramraj asked. “Winning the challenge, winning the mind share of the apps guys, is a unique challenge by itself.”

Assuming all of the previous issues have been solved — infrastructure, support, apps, production workloads — there will come a time where you must need to OpenStack. “Upgrades in recent years have gotten much less intrusive,” said Ramraj, “much more dependable, but still there’s some unique challenges.”

Using managed services for reinforcement

All of these issues can be mitigated with a good managed services provider, said Ramraj. The client does not need to do this alone. A provider brings two things to the table, including the initial consult on the design (which involves choosing the right distribution and architecture, setting up networking and storage block objects, for example).

Secondly, a good managed services provider can offer ongoing services, which can look a lot like traditional IT services management.  “OpenStack does not exist in an island,” said Ramraj. “OpenStack exists in tandem with other services that the client is consuming.” For example, a large enterprise client will likely have to bill individual departments for OpenStack consumption. Setting the billing up correctly is a solid task for a provider.

Another is incident management. A good managed service provider will offer 24/ 7 operations with highly skilled OpenStack people who can help with capacity management, storage optimization, or growth-based compute scaling.

Different clients will need different hand-offs, too, in terms of managing the OpenStack deployment. “If you look at this as a layered graph,” said Ramraj, “typically we say the managed services provider will handle up to the OpenStack layer and then the client owns the retro machines and the applications that they onboard onto the stack.”

Of course, every client is different. Some might only want to own the business outcome, which means the provider manages even the VMs and app onboarding. In addition, handling the VMs might prove another challenge. “One client wanted to back up all those VMs and the rate of the backups and the frequency they wanted to do it caused some issues in terms of configuring Neutron to be able to deal with that network bandwidth,” said Ramraj. The key, however, is having the right conversations between client and provider on how to manage the RACI, or responsibility assignment matrix.

People are key to the whole process, too. There are many roles that are important for a managed services provider to use when implementing OpenStack for a client: the cloud ops support team and the dev ops cloud engineer. The former needs to be a highly skilled OpenStack specialist, not a generalist, while the latter is essential for making the OpenStack implementation work for the client, with apps to support business outcomes. Other roles include account delivery executives and the cloud service delivery managers. “The delivery managers enable governance roles in there, making sure they have regular conversations with the client around, ‘Hey, what’s your forward-looking capacity planning, how are we doing on handling your tickets,’ and making sure the governance is in there.”

Ultimately, Ramraj believes that the battle for OpenStack private clouds can be won. DXC’s work with American Airlines is a prime example as the two worked together to get a critical baggage handling app set up on OpenStack.

Winning the OpenStack battle at American Airlines

American Airlines’ Farhad Sayeed told how his company has been virtualizing things for more than 11 years. Two years ago, they  started with a vision to create a hybrid solution with an internal data center and a public cloud. “During our journey,” said Sayeed, “our focus was to raise open-source usage across the enterprise. We already do quite a few, but we want to focus on them. The future of hosting must support our business agility, some portability, and DevOps culture.”

The journey was admittedly bumpy. First, networking presented a challenge. “We had to work very closely with our network team to set up a repeatable process where we can duplicate that over and over again in various environments,” he said. “In a large enterprise like ours, OpenStack doesn’t stand by itself. It has to integrate with existing networking, backup restore, monitoring, SLAs and so on.” Sayeed’s team ended up testing four different OpenStack distribution types. He recommends finding providers who already have a strong partnership with your company, a solid roadmap and a future.

Originally, American Airlines wanted to create their own platform with networking. “We quickly realized that we were much better off and would get there quicker if we started with the reference architecture and we ended up in a converged platform,” said Sayeed.

American’s OpenStack target workloads standardize on Cloud Foundry for both public and internal clouds. That was the first workload that went into production. “At American Airlines our goal for cloud, both internal and external, is PaaS first, platform as a service,” Sayeed said.

One such app, Bag Tracker, went live recently thanks to the partnership between American and DXC as a managed services provider. Bag Tracker is a web-based app that allows end-to-end tracking, from check-in to delivery, of passenger bags, priority parcels,and aircraft equipment. Their standard Linux distribution today is Red Hat, although his team also sees a rising demand for Ubuntu, which they are testing as well.

Sayeed’s team is still deciding which container to go with. “Various teams are deploying Docker and Kubernetes in various configurations…” he said.

The journey to cloud is a cultural shift, said Sayeed. “Besides all the technical challenges that you will face, you must take care of the culture first,” he said. “All the different silos and disciplines, as we know, in an enterprise, have to be broken down.”

In addition, leadership must support what you’re doing. “My team and I were lucky enough to have full support from all the way from our CIO to my managing director during this journey,” said Sayeed. “That’s important. I cannot stress enough about that. Train and restructure your teams. Just because we work this way today, that doesn’t mean that’s how it needs to be. The cloud journey requires high collaboration. Work hand in hand with networking, security, middleware, application team. Everybody needs to come together.”

Sayeed and American Airlines have been testing four distributions of OpenStack. “We found out that less is more,” he said. “Folks who had done less customization and stayed close to the trunk found OpenStack easy to implement, easy to maintain, easy to upgrade and easy to roll out.”

Finally, bring your operations team in early. “You may have your own IT operations internally. You may have a managed service provider,” Sayeed said. “Don’t wait until you’re ready to go to production to bring them in. Bring them early. You’ll need them, they need you. Partner with them, and partner with them early.”

The toughest battle: Managing client expectations

Rick Mathot runs a global engineering and architecture team that has been deploying private clouds since 2009 (and OpenStack private clouds since 2015) at DXC. While he notes that there are many technical challenged involved in making OpenStack work, his presentation focus was on the human side.

In his keynote, OpenStack CEO Jonathon Bryce noted that OpenStack is no longer a science experiment. Mathot said that two-thirds of deployments are in production. “That means that now is the time that we’re going to be judged on our business value,” said Mathot. “The success of our outcomes will be judged relating to the business. That’s critical, because we need to understand what that value pertains to, (you need to) know the ‘why.’ Understanding your north star, as I call it, is going to get you through those periods where uncertainty prevails.”

It’s even tougher to gain ground when the troops aren’t open to new ideas, said Mathot. “We know how frustrating it is when a business comes to us with a solution and equally and likewise we shouldn’t be coming to them with preconceived solution artifacts (or) jargonized narrative relating to their ask,” he said.

CXOs are usually pretty nervous; business is in a time of significant change and providers need to help them allay their fears with language that they understand. “We’ve got to be able to articulate things simply in terms of what it is,” said Mathot. “For American Airlines, it was a scalable automated platform to host, to perform or provide PaaS to the organization.” In this case, it was the baggage tracking app – it was a simple use case, understood well by the business, and it worked within the OpenStack platform.

Mathot said that partnerships like this exist in a hybrid state — you need to be able to identify all the affected parties, all the stakeholders in a project. “Really, from a human side, bringing them into the tent with you is going to make them a part of the solution and stop some of the problems we’ve seen in the past of creating speed humps just for the sake of that,” he said.

Doing so will ensure that you’ll get support when you need it, which is generally when things are critical. “I think knowing some of the organizations I’ve been into, you may have to put it on a Post-It note and stick it on their screens.” said Mathot. “The reality  is that you’ve got to do what you’ve got to do to get buy-in.” He said you need to lead from the front, don’t stop using mode-two thinking and design principles. “Just always make sure you keep an eye on the rear view mirror,” added Mathot, “to ensure that they’re still with you, the organization is still with you.”

Once you’ve finished the deployment and it’s up and running, you need to quantify what you’ve achieved, even beyond the technical value. “It’s a real simple trick,” said Mathot. “You’ve got to have a story for every level within your organization.” For example, while 20 percent IT efficiency is really great, explaining a 15 percent productivity gain for your application providers is probably a better story for them. Showing a faster speed to market for new useful client-facing apps is what will excite the CXOs, as well. You want to tune your story for each organizational level.

Finally, Mathot cautioned against a Facebook-style “better done than perfect” attitude. “Progress is important and any progress is better than none at all,” he said, “but what you shouldn’t look to do is have that (type of) mantra permeate every level of your deployment.”

Relevance is key, he said. “We all love the challenge and we all want to play with the newest technologies,” said Mathot. “It’s rewarding both personally and professionally, but the (American Airlines) story was a success because your bags arrived at the same airport at the same time as you did. It was useful and it was relevant.”

Catch the whole presentation below.

The post Challenges and battle scars from OpenStack deployments appeared first on OpenStack Superuser.

by Rob LeFebvre at January 22, 2018 09:42 AM

NFVPE @ Red Hat

TripleO AIDE Service

Just a quick post about some patches that have landed in TripleO (upstream Red Hat OSP Director) to deploy AIDE to the overcloud. What’s AIDE? AIDE (Advanced...

by Luke Hinds at January 22, 2018 12:00 AM

January 19, 2018

OpenStack Superuser

Pay it forward: Join Google Summer of Code as an OpenStack mentor

If you’ve been involved with open source projects for more than 29 seconds, you know it takes a village.

The OpenStack community has been helping out with Google’s Summer of Code for college developers since 2014. In January, participating organizations line up mentors and the projects that everyone will work together on for the summer ahead.

That’s where you come in: if you have expertise and want to give back — as well as have a project itch to scratch — get your name in and your project ideas down now.

Applicants may not have ever worked on open-source projects before and come with different levels of competence. The best ideas are ones that can be done by inexperienced contributors, developers or other people from other fields (marketing, communication, graphic design, and anything that may be useful for OpenStack and to include new people in this community).

Ideas can range from Go and container-related projects in OpenStack to writing the “missing manuals” or updating documentation. For example, during the Pike release cycle, interns migrated the identity docs to the Keystone repository. The hope is that now, there will be eager newcomers to consolidate those docs, making sure they’re useful and accurate, not duplicated and use consistent patterns to ease maintenance.

Deadline for mentor applications for the Summer 2018 edition is January 23. (If you’re a student interested in applying, that deadline is in March. You’ll find more information about how you can get started now here.)

If you’re unsure about what mentoring entails, check out the GSoC Mentoring Manual for more information as well as some useful tips and tricks on mentoring.

Once you’ve decided, you can just add your name in the OpenStack Google Summer of Code 2018 wiki page and then add your project ideas here. Make sure you leave your contact information in the OpenStack GSoC 2018 wiki and that you add all of the
salient details about the project idea as well.

If you have questions, comment or concerns, get in touch with coordinators, mentors and students through the openstack-dev mailing list and openstack-internships mailing list or on IRC in #openstack-gsoc at irc.freenode.org.


The post Pay it forward: Join Google Summer of Code as an OpenStack mentor appeared first on OpenStack Superuser.

by Superuser at January 19, 2018 02:40 PM

January 18, 2018


RDO's infrastructure server metrics are now available

Reposted from dev@lists.rdoproject.org post by David Moreau Simard

We have historically been monitoring RDO's infrastructure through Sensu and it has served us well to pre-emptively detect issues and maximize our uptime.

At some point, Software Factory grew an implementation of Grafana, InfluxDB and Telegraf in order to monitor the health of the servers, not unlike how upstream's openstack-infra leverages cacti. This implementation was meant to eventually host graphs such as the ones for Zuul and Nodepool upstream.

While there are still details to be ironed out for the Zuul and Nodepool data collection, there was nothing preventing us from just deploying telegraf everywhere just for the general server metrics. It's one standalone package and one configuration file, that's it.

Originally, we had been thinking about feeding the Sensu metric data to Influxdb … but why even bother if it's there for free in Software Factory ? So here we are.

The metrics are now available here We will use this as a foundation to improve visibility into RDO's infrastructure, make it more "open" and accessible in the future.

We're not getting rid of Sensu although we may narrow it's scope to keep some of the more complex service and miscellaneous monitoring that we need to be doing. We'll see what time has in store for us.

Let me know if you have any questions !

by Rich Bowen at January 18, 2018 09:12 PM

OpenStack Superuser

Why hybrid cloud is the new black for an ad tech company

Amobee is a global marketing technology company whose work includes real‐time bidding for advertising and collaborating with partners in the ad industry for collecting advertising data.

Based in Redwood City, California but with offices stretching from Silicon Valley to Singapore, their client portfolio includes Adidas, Skype and Porsche. Behind all the data crunching – including things like lists of the most-anticipated movies and talked-about TV shows — sits a hybrid environment with both cloud services and physical infrastructure in multiple data centers.

Eric Lakich, director of Amobee’s systems engineering team, says they partnered with Platform9 to move to OpenStack won over by benefits on offer.

“It’s really agile and flexible and we wanted to be able to get that same agile process on our own hardware, while maximizing efficiency,” Lakich says in a case study. “We have a lot of places where we might have a really powerful server running a small application. There was a lot of over-provisioning.”

Now they’re using Platform9’s offerings to break those servers into smaller parts and use VMs to run smaller applications more efficiently. Applications deploy by virtualization, containerization or bare metal; they’re currently using bare metal for software technologies with high machine utilization and heavy resource requirements of compute, memory and network functions.

Lakich says most of the heavy lifting in the migration process was done in the planning phase, since it was the first time Amobee had ever done it.

“We had moved OpenStack around a little bit in our infrastructure and were looking for applications that were a good fit – horizontally scaled, not a lot of hardware‐sticky requirements like persistent storage, etc. Once we identified those applications, the actual migrations were completed within a week, so it was fairly quick.” A couple of UI issues cropped up but were quickly solved, he adds.

Check out the complete case study here and read more about Platform9 in this Superuser profile.


Cover photo: Courtesy Amobee

The post Why hybrid cloud is the new black for an ad tech company appeared first on OpenStack Superuser.

by Superuser at January 18, 2018 03:34 PM

January 17, 2018

OpenStack Superuser

How to get your talk accepted for the next OpenStack Summit

If you’re interested in sharing knowledge with fellow Stackers, you’re in good company: there are typically more than 1,500 submissions for the OpenStack Summit. All community members are encouraged to submit, here are a few tips to improve the chances that your pitch is picked.

Here’s what’s new along with some time-tested tips on what the selection members are looking for. The deadline to submit talks for the Vancouver Summit is February 8.

What’s new for 2018


If you’ve attended an OpenStack Summit, you know there have been dozens of tracks crisscrossing around topics and sectors. For Vancouver these will consolidate into the following eight areas:

  • CI/CD
  • Container infrastructure
  • Edge computing
  • HPC / GPU / AI
  • Open source community
  • Private and hybrid cloud
  • Public cloud
  • Telecom and NFV

“We encourage you to submit proposals covering OpenStack and the open infrastructure tools you’re using, as well as the integration work needed to address these problem domains,” says Lauren Sell, vice president, marketing and community services at the Foundation. “We also encourage you to invite peers from other open source communities to come speak and collaborate.”

Programming Committees

Instead of Track Chairs, there will now be Programming Committees for each track, with both Members and a Chair (or co-chairs). “We’re also recruiting members and chairs from many different open source communities working in open infrastructure, in addition to the many familiar faces in the OpenStack community who will lead the effort,” Sell adds.  If you’re interested in helping out, the deadline for applying to the Committees is January 26.

Landing a talk

Here are a few tips to make your proposal stand out from the pack from veteran speaker Beth Cohen, cloud networking product manager at Verizon.

  • No matter how wonderful you think your innovation is,  product pitch presentations are highly discouraged.
  • Speakers are encouraged to demonstrate past experience with making presentations and speaking to large audiences. If you do not have prior speaking experience, consider participating on a panel.
  • Make the proposal compelling. The community is always interested in real user stories – good and bad, new innovations and details about technical contributions to OpenStack components and of course, the platform itself.
  • Provide any relevant links to additional information, such as code repositories, case studies, papers, blog posts, etc.

More details on designing a successful OpenStack Summit submission can be found in this post from a Women of OpenStack video webinar hosted by Niki Acosta, Anne Gentle and Diane Mueller.

Or you can check out this recent podcast from the Women of OpenStack.

The post How to get your talk accepted for the next OpenStack Summit appeared first on OpenStack Superuser.

by Superuser at January 17, 2018 03:01 PM

Cisco Cloud Blog

Cloud Unfiltered Episode 32: Steve Dake

In this episode of Cloud Unfiltered, Steve Dake discusses the importance of “adjacent community work,” why Zuul should break off from OpenStack, and the future of Istio.

by Ali Amagasu at January 17, 2018 01:55 PM

January 16, 2018

Chris Dent

TC Report 18-03

If a common theme exists in TC activity in the past week, it is the cloudy nature of leadership and governance and how this relates to what the TC should be doing, as a body, and how TC members, as individuals, should identify what they are doing ("I'm doing this with my TC hat on", "I am not doing this with my TC hat on").

It's a bit of a strange business, to me, because I think much of what a TC member can do is related to the relative freedom being elected allows them to achieve. I feel I can budget the time to write this newsletter because I'm a TC member, but I would be doing a bad thing if I declared that this document was an official utterance of OpenStack governance™.

Other TC members probably have a much different experience.

Entropy and Governance

The theme started with a discussion about driving some cleanup of stale repos on https://git.openstack.org/cgit and whether that was an activity that should be associated with the TC role. It is clear there are some conflicts:

  • Because many repositories on git.openstack.org are not official OpenStack projects it would be inappropriate to manage them out of existence. In this case, using OpenStack infra does not indicate volunteering oneself to be governed by the TC. Only being official does that.
  • On the other hand, if being on the TC represents a kind of leadership and presents a form of freedom-to-do, then such cleanups represent an opportunity to, as Sean put it, improve things: "Governed or not, I care about OpenStack and would like to see it not weighed down by entropy." In some sense, the role of the TC is to exercise that caring for OpenStack and what that caring is is context-dependent.

These issues are further complicated by the changing shape of the OpenStack Foundation where there will be things which are officially part of the Foundation (such as Kata), and may use OpenStack infra, but have little to no relationship with the TC. Expect this to get more complicated before it gets less.

That was before office-hours. By the time office hours started, the conversation abstracted (as it often does) into more of a discussion about the role of the TC with me saying:

What I'm upset (mildly) about is our continued effort to sort [of] not have the TC fill the leadership void that I think exists in OpenStack. The details of this particular case are a stimulus for that conversation, but not necessar[il]y relevant.

(I did get a bit cranky in the discussion, my apologies to those who there. This is one of the issues that I'm most passionate about in OpenStack and I let myself run away a bit. My personal feeling has always been that we need an activist and responsive TC if we expect to steward an environment that improves and adapts to change.)

The log is worth reading if this is a topic of interest to you. We delineated some problems that have been left on the floor in the past, some meta-problems with how we identify problems, and even had some agreement on things to try.

OpenStack-wide Goals

Mixed in with the above discussions—and a good example of where the TC does provide some coordination and leadership to help guide all the boats in a similar direction—were efforts to establish sufficient proposals for OpenStack-wide goals to make a fair choice. There are now four reviews pending:

It's quite likely that the StoryBoard goal will move to later, to get increased experience with it (such as by using it for tracking rocky goals). That leaves the other three. They provide a nice balance between improving the user experience, improving the operator experience, and dealing with some technical debt.

If you have thoughts on these goals you should comment on the reviews. There is also a mailing list thread in progress.

Late in the day today, there was discussion of perhaps limiting the number of goals. Some projects are still trying to complete queens goals and were delayed for various reasons, including greater than expected time required to adapt to zuulv3.

Interop Testing

Colleen provided a useful summary of the situation with the location of tests for the interop program. This discussion ground to a bit of a halt but needs to be resolved.

Project Boundaries in Expanded Foundation

Qinling has applied to be official. It is a project to do function as a service. This caused some conversation this morning on what impact the expansion of the Foundation will have on the evaluation of candidate projects.

S Cycle Voting

Also this morning Thierry started the process of making it official that the naming poll for the S cycle will be public. If you have reason to believe this is a bad idea, please comment on the review.

by Chris Dent at January 16, 2018 07:00 PM

Placement Container Playground

I've been experimenting with the OpenStack placement service in a container recently. This is a part of exploratory research to see what will be involved in doing one or both of the following:

  • Making a lightweight, but working, install of the placement service that includes as few imports and as small a configuration as possible with minimal knobs.

  • Eventually extracting placement from the large and convoluted nova repo into its own thing.

What follows are some notes, mostly to myself but published here in case others find them useful, about playing with putting placement in a docker-managed container and peeling off some of the useless bits.

Note: The reason for doing this is not to create a working container for placement. If that were the case I'd be collaborating with Kolla and OpenStack Ansible people. No, the goal here is to do things the manual, crufty, hard way to expose issues and learn some stuff.

I tried to keep track of myself in a git repo, called placedock, where, when I remembered, I would commit some state. I tested myself as I went by having a running devstack where the placement service from devstack was replaced by the one managed by the container. In both cases the requests to placement are proxied through apache2 using mod_proxy_uwsgi so all I had to was change the target of the proxy. Testing was driven by the gabbi-tempest plugin. This worked very well.

The first commit provides just a Dockerfile that uses an Alpine base, installs some binary requirements that allow a pip install -e of nova to succeed.

The nova used there is master plus a long running change that allows the database connection for placement to be explicitly for placement (as [placement_database]/connection in the configuration file). Note that just because there's a different connection configuration setting doesn't mean it has to be a different database. In my tests placement data continues to be stored in the nova_api database.

The second commit results in a running placement service but it uses a very large nova.conf with plenty of stuff that is not needed. Because of a placement bug (now fixed), extra steps of linking the conf file within the container are required. That was cleared in a later commit.

The commit that trims the nova.conf is nice because it removes a lot of meaningless cruft and results in a reasonable conf file that can be the same across several instancesj of the placement service:

debug = True

connection = mysql+pymysql://root:secret@

memcached_servers =
project_domain_name = Default
project_name = service
user_domain_name = Default
password = secret
username = nova
auth_type = password
service_token_roles_required = True
auth_url =
www_authenticate_uri =

Even this is longer than it really should be. Ideally all of this could be communicated via environment variables passed to the container and no file would be needed at all. There are other options too.

The most fun commit is the one that trims the python modules. Nova's requirements.txt is 65 lines long and the cascade of transitive dependencies is huge. This change gets the explicit requirements down to less than 30, but the transitive dependencies remain huge because of the package structure in Nova. It's fairly common in nova for __init__.py to contain imports, which means any packages within suffer those imports too. This makes fine sense if there is only one monolithic entry-point, but in nova there are several different api services and command line tools, running in different processes, not all of which need all that stuff.

There are some short term fixes linked from that bug, but the long term fix to get a lightweight placement is to either change the __init__.py files that impact it, or (much better) extract to its own repo. Extraction comes with its own challenges but there may be some tricks to make it less painful (more to be written about that soon).

The final commit in the repo (thus far) switches the uwsgi configuration to use a TCP socket instead of a unix socket. This makes it possible to distribute multiple placement containers across multiple hosts and address then from a reverse proxy. The commit message describes how things are hooked up to apache2 (the http server usually used with devstack).

Things to do Next

  • Make it smaller. It ought to be possible to incrementally chip away at the imports, but it is quite a tangled mess.
  • Copy the config files into the containers rather than sharing them over a volume.
  • Go a step further and explore the work on etcd backends for oslo.config. This wont help the uwsgi config, but that ought to be fairly static. The dynamic configuration values are database connection strings, and authtoken settings.
  • Try it with more than two placement services (works fine with two).
  • Try it with multiple database servers (galera?).

by Chris Dent at January 16, 2018 04:45 PM

OpenStack Superuser

New TripleO quick start cheatsheet

Carlos Camacho offers this kickstart to TripleO, also known as OpenStack On OpenStack. It’s a program aimed at installing, upgrading and operating OpenStack clouds using OpenStack’s own cloud facilities as the foundations – building on Nova, Neutron and Heat to automate fleet management at datacenter scale.

I have created some cheatsheets for people starting to work on TripleO, mostly to help bootstrap a development environment as soon as possible.

The previous version of this cheatsheet series was used in several community conferences (FOSDEM, DevConf.cz), now, they are deprecated as they way TripleO should be deployed changed considerably last months.

Here you have the last version:

The source code of these bookmarks is available as usual on GitHub

And this is the code if you want to execute it directly:

# 01 - Create the toor user.
sudo useradd toor
echo "toor:toor" | chpasswd
echo "toor ALL=(root) NOPASSWD:ALL" \
  | sudo tee -a /etc/sudoers.d/toor
sudo chmod 0440 /etc/sudoers.d/toor
su - toor

# 02 - Prepare the hypervisor node.
mkdir .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
sudo bash -c "cat .ssh/id_rsa.pub \
  >> /root/.ssh/authorized_keys"
sudo bash -c "echo '' \
  >> /etc/hosts"
export VIRTHOST=
sudo yum groupinstall "Virtualization Host" -y
sudo yum install git lvm2 lvm2-devel -y
ssh root@$VIRTHOST uname -a

# 03 - Clone repos and install deps.
git clone \
chmod u+x ./tripleo-quickstart/quickstart.sh
bash ./tripleo-quickstart/quickstart.sh \
sudo setenforce 0

# 04 - Configure the TripleO deployment with Docker and HA.
export CONFIG=~/deploy-config.yaml
cat > $CONFIG << EOF
  - name: control_0
    flavor: control
    virtualbmc_port: 6230
  - name: compute_0
    flavor: compute
    virtualbmc_port: 6231
node_count: 2
containerized_overcloud: true
delete_docker_cache: true
enable_pacemaker: true
run_tempest: false
extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

# 05 - Deploy TripleO.
export VIRTHOST=
bash ./tripleo-quickstart/quickstart.sh \
      --clean          \
      --release master \
      --teardown all   \
      --tags all       \
      -e @$CONFIG      \

Happy TripleOing!!!

This blog post first appeared on Camacho’s website.

Superuser is always interested in tutorials and community content, get in touch at editorATsuperuser.org

The post New TripleO quick start cheatsheet appeared first on OpenStack Superuser.

by Carlos Camacho at January 16, 2018 03:28 PM

OpenStack Blog

User Group Newsletter – January 2018

Welcome to 2018!


Important Vancouver Summit Updates

The countdown is on until the next Summit in May. Below are some important updates regarding CFP, Travel and Summit Passes

  • CFP is now open!

Call for presentations are now open for the Vancouver Summit. The deadline to submit your proposal is February 8th.

There are some new changes to the structure of the CFP, such as track organisation. Read more here. 

  • Summit Passes and Hotels

These sold quick in 2015 when we last went to Vancouver, don’t miss out and secure yours today. You can do so here

  • Visas

For information regarding Visa applications and invitation letters, read about it here.

As a general planning guideline, if a visa is needed, a foreign traveler should apply for his or her visa as soon as possible, preferably no later than 60 days before the travel date.

  • Travel Support Program

The Travel Support Program (TSP) aims to facilitate participation of key contributors to the OpenStack Summit by covering the costs for their travel and accommodations. The grants will cover a combination of flights, accommodation, and access pass to the Summit. More details here.

All contributors to OpenStack (developers, documentation writers, organizers of user groups around the world, Ask moderators, speakers, translators, etc) are invited to submit a request. You can apply by filling out this form.

  • More Questions?

Got any other questions about the Summit? There is an excellent FAQ here

Ops Meetup Tokyo

The next OpenStack Ops Meetup is happening March 6-7 in Tokyo.

Register here.

Information from Wiki

Project Teams Gathering (PTG)

The Project Teams Gathering (PTG) in Dublin February 26 – March 2. It’s an event for anyone who self-identifies as a member in a specific given project team as well as operators who are specialists on a given project and willing to spend their time giving feedback on their use case and contributing their usage experience to the project team.

Upcoming Industry Events


The Foundation will be present at FOSDEM in Brussels, February 3-4 2018.

OpenStack booth will be in building K, level 1


OpenStack delivers a keynote at Chaosscon

Feb 2, 2018 in Brussels


Upcoming CFP

LF Open Source Leadership

Closing date: January 21, 2018

OSCON 2018

Closing Date: January 30, 2018

LinuxCon ContainerCon | CloudOpen China

Closing Date: March 4, 2018

ONS Europe

Closing Date: June 24, 2018

OpenStack Technical Committee E-office Hours

The TC has e-office hours each week to field any questions you may have.

Find out more details here:

by Sonia Ramza at January 16, 2018 10:50 AM

January 15, 2018


The Intelligent Delivery Manifesto

All of the pieces necessary to create Intelligent Delivery already exist, from Infrastructure as Code to multicloud to machine learning. Now we need to combine them into a single methodology.

by Nick Chase at January 15, 2018 03:59 PM

OpenStack Superuser

Tips for getting travel grants to OpenStack events

OpenStack runs on the power of key contributors.

If you have helped out and want to attend the upcoming Project Team Gathering (PTG) or the  OpenStack Summit but need funds for travel, lodging or a conference pass, the Travel Support Program is here for you.

For every Summit, the OpenStack Foundation funds around 30 dedicated contributors to attend. Deadlines are January 25 (PTG) and March 22 (Summit).

You don’t have to be a code jockey, either. In addition to developers and reviewers, the Support program welcomes documentation writers, organizers of user groups around the world, translators, Operators and Ask moderators. (The Support program doesn’t include university students, however, who are encouraged to apply for a discounted registration pass.)

To boost the odds of getting your application accepted, Superuser talked to Allison Price, marketing coordinator at the OpenStack Foundation who also participates in voting on the applications.

Although applying is a quick process, remember to frame your request clearly. Spend some time answering the question about why you want to attend the Summit. If you make it all about how you’d like to network or visit the town where the summit is taking place, your request is likely to get voted down.

“The biggest common mistake people make is not conveying their value to the community,” says Price. “Focus on what you can contribute to the discussions or pinpoint sessions that would be useful to your business and you have a much better chance.”

She shared some concrete examples of successful applications:

  • “I have contributed to the OpenStack Dashboard (Horizon) and I’d like to attend
    Horizon’s design sessions with the goal of making stronger contributions in
    the next cycle…”
  • “I’ve been involved with the OpenStack project since Bexar days and have played a critical role in developing the OpenStack community in my home country. I’m also the founder and the leader of the OpenStack user group here and an OpenStack Ambassador. I plan to keep
    driving community adoption and involvement in my region.”

And some of those that were too vague to get approved:

  • “I am very interested in cloud computing.”
  • “I want to connect with people in open source community and the OpenStack
    summit provides a great opportunity to do that.”
  • “I would like to attend mostly for the networking and to meet other OpenStack developers…This event will be a great opportunity for me to make new friends.”

What are some other reasons applications get rejected? Keep it friendly, Price says, noting that applications criticizing competitors or trash-talking other applicants definitely strike the wrong note.

Applications are voted on by the Travel Committee, which is made up of three people each from the User Committee, Board of Directors, OpenStack Ambassadors, Project Team Leads/Technical Committee members and Foundation staff. The composition of the committee is refreshed for each Summit.

Asking your company to pay for part of the expenses or finding a buddy to room with won’t influence your chances of getting in, Price says. However, she does recommend at least asking if your company will cover some of the costs — because often your employer is happy to chip in and it allows the Foundation to help to more people.

Price has a final recommendation for applicants: if you need a visa to travel, make sure to request it in time. For each Summit, there have been a number of grantees who haven’t made it because of paperwork.

“It’s too bad when people get accepted but can’t make it to the Summit because their documents don’t arrive in time,” she says. “That’s a spot we could’ve filled with another great contributor.”


Cover Photo by Mario Mancuso // CC BY NC; Recipients of the Barcelona Summit grants.

The post Tips for getting travel grants to OpenStack events appeared first on OpenStack Superuser.

by Nicole Martinelli at January 15, 2018 03:03 PM

January 12, 2018

OpenStack Blog

Developer Mailing List Digest January 5-12th

Success Bot Says

  • e0ne on #openstack-horizon [0]: amotoki runs horizon with django 2.0
  • tristianC on #rdo [1]: review.rdoproject.org is now running sf-2.7
  • mriedem on #openstack-nova [2]: nova merged alternate hosts support for server build
  • mriedem on #openstack-nova [3]: After a week of problems, finally got a volume multiattach test run to actually attach a volume to two instances without melting the world. \o/
  • Tell us yours in OpenStack IRC channels using the command “#success <comment>”

Community Goals for Rocky

So far one goal has been proposed by Kendall Nelson for migrating to Storyboard. It was agreed to postpone the goal until the S cycle, as it could take longer than six months to achieve. There is a good backlog of goals [0], just no champions. It’ll be bad for momentum if we have a cycle with no community wide goal.

PTG Post-lunch Presentations

Feedback received from past PTG session(s) was the lack of situational awareness and missed opportunity for “global” communication at the event. In Dublin we’d used the end of the lunch break to for communications that could be interesting to OpenStack upstream developers and project team members. The idea is not to find a presentation for everyday, but if we find content that is generally useful. Interesting topics include general guidance to make the most of the PTG weeks (good Monday content), development tricks, code review etiquette, new library features you should adopt, lightning talks (good Friday content). We’d like to keep the slot under 20 minutes. If you have ideas please fill out this etherpad [0] in a few weeks.

by Mike Perez at January 12, 2018 08:34 PM

OpenStack Superuser

Building the Middle East’s largest OpenStack-based public cloud

There are plenty of lessons to be learned when deploying a huge OpenStack-based cloud at a massive telecommunications company in the Middle East.

NetApp’s Chris Burnet and Saudi Telecom’s Talal AlBakr discussed their successes and challenges during the process, detailing their reasons for building it in the first place, the services offered through the deployment, and the operational parts of the business, like billing and service management at the OpenStack Summit in Sydney.

NetApp, known for its storage solutions,  has been transitioning to the data management cloud space recently. NetApp helps other companies build OpenStack-based clouds, as well as moving data to and from them. NetApp has been working with OpenStack since the early days, as a charter member that has sponsored every summit.

Saudi Telecommunications Company

Saudi Telecom’s vice president of cloud services, Talal AlBakr, said that his company’s experience with the cloud was an interesting one. “I believe that we’ve really approached it in a unique way,” he said. “We’ve made mistakes. It’s a learning process. It’s a journey. We’ve evolved. The team has really come a long way from where we started in the beginning and where we are today.”

Saudi Telecom Company (STC) is one of the largest companies in Saudi Arabia, said AlBakr. It’s also one of the largest telcos in the world, in the top 20 or 30 globally. “And part of that initiative of course for all telcos is they like to always change and start to venture into new technologies or what we call next generation ICT,” he said. Some of the core competences around information and communication technology are the cloud, the internet of things, big data, and artificial intelligence.

Even just three years ago, said AlBakr, there was no real push for a public cloud. As costs increased, the government wanted to find new ways to move from a capital expenditures model to a operating expenses model. “And that’s when we started to see an opportunity to work on developing what we needed to do and how we could approach the market in a different way,” AlBakr said.

Saudi Arabia doesn’t have many official regulations about data sovereignty, he said. “There are unofficial regulations around data sovereignty, and government entities had to be within the local jurisdiction to be able to benefit from multiple things and to be accessed by different law enforcement agencies,” said AlBakr. “So the idea was to build something around that and to help them with that.”

Recently, a new initiative was announced in the region for a city of the future, called NEOM. “It’s a five hundred trillion dollar initiative,” said AlBakr, “a city (shared) between Saudi Arabia, Jordan and Egypt. It’s supposed to reflect how the push for the future will be. We hope to be a very important aspect in delivering on that future and pushing for that aspect.”

The company aimed for a launch date of late 2015.  The team chose OpenStack to avoid vendor lock-in, said AlBakr, and an extremely attractive price point. STC was getting a lot of government initiatives, which helped. With such a new undertaking, however, STC needed to find a lot of talent, a scarce commodity in the current Saudi tech landscape, said AlBakr. “The open source community in Saudi was extremely small,” he said, “and the development community, or what we had with regards to open source, was around urbanization of traditional technologies and traditional projects..so venturing into this was for us an extremely big risk.”

STC found a partner in Mirantis after analyzing the offerings of multiple distribution vendors. “Mirantis offered a lot of flexibility,” said AlBakr. “Other vendors, or other distributions were becoming more closed, more tied into specific eco-system requirements. We wanted the agility and the flexibility to move in projects as we saw fit, based on business requirements and on needs that the industry pushed our way.” That Mirantis a top contributor to OpenStack and a gold member of the Foundation was also important to STC.

AlBakr and his team began with its first iteration of the STC public cloud in 2015, calling themselves Bluvalt to distinguish it from the main STC eco-system. “We built a cloud based on Mirantis OpenStack six, which was in closed beta mode,” said AlBakr. “We launched, invited specific government agencies and large corporations to come into the cloud for free and started to onboard them.” Blubalt did all of that without building requirements, pricing models, or other traditional startup activities.

A second shift brought STC to a more mature architecture, based on automation and integrated billing with STC. “It was based on an eco-system that we created for partners to support the cloud and build on it,” he said. “Right now, we’re at our third iteration  the cloud and it’s our mass-market production, we split our cloud into two different clouds, virtually.” One cloud serves the enterprise and the government sectors and the other serves small-to-medium enterprises.

The first iteration, however, used Neutron VLAN segmentation, leading-edge at the time. “It caused us some issues as we started to progress,” said AlBakr. “This is one of the other aspects that we see as a work-in-progress in the OpenStack (community).”

The second iteration used Mitaka, which the team found to be fairly stable. Once they moved from Neutron Contrail to OpenContrail, many pieces fell into place. “This is where we started to see our offering mature, and we started to see customers get more adapted into what we started to do,” said AkBakr. “In our iteration in deployment one, we had a lot of issues around networking, around structure, and one of the biggest aspect was, when we started working on our beta and onboarding customers, we reached a situation where we got so much demand that we couldn’t leave beta anymore, and we went full production on that beta environment, which in all honesty was, it was a good and a bad mistake at the same time.” Good because it let STC approach the market very quickly; bad because the design was extremely under-optimized.

The third iteration is set to launch in the first quarter of 2018 with a new Mitaka cloud. The initial cloud is unable to be upgraded anymore and STC has had to build a cloud next to it, then migrate everyone across to it. “This is one of the things that we’re seeing as a draw back within OpenStack,” said AlBakr, “but with the new iterations of the different technologies that we’re seeing with Mirantis, especially on the Mantis cloud platform, it’s becoming a lot more stabilized. Hopefully with containerization we’re going to see things get more streamlined and moving in a better manner.”

The STC cloud runs a ton of projects, according to AlBakr. “We have Swift, we have Neutron,” he said, “we have all these different solutions, OpenID, Keystone, all these different projects running in order to authenticate and to control the whole environment.” As the cloud evolved, the team started moving things around in order to optimize things. OpenContrail and Juniper Contrail were important in that regard, while routing and configuration management are based on the REST API for OpenStack.

The team began to do control analysis when customers started having performance and configuration issues. “With traditional technologies we had problems troubleshooting that,” said AlBakr. “When we started off with our initial customer set, it was manageable because it was a handful of customers, but as we started to progress, it started to blow out of control. And this is where we started to really relying on the updated technologies like OpenContrail.”

NetApp is one of STC’s biggest partners, in terms of OpenStack. Initially, the organization’s entire storage infrastructure was based on FAS and ONTAP technology. As the project progressed, several barriers presented themselves, which caused AlBakr’s team to consider newer technologies. That’s where SolidFire came into play. “SolidFire for us was extremely attractive based on the modular growth capabilities,” said AlBakr. “FAS was an excellent technology but the problem was (how) it limits you to the control of capability. With SolidFire, you just add more nodes as you progress, you get more capacity, more performance, and for a service provider that’s very important.”

A secondary issue arose. “Our Swift Object store was not optimal. It was providing only Swift capabilities. We were getting a lot of requirements from our customers for S3 capabilities and for NFS. So we were facing a lot of different issues around that and this is where we started looking into the StorageGRID.” Removing away from the Swift package to an isolated StorageGRID environment has allowed AlBakr and his team to take care of the S3 and Swift connectivity aspects of their project, offloading it, as it were.

To manage the environment, STC uses a lot of different technologies that come within OpenStack. They monitor their databases, their back end and front end with Grafana and other third-party applications. They have many in-house developed apps that they monitor different aspects of the environment with, including fault tolerance and performance management tools like Elasticsearch, Kibana and others.

One of the most important projects in the STC cloud is its marketplace, what AlBakr calls the “bloodstream of what our cloud is.” As a public cloud, STC needed something it could fully customize to be able to bring in a better price point for its customers, to make it more attractive than, say, Amazon or other providers. “So we launched our marketplace on OpenStack,” he said. “It’s in-house developed. We have a team of around 40 developers working on it continuously, coming up with a lot of updates. We’ve introduced billing per hour, instead of per month and to be more granular, it’s all done within our marketplace. “

The team has also introduced integration with the STC billing system as well as integration with credit card billing, adding even more layers to their system. Both internal teams and external customers utilize the service order management system in place. They also integrated a bus and API environment that can pull all the information from their compute, SDN, object storage, and block storage plans to send it out to STC’s internal or credit billing systems.

The team broke it down into three different segments. The first segment, cloud service providers, got a community for developers to onboard their applications onto the marketplace as a launching pad or app store. That helped empower the local developer community to approach the market with custom-made applications specific to the market. Secondly, STC built a cloud integration partner program that brought system integrated partners get onboard, get certified and then work with customers to build and integrate their own environments. STC even went one step further and do managed services. “Now all of this was under the guidance of our cloud,” said AlBakr. “We took care of the billing. We took care of the project management, and we made sure that the project continued and progressed in the right way. And, it allowed us to really cover a big reach of customers and it allowed us to touch base and to ensure a customer satisfaction ratio that was higher than normal, because we had an additional eco-system to support us.”

Finally, the team defined a segment it calls cloud technology partners. It’s based on technologies that STC has integrated into the marketplace to tackle specific different requirements, like backup, load balancing, and other such services.”This also allows to provide a more complete cloud experience for our customers instead of having them go and buy it from a third party and bring it in,” said AlBakr.

In essence STC created a program. “It gets integrated. It gets streamlined, and it’s all done through our marketplace,” said AlBakr. “Our approach to the marketplace is a little different than traditional approaches. We’re not really trying to just provide different solutions around databases and so on and so forth, we’re trying to provide actual on-premises replacement concepts.”

At the end of the day, STC now has an end-to-end cloud solution. The company provides the mechanism for mechanism, infrastructure, and platform as a service. “We’re launching a lot of services in the next few months, and we’re at a rate of around, I think, five services a month,” AlBakr said. “Most public cloud providers need to keep an eye on the big guys (and) what they’re doing. We’re trying to progress as fast and as quick as we can to keep up with them and to cover as (many) services as they cover.”

As an example, Saudi Telecom Company is able to provide layer three IPVPN. This gave the telco leverage with customers. “So when we sit with customers, we tell them,we’re going to sell you data, internet connectivity and access,” he said. “And in addition to that, we’re going to give you a VPN layer that allows your data center to be part of our data center without going over the internet layer. So this for a lot of customers is very critical because it allows for a more streamlined, integrated solution within their systems. It doesn’t feel like they’re going outside and into the cloud and things like that. For us that’s a very good differentiator and it makes a real different for a lot of our customers.”

OpenStack was a good experience for STC. It allowed the telco to reach extremely attractive price points, with flexibility and plenty of capabilities to offer, which can continue to expand as the company progresses. “We don’t know why people keep saying OpenStack is dying,” said AlBakr. “We see it today flourishing more than ever, and we are all in with OpenStack. We feel it’s the direction of the future. Our market place was a key differentiator for us, and it continues to be within the market. We believe that our solution is extremely unique.”

The marketplace was a huge success, as well, becoming so widely known and praised that STC is looking to commercialize it and sell it to other entities as well as use it for other groups within its own organization. In addition, AlBakr said, storage block, automation and the cloud service provider partnerships have gone very well for the company.


What didn’t go so well was that the initial compute calculations were off and prices were extremely high. “We didn’t understand the intricacies and the different dynamics of how to really price point different aspects in the right way,” said AlBakr. “But as we progressed, we were able to do pricing in the right way to be in line with global prices.” Not that they’re necessarily cheaper, but the prices are much more attractive now and allow a different approach with customers.

The team had issues with migrations around object and block storage, and performance issues as a result. “That’s why we’re addressing it right now with NetApp and StorageGRID,” said AlBakr, “and we’re trying to move to that.” The team is going all-in with Object and has the potential of six petabytes of storage capacity it can grow within the next few months, in addition to the existing storage it already has.

There were also I/O problems, including issues with getting logs. In addition, Microsoft licensing has been a problem as well. “Microsoft licensing, specifically for support is extremely hard on OpenStack,” said AlBakr, “and I think for all cloud vendors.”

Red Hat became another issue because of its lack of support for Mirantis. It took the team aback, said AlBakr. “Why? Red Hat is a leader in the OpenStack community,” he said. “It’s a leader in the open source community but it’s doing all these different aspects to leverage itself. For us it was extremely negative.”

Intially, STC didn’t have availability zones, which caused some problems, thought they’ve started to implement them. There were problems around Boot from image systems as well as customer image and size requirements. VLAN segmentation before they implemented SDN was an issue, too. The upgrade from Liberty to Mitaka wasn’t the smoothest upgrade in the world, said AlBakr, and caused a lot of problems.

A bigger issue, says AlBakr, is the perception of OpenStack as an enterprise play rather than one suited to a public cloud.

“So when we have discussions on downtime, people say, ‘Okay, give me a two or three hour downtime window.’ And I’m like, ‘No. This is a public cloud. I can’t afford a two or three hour downtime window. If I get two, three hour downtime window, that means I lose my customers, because some people have their whole business running on this cloud.’” He’d like to see this perception change and evolve with OpenStack in general.

You can catch the entire talk on the video below.


Cover image courtesy STC Instagram feed

The post Building the Middle East’s largest OpenStack-based public cloud appeared first on OpenStack Superuser.

by Rob LeFebvre at January 12, 2018 02:51 PM

January 11, 2018


Summary of rdopkg development in 2017

During the year of 2017, 10 contributors managed to merge 146 commits into rdopkg.

3771 lines of code were added and 1975 lines deleted across 107 files.

54 unit tests were added on top of existing 32 tests - an increase of 169 % to total of 86 unit tests.

33 scenarios for 5 core rdopkg features were added in new feature tests spanning total of 228 test steps.

3 minor releases increased version from 0.42 to 0.45.0.

Let's talk about the most significant improvements.


rdopkg started as a developers' tool, basically a central repository to accumulate RPM packaging automation in a reusable manner. Quickly adding new features was easy, but making sure existing functionality works consistently as code is added and changed proved to be much greater challenge.

As rdopkg started shifting from developers' powertool to a module used in other automation systems, unevitable breakages started to become a problem and prompted me to adapt development accordingly. As a first step, I tried to practice Test-Driven Development (TDD) as opposed to writing tests after a breakage to prevent specific case. Unit tests helped discover and prevent various bugs introduced by new code, but testing complex behaviors was a frustrating experience where most of development time was spent on writing units tests for cases they weren't meant to cover.

Sounds like using a wrong tool for the job, right? And so I opened a rather urgent rdopkg RFE: test actions in a way that doesn't suck and started researching what cool kids use to develop and test python software without suffering.

Behavior-Driven Development

It would seem that cucumber started quite a revolution of Behavior-Driven Development (BDD) and I really like Gherkin, the Business Readable, Domain Specific Language that lets you describe software's behaviour without detailing how that behaviour is implemented. Gherkin serves two purposes — documentation and automated tests.

After some more research on python BDD tools, I liked behave's implementation, documentation and community the most so I integrated it into rdopkg and started using feature tests. They make it easy to describe and define expected behavior before writing code. New features now start with feature scenario which can be reviewed before writing any code. Covering existing behavior with feature tests helps ensuring they are both preserved and well defined/explained/documented. Big thanks goes to Jon Schlueter who contributed huge number of initial feature tests for core rdopkg features.

Here is an example of rdopkg fix scenario:

    Scenario: rdopkg fix
        Given a distgit
        When I run rdopkg fix
        When I add description to .spec changelog
        When I run rdopkg --continue
        Then spec file contains new changelog entry with 1 lines
        Then new commit was created
        Then rdopkg state file is not present
        Then last commit message is:

            - Description of a change

Proper CI/gating

Thanks to Software Factory, zuul and gerrit, every rdopkg change now needs to pass following automatic gate tests before it can be merged:

  • unit tests (python 2, python 3, Fedora, EPEL, CentOS)
  • feature tests (python 2, python 3, Fedora, EPEL, CentOS)
  • integration tests
  • code style check

In other words, master is now significantly harder to break!

Tests are managed as individual tox targets for convenience.

Paying back the Technical Debt

I tried to write rdopkg code with reusability and future extension in mind, yet in one point of development with big influx of new features/modifications, rdopkg approached critical mass of technical debt where it got into spiral of new functionality breaking existing functionality and with each fix two new bugs surfaced. This kept happening so I stopped adding new stuff and focused on ensuring rdopkg keeps doing what people use it for before extending(breaking) it further. This required quite a few core code refactors, proper integration of features that were hacked in on the clock, as well as leveraging new tools like software factory CI pipeline, and behave described above. But I think it was a success and rdopkg paid its technical debt in 2017 and is ready to face whatever community throws at it in near and far future.


Join Software Factory project

rdopkg became a part of Software Factory project and found a new home alongside DLRN.

Software Factory is an open source, software development forge with an emphasis on collaboration and ensuring code quality through Continuous Integration (CI). It is inspired by OpenStack's development workflow that has proven to be reliable for fast-changing, interdependent projects driven by large communities. Read more in Introducing Software Factory.

Specifically, rdopkg leverages following Software Factory features:

rdopkg repo is still mirrored to github and bugs are kept in Issues tracker there as well because github is accessible public open space.

Did I meantion you can login to Software Factory using github account?

Finally, big thanks to Javier Peña, who paved the way towards Software Factory with DLRN.

Continuous Integration

rdopkg has been using human code reviews for quite some time, and it proved very useful even though I often +2/+1 my own reviews due to lack of reviewers. However, people unevitably make mistakes. There are decent unit and feature tests now to detect mistakes, so we fight human error with computing power and automation.

Each review and thus each code change to rdopkg is gated - all unit tests, feature tests, integration tests and code style checks need to pass before human reviewers consider accepting the change.

Instead of setting up machines and testing environments and installing requirements and waiting for tests to pass, this boring process is now automated on supported distributions and humans can focus on the changes themselves.

Integration with Fedora, EPEL and CentOS

rdopkg is now finally available directly from Fedora/EPEL repositories, so install instructions on Fedora 25+ systems boiled down to:

dnf install rdopkg

On CentOS 7+, EPEL is needed:

yum install epel-release
yum install rdopkg

Fun fact: to update Fedora rdopkg package, I use rdopkg:

fedpkg clone rdopkg
cd rdopkg
rdopkg new-version -bN
fedpkg mockbuild
# testing
fedpkg push
fedpkg build
fedpkg update

So rdopkg is officially packaging itself while also being packaged by itself.

Please nuke jruzicka/rdopkg copr if you were using it previously, it is now obsolete.


rdopkg documentation was cleand up, proof-read, extended with more details and updated with latest information and links.

Feature scenarios are now available as man pages thanks to mhu.

Packaging and Distribution

Python 3 compatibility

By popular demand, rdopkg now supports Python 3. There are Python 3 unit tests and python3-rdopkg RPM package.

Adopt pbr for Versioning

Most of initial patches rdopkg was handling in the very beginning were related to distutils and pbr, the OpenStack packaging meta-library, specifically making it work on a distribution with integrated package management and old conservative packages.

Amusingly, pbr was integrated into rdopkg (well, it actually does solve some problems aside from creating new ones) and in order to release the new rdopkg version with pbr on CentOS/EPEL 7, I had to disable hardcoded pbr>=2.1.0 checks on update of python-pymod2pkg because older version of pbr is available from EPEL 7. I removed the check (in two different places) as I did so many times before and it works fine.

As a tribute to all the fun I had with pbr and distutils, here is a link to my first nuke bogus requirements patch of 2018.

Aside from being consistent with OpenStack related projects, rdopkg adopted strict sematic versioning that pbr uses, which means that releases are always going to have 3 version numbers from now on:

0.45 -> 0.45.0
1.0  -> 1.0.0

And More!

Aside from the big changes mentioned above, large amount of new feature tests and numerous not-so-exciting fixes, here is a list of changes might be worth mentioning:

  • unify rdopkg patch and rdopkg update-patches and use alias
  • rdopkg pkgenv shows more information and better color coding for easy telling of a distgit state and branches setup
  • preserve Change-Id when amending a commit
  • allow fully unattended runs of core actions.
  • commit messages created by all rdopkg actions are now clearer, more consistent and can be overriden using -H/--commit-header-file.
  • better error messages on missing patches in all actions
  • git config can be used to override patches remote, pranch, user name and email
  • improved handling of patches_base and patches_ignore including tests
  • improved handling of %changelog
  • improved new/old patcehs detection
  • improved packaging as suggested in Fedora review
  • improved naming in git and specfile modules
  • properly handle state files
  • linting cleanup and better code style checks
  • python 3 support
  • improve unicode support
  • handle VX.Y.Z tags
  • split bloated utils.cmd into utils.git module
  • merge legacy rdopkg.utils.exception so there is only single module for exceptions now
  • refactor unreasonable default atomic=False affecting action definitions
  • remove legacy rdopkg coprbuild action

Thank you, rdopkg community!

January 11, 2018 06:23 PM

January 10, 2018

Ben Nemec

Zuul Status Page Production-Ready Configuration

About a year ago I got fed up with all the dynamic Zuul status pages (like this one). They're extremely heavyweight pages that make my browser grind almost constantly and there's no way to disable the auto-refresh while you're looking through the list. At least not that I found, and when I asked on openstack-dev the responses agreed.

So I wrote my own.

It was originally designed to keep track of the check-tripleo queue for OVB jobs, but since then I've added better support for the other queues as well. I'm very happy with how it works (shocking!), with one major exception: Periodically the webapp behind it will just stop responding. I suspect it has to do with connectivity issues to the Zuul status JSON endpoint and I've tried to set some timeouts to avoid that, but nothing I've tried has worked. Once a request gets stuck the entire app stops responding.

I suspect a big part of the problem is that I was using the WSGI server built in to Python instead of a production-ready one like uWSGI. It does work, but it seems to be a single-threaded implementation with no ability to handle parallel requests. When a request gets stuck this obviously is a problem. To address this, I've switched to running the app in uWSGI fronted by nginx. This seems to be a popular configuration and it's being used by some OpenStack services as well so it seemed like a useful thing to learn about.

You can see my uWSGI configuration file in the Git repo. I'll include my nginx config file at the end of the post. The command I use to start uWSGI is sudo uwsgi --ini zuul-status.ini

There are a few things I suspect are not ideal (socket in /tmp and running as root), but this is just running in a throwaway vm so if someone does manage to hose it up through some sort of exploit it won't be the end of the world. Be aware that my configuration is not necessarily following best practices though.

I figured I would write this up since I had some trouble finding complete, working configurations for the Python/Nginx/uWSGI combination. Found lots of people with non-working configs though. :-)

A few other environmental notes:

  • I installed nginx on CentOS 7 using the repo provided by nginx. It sounds like it might also be available in EPEL.
  • I also enabled EPEL to get uWSGI.
  • The uwsgi-plugin-python package is also needed to run Python apps.
  • I create a virtualenv for the webapp in the venv directory that contains all of the dependencies for it.
  • I also shut off SELinux. Yeah, yeah, see above about throwaway VM.

nginx conf.d/default.conf:

server {
    listen       80;
    server_name  localhost;

    location / { try_files $uri @zuul-status; }
    location @zuul-status {
        uwsgi_pass unix:///tmp/uwsgi.sock;
        include uwsgi_params;

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;

And that's pretty much it. The app has only been running in uWSGI for a few hours so I can't say whether it will actually help with stability, but hopefully this will be helpful to someone anyway. I'll try to remember to update this post after a while to report how it worked.

by bnemec at January 10, 2018 08:06 PM

Chris Dent

Gabbi Tempest Experiment 1

One of the nice features that's been built into gabbi from near the start is a command line runner that allows gabbi tests to be run against a running web service. Given args.yaml:

- name: check args
  GET: /get?foo=bar
      $.args.foo: bar

You can do:

gabbi-run http://httpbin.org < args.yaml

and get:

... ✓ gabbi-runner.input_check_args

Ran 1 test in 0.379s

In OpenStack development, gabbi is frequently used in functional test for the HTTP services, sometimes paired with wsgi-intercept. In this context it is relatively easy to set up. Using it for integration tests, however, gets a bit more complicated because there's a lot to set up so that gabbi is properly aware of the services in the cloud that's being tested. The gnocchi and heat projects have done a fair bit of work to make tools that work for them.


Based off that work, I've created a plugin for Tempest (an integration testing framework for OpenStack) called gabbi-tempest that tries to take care of some of the necesary set up while preserving the ease of use you get with gabbi-run. The result allows commands like:

GABBI_TEMPEST_PATH=/home/cdent/ds1/gabbits tempest run --regex gabbi

gabbi-tempest itself is responsible for establishing simple credentials and providing gabbi with enough environment variables to be able to find the endpoints for all the services in the service catalog. From there it is possible to write gabbi test files that uses those services.

Trying It Out

I've made a little git repo, gabbi-tempest-sample that shows the results of experimenting with gabbi-tempest to talk to a cloud (created by devstack in this case, but that's not too relevant). You can repeat the experiment yourself with the following instructions.

Install tempest and gabbi-tempest somewhere (if you prefer to work in a virtualenv, please do):

git clone https://git.openstack.org/openstack/tempest
pip install tempest/
pip install gabbi-tempest

Tempest plugins work using entry points, so if tempest and gabbi-tempest are in the same Python environment, tempest will be aware of the plugin without further effort.

Create a workspace from which testing will happen. This will create a basic tempest.conf that will be used to find the target cloud. I've named my workspace ds1 because that's the host where my devstack-based cloud lives.

tempest init ds1

Edit tempest.conf to add information for accessing the Keystone service and identifying an existing user and password in your cloud. In my tests I'm using Keystone v3. If you are not you will need to make somewhat different changes to your configuration. At the moment, the gabbi-tempest plugin requires a user named 'admin' but that will change.

The intent here is have a minimal configuration. Here's what my file looks like:

log_dir = /home/cdent/ds1/logs
log_file = tempest.log

lock_path = /home/cdent/ds1/tempest_lock

auth_version = v3
admin_domain_scope = True
uri_v3 =

admin_domain_name = Default
admin_project_name = admin
admin_password = replace_me
admin_username = admin

uri_v3 is the URI for accessing the keystone service.

Create some YAML to confirm things are working. It's common for gabbi YAML files to be called gabbits. Create a directory with that name and in there create a new file (replace vim with your preferred editor):

mkdir gabbits
vim gabbits/sample.yaml

Gabbi YAML files are a sequence of HTTP requests in a tests attribute, each of which uses any defaults defined in a defaults attribute. We will set some defaults that will be used in every request. The primary purpose here is to make sure that authentication is handled and that the proper content types and microversions are used. For the sake of this experiment we will use the latest microversions, but usually that's not what you want. Add a default section:

        x-auth-token: $ENVIRON['SERVICE_TOKEN']
        content-type: application/json
        accept: application/json
        openstack-api-version: 'compute latest, placement latest'

The SERVICE_TOKEN has been established by the plugin. Any test which does not override these headers will use them.

Now let's create one simple test that lists the servers on the cloud and tests that there are 0 of them:

    - name: list servers
          $.servers.`len`: 0

It also tests that the response status code is 200. If you want to test for a different status each test can take a status field. See gabbi test format docs for more information.

Run that test. To do so we need to tell tempest to only run gabbi tests. We do that with the --regex parameter (but see the caveats, below) By default the plugin will look in the current working directory for a directory named gabbits to find one or more YAML files:

tempest run --regex gabbi

If there are in fact zero servers, you'll see test output that looks something like this:

{0} tempest.scenario.home.cdent.ds1.gabbits.sample_list_servers.test_request [0.260659s] ... ok

Make the test verbose to see what's really happening. By changing the test to add verbose: True we can see the request and response. Make the list servers test look like this:

    - name: list servers
      verbose: True
          $.servers.`len`: 0

Now when you run the tests (as above), you'll also see verbose representations of the request and response bodies and headers. You can make this the default for the file by putting verbose: True in the defaults section instead.

Getting More Complicated

The repo includes a considerably more complex example. This one talks to three services, accepts information from environment variables passed on the command line, and uses advance features of both YAML and gabbi to make reference to data in other responses in the sequence. What it tests is the behavior of the placement service when booting a single server and resizing it.

To try it out, the user in your tempest.conf must have the admin role, as the placement service requires that. Move away your existing sample.yaml and replace it with the complex one. When running it we'll demonstrate some additional features:

GABBI_TEMPEST_PATH=`pwd`/gabbits:/some/other/gabbits FLAVOR_NAME=m1.nano \
   FLAVOR_NAME_ALT=m1.micro IMAGE_NAME=cirros-0.3.5-x86_64-disk \
   tempest run --regex gabbi

GABBI_TEMPEST_PATH contains one or more directory paths in which the gabbi-tempest plugin should look for gabbi YAML files. It is formatted in the same fashion as the standard Unix PATH (colon separated directories). This allows a caller to keep their gabbits in lots of different places, if that's useful.

The NAME variables shows ways in which the caller can pass information into the tests. In this case we are telling the tests which flavors and images to work with, without needing to know their ids (the ids are discovered by tests and referred to in subsequent tests).

Hopefully you can see that this has some potential. I would love some feedback and of course any offers of help are very welcome.

Also, if you've been paying attention, you've probably noticed that tempest isn't really needed here. It's a convenience that allows you to be integrated with the way that tempest accounts for tests and gathers information about the environment. If you don't need that, you could set all the environment variables that are used for substituting service location and auth yourself and use gabbi-run directly.


While the above experiments have worked for me there are some areas of concern that are going to need some additional learning or work:

  • tempest run does test discovery in a way that means the --regex handling does not match on the test names that gabbi generates in its dynamic processing. It is matching on the plugin's code. It may be the case that this situation will change when tests are run in parallel (where tests are listed, grouped, and fed back to the runner) but that introduces other issues.
  • If the --regex parameter is not used, then all tests that tempest can find are run, including the tests provided by the gabbi-tempest plugin and any other plugin active in the environment.
  • In each individual YAML file the test are expected to be, and enforced as, a sequence. In a concurrent environment where care is not taken, this can mean that tests could be run multiple times in separate processes. This can be controlled by manipulating the group_regex (see how this done for nova functional tests) but it is not yet clear how to integrate this cleanly.
  • User management is a bit limited, requiring a user named 'admin' at the moment, but that should be relatively easy to fix.
  • Plenty of other things I don't now but you may. Tell me.

by Chris Dent at January 10, 2018 07:15 PM

OpenStack Superuser

Where to connect with OpenStack in 2018

Every year, thousands of Stackers connect in real life at local meetups, OpenStack Days, the Summits and hackathons.

Start filling out your 2018 calendar at the events page or find your local folks on the User Groups portal.

Right out of the gate in January, user groups from Ankara to Utah are putting their heads together. You’ll also find community members answering questions from a booth at the fantastic FOSDEM in Brussels, Februrary 3-4. (For what to expect, check out our coverage here.)

Next milestone: the Project Teams Gathering (PTG) in Dublin February 26 – March 2. It’s an event for anyone who self-identifies as a member in a specific given project team as well as operators who are specialists on a given project and willing to spend their time giving feedback on their use case and contributing their usage experience to the project team. Wondering what it’s like when the “zoo” of OpenStack projects get together? Check out this write-up.

Coming up in May, the OpenStack Summit will come together again in Vancouver and before heading over to Berlin in November, there’s an entire summer of get-togethers focusing on open infrastucture from Australia to Israel. (The calendar, much like the technology, is constantly evolving so keep an eye on that events page as it fills up for the year.)

Once you’ve saved the date, remember that Superuser is always interested in community content – get in touch at editorATopenstack.org and send us your write-ups (and photos!) of events. Articles posted are eligible for Active User Contributor (AUC) status.

Cover Photo // CC BY NC

The post Where to connect with OpenStack in 2018 appeared first on OpenStack Superuser.

by Superuser at January 10, 2018 03:03 PM

Cisco Cloud Blog

Cloud Unfiltered Episode 31: Talking Kubernetes with Tony Campbell

Tony Campbell has a long history with OpenStack and Rackspace, so his cloud credentials are solid. Now he works at CoreOS, so we talk to him about containers and Kubernetes.

by Ali Amagasu at January 10, 2018 01:49 PM


RDO Community Blogposts

If you've missed out on some of the great RDO Community content over the past few weeks while you were on holiday, not to worry. I've gathered the recent blogposts right here for you. Without further ado…

New TripleO quickstart cheatsheet by Carlos Camacho

I have created some cheatsheets for people starting to work on TripleO, mostly to help them to bootstrap a development environment as soon as possible.

Read more at http://anstack.github.io/blog/2018/01/05/tripleo-quickstart-cheatsheet.html

Using Ansible for Fernet Key Rotation on Red Hat OpenStack Platform 11 by Ken Savich, Senior OpenStack Solution Architect

In our first blog post on the topic of Fernet tokens, we explored what they are and why you should think about enabling them in your OpenStack cloud. In our second post, we looked at the method for enabling these.

Read more at https://redhatstackblog.redhat.com/2017/12/20/using-ansible-for-fernet-key-rotation-on-red-hat-openstack-platform-11/

Automating Undercloud backups and a Mistral introduction for creating workbooks, workflows and actions by Carlos Camacho

The goal of this developer documentation is to address the automated process of backing up a TripleO Undercloud and to give developers a complete description about how to integrate Mistral workbooks, workflows and actions to the Python TripleO client.

Read more at http://anstack.github.io/blog/2017/12/18/automating-the-undercloud-backup-and-mistral-workflows-intro.html

Know of other bloggers that we should be including in these round-ups? Point us to the articles on Twitter or IRC and we'll get them added to our regular cadence.

by Mary Thengvall at January 10, 2018 02:34 AM

January 09, 2018

OpenStack Blog - Swapnil Kulkarni

Setting up the development environment for Kata Containers – proxy

Summarizing the information for setting up the development environment for my first project in Kata-Containers. I have setted up the dev environment for proxy project.

First Things FirstInstall Golang as a prerequisite to the development. Ensure you follow the complete steps to create the required directory structure and test the installation.

Get The Source

This guide assumes you already have forked the proxy project. If not please for the repo. Once you have successfully forked the repo, clone it on your computer

git clone https://github.com/<your-username>/proxy.git $GOPATH/src/github.com/<your-username>/proxy

Add the upstream proxy project as remote to the local clone to fetch up the updates.

$ cd proxy
$ git remote add upstream https://github.com/kata-containers/proxy.git

The proxy project requires following dependencies to be installed prior to build. Use following command to install them.

$ go get github.com/hashicorp/yamux
$ go get github.com/sirupsen/logrus

Do the first build. This will create the executable file kata-proxy in the proxy directory.

$ make
go build -o kata-proxy -ldflags “-X main.version=0.0.1-02a5863f1165b1ee474b41151189c2e1b66f1c40”

To run unit tests run

$ make test
go test -v -race -coverprofile=coverage.txt -covermode=atomic
=== RUN TestUnixAddrParsing
— PASS: TestUnixAddrParsing (0.00s)
=== RUN TestProxy
— PASS: TestProxy (0.05s)
coverage: 44.6% of statements
ok github.com/coolsvap/proxy 1.064s

To remove all generated output files run

$ make clean
rm -f kata-proxy

This is for this time. I am working on setting up the development environment with GolangD IDE. Keep you posted.

by Swapnil Kulkarni at January 09, 2018 04:00 PM

OpenStack Superuser

OpenStack, Spectre and Meltdown: What you need to know

Thierry Carrez, vice president of engineering, OpenStack Foundation and the OpenStack Technical Committee chair offers his thoughts on the security vulnerabilities dubbed “Spectre” and “Meltdown.”

What are Meltdown and Spectre ?

Meltdown and Spectre are the names of a series of vulnerabilities discovered by various security researchers around performance optimization techniques built in modern CPUs. Those optimizations (involving superscalar capabilities, out-of-order execution, and speculative branch prediction) fundamentally create a side-channel that can be exploited to deduce the content of computer memory that should normally not be accessible.

“The answer is always the same: designing defense in depth, keeping track of vulnerabilities found and swiftly applying patches”

Why is it big news?

It’s big news because rather than affecting a specific operating system, it affects most modern CPUs in ways that cannot be completely fixed (since you can’t physically extract the flawed functionality from your CPUs). The real solution is in a new generation of CPU optimizations that will not exhibit the same flaws while reaching the same levels of performance. These are unlikely to come out soon, which means we’ll have to deal with workarounds and mitigation patches for a long time.

Why is it business as usual?

As Bruce Schneier says, “You can’t secure what you don’t understand.” As we build more complex systems (in CPUs, in software, in policies), it’s more difficult to build them securely and they can fail in more subtle ways. There will always be new vulnerabilities and new classes of attacks found and the answer is always the same: designing defense in depth, keeping track of vulnerabilities found and swiftly applying patches. This might be big news, but the solution is still applying well-known techniques and processes.

Are those two or three different vulnerabilities?

There are actually three different exploitation techniques of the same family of vulnerabilities, which need to be protected against separately.

  • CVE-2017-5753 (“bounds check bypass”, or variant 1) is one of the two Spectre variants. It affects specific sequences within compiled applications, which must be addressed on a per-binary basis. Applications that can be made to execute untrusted code (e.g. operating system kernels or web browsers) will need updates as more of those exploitable sequences are found.
  • CVE-2017-5715 (“branch target injection”, or variant 2) is the other Spectre variant. It more generally works by poisoning the CPU branch prediction cache to induce privileged applications to leak small bits of information. This can be fixed by a CPU microcode update or by applying advanced software mitigation techniques (like Google’s Retpoline) to the vulnerable binaries.
  • CVE-2017-5754 (“rogue data cache load”, or variant 3) is also called Meltdown. This technique lets any unprivileged process read kernel memory (and therefore access information and secrets in other processes running on the same system). It’s the easiest to exploit and requires patching the operating system to reinforce isolation of memory page tables at the kernel level.

What’s the impact of those vulnerabilities for OpenStack cloud users?

Infrastructure-as-a-service harnesses virtualization and containerization technologies to present a set of physical, bare-metal resources as virtual computing resources. It relies heavily on the host kernel security features to properly isolate untrusted workloads, especially the various virtual machines running on the same physical host. When those fail (as is the case here), you can have a hypervisor break. An attacker in a hostile VM running on an unpatched host kernel could use those techniques to access data in other VMs running on the same host.

Additionally, if the guest operating system of your VMs is not patched (or you run a vulnerable application) and run untrusted code on that VM (or in that application), that code could leverage those vulnerabilities to access information in memory in other processes on the same VM.

What should I do as an OpenStack cloud provider?

Cloud providers should apply kernel patches (from their Linux distribution), hypervisor software updates (from the distribution or their vendor) and CPU microcode updates (from their hardware vendor) that workaround or mitigate those vulnerabilities as soon as they are made available, in order to protect their users.

What should I do as an OpenStack cloud user?

Cloud users should watch for and apply operating system patches for their guest VMs as soon as they are made available. This advice actually applies to any computer (virtual or physical) you happen to use (including your phone).

Are patches available already?

Some patches are out, some are still due. Kernel patches mitigating the Meltdown attack are available upstream, but they are significant patches with lots of side effects and some OS vendors are still testing them. The coordinated disclosure process failed to keep the secret up to the publication date, which explains why some OS vendors or distributions were not ready when the news dropped.

It’s also important to note that this is likely to trigger a long series of patches, since the workarounds and mitigation patches are refined to reduce side effects and new bugs that those complex patches themselves create. The best recommendation is to keep an eye on your OS vendor patches (and CPU vendor microcode updates) for the coming months and apply all patches quickly.

Is there a performance hit in applying those patches?

The workarounds and mitigation techniques are still being developed, so it’s a little too early to say and it will always depend on the exact workload. However, since the basic flaw here lies in performance optimization techniques in CPUs, most workarounds and mitigation patches should add extra checks, steps and synchronization that will undo some of that performance optimization, resulting in a performance hit.

Is there anything that should be patched on the OpenStack side?

While OpenStack itself is not directly affected, it’s likely that some of the patches that are and will be developed to mitigate those issues will require optimizations in software code to limit the performance penalty. Keep an eye on our stable branches and/or your OpenStack vendor patches to make sure you catch any of those.

Those vulnerabilities also shine some light on the power of side-channel attacks, which shared systems are traditionally more vulnerable to. Security research is likely to focus on such class of issues in the near future, potentially discovering side-channel security attacks in OpenStack that will need to be fixed.

Where can I learn more?

To understand the basic flaw and the CPU technologies involved, I recommend reading Eben Upton’s great post. If that’s too deep or you need a good analogy to tell your less-technical friends, try this one by Robert Merkel.

For technical details on the vulnerability themselves, Jann Horn’s post on Google Project Zero blog should be first on your list. You can also read the Spectre and Meltdown papers.

For more information on the various mitigation techniques, I recommend starting with this article from Google’s Security blog. For information about Linux kernel patches in particular, read Greg Kroah-Hartman’s post.

The post OpenStack, Spectre and Meltdown: What you need to know appeared first on OpenStack Superuser.

by Thierry Carrez at January 09, 2018 03:18 PM

Chris Dent

TC Report 18-02

Welcome to the first normal TC Report of 2018. Last week I did a review of much of 2017 but there was little to report from the first two days of the year, so there is no TC Report 18-01.

Since then there's been a bit of TC business, mostly with people working to get themselves resituated and organized for the upcoming year.


On Thursday there was some effort to recall the salient points from the giant longer development cycle thread. The consensus is that there are some issues to discuss and resolve, but lengthening the cycle does not address them. One standout issue from the thread was managing upgrades. There was an effort to discuss why they are hard, that, in typical fashion, branched broadly to "what are the reasons they matter". There's plenty of awareness that one size is not going to fit all, but not plenty of awareness of what all the sizes are. In part this is because we were unable to stick to any single line of analysis before branching. We need more input and volunteers to help with the work that results from any conclusions.

A few different people have made plans to provide summaries of the development cycle thread or to extract a list of issues so they are not forgotten. Thierry did a TL;DR back in December and has written a temporary conclusion today.

Projects Publishing to PyPI

On Monday the topic of publishing the service projects to PyPI was brought back up. Previous discussion had happened in email. There's general agreement that this would be a good thing assuming two issues can be managed:

  • There's a naming conflict with an existing project on PyPI called keystone. It appears to be stalled but there's no easy process for removing a project. One option is to make the service distributions have names such as openstack-keystone, but they may have cascading effects downstream.

  • We assume (hope?) that there will be very few people doing pip install nova (or any other service) that expect a working system.

The release team is working on it.

Rocky Goals

The main topic from this morning's office hour was establishing some OpenStack-wide goals for the next cycle, Rocky. Today there is only one proposed goal, Migrating to Storyboard, but it has a few issues which leave it without the vast swell of support a good goal ought to have. The community goals etherpad has some suggestions, and based on the discussion in IRC a couple more were added. If you have ideas please propose them in gerrit, post some email, or bring them to #openstack-tc for discussion.

Board Individual Directors Election

This week hosts the election for the OpenStack Foundation Board Individual Directors. Monty has a good tweet storm on why voting (if you can) for these people matters. If you are eligible ("joined the OpenStack Foundation as an Individual Member by July 17, 2017") you should have received a ballot on Monday, the 8th of January.

by Chris Dent at January 09, 2018 01:30 PM

January 08, 2018

Thierry Carrez

OpenStack Spectre/Meltdown FAQ

What are Meltdown and Spectre ?

Meltdown and Spectre are the brand names of a series of vulnerabilities discovered by various security researchers around performance optimization techniques built in modern CPUs. Those optimizations (involving superscalar capabilities, out-of-order execution, and speculative branch prediction) fundamentally create a side-channel that can be exploited to deduce the content of computer memory that should normally not be accessible.

Why is it big news ?

It's big news because rather than affecting a specific operating system, it affects most modern CPUs, in ways that cannot be completely fixed (as you can't physically extract the flawed functionality out of your CPUs). The real solution is in a new generation of CPU optimizations that will not exhibit the same flaws while reaching the same levels of performance. This is unlikely to come soon, which means we'll have to deal with workarounds and mitigation patches for a long time.

Why is it business as usual ?

As Bruce Schneier says, "you can't secure what you don't understand". As we build more complex systems (in CPUs, in software, in policies), it is more difficult to build them securely, and they can fail in more subtle ways. There will always be new vulnerabilities and new classes of attacks found, and the answer is always the same: designing defense in depth, keeping track of vulnerabilities found, and swiftly applying patches. This episode might be big news, but the remediation is still applying well-known techniques and processes.

Are those 2 or 3 different vulnerabilities ?

It is actually three different exploitation techniques of the same famility of vulnerabilities, which need to be protected against separately.

  • CVE-2017-5753 (“bounds check bypass”, or variant 1) is one of the two Spectre variants. It affects specific sequences within compiled applications, which must be addressed on a per-binary basis. Applications that can be made to execute untrusted code (e.g. operating system kernels or web browsers) will need updates as more of those exploitable sequences are found.

  • CVE-2017-5715 (“branch target injection”, or variant 2) is the other Spectre variant. It more generally works by poisoning the CPU branch prediction cache to induce privileged applications to leak small bits of information. This can be fixed by a CPU microcode update or by applying advanced software mitigation techniques (like Google's Retpoline) to the vulnerable binaries.

  • CVE-2017-5754 (“rogue data cache load”, or variant 3) is also called Meltdown. This technique lets any unprivileged process read kernel memory (and therefore access information and secrets in other processes running on the same system). It is the easiest to exploit, and requires patching the operating system to reinforce isolation of memory page tables at the kernel level.

What is the impact of those vulnerabilities for OpenStack cloud users ?

Infrastructure as a service harnesses virtualization and containerization technologies to present a set of physical, bare-metal resources as virtual computing resources. It heavily relies on the host kernel security features to properly isolate untrusted workloads, especially the various virtual machines running on the same physical host. When those fail (like is the case here), you can have a hypervisor break. An attacker in a hostile VM running on an unpatched host kernel could use those techniques to access data in other VMs running on the same host.

Additionally, if the guest operating system of your VMs is not patched (or you run a vulnerable application) and run untrusted code on that VM (or in that application), that code could leverage those vulnerabilities to access information in memory in other processes on the same VM.

What should I do as an OpenStack cloud provider ?

Cloud providers should apply kernel patches (from their Linux distribution), hypervisor software updates (from the distribution or their vendor) and CPU microcode updates (from their hardware vendor) that workaround or mitigate those vulnerabilities as soon as they are made available, in order to protect their users.

What should I do as an OpenStack cloud user ?

Cloud users should watch for and apply operating system patches for their guest VMs as soon as they are made available. This advice actually applies to any computer (virtual or physical) you happen to use (including your phone).

Are patches available already ?

Some patches are out, some are still due. Kernel patches mitigating the Meltdown attack are available upstream, but they are significant patches with lots of side-effects, and some OS vendors are still testing them. The coordinated disclosure process failed to keep the secret up to the publication date, which explains why some OS vendors or distributions were not ready when the news dropped.

It is also important to note that this is likely to trigger a long series of patches, as the workarounds and mitigation patches are refined to reduce side-effects and new bugs that those complex patches themselves create. The best recommendation is to keep an eye on your OS vendor patches (and CPU vendor microcode updates) for the coming months and apply all patches quickly.

Is there a performance hit in applying those patches ?

The workarounds and mitigation techniques are still being developed, so it is a little early to say, and it will always depend on the exact workload. However, since the basic flaw here lies in performance optimization techniques in CPUs, most workarounds and mitigation patches should add extra checks, steps and synchronization that will undo some of that performance optimization, resulting in a performance hit.

Is there anything that should be patched on the OpenStack side ?

While OpenStack itself is not directly affected, it is likely that some of the patches that are and will be developed to mitigate those issues will require optimizations in software code to limit the performance penalty. Keep an eye on our stable branches and/or your OpenStack vendor patches to make sure you catch any of those.

Those vulnerabilities also shine some light on the power of side-channel attacks, which shared systems are traditionally more vulnerable to. Security research is likely to focus on such class of issues in the near future, potentially discovering side-channel security attacks in OpenStack that will need to be fixed.

Where can I learn more ?

You can find lots of explanations over the Internet. To understand the basic flaw and the CPU technologies involved, I recommend reading Eben Upton's great post. If that's too deep or you need a good analogy to tell your less-technical friends, I find this one by Robert Merkel not too bad.

For technical details on the vulnerability themselves, Jann Horn's post on Google Project Zero blog should be first on your list. You can also read the Spectre and Meltdown papers.

For more information on the various mitigation techniques, I recommend starting with this article from Google's Security blog. For information about Linux kernel patches in particular, I recommend Greg Kroah-Hartman's post.

by Thierry Carrez at January 08, 2018 04:03 PM


Is 2018 when the machines take over? Predictions for machine learning, the data center, and beyond

This year has seen a groundswell of technology in ways that seem to be simmering under the surface, and if you look a bit more closely it's all there, just waiting to be noticed.

by Nick Chase at January 08, 2018 03:36 PM

OpenStack Superuser

OpenStack User Group Spotlight: Uganda

In this series of interviews, Superuser takes you around the world to meet our newest user groups. These busy groups act as liaisons between the Foundation and the general community in their regions. You can find out more about a group in your area — or how to start one — at the User Groups portal.

Here we talk to Diarmuid O Briain, network engineer at Uganda Internet Exchange Point, about a new User Group in Uganda based in the capital, Kampala.

What’s the most important OpenStack debate in your region right now?
Right now there is little debate actually. Cloud implementation is confined to organizations like the National Information Technology Authority – Uganda (NITA-U) and the Research and Education Network for Uganda (RENU). At netLabs!UG our interest came from the need to work with Network Functions Virtualisation (NFV) and therefore Openstack as a Virtual Infrastructure Manager (VIM) and the Tacker project as a NFV MANagement and Organisation (MANO) solution.

What’s the key to closing the talent gap in the OpenStack community?
With Openstack in Uganda and in particular the academic world, we’re at the infancy of work with cloud really. Africa did not have the head start of easily accessible public clouds to garner interest due to lack of presence on the continent of the big providers, Amazon, Google, etc.., however with the direction that 5G mobile is taking, the obvious benefits the development of cloud is necessary. Openstack is seen as a key tool for that.

What trends have you seen across your region’s user groups?
We have not had interaction with any other user groups as yet.

What drives cloud adoption in your region?
At the moment adoption is poor except for software-as-a-service from the big providers. Government and Academic organizations like NITA-U and RENU are the current big users.

What’s the biggest obstacle to adoption?
Lack of public accessible carrier neutral data centers.

What types of clouds/organizations are most active in the community?
Private clouds for some service providers with NITA-U offering services to government bodies and RENU to academic bodies.

What’s your vision for your group?
I would like to see what interest there is beyond the academic, what are the potential areas of collaboration between NITA-U, RENU, netLabs!UG and other organizations. What can netLabs!UG do to assist other organizations within Uganda and how we can assist the College of Engineering, Design, Art and Technology (CEDAT) to produce Computer Engineers competent in Cloud infrastructure.
How can people get involved?
Mail openstackATnetlabsug.org and outline their interest and how they want to be involved.

Superuser is always interested in what’s happening in your part of the world with OpenStack, get in touch at editorATopenstack.org.

Cover Photo // CC BY NC


The post OpenStack User Group Spotlight: Uganda appeared first on OpenStack Superuser.

by Sonia Ramza at January 08, 2018 03:18 PM

Rob Hirschfeld

Help OpenStack build more Open Infrastructure communities

Note: OpenStack voting is limited to community members – if you registered by the deadline, you will receive your unique ballot by email.  You have 8 votes to distribute as you see fit.

Vote Now!I believe open infrastructure software is essential for our IT future.  

Open source has been a critical platform for innovation and creating commercial value for our entire industry; however, we have to deliberately foster communities for open source activities that connect creators, users and sponsors.  OpenStack has built exactly that for people interested in infrastructure and that is why I am excited to run for the Foundation Board again.

OpenStack is at a critical juncture in transitioning from a code focus to a community focus.  

We must allow the OpenStack code to consolidate around a simple mission while the community explores adjacent spaces.  It will be a confusing and challenging transition because we’ll have to create new spaces that leave part of the code behind – what we’d call the Innovator’s Dilemma inside of a single company.  And, I don’t think OpenStack has a lot of time to figure this out.

That change requires both strong and collaborative leadership by people who know the community but are not too immersed in the code.

I am seeking community support for my return to the OpenStack Foundation Board.  In the two years since I was on the board, I’ve worked in the Kubernetes community to support operators.  While on the board, I fought hard to deliver testable interoperability (DefCore) and against expanding the project focus (Big Tent).  As a start-up and open source founder, I bring a critical commercial balance to a community that is too easily dominated by large vendor interests.

Re-elected or not, I’m a committed member of the OpenStack community who is enthusiastically supporting the new initiatives by the Foundation.  I believe strongly that our industry needs to sponsor and support open infrastructure.  I also believe that dominate place for OpenStack IaaS code has changed and we also need to focus those efforts to be highly collaborative.

OpenStack cannot keep starting with “use our code” – we have to start with “let’s understand the challenges.”  That’s how we’ll keep building an strong open infrastructure community.

If these ideas resonate with you, then please consider supporting me for the OpenStack board.  If they don’t, please vote anyway!  There are great candidates on the ballot again and voting supports the community.

by Rob H at January 08, 2018 12:31 AM

January 05, 2018

OpenStack Superuser

Inside Kata Containers

Kata Containers is an open-source project and community working to build a standard implementation of lightweight virtual machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs. It combines technology from Intel® Clear Containers and Hyper runV from Hyper.Sh; the name is a nod to the Japanese word “form” or choreographed movement.

Superuser sat down with James Kulina, COO at Hyper.Sh, to find out more.

How did you get involved with OpenStack?

I’ve been involved with OpenStack for five or six years now. First at AT&T, helping them build their OpenStack cloud in all different kinds of use cases. From there, I went to a startup called eNovance that was bought by Red Hat so I was working on OpenStack at Red Hat for awhile…Now I’m at Hyper.sh. Overall, I’m pretty involved with the OpenStack community and knowledgeable about the ecosystem.

How did you get involved in Kata?

We’d been working with Intel for a little over a year and we actually launched our individual projects at around the same time. We started working on merging code and it got to a point where we thought that it’d be better suited to having an open governance, an open model and getting more community and an ecosystem around it, so we came together and decided that Kata should be created and pushed out for the community to consume.

Where do you see this fitting into the landscape?

It’s a foundational piece. It’s going to open up container-native workloads for new kinds of workloads. Because it’s virtualization-based, not hypervisor based, you’ll be able to tie into accelerators and legacy-type systems that people trust and use already. It actually makes a very unique and easy migration path for people who are already using legacy virtualization to start consuming containers and container-native workloads.

What use cases do you see?

Initially, mostly around container-as-a-service. That’s the simplest one that people can understand, it’ll be for people who are already dipping their feet into Kubernetes and using Kubernetes to scale container applications. But it’ll go in a lot of different ways, fairly quickly, I think: a lot of different verticals (financial services, telcos, internet of things, edge) will all benefit from this type of security, performance and integration.

You were at the Kata booth during KubeCon, what kinds of questions did you get?

Mostly just about how these companies came together, and a little bit about how it actually works. Some people were looking for more detail about  how the underpinnings work; there’s a little bit of inertia that we have to get over in terms of people having their own definition of what container native means. We’re trying to work with that definition and expand it.

A couple of questions I overheard were about bare metal and the Clear Container Initiative.  

It’s because you can actually run Kubernetes on bare metal and have full isolation between the pods, so you get a multi-tenant type installation where you can run untrusted code side by side.

So Clear Containers launched the same month that we launched with our Hyper runV project and the vision for both was incredibly the same (laughs) so that’s why it made sense to merge this and go open source.

Were there a lot of questions about how these projects all fit together?

Yes. There were definitely a lot of technical questions around how the projects came together in terms of code base. Some of the code has already been merged, but we’re looking to the community to get involved and help.

What kind of contributions do you need most?

Mostly around integrations with the different virtualization technologies as well as the CRI interface that we’re doing, some work around that could be very beneficial  We’re already talking with a number of community members from Kubernetes and OpenStack about this. There’s a lot of work that people can contribute to, there’s a GitHub repo they can go check out.

Where would you like to see this in a year?

I’d love to see this being used in production. It’s already being used in terms of our technology, but for the Kata project to have a migration path for a production system, maybe on someone’s cloud or in a hybrid scenario. Right now, our goal for the short term is to get people to understand the technology, start contributing and start talking about it. In year from now, hopefully they’ll have stood it up, tested it out, tried it out and maybe gone into production.

Anything else you want people to know?

The one thing we’ve always talked about at Hyper: make containers as simple as possible. To take the headache out going from something working on your laptop to working in the cloud. With Kata, you’ll start to see that become reality.

What’s the best way to get involved right now?




Slack https://katacontainers.slack.com/

The post Inside Kata Containers appeared first on OpenStack Superuser.

by Nicole Martinelli at January 05, 2018 02:41 PM

Carlos Camacho

New TripleO quickstart cheatsheet

I have created some cheatsheets for people starting to work on TripleO, mostly to help them to bootstrap a development environment as quickly as possible.

The previous version of this cheatsheet series was used in several community conferences (FOSDEM, DevConf.cz), now, they are deprecated as the way TripleO should be deployed changed considerably last months.

Here you have the latest version:

The source code of these bookmarks is available as usual on GitHub

And this is the code if you want to execute it directly:

# 01 - Create the toor user.
sudo useradd toor
echo "toor:toor" | chpasswd
echo "toor ALL=(root) NOPASSWD:ALL" \
  | sudo tee -a /etc/sudoers.d/toor
sudo chmod 0440 /etc/sudoers.d/toor
su - toor

# 02 - Prepare the hypervisor node.
mkdir .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
sudo bash -c "cat .ssh/id_rsa.pub \
  >> /root/.ssh/authorized_keys"
sudo bash -c "echo '' \
  >> /etc/hosts"
export VIRTHOST=
sudo yum groupinstall "Virtualization Host" -y
sudo yum install git lvm2 lvm2-devel -y
ssh root@$VIRTHOST uname -a

# 03 - Clone repos and install deps.
git clone \
chmod u+x ./tripleo-quickstart/quickstart.sh
bash ./tripleo-quickstart/quickstart.sh \
sudo setenforce 0

# 04 - Configure the TripleO deployment with Docker and HA.
export CONFIG=~/deploy-config.yaml
cat > $CONFIG << EOF
  - name: control_0
    flavor: control
    virtualbmc_port: 6230
  - name: compute_0
    flavor: compute
    virtualbmc_port: 6231
node_count: 2
containerized_overcloud: true
delete_docker_cache: true
enable_pacemaker: true
run_tempest: false
extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml

# 05 - Deploy TripleO.
export VIRTHOST=
bash ./tripleo-quickstart/quickstart.sh \
      --clean          \
      --release master \
      --teardown all   \
      --tags all       \
      -e @$CONFIG      \

Happy TripleOing!!!

by Carlos Camacho at January 05, 2018 12:00 AM

January 04, 2018

OpenStack Superuser

Resources for getting started with OpenStack

If you’re new to the community, there are plenty of great resources for getting started. Here’s a roundup of some of Superuser’s recent ones.

Learn OpenStack: A new community site

As he was teaching himself, Geoff Rich built a site called Learn OpenStack. “As I was learning, I found a lot of the online material somewhat lacking in places. More as a hobby I decided to make a tutorial website out of my learnings and to also create a set of easily repeatable steps that anyone with a few technical skills could follow. I also wanted a set of steps that anyone can run, regardless of if they run Mac, Windows or Linux.”

OpenStack basics: An overview for the absolute beginner

Ben Silverman, author of “OpenStack for Architects,” gets newbies started with OpenStack. He offered this hour-long intro  hahahathat starts at the very beginning — what cloud is, the origin story of Amazon Web Services, OpenStack and why open source matters — straight through to architecture and managed OpenStack.

Careers in OpenStack: What employers want

Here Silverman examines the most common OpenStack job titles and how to get hired for them. “LinkedIn listed almost 6,000 jobs, Indeed 2,500 jobs and Glassdoor 2,600 jobs with OpenStack as a title or a requirement. Want more good news? The jobs are at all skill levels. According to Glassdoor, OpenStack jobs have an average salary of about $90,000 per year, at a wide range of experience levels, ” he writes.

Making your first contact with OpenStack

Exploring OpenStack? Here’s where to find the First Contact Special Interest Group and how to get involved.

The beginner’s guide to contributing upstream

The Foundation’s Kendall Nelson on steps to contributing upstream and other ways you can get involved.

7 steps to becoming a Certified OpenStack Administrator

Author Matthew Dorn offers a peek into the “Certified OpenStack Administrator Exam” book.

How to get more involved with OpenStack

John Garbutt, works as principal engineer focused on the convergence of OpenStack and high-powered computing at StackHPC shares his thoughts on how to boost your involvement in the community.


The post Resources for getting started with OpenStack appeared first on OpenStack Superuser.

by Superuser at January 04, 2018 01:45 PM

January 03, 2018

OpenStack Superuser

Kickstart your OpenStack know-how with these tutorials

Here are some of our most popular tutorials for 2017, along with a friendly reminder: You get active user contributor (AUC) status by writing for Superuser. Get in touch at editorATsuperuser.org.

Getting started with OpenStack? Check out this cheat sheet

Following the long tradition of one-page cheat sheets for common UNIX applications (remember EMACS, Bash, regular expressions?) Aviv Lichtigstein, head of product evangelism at Loom Systems recently made a nice-looking one for OpenStack where he covers some common commands for services Keystone, Glance, Nova, Neutron and Cinder.

How to set up your work environment to become an OpenStack developer

If you’re a beginning OpenStack developer, contributor Nisha Yadav helps you get set up.

Exploring the OpenStack neighborhood: An offbeat install guide

Nooruddin Abbas, solution architect from GBM, gives you a tour of OpenStack in this three-part series.

Getting to know the essential OpenStack components better

Nooruddin Abbas demonstrates how to get Keystone, Glance, Nova, Neutron and Horizon up and running in part two of this three-part series.

Let’s heat things up with Cinder and Heat

Adding Cinder and Heat in the final part of this three-part OpenStack series.

Kuryr and Kubernetes will knock your socks off

Set up your own Kuryr-Kubernetes instance with this walk through form Red Hat’s Doug Smith.

Manage a multi-node Kubernetes deployment on a bare-metal cluster

This 12-minute demo shows you how to use OpenStack Ironic to manage a multi-node Kubernetes deployment on a bare-metal cluster.

How to: Docker containerized OpenStack Heat Translator

Sahdev P. Zala and Eduardo Patrocinio from IBM show us how they created a Docker container using Heat-Translator.

Enable native Neutron-based networking in Kubernetes with Kuryr

Ilya Chukhnakov and Antoni Segura Puimedon give a sneak peek on how Kuryr brings datacenter-ready OpenStack Neutron networking to Kubernetes.

Deploying and redeploying OpenStack in a physical server

How to use command-line tool Snapper to redeploy OpenStack in this tutorial from Arun Kumar of CloudEnablers.

OpenStack Neutron: an NFV primer

An overview of Neutron for NFV from Rossella Sblendido, core reviewer, and the OpenStack Foundation’s Ildikó Vancsa.

The post Kickstart your OpenStack know-how with these tutorials appeared first on OpenStack Superuser.

by Superuser at January 03, 2018 03:03 PM

Julio Villarreal Pelegrino

What technologies to follow and learn in 2018!

What technologies to follow and learn in 2018!

First of all: 1. I work for Red Hat Inc. 2. Opinions are my own and not the views of my employer.

2017 is gone and with it a great year for technology and innovation. Now, it is time to look a little bit into the future and talk about couple technologies that I believe that will become more relevant in this new year 2018.

1. Managed Security

What technologies to follow and learn in 2018!

Last year was a terrible year for IT Security after multiple high profile breaches like the ones that happened to Equifax, BCBS, Alterix, and a few other companies.
I do believe that this year security will be top-priority for CIOs and CTOs across the world, this will manifest by bringing external vendors to complement in-house security teams as part of the IT security strategies.
Managed Security service providers have a great opportunity in front of them in 2018: to help minimize breaches, drive compliance and innovate in new ways to secure the cloud.

2. Blockchain

What technologies to follow and learn in 2018!

Blockchain is, without doubt, one of my favorite technologies out there. Initially created for the cryptocurrency use case, is a great way to distribute and secure data.
For 2018, I believe that blockchain will become more mainstream, using their “distributed ledger” capabilities for multiple use cases in banking, trading, asset management, decentralized records, government, insurance, and healthcare.

3. Serverless

What technologies to follow and learn in 2018!

Serverless computing will continue to grow in 2018 by making the developer's life more comfortable and the ability for a faster time to market a reality. AWS Lambda should continue to grow this year, but I do believe what will make it serverless an exciting technology to follow is the rise of other players in this arena, like Microsoft Azure Functions, Google Cloud Functions, and the opensource Apache OpenWhisk project.

4. Artificial Intelligence

What technologies to follow and learn in 2018!

Artificial Intelligence will continue to evolve in 2018 with more companies investing in AI initiatives. Here are couple use cases that I believe will be the top priority for AI and Machine Learning this year:

  • Self-driving cars and ships
  • Better personal digital assistants and perfecting the way that we interact with them (voice)
  • More and more healthcare applications
  • IT Security applications of AI will grow

5. IoT will continue growing

What technologies to follow and learn in 2018!

Internet of Things (IoT) should continue to grow in 2018, generating more and more data (that will be used by companies to take better decisions using Machine Learning and AI). For IoT providers, couple things will be in their top priorities:

  • Securing devices and data
  • Market focus will be beyond the consumer
  • More IoT startups will be created
  • Focus on home automation and interoperability with other devices
  • Edge computing application to IoT will grow

6. Kubernetes

What technologies to follow and learn in 2018!

Kubernetes will win the PaaS war in 2018! How will this happen?
Grow of Kubernetes Certified Service providers
Adoption of big players of Kubernetes as the core of their container offerings, like:

  • Red Hat OpenShift Container Platform and OpenShift Online
  • Microsoft Azure Container Service
  • Google Kubernetes Engine
  • Amazon Elastic Container Service for Kubernetes
  • Oracle Container Services
  • Kubernetes will go beyond containers, managing virtualization with KubeVirt.

7. Bare Metal is back!

What technologies to follow and learn in 2018!

Bare Metal should grow in 2018 with more use cases that require bigger, specialized, and dedicated hardware. The comeback of bare metal will be possible thanks to cloud computing: allowing provisioning, configuration, management, and lifecycle as-a-service of physical hardware.
The bare metal needs will be addressed in public and private clouds, with projects like OpenStack Bare Metal Provisioning (Ironic) leading in the private cloud space.

8. Automation and Configuration Management

What technologies to follow and learn in 2018!

Adopting DevOps and Cloud Computing is an almost impossible task without automation and configuration management. In 2018 automation and configuration management tools like Ansible and Ansible Tower by Red Hat will go beyond cloud and datacenter automation. They will drive the adoption of serverless computing and be used in other use cases like application migrations, operating system modernization, and network device orchestration and configuration.

And by the way, Happy New Year 2018

by Julio Villarreal Pelegrino at January 03, 2018 12:37 AM

January 02, 2018

Craige McWhirter

Resolving a Partitioned RabbitMQ Cluster with JuJu

On occasion, a RabbitMQ cluster may partition itself. In a OpenStack environment this can often first present itself as nova-compute services stopping with errors such as these:

ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager._sync_power_states: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c
TRACE nova.openstack.common.periodic_task self._raise_timeout_exception(msg_id)
TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 218, in _raise_timeout_exception
TRACE nova.openstack.common.periodic_task 'Timed out waiting for a reply to message ID %s' % msg_id)
TRACE nova.openstack.common.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 8fc8ea15c5d445f983fba98664b53d0c

Merely restarting the stopped nova-compute services will not resolve this issue.

You may also find that querying the rabbitmq service may either not return or take an awful long time to return:

$ sudo rabbitmqctl -p openstack list_queues name messages consumers status

...and in an environment managed by juju, you could also see JuJu trying to correct the RabbitMQ but failing:

$ juju stat --format tabular | grep rabbit
rabbitmq-server                       false local:trusty/rabbitmq-server-128
rabbitmq-server/0           idle 0/lxc/12 5672/tcp
rabbitmq-server/1   error   idle 1/lxc/8  5672/tcp   hook failed: "config-changed"
rabbitmq-server/2   error   idle 2/lxc/10 5672/tcp   hook failed: "config-changed"

You should now run rabbitmqctl cluster_status on each of your rabbit instances and review the output. If the cluster is partitioned, you will see something like the below:

ubuntu@my_juju_lxc:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@192-168-7-148' ...

You can clearly see from the above that there are two partitions for RabbitMQ. We need to now identify which of these is considered the leader:

maas-my_cloud:~$ juju run --service rabbitmq-server "is-leader"
- MachineId: 0/lxc/12
  Stderr: |
  Stdout: |
  UnitId: rabbitmq-server/0
- MachineId: 1/lxc/8
  Stderr: |
  Stdout: |
  UnitId: rabbitmq-server/1
- MachineId: 2/lxc/10
  Stderr: |
  Stdout: |
  UnitId: rabbitmq-server/2

As you see above, in this example machine 0/lxc/12 is the leader, via it's status of "True". Now we need to hit the other two servers and shut down RabbitMQ:

# service rabbitmq-server stop

Once both services have completed shutting down, we can resolve the partitioning by running:

$ juju resolved -r rabbitmq-server/<whichever is leader>

Substituting <whichever is leader> for the machine ID identified earlier.

Once that has completed, you can start the previously stopped services with the below on each host:

# service rabbitmq-server start

and verify the result with:

$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@192-168-7-148' ...

No partitions \o/

The JuJu errors for RabbitMQ should clear within a few minutes:

$ juju stat --format tabular | grep rabbit
rabbitmq-server                       false local:trusty/rabbitmq-server-128
rabbitmq-server/0             idle 0/lxc/12 5672/tcp 19
rabbitmq-server/1   unknown   idle 1/lxc/8  5672/tcp 19
rabbitmq-server/2   unknown   idle 2/lxc/10 5672/tcp

You should also find the nova-compute instances starting up fine.

by Craige McWhirter at January 02, 2018 10:18 PM

Chris Dent

TC Report: 2017 In Review

The first TC Report was published in late April of 2017. What follows is a review of the issues and persistent themes that distill out of a read through all the reports since then.

As with the reports themselves, this is a biased and opinionated document in which I'll editorialize on my own editorializing. I've had plenty of feedback that says that being biased and opinionated helps to keep the reports interesting, useful, and something other than terribly dry. I also hope that it helps to draw in people who may have different opinions. The point of these things has been to increase awareness and communication, for and from everyone.

If you need a TL;DR, jump ahead to Decoupling.

Shift in TC Membership

Following the rules, there were two TC elections in 2017. In both, especially the second, there was a bit of a shift in the candidate pool and in those eventually elected. Some old timers stepped aside, breaking the usual trend of incumbency equaling persistence. The new membership, along with the change to office hours, has helped to drive a renewed desire for activism that's been tempered a bit by difficulty with gaining traction and trouble finding the necessary levers and fulcrums.

Influencing Corporate Contribution

An area where more action has been desired is influencing the volume and type of corporate contribution, in a more effective fashion. This has been a theme throughout office hours, PTG meetings, and discussions leading up to and following board meetings: How can the TC, and by extension the technical contributors they represent, effectively influence how the corporate Foundation members and other OpenStack corporate stakeholders manage the staff they require or allow to contribute upstream.

The Help most needed list (née "Top 5 help wanted"), OpenStack-wide goals, and TC Vision have helped illuminate some areas that need attention, but the way this topic keeps coming up suggests that more work is needed. The problem is not simply one of getting more resources. It is just as much an issue of achieving some goals that are actually shared, OpenStack-wide, about what OpenStack is or should become.

There's been some agreement that reaching this requires vastly more and more open communication amongst all the people who are involved.

That Vision Thing

The original version of the TC Vision, drafted in the first half of 2017, was edited in the second half to give it a more straightforward style to make it more amenable to skimming and goal creating. This came at the cost of removing some of the reasoning surrounding the goals.

Some progress has been made on the goals implied by the vision. Using bits and pieces of OpenStack independently is on the rise. Relationships and collaboration with adjacent technical communities has been improved. Greater attention is being given to issues with diversity in the community. There is increased awareness of the need to actively encourage and enable new leaders.

However, there is plenty left to do. There was a great deal of positive feedback on the idea of "constellations" but we're some distance away from having the "orion" website that people hoped to see. We've built some very positive bridges with the Kubernetes community, but there were reports from Kubecon that OpenStack is often whipped as the convenient down dog. We're making headway, but have a long way to go before we have anything like truly diverse (gender, ethnic, geographic) representation in the TC and with PTLs and project cores. While new faces are very welcome, it's still often the case that the same heroes are occupying leadership and driver roles.

Maintenance Mode

Midway through the year there was plenty of discussion on the meaning and usefulness of a maintenance-mode tag. It's used to indicate a project that is in a period of low-activity, but is sometimes interpreted as a negative. Some of the debate centered around why that should be: For some, low activity can mean "works so well, no changes are required".

This is relevant as OpenStack reaches a form of maturity. Not all projects will always see a constant flow of contributions, but they may still adequately satisfy their use cases.

Meaning of Supported

There were a few different discussions throughout the year about what it means for a technology to be supported. Questions about support for PostgreSQL led to a current level of community support document and a definition of upstream support.

Meanwhile we realized by the end of the year that the strict interpretation of those rules (especially "if it's not tested in CI, it's not supported") is incredibly limiting. Using PostgreSQL can work. Ways of doing skip-level and decoupled upgrades is possible with sufficient effort. Long Term Support (LTS) exists. But these things are not in CI. They are not what OpenStack is, but they are something other people or organizations can do with OpenStack.

What we do with OpenStack in CI is incredible, but it is not a complete picture of what people can do and want to do with OpenStack. It never can be, and never should be. Being tested in the gate should mean just that: It's tested in the gate. The idea that the OpenStack project, at large, provides support, is misplaced.

See also related discussion on whether we actually enable continuous deployment.

TC Communications

In 2017 there were many changes to how the TC communicates. The weekly TC meetings ended, replaced by office hours in a dedicated IRC channel, #openstack-tc. I do my weekly TC Reports. Thierry does a weekly factual summary (see the most recent one) based on a Tracker wiki page.

This has greatly enhanced the opportunity for engagement amongst the members of the TC and with the community. In reality, however, it appears to be that the members of the TC chat a lot in IRC, there is quite a bit of published information, but not huge amounts of lbi-directional communication between the TC and the entirety of the rest of the community.

Bye Bye "big tent"

2017 dismissed the term "big tent". People were often confused about what it meant and sometimes used that ambiguity with ill intent. A final flushing was made to the governance repository. The commit message includes a bit of history.

The term of art these days is "official project", meaning a project that has voluntarily chosen to have the TC as its governing body. Other projects continue to use and are welcome to use OpenStack infrastructure. These are not the same things.

Glare and Mogan

There were many applications by projects to be official, but two of note stalled out: Glare and Mogan. I wrote about this in the context of the TC's power but eventually the proposals were withdrawn by the applicants.

The balance between protecting and ensuring the state of existing projects while allowing and encouraging a measure of innovation and competition is delicate. In this case protectionism won. For some words on this see Dean's comment on patchset 6 on the Glare review. My comment on patchset 2 on the Mogan Review talks about difficulties when trying to include voices from all participants.


2017 saw the introduction of Special Interest Groups as one way to make including more voices more direct. They are designed to

Replace UC or TC-side "workgroups" by neutral "SIGs" to encourage wide participation --tracker

Some people had been feeling that the governance of a workgroup was a statement of who was allowed to participate. SIGs are an effort to break down those barriers and remove some of the formalisms that have may have impeded progress in the past.

There is a growing list of SIGs. Most are workgroups that have migrated or combined to become a SIG.

Foundation Expansion

The Foundation grew to add a sibling project, Kata Containers, outside the domain of OpenStack. The sibling will have its own governance. Additional projects may be expected in the future. One candidate topic is edge computing.

Release Cycle Length

Thierry mooted the idea of extending the release cycle, birthing a monster email thread. In that thread, and the related IRC discussions vast numbers of ideas and concerns were raised. It's not entirely clear what the outcome was, but my interpretation was that the concerns were valid, but extending the release cycle would not solve them, more investigation needed.


Perhaps it is time we consider decoupling the development cycle from the release cycle and both from the consumption cycle (this is already true in practice)? And consider breaking other forms of coupling such as requiring stabilized projects to release, upgrades of service A requiring upgrades of service B, or downstream support requiring upstream testing.

Use of OpenStack is growing, but development of it is not. We must adapt our habits.

Feedback Loops

If I had to identify a common thread throughout these many topics, it is that in those situations where we have had active feedback loops with keen interest to reach resolutions, we have made progress and improved things. Where feedback has not happened, or has stalled, issues have lingered and often compounded. We are often too distracted by immediate needs to the detriment of long term health and stability. Solving today's tactical crises must be balanced with the hard work of talking through the many layers of cause and effect that make today what it is and control what tomorrow will be. There's a bit more on that in the last report of the year.

Fidelity of communication in terms of requirements and roadblocks, from all parties, in all directions, is low.

We can improve many of the things people have been talking about, but it's going to take a lot of discussion to reach agreement on what each of the problems really are. Historically we have not been that good, as a community, at accepting that broadly ranging conversations are an okay use of time.

They are. They help us develop the shared language that is critical to developing the shared understanding that is required to establish shared goals.

Running for the Board

The need and hope for high-fidelity communication and true bi-directional feedback between contributors and the TC is why I ran for the TC and started writing the TC Report.

The same thing is needed with the Foundation board: rigorous, active, and transparent feedback that represents the concerns of the active technical community. Sean McGinnis and I spoke about this and agreed we should both run to be Individual Member directors on the board. Elections are next week (January 8-12), if you have been an individual member of the Foundation since before July 17, 2017 you should receive a ballot Monday the 8th. There are many excellent candidates. If neither Sean nor I appeal, we won't be offended, but please vote; the Foundation has a significant impact on the shape and activities of the community.

End Note

If you want or need further context on anything here, the entire collection of TC reports is available on the 'tc' tag.

by Chris Dent at January 02, 2018 09:00 PM

OpenStack Superuser

What you’re doing with OpenStack: Top user stories

Superuser is focused on stories about the mix of open technologies building the modern infrastructure stack including OpenStack, Kubernetes, Ceph, Cloud Foundry, OVS, OpenContrail, Open Switch, OPNFV and more. Here are some of our favorite user stories from 2017 — we want to tell yours, too. Get in touch at editorATopenstack.org

OpenStack operator spotlight: CERN

Belmiro Moreira, cloud architect, tells Superuser how the research center went from a few hundred nodes to 280,000 cores.

How OpenStack powers genomics research

“We started using OpenStack because we saw the value of segregating hardware resources and the opportunity to isolate and simplify the infrastructure configuration,” Manuel Sopena Ballesteros says.

OpenStack operator spotlight: Adobe Advertising Cloud

Adobe’s Advertising cloud runs OpenStack in production across six data centers in the US, Europe and Asia says cloud platform manager Joseph Sandoval.

OpenStack delivers services to thousands of Workday customers

The human resources leader deploys private clouds in every data center, with virtual machines (VMs) and bare metal.

How the Gap builds and uses private clouds with OpenStack

How the Gap builds and uses private clouds with OpenStack

A peek inside the Gap’s (server) closet. “From my perspective, it was a wonderful tool for doing infrastructure delivery. Instead of having to buy a server, set up bare metal or get into VMware, find an image, update an image etc. it allowed me to do everything in one place. It literally changed the paradigm. We heard from customers that it could take five, six or seven weeks to get an app- just to get the infrastructure for their application.”

OpenStack operator spotlight: Paddy Power Betfair

The Superuser Awards winner placing is betting on Kubernetes and bare metal.

OpenStack operator spotlight: NTT

The Japanese telecom giant runs over 10,000 virtual machines, 1,000 hypervisors in eight regions. Two team members talk about what’s next and the importance of testing.

OpenStack Verizon case study: The illusion of infinite capacity

Billy Felton and Andrew Hendrickson of Verizon along with Sanjay Mishra CEO of Talligent offer a look into the platform, its architecture and what’s next as they deal with the challenges of edge computing. http://superuser.openstack.org/wp-content/uploads/2017/07/10501804636_be4f2fe0a2_z.jpg

Why large state-owned enterprises are boarding the OpenStack train

100,000 cloud hosts with 800 nodes on the road to virtualization.

How tech giant Tencent uses OpenStack

One of China’s largest tech companies has been running OpenStack in production for 4 years with availability of over 99.99 percent.

The post What you’re doing with OpenStack: Top user stories appeared first on OpenStack Superuser.

by Superuser at January 02, 2018 04:11 PM

December 29, 2017

OpenStack Superuser

Getting a clear view on multi-cloud

Multi-cloud is one of the top search terms on the Superuser website. Here are the top resources — and if there’s more you’d like us to cover, just drop an email to editorATsuperuser.org

Mapping OpenStack’s place in a multi-cloud world

Jonathan Bryce, OpenStack Foundation executive director, gets out his compass.

Hanging in cloud city

You can be an AWS user and an OpenStack user. You can be an Azure user and an OpenStack user. It isn’t an exclusive relationship: it’s multi-cloud, says the OpenStack Foundation’s Anne Bertucio.

Why now is the time for multi-cloud

“It doesn’t matter who delivers it, how it operates or how it’s provisioned – but it matters how it runs,” says Heptio’s Craig McLuckie.

Avoid vendor lock-in with multi-cloud deployments

Vendor lock-in can create headaches over uptime, financial and regulatory risks. Nate Stewart of Cockroach Labs outlines a strategy for avoiding them.

Build it yourself: How a small team deployed OpenStackHow to deploy multi-cloud serverless and Cloud Foundry APIs at scale

“Micro-services and APIs are products and we need to think about them that way,” says IBM’s Ken Parmelee. Parmelee, who leads the API gateway for IBM and Big Blue’s open source projects, has a few ideas about open-source methods for “attacking” the API and how to create micro-services and make them scale.

Why atmail chose OpenStack for email-as-a-service

More than 15 percent of the Australian company’s infrastructure runs on an OpenStack-powered DreamHost Cloud.

How Fujitsu powers ‘human-centric’ artificial intelligence

Imagine a world where the person driving next to you multitasking (eating, checking texts, reading) pays more for insurance than you do by keeping your eyes on the road.

That’s one of areas where Fujitsu Ltd. is concentrating artificial intelligence (AI) research efforts along with signature analysis as-a-service and 3D image modeling. If right now it seems futuristic to control the heat in your house or check the charge of your Tesla from your phone, you’d better hang on.

The post Getting a clear view on multi-cloud appeared first on OpenStack Superuser.

by Superuser at December 29, 2017 01:25 PM

December 28, 2017

OpenStack Superuser

A year in containers: tutorials and user stories

It’s been a busy year in the cloud universe and containers have been one of the buzz words deserving of the term.

One quote to sum it up? “OpenStack and containers will move beyond proof of concept to deploying solutions that solve real-world business problems, providing a comprehensive strategy that allows enterprises and operators to bring new services that are secure, efficient, elastic, and scalable,” says Anand Krishnan of Canonical.

Here are some of Superuser’s top stories and tutorials.

User stories

How a small team keeps Twitter’s Fail Whale at bay

Hundreds of teams running thousands of services on tens of thousands of hosts and in hundreds of thousands of containers are kept afloat by about 10 engineers, says Ian Downes, head of Twitter’s compute team.

Containers on the CERN Cloud

Hints and tips from the CERN OpenStack cloud team

Making containers work in a siloed company: the athenahealth story

Growing healthy container ops in a siloed company isn’t easy — but they are in fine form at athenahealth. Superuser TV talked to Ryan Wallner and Kevin Scheunemann, both on the technical staff of athenahealth, on how they got there.

Containers are easy, running them in production is something else

“Running containers in production requires something more sophisticated,” says David Aronchick, manager at Google Container Engine.

Pinterest’s journey from virtual machines to containers

“We were on the cloud since day one,” says Michael Benedict. “VMs served their purpose but we’re moving everything to Kubernetes.”

What makes OpenStack relevant in a container-driven world

Containers align nicely with OpenStack, providing infrastructure, allowing them to share networking and storage with other types of computer resources in rich environments, says Thierry Carrez.

Why commercial open source is like a voyage to mars: The Kubernetes story

Craig McLuckie and Sarah Novotny on how to navigate a trip to this harsh, beautiful place.


Getting started with Kolla

Try out OpenStack Kolla’s production-ready containers with this tutorial by cloud engineer Saverio Proto with the first of this two-part tutorial.

How to: Docker containerized OpenStack Heat Translator

Sahdev P. Zala and Eduardo Patrocinio from IBM show us how they created a Docker container using Heat-Translator.

How to containerize GPU applications

Nvidia’s Rohit Taneja offers this tutorial.

Deploy Kubernetes on Openstack with Ansible

Increasing demand for container orchestration tools is coming from our users. Kubernetes has currently a lot of hype, and often it comes the question if we are providing a Kubernetes cluster at SWITCH, the foundation that operates the Swiss academic network.

Deploy Kubernetes on Openstack with Ansible

Saverio Proto of SWITCH offers up this tutorial.

Kubernetes on OpenStack: Digging into technical details

Bitnami’s Angus Lees offers up a recommended deployment and advice about future developments.

How to build OpenStack on Kubernetes

How Workday uses Kubernetes to achieve zero downtime for large-scale production software-as-a-service.

How to deploy mariadb on Kubernetes with TripleO

Flavio Percoco offers this screencast to take you the last mile with Kubernetes.

Manage a multi-node Kubernetes deployment on a bare-metal cluster

This 12-minute demo shows you how to use OpenStack Ironic to manage a multi-node Kubernetes deployment on a bare-metal cluster.

How Kubernetes on OpenStack powers DreamHost’s new web builder

Don’t call this a regular jam: Remixer is a click-to-edit website builder that blends Kubernetes and microservices.

The post A year in containers: tutorials and user stories appeared first on OpenStack Superuser.

by Superuser at December 28, 2017 02:39 PM


8 resources for understanding the open source cloud in 2017

2017 was a big year for open source cloud computing tools. Here's a look back at some of the most important things you may have missed.

by Jason Baker at December 28, 2017 08:04 AM

December 27, 2017

OpenStack Superuser

Looking closer at gender diversity in the OpenStack community

A new version of the gender-diversity report brings more points of analysis and updated numbers. Its goal is to spark  more discussion on the topic and open other areas of analysis not covered in the first report. And feedback from the community in general, some project team leads (PTLs) in particular, and the help of the Foundation have been priceless to understand those areas.

This second edition brings attention to two new research areas: attendees representation and non-technical contributions. These studies are added on top of the previous analysis focused on leadership/governance and technical contributions (mainly Git and Gerrit). In the governance area it’s worth mentioning the addition of two women to the Technical Committee, that’s 15 percent of the total members.

Attendees at the most recent Summits and project team gatherings (PTGs) are in line with the technical contribution numbers, where women represent between ten and 12 percent. Keynote speakers now account for over 25 percent at the OpenStack Summits.

Non-technical contributions, the new area of analysis, offers insight into how gender diversity has evolved, including contributions to the mailing lists. The participation of women in this case is around 13 percent. The addition of these two new areas of analysis provides a broader understanding of the efforts to bring more diverse working group under the Foundation umbrella.There’s a remarkable amount of diversity at the Summits and PTGs.

One of the goals of this analysis is not to focus on specific numbers, but to discuss where there’s room for improvement. Ideally, this data also helps make decisions and evaluate the effectiveness of  initiatives.

The data compiled in this report is simply a first step and should be seen as a tool that can be easily used by the community. More in-depth analysis can be done, by project,  by organization or by other criteria.

You can download the 37-page the report in .PDF here.

But to keep improving, we need your feedback as member of the OpenStack community. If you have any ideas, comments or concerns about how to help with gender diversity, let us know!

The post Looking closer at gender diversity in the OpenStack community appeared first on OpenStack Superuser.

by Daniel Izquierdo at December 27, 2017 02:43 PM


5 new guides and tutorials for OpenStack

Want to keep up with OpenStack? Here are some great community-created resources to help you out.

by Jason Baker at December 27, 2017 08:00 AM

December 26, 2017

OpenStack Superuser

The skinny on edge computing

Edge was one of the those topics in cloud computing in 2017 that was hard to get away from.

In a report from SDxCentral, 87 percent of respondents said that OpenStack would be used to manage edge infrastructure. Here are some of our top resources for ramping up your knowledge.

Navigating Kubernetes and edge computing

“Containers are what the developers are using,” said Jeremy Huylebroeck of Orange Silicon Valley. “It’s way more convenient for them to actually publish their code and try things faster.”

Paths to autonomous workload management at the edge

“We can’t assume that tech developed for data center operations can directly solve edge computing use cases,” says Inmarsat’s Gregory Katsaros. Here are his thoughts on moving forward.

How edge computing can take the pain out of daily life

From eldercare to taking the anguish from building Ikea furniture, one computer science pioneer outlines how edge computing will change our days for the better. Some day soon, when you assemble a piece of Ikea furniture without tears you’ll have edge computing to thank. That’s the vision of Mahadev Satyanarayanan, known as Satya, a computer science faculty member at Carnegie Mellon University. While the word “disruptive” gets passed around more than bitcoin in the Bay Area, he means it, expecting that edge computing will touch everything from troubleshooting equipment repair to helping people with Alzheimer’s live at home for longer.

Pushing the edges with OpenStack

Verizon’s cloud-in-a-box offering based on OpenStack delivers computing to the edge — and your local coffee shop. “We need to be in the edge, the data center, all over the place,” says Beth Cohen, distinguished member of technical staff at Verizon. If you take a look at the the scope of their business, it’s easy to see why they need an edge: Verizon has a 4LTE network available to 98 percent of the US population, a global IP network covering six continents, IP and data services in over 150 countries and seven security operations centers on four continents.

Clearing up why fog computing is important

Superuser talks to Adrien Lebre, who co-chairs the OpenStack Fog, Edge and Distributed Computing Working Group. “During the last couple of years, there’s been a trend towards building mega data centers by companies like Google, Microsoft, and Amazon. The idea is that you build a mega data center with thousands of servers to cope with the demands of cloud computing. Unfortunately, with the new usage trends — internet of things, tactical internet — these mega data centers cannot satisfy the latency needs for these applications. So,we need to propose a new model that will be able to satisfy all this latency-critical requirements.”

OpenStack Verizon case study: The illusion of infinite capacity

At the OpenStack Summit Boston, Billy Felton and Andrew Hendrickson of Verizon along with Sanjay Mishra CEO of Talligent offered a look into the platform, its architecture and what’s next as they deal with the challenges of edge computing.

The post The skinny on edge computing appeared first on OpenStack Superuser.

by Superuser at December 26, 2017 01:42 PM

December 22, 2017

OpenStack Superuser

How to move a massive hyperscale insfrastructure to OpenStack

What do you do when you work for one of the original internet giants, Oath (formerly Yahoo! and AOL) and you want to update their infrastructure? Migrating hyperscale enterprise from a legacy, bespoke system to OpenStack may seem like a daunting task.James Penick managed this transition, from the initial demonstrations of the power of OpenStack to the nitty gritty of implementation. His team manages hundreds of thousands of compute resources with OpenStack software. This comes to about 70 percent of Oath’s infrastructure. Penick spoke at the recent OpenStack Summit in Sydney, Australia about his team’s migration strategy in a talk called “Migrating hyperscale enterprise to OpenStack.”

Moving such a massive infrastructure to OpenStack is not an overnight transition. “I liken it to be like a tug boat sitting at the front of a ship pushing an aircraft carrier to turn around,” Penick said. “It takes time, but if you’re relentless, and you push and you push and you push, you will get there.” As one of the first huge internet companies out there, all of the tools needed to manage the company’s infrastructure didn’t exist. That led to a lot of manual labor and an attitude of ownership before all comers.  “I actually joked that it (was) humans-as-a-service,” Penick said. “If you needed to requisition compute resources, you follow the ticket, and people would scurry around and take care of it for you.”

This sort of IT environment led to hoarding behavior, too. If specific resources — hardware or software — were created there, people tended to hang on to them, and take them with them when they leave. “The day I leave the company,” said Penick, “I’m gonna tuck that rack-mounted server under my arm and step out the door.”

The initial infrastructure at Oath consisted of dozens of layer-two backplanes in every data center, which split the entire system across many small pools of compute. “This made it very difficult to transition to more of a commodity compute approach,” Penick said, “because where do you go if you want to build a pool of compute resources when everyone’s separated into little puddles?”

The first thing Penick had to do, of course, was convince the powers that be that OpenStack was the right solution. He said that he needed to be honest with the decision makers and tell them what the platform can and cannot do. “When you’re honest and direct with people and you’re trying to influence them to move to this new thing,” he said, “they are a lot more compelled to do that when you actually show them the warts and acknowledge that they’re there.”

In addition, Penick said, you need to be able to address root needs of the organization instead of how OpenStack is supposed to work. Don’t try to solve every problem up front, either. “Premature optimization is the death of good software,” said Penick. “Don’t try and solve every problem right out of the gate. Focus on the big rocks in the jar.”

The first step to building a cloud-based, private infrastructure with a technology like OpenStack is to figure out the what and the why of the project. Penick says that his own reason was to make it easy for people to do their jobs, and to try and make the organization more agile to meet its business goals. “I want to save money, which (is) all (it) really comes down to: you want to make money and save money,” he said. Focusing on these basic tenets helps teams make decisions along the way, too. Every choice can be centered on meeting business goals.

The next step is to find friendly customers willing to pilot the new infrastructure. Ask these customers what they want. Some of the things they will ask for will be reasonable; others will not. Integrating with an existing internal tool is a good example of a reasonable request, while wanting dedicated hypervisors that they can have root access on may not be. “It’s important to pick the things that matter,” said Penick. “But hold your ground on the areas that don’t actually meet the use cases you’re trying to solve. You’re trying to present a pool, like a private cloud, for the organization, not for a specific team. So it’s important to stick to your guns on that.”

Now you’re ready to build a cluster and let your pilot customers in there, letting them boot and use compute resources. “If your pilot customer is actually in a production kind of security zone and you can surf traffic, that’s great,” said Penick. “Make sure that you can work with your customers, get that data, learn how they’re using it, and then use it to evangelize other users throughout the organization.”

It’s likely you’ll see some challenging user behavior, so be ready for it. “You’re gonna give them quotas, you’re gonna build this new cluster, and I’ve got my pilot properties and I’m gonna give them quota, and they’ve run out and they’ve booted as many VMs as they can to make sure that they’ve earmarked all this compute resources so no one will take it away from them.”  Working through it, and being able to measure what’s going on, will help you document your success.

Next, you’ll want to move forward, creating more clusters, building up your infrastructure and adding in additional users. You’ll want to focus on the perception of what you’re doing here. “Don’t try and run out the door and say, ‘I’m gonna save so much money. I’m gonna run massive overcommit. I’m gonna try to show how much money I can save,’” Penick said. “Focus on building a positive perception of using infrastructure as a service, because that’s really one of the big problems we try and solve when you move a huge company, because you’re trying to get people to buy into the idea that they can actually do their day to day jobs with an API and it’s better and faster. Saving money is something that is sort of a side effect of that.”

Penick was able to get more users on board with five clusters around the world, which they call Openhouse. “Every single person at the company can boot up to five VMs in this environment,” he said. “This allows them to build development environments on the fly. As a side effect of this, we have actually managed to kill Windows laptops and desktops under the desk. They’re gone. Except for mine. I still have mine because I use it as a footrest, so I didn’t want to give it up.” The implementation has effectively saved tons of money on hardware just by making employees’ lives better.

Now people at the organization were using VM for their sandbox environments. They were doing development work, innovating, seeing some agility. The company, however, was still focused on (“addicted to”) bare metal. Penick told them that his team would support Bare Metal, but only by integrating with the existing processes that the company used to acquire Bare Metal. “It took about thirty to ninety days to get physical compute resources, so we’re gonna slide into that and we’re gonna become a part of that,” he said. “And this means that you have to sit and you have to work very closely with supply chain teams, property architects, (and) application architects to influence them and convince them to want to use this thing.” No matter what, said Penick, there needed to be an OpenStack API between the user and the infrastructure.

Ultimately, what convinced people was the way Penick’s OverStack implementation made it even easier to use Bare Metal. “If you need to request requisition compute resources, what we’re going to do is take the ten most commonly used configurations of hardware and we’re gonna build those out into the network back planes we want to see you in. We’re going to build large pools of that hardware that you can come to a forum and say, ‘Hey, I want five of these things.’ Approved. Done. You’re off and moving.”

This encouraged the right sort of behavior by offering people an easy, simple path to do it, said Penick. Eighty to ninety percent of the individual hardware requests are now coming through a simple software tool with a few simple questions. “The other form that we offered actually had something like forty six to sixty questions, most of which could actually send you off onto a pretty dark path,” joked Penick. “So we’ve reduced this down to: who are you, what do you want, where do you want it, how many do you want?”

The new tool shows requesters how many of each type of configuration is available. “Open and honest,” said Penick. “We’re all part of the same company. We all trust each other. Give people the information so they can make an informed decision so they’re not making a stab in the dark.”

While using OpenStack infrastructure was optional at first, the managed transition led to a point where Penick could make it required. “But at this part,” said Penick, “we’ve actually had sufficient momentum to really encourage people to do the right thing.”

Finally, Penick and his team took the existing hardware and began to manage it with OpenStack. He calls this “horizontal migration” where all hardware can be deleted or re-imaged through the OpenStack system. Oath is now about seventy percent on OpenStack. “The remaining thirty percent, we’ll have most of that covered by the end of 2018,” he asserted.

It took Penick and his team about five years to manage this hyperscale transition. Now, even the hardcore folks that were initially against OpenStack automatically think of it when planning new projects. “If we were to do it all over again,” said Penick, “it would probably only take two. There’s been a lot of enhancements made to OpenStack, and there (are) a lot of perceptions that have changed in the industry.”

“But be aware,” Penick concluded, “you’re not gonna get this done in six months. And if you do, that’s amazing. But have realistic expectations of what you’ll be able to accomplish.”

Catch the entire 14-minute talk below.


Cover Photo // CC BY NC

The post How to move a massive hyperscale insfrastructure to OpenStack appeared first on OpenStack Superuser.

by Rob LeFebvre at December 22, 2017 02:33 PM

December 21, 2017

OpenStack Superuser

OpenStack: Picks for your bookshelf

Yeah, we know: books are #oldschool. They can come in handy (and we don’t mean as a doorstop, either) when you’re trying to pick up knowledge, though. Here are some picks from the most recent crop of OpenStack books.

7 steps to becoming a Certified OpenStack Administrator

The Certified OpenStack Administrator Exam prepares students for the upcoming Newton version of the exam on Ubuntu 16.04. Author Matthew Dorn focuses on providing a clear, concise strategy so you gain the specific skills required to pass the Certified OpenStack Administrator exam. The book includes exercises and performance-based tasks to ensure all exam objectives can be completed via the Horizon dashboard and command-line interface


How to design and implement successful private clouds with OpenStack

Michael Solberg and Ben Silverman wrote “OpenStack for Architects,” a guide to walk you through the major decision points to make effective blueprints for an OpenStack private cloud.  Solberg, chief architect at Red Hat, and Silverman principal cloud architect for OnX Enterprise Solutions, penned the 214-page book available in multiple formats from Packt Publishing.

“OpenStack Application Essentials”

This booklet by the Enterprise Working Group focuses on designing, migrating and deploying applications. It’s available gratis online, print on demand. It aims to help application developers the most, focusing on the factors they should keep in mind to best leverage OpenStack to increase the value of a new or an existing product.

Tips for keeping your OpenStack ops moving

This eBook from Canonical offers up a four-phase plan to remobilizing the what it calls “StuckStack infrastructure” to regain organizational agility.

Here are some salient excerpts from the book titled “How to escape StuckStack and profit from your OpenStack investment” which you can download for free with registration here.

OpenStack for the cloud challenged

Need to explain cloud to your client? “OpenStack for Dummies,” a free ebook from vScaler, sets out to do just that.

All clients are newbies. Or “dummies” to use the term that publisher Wiley made popular. That’s why London-based vScaler wrote a book for potential clients who may be cloud-challenged called “OpenStack for Dummies.” Written in collaboration with Wiley, the eBook is offered free with registration on their website and serves as a “gentle starting point” for newcomers.

More space on the shelf? Head to the Marketplace

And if you want to fill out your shelf more, remember that the OpenStack Marketplace — your one-stop shop for training, distros, private-cloud-as-a-service and more — now offers a selection of technical publications, too. The listings are not affiliate links, but offered as a way to highlight the efforts of community members.

Under the “books” heading, you’ll find titles by Stackers including “Mastering OpenStack,” “OpenStack Networking Essentials,” and “OpenStack: Building a Cloud Environment.”

The post OpenStack: Picks for your bookshelf appeared first on OpenStack Superuser.

by Superuser at December 21, 2017 05:21 PM

Red Hat Stack

Using Ansible for Fernet Key Rotation on Red Hat OpenStack Platform 11

In our first blog post on the topic of Fernet tokens, we explored what they are and why you should think about enabling them in your OpenStack cloud. In our second post, we looked at the method for enabling these

Fernet tokens in Keystone are fantastic. Enabling these, instead of UUID or PKI tokens, really does make a difference in your cloud’s performance and overall ease of management. I get asked a lot about how to manage keys on your controller cluster when using Fernet. As you may imagine, this could potentially take your cloud down if you do it wrong. Let’s review what Fernet keys are, as well as how to manage them in your Red Hat OpenStack Platform cloud.

Photo by Freddy Marschall on Unsplash


  • A Red Hat OpenStack Platform 11 director-based deployment
  • One or more controller nodes
  • Git command-line client

What are Fernet Keys?

Fernet keys are used to encrypt and decrypt Fernet tokens in OpenStack’s Keystone API. These keys are stored on each controller node, and must be available to authenticate and validate users of the various OpenStack components in your cloud.

Any given implementation of keystone can have (n)keys based on the max_active_keys setting in /etc/keystone/keystone.conf. This number will include all of the types listed below.

There are essentially three types of keys:


Primary keys are used for token generation and validation. You can think of this as the active key in your cloud. Any time a user authenticates, or is validated by an OpenStack API, these are the keys that will be used. There can only be one primary key, and it must exist on all nodes (usually controllers) that are running the keystone API. The primary key is always the highest indexed key.


Secondary keys are only used for token validation. These keys are rotated out of primary status, and thus are used to validate tokens that may exist after a new primary key has been created. There can be multiple secondary keys, the oldest of which will be deleted based on your max_active_keys setting after each key rotation.


These keys are always the lowest indexed keys (0). Whenever keys are rotated, this key is promoted to a primary key at the highest index allowable by max_active_keys. These keys exist to allow you to copy them to all nodes in your cluster before they’re promoted to primary status. This avoids the potential issue where keystone fails to validate a token because the key used to encrypt it does not yet exist in /etc/keystone/fernet-keys.

The following example shows the keys that you’d see in /etc/keystone/fernet-keys, with max_active_keys set to 4.

0 (staged: the next primary key)
1 (primary: token generation & validation)

Upon performing a key rotation, our staged key (0), will be the new primary key (2), while our old primary key (1), will be moved to secondary status (1).

0 (staged: the next primary key)
1 (secondary: token validation)
2 (primary: token generation & validation)

We have three keys here, so yet another key rotation will produce the following result:

0 (staged: the next primary key)
1 (secondary: token validation)
2 (secondary: token validation)
3 (primary: token generation & validation)

Our staged key (0), now becomes our primary key (3). Our old primary key (2), now becomes a secondary key (2), and (1) remains a secondary key.

We now have four keys, the number we’ve set in max_active_keys. One more final rotation would produce the following:

0 (staged: the next primary key)
1 (deleted)
2 (secondary: token validation)
3 (secondary: token validation)
4 (primary: token generation & validation)

Our oldest key, secondary (1), is deleted. Our previously staged key (0), is moved to primary (4) status.  A new staged key (0) is created. And finally our old primary key (3) is moved to secondary status.

If you haven’t noticed this by now, rotating keys will always remove the key with the lowest index, excluding 0 — up to your max_active_keys. Additionally, note that you must be careful to set your max_active_keys configuration setting to something that makes sense, given your token lifetime and how often you plan to rotate your keys.

When to rotate?

Photo by Uroš Jovičić on Unsplash

The answer to this question would probably be different for most organizations. My take on this is simply: if you can do it safely, why not automate it and do it on a regular basis? Your threat model and use-case would normally dictate this or you may need to adhere to certain encryption and key management security controls in a given compliance framework. Whatever the case, I think about regular key rotation as a best-practices security measure. You always want to limit the amount of sensitive data, in this case Fernet tokens, encrypted with a single version of any given encryption key. Rotating your keys on a regular basis creates a smaller exposure surface for your cloud and your users.

How many keys do you need active at one time? This all depends on how often you plan to rotate them, as well as how long your token lifetime is. The answer to this can be expressed in the following equation:

fernet-keys = token-validity(hours) / rotation-time(hours) + 2

Let’s use an example of rotation every 8 hours, with a default token lifetime of 24 hours. This would be

24 hours / 8 hours + 2 = 5

Five keys on your controllers would ensure that you always had an active set of keys for your cloud. With this in mind, let’s look at way to rotate your keys using Ansible.

Rotating Fernet keys

So you may be wondering, how does one automate this process? You can image that this process can be painful and prone to error if done by hand. While you could use the fernet_rotate command to do this on each node manually, why would you?

Let’s look at how to do this with Ansible, Red Hat’s awesome tool for automation. If you’re new to Ansible, please do yourself a favor and check out this quick-start video.

We’ll be using an Ansible role, created by my fellow Red Hatter Juan Antonio Osorio (Ozz), one of the coolest guys I know. This is just one way of doing this. For a Red Hat OpenStack Platform install you should contact Red Hat support to review your options and support implications. And of course, your results may vary so be sure to test out on a non-production install!

Let’s start by logging into your Red Hat OpenStack director node as the stack user, and creating a roles directory in /home/stack:

$ cat << EOF > ~/rotate.yml
- hosts: controller 
  become: true 
    - tripleo-fernet-keys-rotation

We need to source our stackrc, as we’ll be operating on our controller nodes in the next step

$ source ~/stackrc

Using a dynamic inventory from /usr/bin/tripleo-ansible-inventory, we’ll run this playbook and rotate the keys on our controllers

$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory rotate.yml

Ansible Role Analysis

What happened? Looking at Ansible’s output, you’ll note that several tasks were performed. If you’d like to see these tasks, look no further than /home/stack/roles/tripleo-fernet-keys-rotation/tasks/main.yml:

This task runs a python script, generate_key_yaml.py, in ~/roles/tripleo-ansible-inventory/files, that creates a new fernet key:

- name: Generate new key
 script: generate_key_yaml.py
 register: new_key_register
 run_once: true

This task will take the output of the previous task, from stdout, and register it as the new_key.

- name: Set new key fact
 new_key: "{{ new_key_register.stdout }}"

Next, we get a sorted list of the keys that currently exist in /etc/keystone/fernet-keys

- name: Get current primary key index
 shell: ls /etc/keystone/fernet-keys | sort -r | head -1
 register: current_key_index_register

Let’s set the next primary key index

- name: Set next key index fact
 next_key_index: "{{ current_key_index_register.stdout|int + 1 }}"

Now we’ll move the staged key to the new primary key

- name: Move staged key to new index
 command: mv /etc/keystone/fernet-keys/0 /etc/keystone/fernet-keys/{{ next_key_index }}

Next, let’s set our new_key to the new staged key

- name: Set new key as staged key
 content: "{{ new_key }}"
 dest: /etc/keystone/fernet-keys/0
 owner: keystone
 group: keystone
 mode: 0600

Finally, we’ll reload (not restart) httpd on the controller, allowing keystone to load the new keys

- name: Reload httpd
 name: httpd
 state: reloaded


Now that we have a way to automate rotation of our keys, it’s time to schedule this automation. There are several ways you could do this:


You could, but why?

Systemd Realtime Timers

Let’s create the systemd service that will run our playbook:

cat << EOF > /etc/systemd/system/fernet-rotate.service
Description=Run an Ansible playbook to rotate fernet keys on the overcloud
ExecStart=/usr/bin/ansible-playbook \
  -i /usr/bin/tripleo-ansible-inventory /home/stack/rotate.yml

Now we’ll create a timer with the same name, only with .timer as the suffix, in /etc/systemd/system on the director node:

cat << EOF > /etc/systemd/system/fernet-rotate.timer
Description=Timer to rotate our Overcloud Fernet Keys weekly

Ansible Tower

I like how your thinking! But that’s a topic for another day.

Red Hat OpenStack Platform 12

Red Hat OpenStack Platform 12 provides support for key rotation via Mistral. Learn all about Red Hat OpenStack Platform 12 here.

What about logging?

Ansible to the rescue!

Ansible will use the log_path configuration option from /etc/ansible/ansible.cfg, ansible.cfg in the directory of the playbook, or $HOME/.ansible.cfg. You just need to set this and forget it.

So let’s enable this service and timer, and we’re off to the races:

$ sudo systemctl enable fernet-rotate.service
$ sudo systemctl enable fernet-rotate.timer

Credit: Many thanks to Lance Bragstad and Dolph Matthews for the key rotation methodology.

by Ken Savich, Senior OpenStack Solution Architect at December 21, 2017 02:08 AM

December 20, 2017

OpenStack Superuser

How the French government uses OpenStack

The call came unexpectedly. The Paris-based Osones team had jumped the pond to participate in the 2014 OpenStack Atlanta Summit.  On the line was a partner for France’s Interior Ministry, who said the government was looking to use OpenStack and sought an experienced team. They’d made the right call: the cloud experts at Osones had been evangelists, contributors, Summit attendees and active members of the OpenStack French-speaking User Group since 2013.

Osones’ main mission was to transfer knowledge to Ministry IT engineers, to advocate OpenStack best practices and to help in choosing tools and deployment strategies. The Osones team took part in building the OpenStack platform itself as well, until it was up and running and ready to host the first applications, Jean Francois Taltavull and Adrien Cunin, both cloud engineers at Osones told Superuser.

Some of the current applications using OpenStack are part of the larger push for bureaucratic reforms started by president Emmanuel Macron back when he was finance minister. These uses of OpenStack include processing requests from asylum seekers — for an idea of scale, there were 75,000 first-time asylum applicants in France in 2016,  over 6 percent of the European Union total — and fast-tracking the notoriously slow driver’s license process.

Superuser talked to Nicolas Duffour at the Interior Ministry about what they’re doing now and planning on for the future.

How are you using OpenStack? What kinds of applications or workloads
are you currently running on OpenStack?

OpenStack was chosen by the French government and promoted by DINSIC (the national IT shop and open data initiative formed in 2015) because it’s open source and efficient as a cloud stack.

At the moment, we’re running two applications in production: ASILE to process asylum requests and ET GOA to process the Macron law initiative for privatizing the written driver’s license test and making it available thanks to a web server in public places such as the post office.

About 20 projects are under consideration to go to our cloud, PI (Produits de l’Interieur), and we plan to migrate 150 applications in the next three to five years.

What have been the biggest benefits to your organization as a result of using OpenStack? How are you measuring the impact?

The main benefit for our organization is to offer a real digital platform to accompany the digital transformation and modernization of the administration: secure (meeting the national standard for providers, SecNum cloud ANSSI), robust, efficient for our clients of the Interior Ministry and the other administrations. Being able to develop in agile/dev ops mode and gain in continuous delivery and in integration; concentrate infrastructure resources available for the applications and make resources available interactively.

We are defining the dashboard with our clients to measure more precisely the impact. A 30-person transformation team has been created to help and accompany our clients to manage and use our cloud.

What future plans can you share with us?

Our short-term plan is to open our platform to the other administrations at the beginning of 2018 and to deploy our cloud on two sites to be fully resilient and take steps with our clients to evolve our services catalog.


Superuser is always interested in user stories, get in touch at editorATopenstack.org

Cover Photo // CC BY NC

The post How the French government uses OpenStack appeared first on OpenStack Superuser.

by Nicole Martinelli at December 20, 2017 05:20 PM

OpenStack Blog

User Group Newsletter December 2017

Summit recap

With 2300+ attendees and 54 countries represented, it was a fantastic week in Sydney. Thank you to all who attended!

You can catch all the action from the Keynotes here. The Summit talk videos are live, find them here. Superuser also covered the Summit with an awesome article covering 50 things you need to know.

Next Summit

Our next Summit destination is in Vancouver. Registration and Hotel booking are now available. Last time we went to Canada, these sold out quick. Don’t miss out and secure yours today.

Congratulations to SuperUser award winners!

On the final day of the Sydney Summit, the Community Contributor Awards gave a special tip of the hat to those who might not know how much they are valued. Read more about the ceremony here.  

Stay tuned to Superuser and OpenStack’s social channels for when nominations open for Vancouver.

2018 CFP dates

Cloud Native Con/KubeCon Europe

Closing date: January 12, 2018

ONS 2018

Closing date: January 14, 2018

LF Open Source Leadership

Closing date: January 21, 2018

OSCON 2018

Closing Date: January 30, 2018

LinuxCon ContainerCon | CloudOpen China

Closing Date: March 4, 2018

ONS Europe

Closing Date: June 24, 2018

Superuser articles of interest

Some fantastic Superuser articles have been published in recent weeks. They include tutorials, information about a new working group and project updates.

Making your first contact with OpenStack

Check out these OpenStack project updates

Working together: OpenStack and Kubernetes

Travel grants support global community to attend OpenStack Summit

How to use OpenStack Glance Image Import

Launch of Kata Containers

The Foundation is excited to announce Kata Containers. This project unites the security advantages of virtual machines with the speed and manageability of containers for running container management tools directly on bare metal without sacrificing workload isolation. Kata Containers delivers increased performance, faster boot time and cost efficiencies. Kata Containers will have its own governance, community and communication channels.

To find out more and get involved:
Sign up for Kata Containers updates at katacontainers.io

Get involved on the Kata Containers Mailing List, Slack* and IRC* (#kata-dev)

Follow Kata Containers on Twitter: @katacontainers

HPC book update

The Scientific Working Group has updated the Crossroads of Cloud and HPC: OpenStack for Scientific Research. Version 2 of the popular book details multiple enhancements made to OpenStack for HPC workloads over the past year and added several new in-depth case studies on how OpenStack supports radio astronomy, bioinformatics, cancer and genome research and more. Read or order the book at https://www.openstack.org/science/ to find out exactly how OpenStack allows researchers to spend less time managing infrastructure and more time on research that matters. To learn more or get involved in the Scientific Working Group, visit https://wiki.openstack.org/wiki/Scientific_SIG


It has been a fantastic 2017!!! Thank you to all community members for your industrious efforts and contributions! Wish you all a Happy Holiday break and a great New Year.


by Sonia Ramza at December 20, 2017 01:51 PM

December 19, 2017

Adam Spiers

Abstraction As A Service

The birth of abstraction layers

The last five decades of computing have seen a gradual progression of architectural abstraction layers. Around 50 years ago, IBM mainframes gained virtualization capabilities. Despite explosive progress in the sophistication of hardware following Moore’s Law, there wasn’t too much further innovation in abstraction layers in server computing until well after the dawn of the microcomputer era, in the early 2000s, when virtualization suddenly became all the rage again. (I heard a rumour that this was due to certain IBM patents expiring, but maybe that’s an urban myth.) Different types of hypervisors emerged, including early forms of containers.

Then we started to realise that a hypervisor wasn’t enough, and we needed a whole management layer to keep control of the new “VM sprawl” problem which had arisen. A whole bunch of solutions appeared, including the concept of “cloud”, but many were proprietary, and so after a few years OpenStack came along to the rescue!

The cloud era

But then we realised that managing OpenStack itself was a pain, and someone had the idea that rather than building a separate management layer for managing OpenStack, we could just use OpenStack to manage itself! And so OpenStack on OpenStack, or Triple-O as it’s now known, was born.

Within and alongside OpenStack, several other new exciting trends emerged: Software-Defined Networking (SDN), Software-Defined Storage (e.g. Ceph), etc. So the umbrella term Software-Defined Infrastructure was coined to refer to this group of abstraction layers.

The container era

Whilst OpenStack was busy growing up and moving past the Peak of Inflated Expectations, all of a sudden Docker and containers burst onto the scene and provided a lot of new buzzwords to get everyone excited again. But after the excitement started to fade, that familiar sinking feeling came back with the realisation that just like VMs, containers need something to manage them.

But then Kubernetes leapt in to the rescue! And all the excitement returned. Except that of course then you need something to manage Kubernetes, but fortunately we already had OpenStack, so we could just use that! And so Magnum was born. And since Kubernetes is so awesome, we realised that we could also use it as the basis for deploying OpenStack. At the recent OpenStack Summit in Sydney, we saw the continued rise in popularity of running both Kubernetes on OpenStack, and OpenStack on Kubernetes.

Looking to the future

But that still leaves the pesky job of managing raw hardware to put all this stuff on top. Fortunately there are services you can pay for so that other people do that for you, and they even have APIs you can hook into! This is called public cloud. And even better, many companies use OpenStack to drive their public clouds.

So now we’re running Kubernetes on OpenStack on Kubernetes on OpenStack. And now we’re done! Right?

Well … if we’re to learn anything from this history, it should be that we’ll always find more good reasons for new abstraction layers. I mean, there’s already a huge amount of work going into things like Cloud Foundry on the PaaS layer, NFV in the telco space, serverless computing, … And I hear that Kubernetes is a great platform for running Cloud Foundry, just like it is for OpenStack. And wouldn’t OpenStack be a cool platform to provide inside Cloud Foundry, e.g. for people who just want to quickly try it out? So who knows, maybe in the next few years we’ll have OpenStack on Cloud Foundry on Kubernetes on OpenStack on Kubernetes on OpenStack.

A proposal to make things simpler

Of course this starts getting a bit unwieldly. Every time we introduce a new abstraction layer there’s extra complexity to deal with. But that’s OK, because we can always deal with complexity by abstracting it away! It’s a bit like the cyber-equivalent of delegating difficult tasks to someone else. So I’d like to propose a new concept, and corresponding meta-component of the overall architecture:

Whenever we realise we need a new abstraction layer, rather than having to deal with the complexity of deploying and managing this layer, we could just invoke APIs to a central service which takes care of this complexity for us. We could call this new concept (drum roll, please…) Abstraction As A Service, or AaaS. (If you’re British or Australian you may prefer to pronounce this as if there was an “r” in the middle, to distinguish from similar-sounding existing words such as “as”. Or maybe we should call it Sweeping Stuff Under The Carpet As A Service? Or SSUtCaaS for short, which can be pronounced “suitcase” (thanks to Florian for pointing this out).

With AaaS, if we wanted say, NFV on serverless on CF on Kubernetes on OpenStack on Kubernetes on OpenStack on public cloud on COBOL, we could simply write some declarative YAML or JSON describing the stack we want, push it to the AaaS REST API endpoint via an HTTP POST, and it would set the whole thing up for us automatically. We could build any number and combination of abstraction layers we would possibly need, so at this point the job could be considered well and truly done, once and for all!

Except, ya know, we’d need a way to deploy and manage our AaaS service, of course. Maybe we could build an AaaSaaS service for that …

Sheepish postscript / disclaimer

P.S. I know, it’s a terrible joke if you have to explain it, but based on previous experiences of my dry British humour being misunderstood (especially given the inevitably international audience), I feel the need to point out that this blog post was intended as nothing more than poking a bit of gentle fun at the cloud software industry. I’m a huge fan of all the technologies mentioned here, and yes, I’m even in favour of multiple abstraction layers, despite occasionally wondering if we’ve all gone a bit insane 😉 Thanks to Florian, Andrew, and Dirk for reviewing an earlier draft, but I take responsibility for any mistakes or any offence unintentionally caused!


The post Abstraction As A Service appeared first on Structured Procrastination.

by Adam at December 19, 2017 06:58 PM

OpenStack Superuser

Composing open source cloud technologies in the “wild”

OpenStack survey data shows that many industries beyond IT are finding the open source cloud-computing management platform an essential component of their businesses. Along with telecommunications, government, and retail, financial services is one of the industries showing the most significant growth in deployments of OpenStack technology.

As part of the recent OpenStack Summit in Sydney, Australia, Commonwealth Bank’s Quinton Anderson gave a short rundown of his company’s basic OpenStack deployment, which helps Commonwealth manage the data to help figure out what its customers really want and need.

Commonwealth is one of the larger banks in Australia, said Anderson and one of the largest banks in the world by market cap. His team started to look at maturing its approach to data management and machine learning four or five years ago and found that there were three levels of maturity their system needed to advance through. “You start with the basic process of experimentation, a first level of maturity,” he said. “You get some data in the correct place and you start to generate some experiments.”

The experimental process was interesting, said Anderson, and it helped his team learn a lot about data management. Taking the often brittle and scientific nature of these early experiments from lab to factory, he said, is the next step of maturity. “How do I get to the state where my experiments result in predictions, result in customizations of the way I engage with my customer and the way my product systems behave, in a way that is reliable, operational, (and) productionized?”

Anderson’s team spent a lot of time figuring out how to deal with data labels as well as with data and model monitoring. The third level of maturity, then, is deciding how to impact the customer experience and the way the business operates as a result. “We call this third level of maturity decisioning,” Anderson said.

The logical representation of Commonwealth’s decisioning system is above. It has two major use cases, one for closed loop, automated decisions and one that focuses on information for people. The automated system integrates analytical and data management processes into the product system that impacts what the customers see, when they see it, and how the company engages with those customers as a result. The second use case is for internal staff to know what’s going on with the business and its customers’ businesses and lives to help staff make informed decisions.

The team started with some fairly large Hadoop clusters, with some fairly traditional data warehousing technology. The Hadoop clusters were initially based on physical servers in Commonwealth’s data centers. “That worked in the beginning,” Anderson said, “but as you can imagine, (it) comes with a bunch of inherent constraints, which we needed to work through.”

Obviously, said Anderson, the cost of upgrading and changing this initial setup was not good for the company or developers. “We departed on a journey to try and cloud-enable some of these technologies, said Anderson, “so that the process of experimentation was a lot faster for our business, but also a lot cheaper, and a lot more deterministic.”

The technology stack Commonwealth decided on is fairly typical. “We’ve got OpenStack at the bottom, and we run Ironic for that,” Anderson said. “The reason (we used) Ironic, for the initial set of use cases, is the (ability to do) big data processing; staple services want to be close to the disk. Also, increasingly as we move into micro-services, the VM abstraction is, in a lot of ways, uncomfortable if you want to drive immutability.”

The team runs Ubuntu on top of that, with Docker for containers and Calico for container networking. They also run Vault for secrets-management, as well as Mesos and Marathon for resource orchestration with a platform-as-a-service API. “On top of that, we run our big data applications, and finally end applications on top of that,” Anderson said. “This is a fairly typical stack… It’s a container cluster on top of a cloud.”

Anderson feels that what makes their implementation unique is how it focuses on continuous delivery at all layers of the stack. They also focused on versioning through codification, along with immutability through all layers. “For us, an upgrade of an operating system, or even changing the permissions on a folder inside an operating system in an underlying host is not a matter of SSH-ing into a box,” he said, “but rather updating a declarative form of the environment, and issuing a pull request, and allowing the tooling to make all the changes on your behalf.

That is true for the operating system, for containers and container clusters, for monitoring, orchestrators, alerting and absolutely everything in the system. “We use the same basic interaction pattern,” Anderson said, “which is versioning through codification, automation through codification, and continuous delivery, applying the changes off the back of that.”

The workflow separates out a module repository, which has the source code, the tests, the pipeline, the pool, and so on, for each particular module (which can be a container in the control tier, an app, or anything executing in an environment. “We separate those module repos from our environment repos,” said Anderson. “The environment repos contain declarative forms of the environment… (and) we try and ensure that these things are strongly versioned.”

There’s a declarative form of the deployment’s environment sitting on Master, which represents what’s running in the designated environment. Anderson can fork the repository from that point, making changes like upgrades, host introduction, deploying new clusters or applications, and the like, and then issue a pool request against the deploy branch. “The pool tooling will then generate a change plan,” he said, “and attach that to the pool request for me. You can think about this workflow as being pretty similar to HashiCorp’s Terraform, where it’s going to generate a change plan, but we’ve integrated this very strongly into GitHub, and therefore into version control.”

Several “bots” make reviews of any requested changes to make sure there’s enough environmental capacity, that no changes are too risky, and that they’re being deployed into the correct environment. They can also check for any junior developers who shouldn’t be pooling against production, said Anderson. There are also manual pool checks, too, where an actual human will look at proposed changes. “When the pool request merges,” he said, “the tooling goes off and applies the changes into the environment. You can think about what’s sitting on the deploy branch as being, effectively, the desired state, what’s sitting on master as being the actual state or the previous state. The tooling is going to go off and apply those changes to the environment by making API calls to Nova or Marathon, HashiCorp Vault, so forth. Once the changes have completed, it will then merge the new declarative form back onto Master.”

If there are any failures in the deployment, it’s fairly easy to get back from that point. “You’ve got desired state,” said Anderson. “You’ve got actual state. You’ve got an interim desired state convergence, which are in your logs. Of course, as you do smaller and smaller changes, that problem becomes less and less.”

What the team has done, then, is compose a large number of open-source projects together, with OpenStack as the foundation, in order to generate Commonwealth’s business outcomes. “We believe strongly that versioning through codification gives us an absolutely best-case we have to change all of those different levels of compositions, and therefore services and systems, in a sustainable manner over time,” said Anderson. “It allows us to recreate environments. It allows to build automation in a scalable manner. It allows us to recreate the environments, and therefore have automated testing, which is the basis on which safe change can take place.”

Anderson started with OpenStack as the basis for this approach, in order to even just do immutable and codified access of their infrastructure. “OpenStack Ironic is the first place you start,” he said. “Then, it’s a buildup from there.”

Over time, though, Anderson’s team will take the same approach to its public cloud environments to match it’s overall hybrid cloud strategy. “In the analytics space then, R is a major tool of use from our data scientists,” he said. “(We’re using) Hadoop, (and) increasingly Spark, as a part of the ecosystem. We still have a fair amount of HD affairs from a stateful workload perspective, and increasingly Cassandra. (We’re) starting small, but in increasing amount of TensorFlow coming in.”

The team uses Mesos primarily in the stateful space right now, and increasingly in the stateless space Kubernetes, as well. Calico offers container networking in a simple, well-integrated way with the various existing networking environments across different cloud providers. The team uses Docker as their containers, and Ubuntu as the hosts. “This provides us not only a nice environment for our stateful and analytics workloads,” said Anderson. “It also, increasingly, is providing us with a nice environment to host different types of workloads (and) pure micro-services. It started with just data science dashboards, Python user interfaces, Python dashboards, etc. and increasingly is going into our more web-facing workloads and micro-services.”

Anderson calls OpenStack an important foundation for building Commonwealth Bank’s internal systems. “It’s been a really interesting journey to start with composing OpenStack and some basic container environments,” he said, “and work through the entire process of composing the rest of these open source technologies, using a strong approach of getting the basics right first.”

Catch the whole keynote talk below.

The post Composing open source cloud technologies in the “wild” appeared first on OpenStack Superuser.

by Rob LeFebvre at December 19, 2017 05:27 PM

Chris Dent

TC Report 51

This will be the last TC Report of the year, expect the next one around the 9th of January.

We're closing out the year with a series of discussions about making changes to deal with various frustrations in the contributor community. They were triggered by a proposal to consider extending the release cycle as a way to adapt to changes in the rhythm of the community. This exploded into a monster email thread and several episodes of lengthy discussion in IRC.

Thierry made a TL;DR of the email thread, incorporating some of what he heard from the IRC discussion. It provides a pretty good summary, touching on many of the major facets.

An additional idea, changing from PTL to shared leaders, was also mooted. This one a sort of strawman to hear what people care about; a less well-formed idea to contemplate the issues surrounding overloaded PTLs.

There is a great deal in IRC to look at, many perspectives and insights, if you have the time and energy to read through IRC logs:

Throughout all of this it is very clear that there are a huge number of partially related frustrations that no single solution is going to resolve.

There was a bit of a tangent this morning where I wanted us to zero in on a foundational cause but that led to a paralysis of "all our problems are probably due to capitalism". This suggests that we have to approach this using analysis and resolution of second order problems.

What we haven't got, though, is an aggregation of what people consider to be the problems. Instead we have lurking fear or FUD. And if we did have the list of problems, getting agreement on priority would be difficult. This suggests we need, in part, to achieve some measure of decoupling so that different projects can apply different solutions. One size will not fit all.

A constant throughout all this discussion is the question of "who will do the additional work" and the need for a vastly more effective feedback loop with the corporations that support the humans doing work in the community. Not just with the foundation board but also with the internal groups doing the work and/or managing the humans. Fidelity of communication in terms of requirements and roadblocks, from all parties, in all directions, is low.

We can improve many of the things people have been talking about, but it's going to take a lot of discussion to reach agreement on what each of the problems really are. Historically we have not been that good, as a community, at accepting that broadly ranging conversations are an okay use of time.

They are. They help us develop the shared language that is critical to developing the shared understanding that is required to establish shared goals.

Until we crush capitalism, we work with what we've got. Happy New Year, comrades.

by Chris Dent at December 19, 2017 04:45 PM

NFVPE @ Red Hat

Kubernetes multiple network interfaces — but! With different configs per pod; Multus CNI has your back.

You need multiple network interfaces in each pod – because you, like me, have some more serious networking requirements for Kubernetes than your average bear. The thing is – if you have different specifications for each pod, and what network interfaces each pod should have based on its role, well… Previously you were fairly limited. At least using my previous (and somewhat dated) method of using Multus CNI (a CNI plugin to help enable you to have multiple interfaces per pod), you could only apply to all pods (or at best, with multiple CNI configs per box, and have it per box). Thanks to Kural and crew, Multus includes the functionality to use Kubernetes Custom Resources (Also known as “CRDs”). These “custom resource definitions” are a way to extend the Kubernetes API. Today we’ll take advantage of that functionality. The CRD implementation in Multus allows us to specify exactly what multiple network interfaces each pod has based on annotations attached to each pod. Our goal here will be to spin up a Kubernetes cluster complete with Multus CNI (including the CRD functionality), and then we’ll spin up pods where we have some with a single interface, and some with multiple interfaces, and then we’ll inspect those.

by Doug Smith at December 19, 2017 04:00 PM

December 18, 2017

OpenStack Superuser

Rio de Janeiro hackers build smarter cities

Three days, 14 teams and the grand prize of a fully funded trip to the OpenStack Summit Vancouver at stake—participants at the recent OpenStack Hackathon Rio de Janeiro rose to the challenge to develop projects around a smart cities theme.

In early December, some 67 hackers were tasked with solving real problems identified by the city’s innovation department around accessibility, public health, childhood education, safety and mobility.

Over 160 people participated in the Rio de Janeiro Smart Cities Hackathon.

Inspired by the visually impaired, the winning team of four created a platform dubbed Alirou to conquer accessibility challenges with a high-contrast color palette and shapes so that printed materials can be interpreted.

Alirou is a collaborative platform where people are visually impaired can have a better experience, with audio descriptions, access ramps and tactile floors.

The winning team who created Alirou, a platform to improve accessibility for the visually impaired.

Team members Alexandre Werneck, Jesiel Conceição, Luíza Valério and Pedro Mendes, represented the cities of Rio de Janeiro, Florianópolis, Campos and Campos, respectively.

The name is almost as innovative as the project: they hacked together ‘aliro’  which means access in Esperanto and ‘iom’ which means a little, in a nod to how a small gesture to gain relevant information can make a big impact on people’s lives.

In second place came a public health project by team SUSi, in third place a recyclable trash project by Rio Descarte Inteligente.

The winning team received a fully funded trip to the OpenStack Summit Vancouver where they will have the opportunity to network and attend hundreds of sessions around open infrastructure.

It was made possible with by generous support of sponsors. They include: Neopath, Assespro-RJ, Rio’s association of ICT companies, the national agency to promote entrepreneurship, Sebrae,  university student association UFRJ and the city’s agency for development, employment and innovation, SMDEIRJ.

Organizer João C D Freire Ribeiro  gave a special shout-out to that global fuel for hackathons, pizza, in this case provided “by the hundreds” from Domino’s.

Cover photo // CC by NC

The post Rio de Janeiro hackers build smarter cities appeared first on OpenStack Superuser.

by Allison Price at December 18, 2017 06:16 PM

James Page

Ubuntu OpenStack Dev Summary – 18th December 2018

Welcome to the Ubuntu OpenStack development summary!

This summary is intended to be a regular communication of activities and plans happening in and around Ubuntu OpenStack, covering but not limited to the distribution and deployment of OpenStack on Ubuntu.

If there is something that you would like to see covered in future summaries, or you have general feedback on content please feel free to reach out to me (jamespage on Freenode IRC) or any of the OpenStack Engineering team at Canonical!

OpenStack Distribution

Stable Releases

Current in-flight SRU’s for OpenStack related packages:

Ceph 12.2.1

OpenvSwitch 2.8.1

nova-novncproxy process gets wedged, requiring kill -HUP

Horizon Cinder Consistency Groups

Recently released SRU’s for OpenStack related packages:

Percona XtraDB Cluster Security Updates

Pike Stable Releases

Ocata Stable Releases

Ceph 10.2.9

Development Release

Since the last dev summary, OpenStack Queens Cloud Archive pockets have been setup and have received package updates for the first and second development milestones – you can install them on Ubuntu 16.04 LTS using:

sudo add-apt-repository cloud-archive:queens[-proposed]

OpenStack Queens will also form part of the Ubuntu 18.04 LTS release in April 2018, so alternatively you can try out OpenStack Queens using Ubuntu Bionic directly.

You can always test with up-to-date packages built from project branches from the Ubuntu OpenStack testing PPA’s:

sudo add-apt-repository ppa:openstack-ubuntu-testing/queens

Nova LXD

No significant feature work to report on since the last dev summary.

The OpenStack Ansible team have contributed an additional functional gate for nova-lxd – its currently non-voting, but does provide some additional testing feedback for nova-lxd developers during the code review process.  If it proves stable and useful, we’ll make this a voting check/gate.

OpenStack Charms

Ceph charm migration

Since the last development summary, the Charms team released the 17.11 set of stable charms; this includes a migration path for users of the deprecated ceph charm to using ceph-mon and ceph-osd. For full details on this process checkout the charm deployment guide.

Queens Development

As part of the 17.11 charm release a number of charms switched to execution of charm hooks under Python 3 – this includes the nova-compute, neutron-{api,gateway,openvswitch}, ceph-{mon,osd} and heat charms;  once these have had some battle testing, we’ll focus on migrating the rest of the charm set to Python 3 as well.

Charm changes to support the second Queens milestone (mainly in ceilometer and keystone) and Ubuntu Bionic are landing into charm development to support ongoing testing during the development cycle.  OpenStack Charm deployments for Queens and later will default to using the Keystone v3 API (v2 has been removed as of Queens).  Telemetry users must deploy Ceilometer with Gnocchi and Aodh as the Ceilometer API has now been removed from charm based deployments and from the Ceilometer codebase.  You can install the current tip of charm development using the the openstack-charmers-next prefix for charmstore URL’s – for example:

juju deploy cs:~openstack-charmers-next/neutron-api

ZeroMQ support has been dropped from the charms; having no know users and no functional testing in the gate and having issued deprecation warnings in release notes it was time to drop the associated code from the code base.  PostgreSQL and deploy from source are also expected to be removed from the charms this cycle.

You can read the full list of specs currently scheduled for Queens here.


The last stable charm release went out at the end of November including the first stable release of the Gnocchi charm – you can read the full details in the release notes.  The next stable charm release will take place in February alongside OpenStack Queens, with a release shortly after the Ubuntu 18.04 LTS release in May to sweep up any pending LTS support and fixes needed.

IRC (and meetings)

As always, you can participate in the OpenStack charm development and discussion by joining the #openstack-charms channel on Freenode IRC; we also have a weekly development meeting in #openstack-meeting-4 at either 1000 UTC (odd weeks) or 1700 UTC (even weeks) – see http://eavesdrop.openstack.org/#OpenStack_Charms for more details.  The next IRC meeting will be on the 8th of January at 1700 UTC.

And finally – Merry Christmas!



by JavaCruft at December 18, 2017 04:18 PM

Carlos Camacho

Automating Undercloud backups and a Mistral introduction for creating workbooks, workflows and actions

The goal of this developer documentation is to address the automated process of backing up a TripleO Undercloud and to give developers a complete description about how to integrate Mistral workbooks, workflows and actions into the Python TripleO client.

This tutorial will be divided into several sections:

  1. Introduction and prerequisites
  2. Undercloud backups
  3. Creating a new OpenStack CLI command in python-tripleoclient (openstack undercloud backup).
  4. Creating Mistral workflows for the new python-tripleoclient CLI command.
  5. Give support for new Mistral environment variables when installing the undercloud.
  6. Show how to test locally the changes in python-tripleoclient and tripleo-common.
  7. Give elevated privileges to specific Mistral actions that need to run with elevated privileges.
  8. Debugging actions
  9. Unit tests
  10. Why all previous sections are related to Upgrades?

1. Introduction and prerequisites

Let’s assume you have a TripleO development environment healthy and working properly. All the commands and customization we are going to run will run in the Undercloud, as usual logged in as the stack user and having sourced the stackrc file.

Then let’s proceed by cloning the repositories we are going to work with in a temporary folder:

mkdir dev-docs
cd dev-docs
git clone https://github.com/openstack/python-tripleoclient
git clone https://github.com/openstack/tripleo-common
git clone https://github.com/openstack/instack-undercloud
  • python-tripleoclient: Will define the OpenStack CLI commands.
  • tripleo-common: Will have the Mistral logic.
  • instack-undercloud: Allows to update and create mistral environments to store configuration details needed when executing Mistral workflows.

2. Undercloud backups

Most of the Undercloud back procedure is available in the TripleO official documentation site.

We will focus on the automation of backing up the resources required to restore the Undercloud in case of a failed upgrade.

  • All MariaDB databases on the undercloud node
  • MariaDB configuration file on undercloud (so we can restore databases accurately)
  • All glance image data in /var/lib/glance/images
  • All swift data in /srv/node
  • All data in stack users home directory

For doing this we need to be able to:

  • Connect to the database server as root.
  • Dump all databases to file.
  • Create a filesystem backup of several folders (and be able to access folders with restricted access).
  • Upload this backup to a swift container to be able to get it from the TripleO web UI.

3. Creating a new OpenStack CLI command in python-tripleoclient (openstack undercloud backup).

The first action needed is to be able to create a new CLI command for the OpenStack client. In this case, we are going to implement the openstack undercloud backup command.

cd dev-docs
cd python-tripleoclient

Let’s list the files inside this folder:

[stack@undercloud python-tripleoclient]$ ls
AUTHORS           doc                            setup.py
babel.cfg         LICENSE                        test-requirements.txt
bindep.txt        zuul.d                         tools
build             README.rst                     tox.ini
ChangeLog         releasenotes                   tripleoclient
config-generator  requirements.txt               
CONTRIBUTING.rst  setup.cfg

Once inside the python-tripleoclient folder we need to check the following file:

setup.cfg: This file defines all the CLI commands for the Python TripleO client. Specifically, we will need at the end of this file our new command definition:

undercloud_backup = tripleoclient.v1.undercloud_backup:BackupUndercloud

This means that we have a new command defined as undercloud backup that will instantiate the BackupUndercloud class defined in the file tripleoclient/v1/undercloud_backup.py

For further details related to this class definition please go to the gerrit review.

Now, having our class defined we can call other methods to invoke Mistral in this way:

clients = self.app.client_manager

files_to_backup = ','.join(list(set(parsed_args.add_files_to_backup)))

workflow_input = {
    "sources_path": files_to_backup
output = undercloud_backup.prepare(clients, workflow_input)

So forth, we will call the undercloud_backup.prepare method defined in the file tripleoclient/workflows/undercloud_backup.py wich will call the Mistral workflow:

def prepare(clients, workflow_input):
    workflow_client = clients.workflow_engine
    tripleoclients = clients.tripleoclient
    with tripleoclients.messaging_websocket() as ws:
        execution = base.start_workflow(
        for payload in base.wait_for_messages(workflow_client, ws, execution):
            if 'message' in payload:
                return payload['message']

In this case, we will create a loop within the tripleoclient and wait until we receive a message from the Mistral workflow tripleo.undercloud_backup.v1.prepare_environment that indicates if the invoked workflow ended correctly.

4. Creating Mistral workflows for the new python-tripleoclient CLI command.

The next step is to define the tripleo.undercloud_backup.v1.prepare_environment Mistral workflow, all the Mistral workbooks, workflows and actions will be defined in the tripleo-common repository.

Let’s go inside tripleo-common

cd dev-docs
cd tripleo-common

And see it’s conent:

[stack@undercloud tripleo-common]$ ls
AUTHORS           doc                README.rst        test-requirements.txt
babel.cfg         HACKING.rst        releasenotes      tools
build             healthcheck        requirements.txt  tox.ini
ChangeLog         heat_docker_agent  scripts           tripleo_common
container-images  image-yaml         setup.cfg         undercloud_heat_plugins
contrib           LICENSE            setup.py          workbooks
CONTRIBUTING.rst  playbooks          sudoers           zuul.d

Again we need to check the following file:

setup.cfg: This file defines all the Mistral actions we can call. Specifically, we will need at the end of this file our new actions:

tripleo.undercloud.get_free_space = tripleo_common.actions.undercloud:GetFreeSpace
tripleo.undercloud.create_backup_dir = tripleo_common.actions.undercloud:CreateBackupDir
tripleo.undercloud.create_database_backup = tripleo_common.actions.undercloud:CreateDatabaseBackup
tripleo.undercloud.create_file_system_backup = tripleo_common.actions.undercloud:CreateFileSystemBackup
tripleo.undercloud.upload_backup_to_swift = tripleo_common.actions.undercloud:UploadUndercloudBackupToSwift

4.1. Action definition

Let’s take the first action to describe it’s definition, tripleo.undercloud.get_free_space = tripleo_common.actions.undercloud:GetFreeSpace

We have defined the action named as tripleo.undercloud.get_free_space which will instantiate the class GetFreeSpace defined in the file tripleo_common/actions/undercloud.py file.

If we open tripleo_common/actions/undercloud.py we can see the class definition as:

class GetFreeSpace(base.Action):
    """Get the Undercloud free space for the backup.

       The default path to check will be /tmp and the default minimum size will
       be 10240 MB (10GB).

    def __init__(self, min_space=10240):
        self.min_space = min_space

    def run(self, context):
        temp_path = tempfile.gettempdir()
        min_space = self.min_space
        while not os.path.isdir(temp_path):
            head, tail = os.path.split(temp_path)
            temp_path = head
        available_space = (
            (os.statvfs(temp_path).f_frsize * os.statvfs(temp_path).f_bavail) /
            (1024 * 1024))
        if (available_space < min_space):
            msg = "There is no enough space, avail. - %s MB" \
                  % str(available_space)
            return actions.Result(error={'msg': msg})
            msg = "There is enough space, avail. - %s MB" \
                  % str(available_space)
            return actions.Result(data={'msg': msg})

In this specific case this class will check if there is enough space to perform the backup. Later we will be able to inkove action as

mistral run-action tripleo.undercloud.get_free_space

or use it workbooks.

4.2. Workflow definition.

Once we have defined all our new actions, we need to orchestrate them in order to have a fully working Mistral workflow.

All tripleo-comon workbooks are defined in the workbooks folder.

In the next example we have a workbook definition with all actions inside it, in this case we put in the example the first workflow with all the tasks involved.

version: '2.0'
name: tripleo.undercloud_backup.v1
description: TripleO Undercloud backup workflows


    description: >
      This workflow will prepare the Undercloud to run the database backup
      - tripleo-common-managed
      - queue_name: tripleo
      # Action to know if there is enough available space
      # to run the Undercloud backup
        action: tripleo.undercloud.get_free_space
            status: <% task().result %>
            free_space: <% task().result %>
        on-success: send_message
        on-error: send_message
          status: FAILED
          message: <% task().result %>

      # Sending a message that the folder to create the backup was
      # created succesfully
        action: zaqar.queue_post
        retry: count=5 delay=1
          queue_name: <% $.queue_name %>
              type: tripleo.undercloud_backup.v1.launch
                status: <% $.status %>
                execution: <% execution() %>
                message: <% $.get('message', '') %>
          - fail: <% $.get('status') = "FAILED" %>

The workflow its self explanatory, the only not so clear part might be the last one as the workflow uses an action to send a message stating that the workflow ended correctly. Passing as the message the output of the previous task, in this case the result of the create_backup_dir.

5. Give support for new Mistral environment variables when installing the undercloud.

Sometimes is needed to use additional values inside a Mistral task. For example, if we need to create a dump of a database we might need another that the Mistral user credentials for authentication purposes.

Initially when the Undercloud is installed it’s created a Mistral environment called tripleo.undercloud-config. This environment variable will have all required configuration details that we can get from Mistral. This is defined in the instack-undercloud repository.

Let’s get into the repository and check the content of the file instack_undercloud/undercloud.py.

This file defines a set of methods to interact with the Undercloud, specifically the method called _create_mistral_config_environment allows to configure additional environment variables when installing the Undercloud.

For additional testing, you can use the Python snippet to call Mistral client from the Undercloud node available in gist.github.com.

6. Show how to test locally the changes in python-tripleoclient and tripleo-common.

If it’s needed a local test of a change in python-tripleoclient or tripleo-common, the following procedures allow to test it locally.

For a change in python-tripleoclient, assuming you already have downloaded the change you want to test, execute:

cd python-tripleoclient
sudo rm -Rf /usr/lib/python2.7/site-packages/tripleoclient*
sudo rm -Rf /usr/lib/python2.7/site-packages/python_tripleoclient*
sudo python setup.py clean --all install

For a change in tripleo-common, assuming you already have downloaded the change you want to test, execute:

cd tripleo-common
sudo rm -Rf /usr/lib/python2.7/site-packages/tripleo_common*
sudo python setup.py clean --all install
sudo cp /usr/share/tripleo-common/sudoers /etc/sudoers.d/tripleo-common
# this loads the actions via entrypoints
sudo mistral-db-manage --config-file /etc/mistral/mistral.conf populate
# make sure the new actions got loaded
mistral action-list | grep tripleo
for workbook in workbooks/*.yaml; do
    mistral workbook-create $workbook

for workbook in workbooks/*.yaml; do
    mistral workbook-update $workbook
sudo systemctl restart openstack-mistral-executor
sudo systemctl restart openstack-mistral-engine

If we want to execute a Mistral action or a Mistral workflow you can execute:

Examples about how to test Mistral actions independently:

mistral run-action tripleo.undercloud.get_free_space #Without parameters
mistral run-action tripleo.undercloud.get_free_space '{"path": "/etc/"}' # With parameters
mistral run-action tripleo.undercloud.create_file_system_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}'

Examples about how to test a Mistral workflow independently:

mistral execution-create tripleo.undercloud_backup.v1.prepare_environment # No parameters
mistral execution-create tripleo.undercloud_backup.v1.filesystem_backup '{"sources_path": "/tmp/asdf.txt,/tmp/asdf", "destination_path": "/tmp/"}' # With parameters

7. Give elevated privileges to specific Mistral actions that need to run with elevated privileges.

Sometimes its is not possible to execute some restricted actions from the Mistral user, for example, when creating the Undercloud backup we won’t be able to access the /home/stack/ folder to create a tarball of it. For this cases it’s possible to execute elevates actions from the Mistral user:

This is the content of the sudoers in the root of the tripleo-common repository at the time of the creatino of this guide.

Defaults!/usr/bin/run-validation !requiretty
Defaults:validations !requiretty
Defaults:mistral !requiretty
mistral ALL = (validations) NOPASSWD:SETENV: /usr/bin/run-validation
mistral ALL = NOPASSWD: /usr/bin/chown -h validations\: /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
        /usr/bin/chown validations\: /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
        !/usr/bin/chown /tmp/validations_identity_* *, !/usr/bin/chown /tmp/validations_identity_*..*
mistral ALL = NOPASSWD: /usr/bin/rm -f /tmp/validations_identity_[A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_][A-Za-z0-9_], \
        !/usr/bin/rm /tmp/validations_identity_* *, !/usr/bin/rm /tmp/validations_identity_*..*
mistral ALL = NOPASSWD: /bin/nova-manage cell_v2 discover_hosts *
mistral ALL = NOPASSWD: /usr/bin/tar --ignore-failed-read -C / -cf /tmp/undercloud-backup-*.tar *
mistral ALL = NOPASSWD: /usr/bin/chown mistral. /tmp/undercloud-backup-*/filesystem-*.tar
validations ALL = NOPASSWD: ALL

Here you can grant permissions for specific tasks in when executing Mistral workflows from tripleo-common

7. Debugging actions.

Let’s assume the action is written, added to setup.cfg but not appeared. Firstly, check if action was added by sudo mistral-db-manage populate. Run

mistral action-list -f value -c Name | grep -e '^tripleo.undercloud'

If you don’t see your actions check output of sudo mistral-db-manage populate as

sudo mistral-db-manage populate 2>&1| grep ERROR | less

The following output may indicate issues in code. Simply fix code.

2018-01-01:00:59.730 7218 ERROR stevedore.extension [-] Could not load 'tripleo.undercloud.get_free_space': unexpected indent (undercloud.py, line 40):   File "/usr/lib/python2.7/site-packages/tripleo_common/actions/undercloud.py", line 40

Execute single action, execute workflow from workbook to make sure it works as designed.

8. Unit tests

Writing Unit test is essential instrument of Software Developer. Unit tests are much faster that running Workflow itself. So, let’s write unit tests for written action. Let’s add tripleo_common/tests/actions/test_undercloud.py file with the following content in tripleo-comon repositiry.

import mock

from tripleo_common.actions import undercloud
from tripleo_common.tests import base

class GetFreeSpaceTest(base.TestCase):
    def setUp(self):
        super(GetFreeSpaceTest, self).setUp()
        self.temp_dir = "/tmp"

    def test_run_false(self, mock_statvfs, mock_isdir, mock_gettempdir):
        mock_gettempdir.return_value = self.temp_dir
        mock_isdir.return_value = True
        mock_statvfs.return_value = mock.MagicMock(
            spec_set=['f_frsize', 'f_bavail'],
            f_frsize=4096, f_bavail=1024)
        action = undercloud.GetFreeSpace()
        action_result = action.run(context={})
        self.assertEqual("There is no enough space, avail. - 4 MB",

    def test_run_true(self, mock_statvfs, mock_isdir, mock_gettempdir):
        mock_gettempdir.return_value = self.temp_dir
        mock_isdir.return_value = True
        mock_statvfs.return_value = mock.MagicMock(
            spec_set=['f_frsize', 'f_bavail'],
            f_frsize=4096, f_bavail=10240000)
        action = undercloud.GetFreeSpace()
        action_result = action.run(context={})
        self.assertEqual("There is enough space, avail. - 40000 MB",


tox -epy27

to see any unit tests errors.

  • Undercloud backups are an important step before runing an Upgrade.
  • Writing developer docs will help people to create and develope new features.

9. References

  • http://www.dougalmatthews.com/2016/Sep/21/debugging-mistral-in-tripleo/
  • http://blog.johnlikesopenstack.com/2017/06/accessing-mistral-environment-in-cli.html
  • http://hardysteven.blogspot.com.es/2017/03/developing-mistral-workflows-for-tripleo.html

by Carlos Camacho at December 18, 2017 12:00 AM

December 16, 2017

OpenStack Blog

Developer Mailing List Digest December 9-15th

Success Bot Says

  • mordred on #openstack-sdks: “Finished the shade transition to pure REST away from client libs” [0]
  • Tell us yours in OpenStack IRC channels using the command “#success <comment>”

Community Summaries

  • TC report 50 [0]
  • POST /api-sig/news [1]
  • Release countdown [2]
  • Technical Committee Status Update [3]
  • Resource Providers Update 45 [4]

Switching to Longer Development Cycles

Our self-imposed rhythm no longer matches our natural pace. Our elections, feature freezes feel like they’re around the corner, and due to that, we’re losing time from them, as multiple people have stated. The pace was designed around more people contributing full time to OpenStack, but lately, we have composite jobs, or were participating in multiple communities (which is great!).
It means:
  • One OpenStack coordinated release per year.
  • Maintain one stable branch per year.
  • Elect PTLs once a year.
  • One set of community goals per year.
  • One PTG per year.
Any project that still wants to release often can use the cycle-with-intermediary release model [0]. At a minimum, we’ll release once a year. Teams can choose to revive midcycle if they need to meet more than the one PTG each year. We’ll have more time for community-wide goals, so we will have more time to complete or even set more ambitious goals.
This doesn’t simplify upgrades. While upgrading every year is incrementally better than being forced to upgrade every 6 months. The real solution is better support for skipping releases.
It also doesn’t give us LTS. The cost of maintaining branches is not really due to the number of them in parallel, it’s due to the age of the oldest one. The real solution here is being discussed by the (still forming) LTS SIG. Extending stability periods is per-project at this point, and the proposal has yet to address a coordinated stability period.
We’re doing a year because of the way the events are organized. It’s suggested to start for the Rocky release and have the single PTG in 2018 in Dublin. The release team is opening the discussion to the community, and the TC will give a final decision.
Various cons were expressed, like this causing a rush to get in code at the end of a release to avoid having to wait an entire year. Projects are also forced to do compatibility support for versioned APIs, config files, etc since projects would be unable to drop compatibility in an intermediate release. Projects like Grenade will have to support the cases of some projects doing intermediate releases, and those doing yearly.
For the people that spend 20% of their time contributing to OpenStack, it has been voiced that it takes time to get a feature merged and the current cycle makes it impossible. This longer development cycle could help with allowing more time for those people. Various people expressed that we should be looking at the root cause of helping our part-time contributors, as the length of the cycle is unlikely the cause. People who can only contribute 20% of their time are also likely to deal with rebase conflicts, instead of focusing on their code.
Having a year could also impose having intermediate releases so specifications that can have multiple times a year to be approved. If that’s the idea, however, then it’s causing stress on the core team by having to match these precise schedules, in which the proposal was aiming to ease. It was a concern that this more solves the problem of giving more opportunities to people to get their spec/new feature considered.
This topic has also been brought up by various times in the past. Someone noted Daniel Berrange’s thread [1]

Zuul Dashboard Available

In addition to the Zuul dashboard [0] showing the “Status”, there are additional tabs added. “Jobs” page shows a list of all jobs in the system and their descriptions. The “Builds” page lists the most recent runs. You can query pipeline, job, and project.

Security SIG

Following previous mailing list discussions [0], the Security Project will be changing into a Special Interest Group (SIG). SIGs have shown to be a good match for the activity the group does around a topic or practice that spans around the community of developers, operators, and users. The group will continue to manage and care for the Security Guide, OSSNs, Bandit, Thread Reviews, Syntribos as well as encourage and incubate new security projects. The group will continue to work with the VMT and will keep a Sec-core group for launchpad that can work with the embargoes issues.

Cycle Highlights Reminder

As we get closer to Queens-3 and our final RCs, a reminder is given about the new ‘cycle-highlights’ that has been added to deliverable info. Background of why this was added, some PTLs were being pinged multiple times every release cycle by various people like journalists, product manager and other to compile the same information. To mitigate that, we have a way to capture highlights as part of the release. This will give basic information, but not as a complete marketing message.
This is done in the openstack/releases repository in the deliverables/queens/$PROJECT.yaml file formatted like so:
        – Introduced new service to use the unsed host to mine bitcoin.
We have three different places that document activities for three different audiences:
Commit messages: Developer documentation
Release notes: End-user and deployer documentation
Cycle highlights: Journalists, product manager, and others.

Community Goals For Rocky

Some questions to ask ourselves: What common challenges do we have, and who is willing to drive that community-wide goal (aka champion).
A champion is someone who drives a goal but doesn’t commit to writing code necessarily. The champion will communicate with project PTLs about the goal, and make the liaison if needed.
The list of ideas for community-wide goals is collected on this etherpad [0]. Propose some ideas now!

by Mike Perez at December 16, 2017 07:48 PM

December 15, 2017

Major Hayden

Ensuring keepalived starts after the network is ready

Wait on pavementAfter a recent OpenStack-Ansible (OSA) deployment on CentOS, I found that keepalived was not starting properly at boot time:

Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: VRRP_Instance(internal) Unknown interface !
systemd[1]: Started LVS and VRRP High Availability Monitor.
Keepalived_vrrp[801]: Stopped
Keepalived[799]: Keepalived_vrrp exited with permanent error CONFIG. Terminating

OSA deployments have a management bridge for traffic between containers. These containers run the OpenStack APIs and other support services. By default, this bridge is called br-mgmt.

The keepalived daemon is starting before NetworkManager can bring up the br-mgmt bridge and that is causing keepalived to fail. We need a way to tell systemd to wait on the network before bringing up keepalived.

Waiting on NetworkManager

There is a special systemd target, network-online.target, that is not reached until all networking is properly configured. NetworkManager comes with a handy service called NetworkManager-wait-online.service that must be complete before the network-online target can be reached:

# rpm -ql NetworkManager | grep network-online

Start by ensuring that the NetworkManager-wait-online service starts at boot time:

systemctl enable NetworkManager-wait-online.service

Using network-online.target

Next, we tell the keepalived service to wait on network-online.target. Bring up an editor for overriding the keepalived.service unit:

systemctl edit keepalived.service

Once the editor appears, add the following text:


Save the file in the editor and reboot the server. The keepalived service should come up successfully after NetworkManager signals that all of the network devices are online.

Learn more by reading the upstream NetworkTarget documentation.

The post Ensuring keepalived starts after the network is ready appeared first on major.io.

by Major Hayden at December 15, 2017 09:18 PM

OpenStack Superuser

Pinterest’s journey from virtual machines to containers

AUSTIN — Around the globe, Pinterest’s 200 million active users pop digital pins on images that inspire them. Another billion scroll through those boards looking for ideas. All that pinning requires a lot of computing power and until now the company, which launched in 2010, has done it all on Amazon Web Services and virtual machines.

Recently, though, they made a plan to stick with a new strategy: containers. The San Francisco-based unicorn, one of the newest members of the Cloud Native Computing Foundation, came to KubeCon + CloudNativeCon North America to talk about this change of tack.

Michael Benedict at the press briefing.

During a press briefing, Micheal Benedict who manages data and cloud infrastructure at Pinterest, spoke about the challenges the engineering team faced and what’s up next.

“We were on the cloud since day one,” he told journalists. The scale of the seven-year-old company is impressive: tens of thousands of instances, processing tens and hundreds of petabytes of data. Everything was on AWS — they have never had private clouds or bare metal in the underpinnings, he clarifies. While “VMS have served their purpose,” he says, they’re transitioning in the next 18 months to containers with Kubernetes. Some 75 percent of their stateless workloads have already migrated. “It’s a huge thing for us.”

They didn’t arrive at Kubernetes and the CNCF by chance. Benedict admits there were a “whole host of challenges” around exploring the different systems currently available (including Mesos and a number of hosted systems) that always seemed to leave more questions than they answered. “Our concerns were around ‘how much do you want to commit to a provider?’” and not just for ops but also for future tie-ins. “Given our scale, it’s really important.”

The four main drivers for the switch were: speeding things up for devs, reliability, infrastructure governance and efficiency. With the previous set-up, launching a service involved multiple Amazon Machine Images, required end users to perform Puppet authoring and testing (and the unpredictability around those Puppet runs) and disparate process management (monit, upstart, supervisor). With containers, there will be a single AMI, no Puppet authoring, unified process management and immutable infrastructure with deterministic behavior.

“We want to thing more about consolidating the process from idea to production without thinking about underlying infrastructure,” he says.

Benedict and Lida Li of the cloud platform at Pinterest also gave a public talk at KubeCon. You can check out the slides here and we’ll update this post with that video when it’s available.

The post Pinterest’s journey from virtual machines to containers appeared first on OpenStack Superuser.

by Nicole Martinelli at December 15, 2017 02:46 PM


Blog Round-up

It's time for another round-up of the great content that's circulating our community. But before we jump in, if you know of an OpenStack or RDO-focused blog that isn't featured here, be sure to leave a comment below and we'll add it to the list.

ICYMI, here's what has sparked the community's attention this month, from Ansible to TripleO, emoji-rendering, and more.

TripleO and Ansible (Part 2) by slagle

In my last post, I covered some of the details about using Ansible to deploy with TripleO. If you haven’t read that yet, I suggest starting there: http://blog-slagle.rhcloud.com/?p=355

Read more at http://blog-slagle.rhcloud.com/?p=369

TripleO and Ansible deployment (Part 1) by slagle

In the Queens release of TripleO, you’ll be able to use Ansible to apply the software deployment and configuration of an Overcloud.

Read more at http://blog-slagle.rhcloud.com/?p=355

An Introduction to Fernet tokens in Red Hat OpenStack Platform by Ken Savich, Senior OpenStack Solution Architect

Thank you for joining me to talk about Fernet tokens. In this first of three posts on Fernet tokens, I’d like to go over the definition of OpenStack tokens, the different types and why Fernet tokens should matter to you. This series will conclude with some awesome examples of how to use Red Hat Ansible to manage your Fernet token keys in production.

Read more at https://redhatstackblog.redhat.com/2017/12/07/in-introduction-to-fernet-tokens-in-red-hat-openstack-platform/

Full coverage of libvirt XML schemas achieved in libvirt-go-xml by Daniel Berrange

In recent times I have been aggressively working to expand the coverage of libvirt XML schemas in the libvirt-go-xml project. Today this work has finally come to a conclusion, when I achieved what I believe to be effectively 100% coverage of all of the libvirt XML schemas. More on this later, but first some background on Go and XML…

Read more at https://www.berrange.com/posts/2017/12/07/full-coverage-of-libvirt-xml-schemas-achieved-in-libvirt-go-xml/

Full colour emojis in virtual machine names in Fedora 27 by Daniel Berrange

Quite by chance today I discovered that Fedora 27 can display full colour glyphs for unicode characters that correspond to emojis, when the terminal displaying my mutt mail reader displayed someone’s name with a full colour glyph showing stars:

Read more at https://www.berrange.com/posts/2017/12/01/full-colour-emojis-in-virtual-machine-names-in-fedora-27/

Booting baremetal from a Cinder Volume in TripleO by higginsd

Up until recently in TripleO booting, from a cinder volume was confined to virtual instances, but now thanks to some recent work in ironic, baremetal instances can also be booted backed by a cinder volume.

Read more at http://goodsquishy.com/2017/11/booting-baremetal-from-a-cinder-volume-in-tripleo/

by Mary Thengvall at December 15, 2017 06:20 AM

December 14, 2017

OpenStack Superuser

Travel grants support global community to attend OpenStack Summit

Key contributors from 14 countries attended the recent OpenStack Summit in Sydney. Find out how the Travel Support Program could be your ticket to Vancouver.

Some Summit participants have to travel great distances to attend, but may not always have the resources or support to do so. The OpenStack Foundation helps participants reach their attendance goal via the Travel Support Program.

Winners of travel support grants at the Summit in Sydney.

The Foundation supported 30 people from 14 different countries to come participate in OpenStack Summit in Sydney.

The Travel Support Program is based on the premise of Open Design a commitment to an open design process that welcomes the public, including users, developers and upstream projects. This year the program also included individual supporters who chose to donate frequent flyer miles or funds to assist the program’s efforts.

The Summit is a great opportunity for participants to network and have important discussions regarding OpenStack contributions. Core contributor to the Manila Project, Rodrigo Barbieri made his first trip to Australia and thanks to the program, was able to have key meetings with fellow operators.

Amy Marrich also made her first trip to Australia. Through the support of the program, she was able to give back to the community, mentoring and instructing at the Upstream Institute and at a number of Women of OpenStack sessions. In addition, Marrich participated in the Forum, joining OpenStack-Ansible conversations and attended User Committee sessions, allowing the beginnings of reactivating the Diversity Working Group.

Tony Breeds, OpenStack software engineer, talked highly of the Summit and project team gatherings (PTGs), stating they are “invaluable if you need to be productive in OpenStack and the travel support program is a fantastic way of assisting community to attend.”

The deadline to apply for Travel Support to the Vancouver Summit is March 22, 2018. You can submit your application here.

Read these on how to apply for Travel support.

https://www.flickr.com/photos/billward/2858932924/ // CC BY

The post Travel grants support global community to attend OpenStack Summit appeared first on OpenStack Superuser.

by Sonia Ramza at December 14, 2017 12:48 PM

Red Hat Stack

Red Hat OpenStack Platform 12 Is Here!

We are happy to announce that Red Hat OpenStack Platform 12 is now Generally Available (GA).

This is Red Hat OpenStack Platform’s 10th release and is based on the upstream OpenStack release, Pike.

Red Hat OpenStack Platform 12 is focused on the operational aspects to deploying OpenStack. OpenStack has established itself as a solid technology choice and with this release, we are working hard to further improve the usability aspects and bring OpenStack and operators into harmony.

Logotype_RH_OpenStackPlatform_RGB_Black (1)

With operationalization in mind, let’s take a quick look at some the biggest and most exciting features now available.


As containers are changing and improving IT operations it only stands to reason that OpenStack operators can also benefit from this important and useful technology concept. In Red Hat OpenStack Platform we have begun the work of containerizing the control plane. This includes some of the main services that run OpenStack, like Nova and Glance, as well as supporting technologies, such as Red Hat Ceph Storage. All these services can be deployed as containerized applications via Red Hat OpenStack Platform’s lifecycle and deployment tool, director.

Photo by frank mckenna on Unsplash

Bringing a containerized control plane to OpenStack is important. Through it we can immediately enhance, among other things, stability and security features through isolation. By design, OpenStack services often have complex, overlapping library dependencies that must be accounted for in every upgrade, rollback, and change. For example, if Glance needs a security patch that affects a library shared by Nova, time must be spent to ensure Nova can survive the change; or even more frustratingly, Nova may need to be updated itself. This makes the change effort and resulting change window and impact, much more challenging. Simply put, it’s an operational headache.

However, when we isolate those dependencies into a container we are able to work with services with much more granularity and separation. An urgent upgrade to Glance can be done alongside Nova without affecting it in any way. With this granularity, operators can more easily quantify and test the changes helping to get them to production more quickly.

We are working closely with our vendors, partners, and customers to move to this containerized approach in a way that is minimally disruptive. Upgrading from a non-containerized control plane to one with most services containerized is fully managed by Red Hat OpenStack Platform director. Indeed, when upgrading from Red Hat OpenStack Platform 11 to Red Hat OpenStack Platform 12 the entire move to containerized services is handled “under the hood” by director. With just a few simple preparatory steps director delivers the biggest change to OpenStack in years direct to your running deployment in an almost invisible, simple to run, upgrade. It’s really cool!

Red Hat Ansible.

Like containers, it’s pretty much impossible to work in operations and not be aware of, or more likely be actively using, Red Hat Ansible. Red Hat Ansible is known to be easier to use for customising and debugging; most operators are more comfortable with it, and it generally provides an overall nicer experience through a straightforward and easy to read format.


Of course, we at Red Hat are excited to include Ansible as a member of our own family. With Red Hat Ansible we are actively integrating this important technology into more and more of our products.

In Red Hat OpenStack Platform 12, Red Hat Ansible takes center stage.

But first, let’s be clear, we have not dropped Heat; there are very real requirements around backward compatibility and operator familiarity that are delivered with the Heat template model.

But we don’t have to compromise because of this requirement. With Ansible we are offering operator and developer access points independent of the Heat templates. We use the same composable services architecture as we had before; the Heat-level flexibility still works the same, we just translate to Ansible under the hood.

Simplistically speaking, before Ansible, our deployments were mostly managed by Heat templates driving Puppet. Now, we use Heat to drive Ansible by default, and then Ansible drives Puppet and other deployment activities as needed. And with the addition of containerized services, we also have positioned Ansible as a key component of the entire container deployment. By adding a thin layer of Ansible, operators can now interact with a deployment in ways they could not previously.

For instance, take the new openstack overcloud config download command. This command allows an operator to generate all the Ansible playbooks being used for a deployment into a local directory for review. And these aren’t mere interpretations of Heat actions, these are the actual, dynamically generated playbooks being run during the deployment. Combine this with Ansible’s cool dynamic inventory feature, which allows an operator to maintain their Ansible inventory file based on a real-time infrastructure query, and you get an incredibly powerful troubleshooting entry point.

Check out this short (1:50) video showing Red Hat Ansible and this new exciting command and concept:

Network composability.

Another major new addition for operators is the extension of the composability concept into networks.

As a reminder, when we speak about composability we are talking about enabling operators to create detailed solutions by giving them basic, simple, defined components from which they can build for their own unique, complex topologies.

With composable networks, operators are no longer only limited to using the predefined networks provided by director. Instead, they can now create additional networks to suit their specific needs. For instance, they might create a network just for NFS filer traffic, or a dedicated SSH network for security reasons.

Photo by Radek Grzybowski on Unsplash

And as expected, composable networks work with composable roles. Operators can create custom roles and apply multiple, custom networks to them as required. The combinations lead to an incredibly powerful way to build complex enterprise network topologies, including an on-ramp to the popular L3 spine-leaf topology.

And to make it even easier to put together we have added automation in director that verifies that resources and Heat templates for each composable network are automatically generated for all roles. Fewer templates to edit can mean less time to deployment!

Telco speed.

Telcos will be excited to know we are now delivering production ready virtualized fast data path technologies. This release includes Open vSwitch 2.7 and the Data Plane Development Kit (DPDK) 16.11 along with improvements to Neutron and Nova allowing for robust virtualized deployments that include support for large MTU sizing (i.e. jumbo frames) and multiple queues per interface. OVS+DPDK is now a viable option alongside SR-IOV and PCI passthrough in offering more choice for fast data in Infrastructure-as-a-Service (IaaS) solutions.

Operators will be pleased to see that these new features can be more easily deployed thanks to new capabilities within Ironic, which store environmental parameters during introspection. These values are then available to the overcloud deployment providing an accurate view of hardware for ideal tuning. Indeed, operators can further reduce the complexity around tuning NFV deployments by allowing director to use the collected values to dynamically derive the correct parameters resulting in truly dynamic, optimized tuning.

Serious about security.


Helping operators, and the companies they work for, focus on delivering business value instead of worrying about their infrastructure is core to Red Hat’s thinking. And one way we make sure everyone sleeps better at night with OpenStack is through a dedicated focus on security.

Starting with Red Hat OpenStack Platform 12 we have more internal services using encryption than in any previous release. This is an important step for OpenStack as a community to help increase adoption in enterprise datacenters, and we are proud to be squarely at the center of that effort. For instance, in this release even more services now feature internal TLS encryption.

Let’s be realistic, though, focusing on security extends beyond just technical implementation. Starting with Red Hat OpenStack Platform 12 we are also releasing a comprehensive security guide, which provides best practices as well as conceptual information on how to make an OpenStack cloud more secure. Our security stance is firmly rooted in meeting global standards from top international agencies such as FedRAMP (USA), ETSI (Europe), and ANSSI (France). With this guide, we are excited to share these efforts with the broader community.

Do you even test?

How many times has someone asked an operations person this question? Too many! “Of course we test,” they will say. And with Red Hat OpenStack Platform 12 we’ve decided to make sure the world knows we do, too.

Through the concept of Distributed Continuous Integration (DCI), we place remote agents on site with customers, partners, and vendors that continuously build our releases at all different stages on all different architectures. By engaging outside resources we are not limited by internal resource restrictions; instead, we gain access to hardware and architecture that could never be tested in any one company’s QA department. With DCI we can fully test our releases to see how they work under an ever-increasing set of environments. We are currently partnered with major industry vendors for this program and are very excited about how it helps us make the entire OpenStack ecosystem better for our customers.

So, do we even test? Oh, you bet we do!

Feel the love!

Photo by grafxart photo on Unsplash

And this is just a small piece of the latest Red Hat OpenStack Platform 12 release. Whether you are looking to try out a new cloud, or thinking about an upgrade, this release brings a level of operational maturity that will really impress!

Now that OpenStack has proven itself an excellent choice for IaaS, it can focus on making itself a loveable one.

Let Red Hat OpenStack Platform 12 reignite the romance between you and your cloud!

Red Hat OpenStack Platform 12 is designated as a “Standard” release with a one-year support window. Click here for more details on the release lifecycle for Red Hat OpenStack Platform.

Find out more about this release at the Red Hat OpenStack Platform Product page. Or visit our vast online documentation.

And if you’re ready to get started now, check out the free 60-day evaluation available on the Red Hat portal.

Looking for even more? Contact your local Red Hat office today.


by August Simonelli, Technical Marketing Manager, Cloud at December 14, 2017 01:49 AM


Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


Last updated:
January 23, 2018 01:52 AM
All times are UTC.

Powered by: