May 23, 2018

OpenStack Superuser

OpenStack in production at CERN: Mysteries, challenges, setbacks and achievements

VANCOUVER — The European Organization for Nuclear Research, known as CERN, is on a mission to unravel the mysteries of the universe.

Driven by the demand to support research at the world’s largest particle collider, the CERN IT department decided in 2012 to build an agile infrastructure centered around an OpenStack- based private cloud. CERN provides several facilities and resources to scientists all around the world, including compute and storage resources, for their fundamental research.

Internally, CERN’s data center provides roughly 20 to 25 percent of these resources, says Arne Wiebalck, computing engineer. The analysis is done in a worldwide distributed grid with 170 data centers on about 800,000 cores  and a total 900 petabytes of data on disk on tape. Some 90 percent of the CERN resources are delivered on top of OpenStack.

Running OpenStack in production has also lead to some mysteries, challenges setbacks and achievements, much like those faced by its researchers Wiebalck says.

One of these head-scratchers was the baffling case of innocent instances being killed randomly, VMs “just disappearing.” The culprit was a random independent service that was supposed to remember which PIDs to kill — but instead just killed randomly here and there. “It took us quite a while to identify the murderer,” he adds.

Then there were hosts shutting down right after boot, deletions grinding Cinder to a halt and the bare metal database losing entries.

Challenges included scaling a booming service — more than 300,000 cores over 9,000 hypervisors spanning over 4,000 projects — continuous upgrades, staff turnover and bridging the physical-to-virtual performance gap.

Fortunately the CERN team can count a vast number of achievements, too. In addition to the numbers cited above, they include deployment across two data centers, migration of thousands of VMS due, two million requests handled by a Magnum Kubernetes cluster and many contributions upstream.

Catch his entire talk below or on the OSF’s YouTube channel.

The post OpenStack in production at CERN: Mysteries, challenges, setbacks and achievements appeared first on Superuser.

by Nicole Martinelli at May 23, 2018 06:02 PM

Gorka Eguileor

Don’t rewrite your driver. 80 storage drivers for containers rolled into one!

Do you work with containers but your storage doesn’t support your Container Orchestration system? Have you or your company already developed an Openstack/Cinder storage driver and now you have to do it again for containers? Are you having trouble deciding how to balance your engineering force between storage driver development in OpenStack, Containers, Ansible, etc? […]

by geguileo at May 23, 2018 12:06 PM

Ansible Storage Role: automating your storage solutions

Were you in the middle of writing your Ansible playbooks to automate your software provisioning, configuration, and application deployment when you realized you had to manage your storage as well? And it turns out that each of your storage solutions has a completely different Ansible module. Now you have to figure out how each module […]

by geguileo at May 23, 2018 12:05 PM

Cinderlib: Every storage driver on a single Python library

Wouldn’t it be great if we could manage any storage array using a single Python library that provided the right storage management abstraction? Imagine writing your own Python storage management software and not caring about the backend or the connection. What if your code was backend and connection agnostic so the same method was used […]

by geguileo at May 23, 2018 12:03 PM

What's new in OpenStack?

The OpenStack global community is gathering together in Vancouver, British Columbia this week to collaborate, learn, and build the future of open source cloud computing.

by Jason Baker at May 23, 2018 07:00 AM

May 22, 2018

Red Hat Stack

“Ultimate Private Cloud” Demo, Under The Hood!

At the recent Red Hat Summit in San Francisco, and more recently the OpenStack Summit in Vancouver, the OpenStack engineering team worked on some interesting demos for the keynote talks.

I’ve been directly involved with the deployment of Red Hat OpenShift Platform on bare metal using the Red Hat OpenStack Platform director deployment/management tool, integrated with openshift-ansible. I’ll give some details of this demo, the upstream TripleO features related to this work, and insight around the potential use-cases.

TripleO & Ansible, a Powerful Combination!

For anyone that’s used Red Hat OpenStack Platform director (or the upstream TripleO project, upon which it is based), you’re familiar with the model of deploying a management node (“undercloud” in TripleO terminology), then deploying and managing your OpenStack nodes on bare metal.  However, TripleO also provides a very flexible and powerful combination of planning, deployment, and day-2 operations features. For instance, director allows us to manage and provision bare metal nodes, then deploy virtually any application onto those nodes via Ansible!

The “undercloud” management node makes use of several existing OpenStack services, including Ironic for discovery/introspection and provisioning of bare metal nodes, Heat, a declarative orchestration tool, and Mistral, a workflow engine.  It also provides a convenient UI, showcased in the demo, along with flexible CLI interfaces and standard OpenStack ReST APIs for automation.

As described in the demo, director has many useful features for managing your hardware inventory – you can either register or auto-discover your nodes, then do introspection (with optional benchmarking tests) to discover the hardware characteristics via the OpenStack ironic-inspector service.  Nodes can then be matched to a particular profile either manually or via rules implemented through the OpenStack Mistral workflow API. You are then ready to deploy an Operating System image onto the nodes using the OpenStack Ironic “bare metal-as-a-service” API.

When deciding what will be deployed onto your nodes, director has the concept of a “deployment plan,” which combines specifying which nodes/profiles will be used and which configuration will be applied, known as “roles” in TripleO terminology.

This is a pretty flexible system enabling a high degree of operator customization and extension through custom roles where needed, as well as supporting network isolation and custom networks (isolated networks for different types of traffic), declarative configuration of  network interfaces, and much more!

Deploying Red Hat OpenShift Container Platform on bare metal

What was new in the Summit demo was deploying OpenShift alongside OpenStack, both on bare metal, and both managed by Red Hat OpenStack Platform  director. Over the last few releases we’ve made good progress on ansible integration in TripleO, including enabling integration with “external” installers.  We’ve made use of that capability here to deploy OpenShift via TripleO, combining the powerful bare-metal management capabilities of TripleO with existing openshift-ansible management of configuration.

Integration between Red Hat OpenStack Platform and Red Hat OpenShift Container Platform

Something we didn’t have time to get into in great detail during the demo was the potential for integration between OpenStack and OpenShift – if you have an existing Red Hat OpenStack Platform deployment you can choose to deploy OpenShift with persistent volumes backed by Cinder (the OpenStack block storage service). And for networking integration, the Kuryr project, combined with OVN from OpenvSwitch, enables the sharing of a common overlay network between both platforms, without the overhead of double encapsulation.

This makes it easy to add OpenShift managed containers to your infrastructure, while almost seamlessly integrating them with VM workloads running on OpenStack. You can also take advantage of existing OpenStack capacity and vendor support while using the container management capabilities of OpenShift.

Container-native virtualization

After we deployed OpenShift we saw some exciting demos focussed on workloads running on OpenShift, including a preview of the new container native virtualization (CNV) feature. CNV uses the upstream KubeVirt project to run Virtual Machine (VM) workloads directly on OpenShift.

Unlike the OpenShift and OpenStack combination described above, here OpenShift manages the VM workloads, providing an easier way to  transition your VM workloads where no existing virtualization solution is in place. The bare-metal deployment capabilities outlined earlier are particularly relevant here, as you may want to run OpenShift worker nodes that host VMs on bare metal for improved performance. As the demo has shown,  the combination of director and openshift-ansible makes deploying, managing, and running OpenShift and OpenStack easier to achieve!


by Steven Hardy, Senior Principal Software Engineer at May 22, 2018 11:56 PM

OpenStack Superuser

Say hello to Kata Containers 1.0

Today the Kata Containers project launched the 1.0 release, successfully merging Intel® Clear Containers and Hyper runV technologies to provide a single runtime for virtualized containers, delivering the speed of containers with the security of virtual machines (VMs).

Get started with Kata Containers on Kata’s GitHub.

Kata Containers addresses the security drawbacks of traditional containers

Kata Containers bridges the gap between traditional VM security and the lightweight benefits of traditional Linux* containers. Traditional containers share the same underlying Linux kernel. There are known exploits that allow a malicious user to “escape” a container and gain access to the kernel and the shared containers. In a multi-tenant environment where workloads are running with unknown levels of trust, significant efforts are required to ensure a secure system.

Traditional containers use Linux control groups, referred to as cgroups, for managing and allocating resources and namespaces to provide container isolation. Further security isolation is provided by dropping Linux capabilities, using read-only mount points, mandatory access controls (MAC) security measures like those in SELinux and AppArmor*, dropping syscalls using SECCOMP, etc. It’s difficult, if not impossible, to effectively apply these security policies to complex applications.

As a result, most often containers end up being provisioned in their own full VM, negating the performance promises of containers. Protecting against security breaches in these environments is one of the drivers behind the Kata Containers project.

Traditional containers with a shared kernel

Kata Containers provides container isolation by using hardware virtualization. In the case of Docker*, kata-runtime provides VM isolation at the container level. For Kubernetes, VM isolation is provided at the pod level. (In the rest of this post, the term container/pod refers to for container for Docker and pod for Kubernetes.)

For Kata Containers, each container/pod is booted as a lightweight VM with its own unique kernel instance. Since each container/pod is now running with its own VM, they no longer gain access to the host kernel and get the full security benefits of a VM. This simplifies the security policies you need to put in place to protect the host kernel against container exploits.

Kata Containers use lightweight virtual machines for isolation

Kata Containers also makes it possible for container-as-a-service (CaaS) providers to offer containers running on bare metal. Kata Containers allows mutually untrusting tenants to use the same cluster thanks to the hardware isolation between containers. This assumes there are also network security policies in place to provide network isolation between tenants within the cluster.

How Kata Containers fits into the container ecosystem

A container runtime is the component that handles the life cycle of a container, implementing basic concepts such as creating, starting, stopping and removing a container workload. The Open Container Initiative (OCI) created a runtime specification that details the API for an OCI-compatible runtime. runC is the canonical OCI runtime solution, which is described as a “CLI tool for spawning and running containers according to the OCI specification.” runC uses Linux cgroups and namespaces to provide isolation.

Kata Containers is a member of OCI and the Kata Containers runtime — known as kata-runtime — will be OCI-compatible.

Another place the term runtime is used is in the Container Runtime Interface (CRI) provided in Kubernetes. CRI runtimes are at a higher level of abstraction and should not be confused with an OCI-compatible runtime.

Interacting with Docker Engine

For Docker, kata-runtime is just one more OCI-compatible runtime option that can be used.

In a default configuration, if you install and run Docker, the Docker engine will:

  1. Create a container configuration.
  2. Pass this configuration to runC.
  3. runC will create a container based on the configuration and workload provided from the Docker engine.

If you install Kata Container’s runtime, kata-runtime, you can configure Docker to be aware of both container runtimes, giving users the choice of which to use on a per-container granularity. kata-runtime complements runC and enhances the solution provided by Docker. (See Docker’s runtime documentation for more details.) When using kata-runtime, each Docker container will run within its own lightweight VM.

Kata Containers is compatible with Docker

Kata Containers and Kubernetes

Kubernetes 1.5 introduced the CRI (Container Runtime Interface), which enables a variety of container runtimes to be plugged in easily. Prior to this, Kubernetes only made use of the default Docker image repository and its default OCI-compatible runtime, runC. Since “runtime” continues to be an overloaded term, in this discussion we’ll call the CRI runtime a CRI shim and use “runtime” to describe an OCI-compatible runtime.

Since the introduction of CRI, a number of CRI shims have been introduced, including cri-containerd, CRI-o, dockershim, and frakti. Some of these call into an OCI-based runtime, while others are a monolithic solution. A high-level overview of how these implement a solution via CRI is shown below. Note that dockershim currently only supports runC, not kata-runtime.

Kata Containers is compatible with Kubernetes

Kata Containers provides two interfaces for CRI shims to manage hardware virtualization based Kubernetes pods:

  1. An OCI-compatible runtime, kata-runtime. This is currently usable with the CRI solutions, cri-containerd and CRI-O.
  2. A hardware virtualization runtime library API for CRI shims to consume and provide a more CRI-native implementation. Frakti is an example CRI shim here.

While the work of defining the concept of a secure sandbox continues at the Kubernetes level, some of the CRI implementations already support the concept of multiple runtimes running on a single node. For example, CRI-O supports the concept of a trusted and an untrusted sandbox. Based on pod annotations and default CRI-O configuration, you can run a mix of VM and namespace-based pods. This article goes into depth on how this currently works with CRI-O.

Kata Containers with kubelet

VM isolation is provided at the pod level for kata-runtime. Containers running inside a Kata Containers pod are isolated and managed via namespaces and cgroups, similar to what is done by runC.

How to try out and get involved with Kata Containers

Kata Containers 1.0 is now ready! Kata Containers is a fully open-source project––check out Kata Containers on GitHub and join the channels below to find out how you can contribute.


Slack: link: ; invite:

IRC: #kata-dev on Freenode

Mailing list:

The post Say hello to Kata Containers 1.0 appeared first on Superuser.

by Anne Bertucio at May 22, 2018 04:05 PM

StackHPC Team Blog

HPC Container Orchestration with Bare Metal Magnum

Our project with the Square Kilometre Array includes a requirement for high-performance containerised runtime environments. We have been building a system with bare metal infrastructure, multiple physical networks, high-performance data services and optimal integrations between OpenStack and container orchestration engines (such as Kubernetes and Docker Swarm).

We have previously documented our upgrade of OpenStack deployment from Ocata to Pike. This upgrade impacted Docker Swarm and Kubernetes: provisioning of both COEs in a bare metal environment failed after the upgrade. We resolved the issues with Docker Swarm but left Kubernetes for patching over a major release upgrade.

A fix was announced with Queens release, along with swarm-mode support for Docker. This strengthened the case to upgrade Magnum to Queens on an underlying Openstack Pike. The design ethos of Kolla-Ansible and Kayobe, using containerisation to avoid the nightmares of dependency interlock, made the targeted upgrade of Magnum a relatively smooth ride.

Fixing Magnum deployment by upgrading from Pike to Queens

We use Kayobe to manage the configuration of our Kolla deployment. Changing the version of a single OpenStack service (in this case, Magnum) is as simple as setting the version of a specific Docker container tag, as follows:

  • Prepare the Kaybobe environment (assuming it is already installed):

    cd src/kayobe-config
    git checkout BRANCH-NAME
    git pull
    source kayobe-env
    cd ../kayobe
    source ../../venv/kayobe/bin/activate
    export KAYOBE_VAULT_PASSWORD=**secret**
  • Add magnum_tag: to kayobe-config/etc/kayobe/kolla/globals.yml.

  • Finally, build and deploy the new version of Magnum to the control plane. To ensure that other OpenStack services are not affected during the deployment, we use --kolla-tags and --kolla-skip-tags:

    kayobe overcloud container image build magnum --push
    -e kolla_source_version=stable/queens
    -e kolla_openstack_release=
    -e kolla_source_url=
    kayobe overcloud container image pull --kolla-tags magnum
    --kolla-skip-tags common
    kayobe overcloud service upgrade --kolla-tags magnum
    --kolla-skip-tags common

That said, the upgrade came with a few unforeseen issues:

  • We discovered that Kolla Ansible, a tool that Kayobe uses to deploy Magnum containers, assumes that host machines running Kubernetes are able to communicate with Keystone on an internal endpoint, not an option in our case since the internal endpoints were internal to the control plane, which does not include tenant networks and instances (which could be baremetal nodes or VMs). Since this is generally an invalid assumption, a patch was pushed upstream which has been quickly approved in the code review process. After applying this patch, it is necessary to reconfigure default configuration templates for Heat, made possible by a single Kayobe command:

    kayobe overcloud service reconfigure --kolla-tags heat
    --kolla-skip-tags common
  • Docker community edition (v17.03.0-ce onwards) uses cgroupfs as the native.cgroupdriver. However, Magnum assumes that this is systemd and does not explicitly demand this be the case. As a result, deployment fails. This was addressed in this pull request.

  • By default, Magnum's behaviour is to assign a floating IP to each server in a container infrastructure cluster. This means that all the traffic flows through the control plane (when accessing the cluster from an external location; internal traffic is direct). Disabling floating IP appeared to have no effect which we filed as a bug on launchpad. Patch to fix Magnum to correctly handle disabling of floating IP in swarm mode is currently under way.

  • Patch kayobe-config to update magnum_tag to as well as point magnum_conductor_footer and magnum_api_footer to a patched Magnum Queens fork stackhpc/queens on our Github account.

Fedora Atomic 27 image for containers

A recently recently released Fedora Atomic 27 image (download link comes packaged with baremetal and Mellanox drivers therefore it is no longer necessary to build custom image using diskimage-builder to incorporate these drivers. However, it was necessary to make a few one-off manual changes to the image which we achieved by making changes to the image through virsh console:

  • First, boot into the image using cloud-init credentials defined within init.iso (built from these instructions):

    sudo virt-install --name fa27 --ram 2048 --vcpus 2 --disk
    --os-type linux --os-variant fedora25 --network bridge=virbr0
    --cdrom /var/lib/libvirt/images/init.iso --noautoconsole
    sudo virsh console fa27
  • The images are shipped with Docker v1.13.1 which is 12 releases behind the current stable release, Docker v18.03.1-ce (note that versioning scheme changed after v1.13.1 to v17.03.0-ce). To obtain up-to-date features required by our customers, we upgraded this to the latest release.

    sudo su
    cd /etc/yum.repos.d/
    curl -O
    rpm-ostree override remove docker docker-common cockpit-docker
    rpm-ostree install docker-ce -r
  • Fedora Atomic 27 comes installed with packages for a scalable network file system called GlusterFS. However, one of our customer requirements was to support RDMA capability for GlusterFS in order to maximise IOPs for data intensive tasks compared to IP-over-Infiniband. The package was available on rpm-ostree repository as glusterfs-rdma. It was installed and enabled as follows:

    # installing glusterfs
    sudo su
    rpm-ostree upgrade
    rpm-ostree install glusterfs-rdma fio
    systemctl enable rdma
    rpm-ostree install perftest infiniband-diags
  • Write cloud-init script that runs once to resize root volume partition due to the fact that root volume is mounted as LVM and conventional cloud-init script to grow this partition fails leading to containers deployed inside swarm cluster to quickly fill up. The following script placed under /etc/cloud/cloud.cfg.d/99_growpart.cfg did the trick in our case which generalises to different types of root block devices:

    # resize volume
      - lsblk
      - PART=$(pvs | awk '$2 == "atomicos" { print $1 }')
      - echo $PART
      - /usr/bin/growpart ${PART:: -1} ${PART: -1}
      - pvresize ${PART}
      - lvresize -r -l 100%FREE /dev/atomicos/root
      - lsblk
  • Cleaning up the image makes it lighter but prevents users rolling back the installation, an action we do not anticipate users to need to perform. Removing cloud-init config allows it to run again and removes authorisation details. Cleaning service logs gives the image a fresh start.

    # undeploy old images
    sudo atomic host status
    # current deployment <0/1> should have a * next to it
    sudo ostree admin undeploy <0/1>
    sudo rpm-ostree cleanup -r
    # image cleanup
    sudo rm -rf /var/log/journal/*
    sudo rm -rf /var/log/audit/*
    sudo rm -rf /var/lib/cloud/*
    sudo rm /var/log/cloud-init*.log
    sudo rm -rf /etc/sysconfig/network-scripts/ifcfg-*
    # auth cleanup
    sudo rm ~/.ssh/authorized_keys
    sudo passwd -d fedora
    # virsh cleanup
    # press Ctrl+Shift+] to exit virsh console
    sudo virsh shutdown fa27
    sudo virsh undefine fa27

Ansible roles for managing container infrastructure

There are official Ansible modules for various OpenStack projects like Nova, Heat, Keystone, etc. However, Magnum currently does not have one, especially those that concern creating, updating and managing container infrastructure inventory. Magnum currently lacks certain useful features possible indirectly through Nova API like attaching multiple network interfaces to each node in the cluster that it creates. The ability to generate and reuse an existing cluster inventory is further necessitated by a specific requirement of this project to mount GlusterFS volumes to each node in the container infrastructure cluster.

In order to lay the foundation for performing preliminary data consumption tests for the Square Kilometre Array's (SKA) Performance Prototype Platform (P3), we needed to attach each node in the container infrastructure cluster to multiple high speed network interfaces:

  • 10G Ethernet
  • 25G High Throughput Ethernet
  • 100G Infiniband

We have submitted a blueprint to support multiple networks using Magnum API since Nova already allows multiple network interfaces to be attached. In the meantime, we wrote an Ansible role to drive Magnum and generate an Ansible inventory from the cluster deployment. Using this inventory, further playbooks apply our enhancements to the deployment.

The role allows us to declare specification of container infrastructure required including a variable that is a list of networks to attach to cluster nodes. A bespoke ansible module os_container_infra creates/updates/deletes cluster as specified using python-magnumclient. Another module called os_stack_facts then gathers facts about container infrastructure using python-heatclient allowing us to generate an inventory of the cluster. Finally, a module called os_server_interface uses python-novaclient to attach each node in the container infrastructure cluster to additional network interfaces declared in the specifications.

We make use of the recently announced openstacksdk Python module for talking to OpenStack which was conceived with an aim to assimilate shade and os_client_config projects which have been performing similar functions under separate umbrellas. We enjoyed the experience of using openstacksdk API that is largely consistent with the parent projects. Ansible plans to eventually transition to openstacksdk but they do not currently have specific plans to support plugin libraries like python-magnumclient, python-heatclient and python-novaclient which provide wider coverage in terms of the range of interaction with their API compared to openstacksdk, which only offers a set of common-denominators across various OpenStack cloud platforms.

With Magnum playing an increasing more important role in the OpenStack ecosystem by allowing users to create and manage container orchestration engines like Kubernetes, we expect this role will make lives easier for those of us who regularly use Ansible to manage complex and large scale HPC infrastructure.

by Bharat Kunwar at May 22, 2018 09:00 AM

May 21, 2018

OpenStack Superuser

And the Superuser Award goes to…

VANCOUVER — The OpenStack community and Superuser editorial advisors have weighed in on of all the stellar finalists and chosen the Ontario Institute of Cancer Research (OICR) as the winner of the Vancouver 2018 Superuser Award, sponsored by Zenko.

Previous winners Tencent TStack presented the award on stage by  during Monday’s keynotes.

“OpenStack made it possible for OICR to build the Cancer Genome Collaboratory, a cloud that enables research on the world’s largest and most comprehensive cancer genome dataset…Researchers can run complex analysis across a large repository of cancer genome sequences. Instead of spending weeks to months downloading hundreds of terabytes of data from a central repository before computations can begin, researchers can upload their analytic software into the Collaboratory cloud, run it and download the computed results in a secure fashion.”

The Collaboratory has 2,600 cores and 18 terabytes of RAM, as well as 7.3 petabytes of storage managed by Ceph. More than 40 research labs and 95 researchers across four continents are using Collaboratory to access 670 terabytes of protected cancer genome data. The Collaboratory has contributed to the research underlying 43 peer-reviewed papers, with an additional 50 papers from the PanCancer Analysis of Whole Genomes project currently in preparation or review.

The Collaboratory team also developed two open-source applications:

They have also developed and open-sourced metadata and storage software used to provide granular and time-limited access to the protected data sets only to approved researchers. In addition, they also created Dockstore which allows researchers to package and share bio-infomatics tools and workflows.

The winners competed against impressive finalists: City Network, VEXXHOST and T-Mobile. Previous winners include AT&T, CERN, China Mobile, Comcast, NTT Group, Paddy Power Betfair and UKCloud.

The Superuser Awards launched in 2014 to recognize organizations that have used OpenStack to meaningfully improve their business while contributing back to the community.

Stay tuned for the next cycle of nominations as the Superuser Awards head to the OpenStack Berlin Summit!

The post And the Superuser Award goes to… appeared first on Superuser.

by Superuser at May 21, 2018 06:31 PM


Admit it. VNF Certification is Marketing.

Generic VNF "certification" programs sound great and are terrific for marketing. Unfortunately they don't add value for customers.

by Boris Renski at May 21, 2018 04:00 PM

OpenStack Superuser

Zuul case study: GoDaddy

Internet domain registrar and web hosting company GoDaddy has more than 17 million customers and over 7,000 employees worldwide. A former Superuser Awards finalist, they’re longtime proponents of OpenStack and active members of the OSF.

Here Superuser talks to Clint Byrum, a senior cloud software engineer at GoDaddy, about how they’re currently using Zuul,  the importance of community and the perils of neophobia.

If you’re at the Vancouver Summit, catch him moderating an collaborative session on CI/CD and all the sessions on Zuul here.

The days of annual, monthly or even weekly releases are long gone. How is CI/CD defining new ways to develop and manage software with open infrastructure?

We have a lot of free/open software and a lot of physical infrastructure to manage underneath it. Every day, hundreds of GoDaddy developers and tens of thousands of GoDaddy customers rely on our systems to stay usable while we add features, fix bugs and generally deal with the problem of change. Without a strong investment in CI/CD, we’d just be pulling manual levers, sitting on already-complete work and generally never make any progress.

Are there specific Zuul features that drew you to it?

The community that built Zuul was probably the first feature. We have a very large OpenStack installation at GoDaddy and a few of us have been using Zuul since its very early days as OpenStack developers. This experience has always been pleasant and the folks in charge of it are extremely responsive and welcoming.

Beyond that, cross-repo gating allows us to keep our concerns separate and build CI/CD pipelines out of a combination of upstream free/open software, such as OpenStack, in concert with our more custom GoDaddy-specific integrations. Having built-in multi-node testing is a massively important feature for us. We are trying to test deployment that must orchestrate several pieces of infrastructure, such as databases, message queues, and hypervisor software like libvirt/KVM. It just doesn’t work to do this all on one VM, and containers lack the ability to accurately simulate our production bare metal environments. It doesn’t hurt that Zuul makes use of Ansible and our specific team is very well versed in Ansible, making it easy to get up to speed quickly writing test cases in Zuul, vs. Jenkins, which requires people to learn Groovy or the pipeline DSL to write complicated test cases.

Finally, the fact that it can scale your test bandwidth as big as your available OpenStack is appealing to us, since we have quite a bit of OpenStack capacity available to us.

How are you using Zuul at GoDaddy now?

Zuul is still in somewhat of a pilot for us, primarily in use for testing our OpenStack deployment automation and also for managing changes to some of our Kubernetes deployment automation. We also use it to test and deploy itself. Hopefully being self-aware won’t turn it into SkyNet.

As for nuts and bolts, we have a small-ish team of about eight developers using it every day, with a few other devs from other teams contributing as well. The Zuul control plane runs on a single 16GB VM, and we currently run between 40 and 75 8GB VMs for tests across two regions of our internal OpenStack cloud. We use the GitHub driver to interface with our GitHub Enterprise installation, which allows teams to partially adopt Zuul while still using the code review workflow they’re used to.

What benefits have you seen so far?

We’ve replaced a lot of slow, brittle Jenkins jobs with highly parallelized, more realistic Zuul test jobs. We’ve also been able to accelerate development on some new projects by setting up rapid proof-of-concepts in PR’s to GitHub. Some of these go nowhere, and some of them eventually become the deployment automation that goes into production.

In addition, since all of Zuul’s job and project configuration is done in YAML files stored in Git repositories, we’re able to treat our testing infrastructure as code and reap all the benefits of that without having to take notes and repeat manual steps.

What challenges have you overcome?

Zuul v3.0 was only just released with a stable interface. We were early adopters and thus had to deal with tracking the extremely aggressive pace of development in Zuul. There’s also substantial inertia around Jenkins and it’s not always 100 percent obvious why Zuul is a better fit in some cases when Jenkins is just so popular. However, once shown the power of it, we’ve had no complaints from members of the team about Zuul, so this was largely just a case of neophobia.

What can you tell us about your future plans?

Now that 3.0 is released, it’s a much easier conversation to have with internal dev teams about potentially adopting it. I plan to personally evangelize Zuul inside GoDaddy and outside GoDaddy in the open-source communities where I’m active.

We also want to help steer the future of Zuul and make sure we can use it in conjunction with Kubernetes more smoothly. We have a lot of institutional knowledge of Kubernetes, but Zuul just doesn’t have much to offer when all you need is a container.

What are you hoping the Zuul community continues to focus on or deliver?

In addition to container integration, I’d really like to see a focus more on the CD part. The CI part is incredibly strong in Zuul, but there are some real challenges to deploying things at scale with Zuul. I’d also like us to see about integrations with some other tools, like Spinnaker, in this area, where Zuul might be handy for the dev-> deploy hand-off.

Anything else you’d like to add?

So many great people worked very hard on Zuul, it’s really an honor to be able to speak about what they’ve done. I just want to say thank you to the OpenStack community and the Zuul community for making our lives a little better each day.

Superuser wants to hear what you’re up to with open infrastructure, get in touch:

Cover photo: GoDaddy Tempe office, courtesy Go Daddy.

The post Zuul case study: GoDaddy appeared first on Superuser.

by Nicole Martinelli at May 21, 2018 02:17 PM

How OpenLab is using Zuul

Zuul is a program that drives continuous integration, delivery and deployment systems with a focus on project gating and interrelated projects. In this series of interviews, we’re talking to users about the problems it’s solving and the challenges they’ve encountered.

Superuser talked to Melvin Hillsman, chair of the OpenStack User Committee who works at Huawei on the open source project OpenLab, about how he’s using Zuul and how you can get involved.

OpenLab is a community-led program to test and improve support for software development kits (SDKs)—as well as platforms like Kubernetes, Terraform, Cloud Foundry and more—in hybrid and multi-cloud OpenStack environments.

How has CI/CD defined new ways to develop and manage software within open infrastructure?

I could be focusing too much on the word “defined” but here are my thoughts. For my sanity, I use the abbreviations CoINT (integration), CoDEL (delivery), and CoDEP (deployment) with CoARC (architecture). I look at these as philosophies that have suggested habits for reduced friction and improved quality of the code developed, released and used as software and/or applications.

Clearly defined new ways of software development and management within open infrastructure can be extracted from the habits they suggest, but these habits are still not exactly common practice. The other issue is that practitioners who implement these habits haven’t established standards about expected outputs or inputs, so it’s difficult to gauge the impact on open infrastructure.

We’re getting there, yet one might argue that open infrastructure is still being defined as well — though we may think we have pinned it down.

To be honest, it’s hard to stick with these habits when you’re starting from nothing and even more so for established small/medium/large companies — even if you’re trying to adhere to them for one software or application in a single business unit.

So tl;dr, we can still use some clear definitions!  However, the habits of upstream first, check code in early and often, automate everything, test-driven development, strive for 100 percent test coverage, work in small chunks, test against production-like environments are what folks strive for. It’s evident that confusion is common — there are still articles being written today explaining the differences between CoINT, CoDEL, and CoDEP…

How are you currently using Zuul at OpenLab?

Zuul is currently being used as an offering within OpenLab for projects/applications/tools that need CI gating and/or automation around testing. With the companion tool Nodepool we’re able to keep OpenStack VMs available, speeding up the process for developers of testing code changes.

What benefits have you seen so far?

Primarily the benefit we see is the ease of integration with other projects. Along those lines, Zuul is beneficial in general for open source users who utilize Ansible for other problem domains because it allows for a common language/workflow to be utilized both within the CI/CD system and outside it, opening the door for other tools to be more easily integrated.

What challenges have you overcome?

With Ansible acting as the component Zuul uses to facilitate job definitions it has become easier to onboard. We haven’t yet explored using Ansible Galaxy roles, but we expect to be able to overcome the challenge of having centralized job definitions available outside of the jobs defined within our Github repositories.

What can you tell us about your future plans?

In particular with CI/CD and automating infrastructure/architecture, we’re working with an effort called openci which is about cross-community/foundation/organization integration, deployment, delivery and automation.

Collaboration in this space appears to be still very new as it has been quite varied regarding maturity of tooling, availability of artifacts, common vocabulary and standardized approaches and workflows. With OpenLab, we have the opportunity to explore many aspects of these efforts, helping those who are deploying production environments.

What help could you use from community members (pain points, testing etc.)?

Regarding Zuul – and, as a byproduct in some instances, Nodepool – it would be great to see multi-tenancy land, as well as some additional connection support (Gitlab, Kubernetes Prow, etc.), pushing local repository/code to be included with job run(s) via zuul cli and more robust documentation, to name a few things. We – and others – get great support from the Zuul team via #zuul on freenode. They’re very attentive and extremely knowledgeable — not only about Zuul but also the problem domain it sits in.


The post How OpenLab is using Zuul appeared first on Superuser.

by Nicole Martinelli at May 21, 2018 02:15 PM

May 20, 2018

Aija Jauntēva

Outreachy: How I got started with OpenStack development

The first week has passed and in this post I will cover what I did to get my first contribution during application phase and some follow up reading afterwards. Here I will focus on general tools used in OpenStack development workflow. In next post I will write about the project I'm working on.

It was easy to start developing for OpenStack projects and almost everything I needed to know is well documented in one long page[1]. The page contains some links to follow for more detailed information. One of such pages that I find useful is about commit messages[2] that not only writes about messages themselves, but about splitting commits in smaller patches where split is reasonable and somewhat isolated rather than creating one large patch which is harder to review for everyone involved. Also commit messages should be self-contained and hold enough information to understand what's being done and why without accessing other systems like issue/bug tracker. This also says that commit message bodies should be wrapped at 72 characters. The main page[1] only mentions commit message headers that should be 50 characters.

Then there are specific OpenStack code style guidelines for Python[3] that also references the general Python style guide, PEP 8[4]. Here I'm relying on tox to catch any violations.

To write some code, I cloned the necessary projects, branched, and made some changes. There were some tools missing from my development environment that I installed as I went by, and I did not encounter any setup related issues when running projects.

Once the changes are done, the next step is to submit it for code review.

Gerrit is used for code review, it used to be written in Python, but was rewritten in Java. Wikipedia says[5] that it was done to make it easier to run on different Linux systems, but it does not give any reference to source and a quick search online did not help me to find any historical discussions, but this is not so important here, was just curious. More about Gerrit it is written at [6]. Gerrit distinguishes between core reviewers and regular reviewers, something that I have not seen in other tools where coders enforce it manually. In such cases, it is not like they try to bypass core reviewer requirement (as system does not enforce it), but it makes hard to identify open code reviews that are missing core reviewers, so it is nice that Gerrit has this built-in.

There is a handy tool git-review[7] to make work with Gerrit easier - to submit code review just run git review instead of following instructions in Gerrit's quick intro[6].

Once code review submitted Zuul[8] gets involved. Don't have to interact with this much, but Zuul will be the one who will tell if there are tests failing and PEP 8 guidelines violated. This should have been checked locally before submitting code review, but there are always more integrations to run and other issues that might not show up in local environment. As there are many patches submitted within the same time frame, Zuul will queue them and test queued patches together according to their place in queue to catch any conflicts between patches before they get merged to master. Then when code review is done and usually it requires 2 core code reviewers to approve, the Zuul will merge the changes to master.

That's all for now, if I encounter something new or something I misunderstood in relation to workflow and tools, will share it in further posts.

by ajya at May 20, 2018 03:00 PM

Jesse Pretorius

Ansible 2.5 and OpenStack

I spent some time today figuring out how the new mechanism for inventory plugins for Ansible 2.5 works, and also spent quite a bit of time scratching my head at an error I encountered when executing a test playbook against Rackspace Public Cloud.

Inventory Plugin for OpenStack

The inventory plugin system was introduced into Ansible 2.4. The old dynamic inventory mechanism still works, so until now I’ve never tried to use the new system. The documentation is a little confusing and I found the inventory samples by Alan Rominger very useful to get me on the right track.

The minimal requirement to make the OpenStack inventory plugin work is to put a file named openstack.yml or openstack.yaml in your inventory folder.

For example, I have the following in ansible.cfg:

inventory = inventory

And the contents of inventory/openstack.yml is:

plugin: openstack
fail_on_errors: yes

This assumes that you already have a clouds.yaml file in your ~/.config folder, and have the shade library installed into the same python environment as Ansible.

That’s it! Thanks to the default auto inventory plugin being enabled, it will find the openstack.yml file and enable the OpenStack inventory plugin.

Using the Ansible os_server module against Rackspace Public Cloud

When trying to use the os_server module against Rackspace Public Cloud I got this error:

TASK [Create/delete OpenStack Instance] ************************************************************************************************************************
fatal: [localhost]: FAILED! => changed=false
  extra_data: null
  msg: 'Unavailable feature: security groups'

This is odd, because the same playbook works just fine in Ansible 2.3. I spent quite a bit of time trying to hunt down what changed to precipitate this problem but was unsuccessful. I did manage to find a way to make it work though - just provide an empty list for the security_groups argument.

- name: Create/delete OpenStack Instance
    name: "setup-instance-test1"
    flavor: "general1-1"
    state: "present"
    cloud: "rackspace"
    region_name: "IAD"
    image: "Ubuntu 18.04 LTS (Bionic Beaver) (PVHVM)"
    key_name: superduper
    security_groups: []
    config_drive: yes
      build_config: core
    wait: yes
    timeout: 900

The default value the module has is [ 'default' ] and the module therefore tries to implement this when building the instance. However it seems that this capability is not available for Rackspace Public Cloud, and it fails the task.

I hope that figuring this out will help someone else save some time.

May 20, 2018 02:31 PM

May 18, 2018

OpenStack @ NetApp

NetApp, Cisco and Red Hat announce OpenStack on FlexPod SolidFire!

I am happy to share with you the publishing of the second document of the FlexPod SolidFire with Red Hat OpenStack Platform 10 – Cisco Validated Design (CVD) solution. Paired with its partner Design Guide, this CVD enables an enterprise-grade OpenStack deployment, in an accelerated fashion, with little risk. Deployment Guide (NEW) – is a ... Read more

The post NetApp, Cisco and Red Hat announce OpenStack on FlexPod SolidFire! appeared first on thePub.

by Amit Borulkar at May 18, 2018 06:49 PM

OpenStack Superuser

Inside public cloud: Must-see sessions at the Vancouver Summit

Join the people building and operating open infrastructure at the OpenStack Summit Vancouver in May.  The Summit schedule features over 25 events — presentations, workshops, fishbowls, lightning talks — covering public cloud.

Here we’re highlighting some of those sessions; be sure to catch sessions on the OpenStack Global Passport Program, a collaborative effort between OpenStack public cloud providers that allow users to experience the freedom, performance and interoperability of open source infrastructure.

Check out all the sessions, workshops and lightning talks focusing on public cloud here.

Stretching your application from OpenStack into public cloud

Multiple clouds are a must in today’s IT world to prevent vendor lock-in, provide redundancy, a migration path and other benefits. Developers and IT admins demand that applications and the micro-services that comprise them prove capable of running in multiple environments. A service mesh, instantiated by Istio, provides a new paradigm for connecting, managing and securing micro-services. With services running in multiple clouds, service mesh has to be established across the clouds. In this presentation, Cisco’s John Joyce and Timothy Swanson will show how an application can be stretched between two disparate cloud environments–public cloud and an OpenStack based cloud. Details here.

OpenStack for AWS architects: Similarities, differences and bridging the gap

Architects often use only use one tool and to a carpenter with only a hammer, everything looks like a nail. Adding OpenStack to the tool belt opens up a whole new world of possibilities and solutions for on-premise cloud challenges. In this beginner-level presentation, Ben Silverman of OnX Enterprise Solutions will talk about how to bridge the two platforms by making direct connections from OpenStack projects to AWS products and discuss the commonalities and differences in product features and architecture. He’ll also dive into the ways both platforms solve complex business challenges and how each platform is uniquely suited to solving certain use cases. Of course, no AWS/OpenStack presentation would be complete without discussing hybrid and multi-cloud scenarios and how to use some of the products we discuss in a multi-modal architecture. Details here.

Closing automation and DNS security gaps in today’s dynamic world

Even though security is top of mind for every telco and organization across the world, security holes for DNS are still a major concern. Over 95% of tested organizations allowed data exfiltration via DNS tunnels because existing tools allow open DNS communications.Two critical DNS gaps occur in traditional NFV and containers deployments – lack of IP address automation and DNS-based security risks. In this demo, Infoblox’s Philip Miller and Matt Gowarty will highlight how carrier-grade DNS and IP address management (IPAM) closes these gaps. The demo will highlight how data can easily be stolen via DNS, and more importantly, how to prevent this risk. Details here.

Is the public cloud really eating OpenStack’s lunch?

This session from Red Hat’s Margaret Dawson will walk through what’s real and what’s hype in the public cloud world and show how customers are using OpenStack to solve real-world problems and power next-generation applications. Attendees will learn how to align workloads to the right infrastructure and how to develop a cloud strategy. This is vendor-agnostic session will include customer case studies and global research results. Details here.

Enabling cloud-native applications with Application Credentials in Keystone

Keystone has long been the barrier blocking cloud-native applications from truly automation-compatible support. Applications, deployment tools and OpenStack services have all required dedicated keystone users whose passwords are committed to configuration files. This talk from Suse’s Colleen Murphy introduces Application Credentials, a new feature in the Queens release of keystone that enables applications and automation to authenticate with keystone without requiring a dedicated keystone user, allows graceful rotation of credentials with minimal downtime, and encourages restrictive permissions delegation to applications. Details here.

Encrypt your volumes with Barbican

If you’re interested in volume encryption to secure your environment, this advanced session will show how to design and implement a volume encryption service based on the Barbican project. Using lessons learned from OnRamp’s VPC, engineer Duncan Wannamaker will review the design journey from soup to nuts, including outlining requirements, choosing from the available open source platforms, developing the architecture and implementing the volume encryption. OnRamp built and deployed a volume encryption service for their virtual private cloud, allowing users to encrypt their volumes. Each volume the VPC users provision is encrypted with a unique key, ensuring that their data is not readable by other tenants in the cloud. Wannaker will offer up a demo of how to create an encryption key, integrate Cinder with Barbican, and create and attach encrypted volumes. Details here.

See you at the OSF Summit in Vancouver, May 21-24, 2018! Register here.

The post Inside public cloud: Must-see sessions at the Vancouver Summit appeared first on Superuser.

by Nicole Martinelli at May 18, 2018 04:16 PM

SWITCH Cloud Blog

Openstack Horizon runs on Kubernetes in production at SWITCH

In April we upgraded the SWITCHengines OpenStack Horizon dashboard to the OpenStack Pike version. But this upgrade was a little bit special, it was more than an Horizon upgrade from Newton to Pike.

Our Horizon deployment is now hosted on a Kubernetes cluster. The cluster is deployed using the playbook k8s-on-openstack that we actively develop. We have been testing this Kubernetes deployment for a while, but it is only when you have to deploy an application on top of it in production that you really learn and you fix real problems.

Horizon is a good application to start learning Kubernetes, because it is completely stateless and it does not require any persistent storage. It is just a GUI to the OpenStack API. The user logs in with his credentials, and Horizon will get a token and will start making API calls with the user’s credentials.

Running Horizon in a single Kubernetes pod for a demo takes probably 5 minutes, but deploying for production usage is far more complex. We needed to address the following issues:

  • Horizontally scale the number of pods, keeping a central memcached or redis cache
  • Allow both IPv4 and IPv6 access to
  • Define the Load Balancing architecture
  • Implement a persistent logging system

If you want to run to the solution of all these problems, you can have a look at the project SWITCH-openstack-horizon-k8s-deployment where we have published all the Dockerfiles and the Kubernetes descriptors to recreate our deployment.

Scale Horizontally

Horizon performs much faster when it accesses a memory cache, it is the recommended way to deploy in production. We decided to go for Redis cache.

Creating a Redis service in our namespace with the name redis-master we are able to use the special environment variable ${REDIS_MASTER_SERVICE_HOST} when booting the Horizon container, to make sure all the instances point to the same cache server.

This is a good example of how you combine two services together in a Kubernetes namespace. We can horizontally scale the Horizon pods, but the Horizon deployment is independent from the Redis deployment.

IPv4 and IPv6

We always publish our services on IPv6. In our previous Kubernetes demos we used the OpenStack LBaaS to expose services to the outside world. Unfortunately in the Newton version of OpenStack, the LBaaS lacks proper IPv6 integration. To publish a production service on Kubernetes, we suggest to use an ingress controller. There are several kinds available, but we used the standard Nginx ingress controller. The key idea is that we have a K8s node with an interface exposed to the public Internet where a privileged Docker container is running with –net=host. The container runs Nginx that can bind to IPv6 and IPv4 on the node, but of course it can also reach any other pod on the cluster network.

Define the Load Balancing architecture

I already wrote above that if you need IPv6, you should not use the Openstack LBaaSv2. However I am going to explain why I would not use that kind of load balancer even for IPv4.

The first picture shows the network diagram of a LBaaSv2 deployment. The LoadBalancer is implemented as a network namespace on the network node, called qlbaas-<uuid>, in which a HAProxy process is running. This is a L4 LoadBalancer. The bad thing of this architecture is that when an instance boots, the default gateway configured via DHCP will be the IP address of the neutron router. When we expose a service with the floating IP configured on the outer interface of the LBaaS, in order to force the traffic to follow a symmetric return path, the Load Balancer must perform a DNAT and SNAT operation. This means that the IP packets hitting the Pod have completely lost the information about the source IP address of the original client. Because it is a pure L4 load balancer, we don’t have the possibility to carry this lost information on in a HTTP header. This prevents the operator from building any useful logging system, because once the traffic arrives at the pod, the information about the client is filtered out.

In the next picture we have a look on how the Nginx ingress works. In this case the external traffic is received on a public floating IP that is configured on the virtual machine running the ingress pod, in this case on the master. We terminate the TLS connection at the nginx-ingress. This is necessary because the ingress also has to perform a SNAT and DNAT but it adds to the HTTP requests the X-Forwarded-For header that we use to populate our log files. We could not add the header if we were just moving encrypted packets around.

Another advantage of this solution is that it uses just a normal instance to implement the ingress, this means that you can use in a totally independent way from the version of OpenStack you are running on.

In the future you might be able to use the newer OpenStack Octavia Load Balancer, but at the moment I did not investigate that. All I know is that the solution is really similar, but you will have an OpenStack service VM running an Nginx instance.

Implement a persistent logging system

Pods are short lived and distributed over different VMs that are also ephemeral. To collect the logs, we run docker with the log-driver journald. Once this is set up, all the docker containers running on the host will send their logging output to journald. We then collect this information with journalbeat to send the data to our elastic search cluster. This part is not yet released into our public playbook because is not very portable. If you don’t have a ready-to-use ELK cluster, you would have no benefit from running journalbeat.


It is now almost a month that we have been running in production, and we found the system to be robust and stable. We had no complaints from our users, so we can say that the migration was seamless for our users. We have learned a lot from this experience.

In the next blog post we will describe how we implemented the metrics monitoring, to observe how much memory and CPU cores each pod is consuming. Make sure you keep an eye on our blog for updates.

by Saverio Proto at May 18, 2018 03:54 PM

Mastering CI/CD at OpenDev

After launching in 2017, the OpenDev Conference is now an annual event. At the inaugural event last September, the conference focus was on edge computing. This year's event, taking place May 22-23, will be focused on Continuous Integration/Continuous Deployment (CI/CD) and will be co-located with the OpenStack Summit in Vancouver.

by pleia2 at May 18, 2018 07:00 AM


What is Open Networking?

Aptira Open Networking

Ask someone, or search, for a definition of Open Networking and you’ll get a variety of answers.  It’s one of those “obvious” terms that everyone understands; but when you dig down, everyone understands differently. 

The most narrow definition would focus on Software Defined Networks and the solutions enabled by OpenFlow. But is that all there is? 

Other definitions would focus on interoperability and the ability to substitute one component for another in your platform.  But anyone who has worked in solutions integration for even a short while understands that the reality of interoperability and component substitution falls a long way short of the aspiration. 

Here at Aptira, we have a much more holistic and inclusive definition: Open Networking is simply how you build and manage highly performant, high-quality and high-value computing systems in the 21st Century. 

At Aptira we believe Open Networking to be a set of concepts, practices and tools for the realisation and management of highly performant, high-quality and high-value computing systems.  At the current time, Open Networking is enabled by: 

  • Open Source Software 
  • Open Network Devices 
  • Open Compute Hardware 
  • Software Defined Networks 
  • Network Function Virtualisation 
  • Cloud Computing 
  • Automation 
  • Agile Methods & Processes 

This is a much broader definition of Open Networking than others, but it’s the only definition that doesn’t create more solution silos or bend the solution outcome to a buzzword trend or one or more competing technologies.  A holistic and inclusive definition of Open Networking is necessary and produces the best results. 

These technologies represent the transition from hardware-based, specific-function and proprietary network and computing components to a combination of more generic and simpler hardware and a significant migration of functionality to software.  To riff off Marc Andressen’s famous comment, you could say that “Software is eating the network”. 

Aptira Open Networking Hexagon Grid

The progressive disaggregation of componentry and the increased role of software results in more product options, driven by the entry of more vendors and vendor business models (think Open Source projects) which enables rapid product evolution and innovation.  But it also requires a holistic approach to solution components that have long been treated differently. 

Leaving all the buzzwords aside, what does this mean? It means that for a given function, more component options are available that are more interchangeable, and have a faster innovation and enhancement cycle. 

The result is solutions that are more ‘open’ in the sense of component compatibility, inter-operability, expandability and extendibility. 

When solution integrators and customers have a broadest range of options for sourcing, partnering and in-house development and operation at all stages of the solution’s life, we end up with more cost-effective solutions, delivered faster and more able to evolve. 

This all sounds very cool, and the trend is accelerating: there’s no going back to the old concept of compute and networks.  But there are many considerations for adopters: many potential benefits but also many risks. 

What are the implications of these characteristics of Open Networking?  What new skills are needed by vendors, customers and systems integrators?  What potential pitfalls exist? What is the future? 

As expert practitioners in the field of Open Networking, including ONAP, SDN/NFV and more, Aptira can help guide you through these considerations to a desired outcome. 

And we will expand on these topics and more in future articles.  Stay tuned… 

The post What is Open Networking? appeared first on Aptira.

by Adam Russell at May 18, 2018 02:06 AM

Carlos Camacho

Testing Undercloud backup and restore using Ansible

Testing the Undercloud backup and restore

It is possible to test how the Undercloud backup and restore should be performed using Ansible.

The following Ansible playbooks will show how can be used Ansible to test the backups execution in a test environment.

Creating the Ansible playbooks to run the tasks

Create a yaml file called uc-backup.yaml with the following content:

- hosts: localhost
  - name: Remove any previously created UC backups
    shell: |
      source ~/stackrc
      openstack container delete undercloud-backups --recursive
    ignore_errors: True
  - name: Create UC backup
    shell: |
      source ~/stackrc
      openstack undercloud backup --add-path /etc/ --add-path /root/

Create a yaml file called uc-backup-download.yaml with the following content:

- hosts: localhost
  - name: Print destroy warning.
      msg: |
        We are about to destroy the UC, as we are not
        moving outside the UC the backup tarball, we will
        download it and unzip it in a temporary folder to
        recover the UC using those files.
      msg: ""
  - name: Make sure the temp folder used for the restore does not exist
    become: true
      path: "/var/tmp/test_bk_down"
      state: absent
  - name: Create temp folder to unzip the backup
    become: true
      path: "/var/tmp/test_bk_down"
      state: directory
      owner: "stack"
      group: "stack"
      mode: "0775"
      recurse: "yes"
  - name: Download the UC backup to a temporary folder (After breaking the UC we won't be able to get it back)
    shell: |
      source ~/stackrc
      cd /var/tmp/test_bk_down
      openstack container save undercloud-backups
  - name: Unzip the backup
    become: true
    shell: |
      cd /var/tmp/test_bk_down
      tar -xvf UC-backup-*.tar
      gunzip *.gz
      tar -xvf filesystem-*.tar
  - name: Make sure stack user can get the backup files
    become: true
      path: "/var/tmp/test_bk_down"
      state: directory
      owner: "stack"
      group: "stack"
      mode: "0775"
      recurse: "yes"

Create a yaml file called uc-destroy.yaml with the following content:

- hosts: localhost
  - name: Remove mariadb
    become: true
    yum: pkg=
      - mariadb
      - mariadb-server
  - name: Remove files
    become: true
      path: ""
      state: absent
      - /root/.my.cnf
      - /var/lib/mysql

Create a yaml file called uc-restore.yaml with the following content:

- hosts: localhost
    - name: Install mariadb
      become: true
      yum: pkg=
        - mariadb
        - mariadb-server
    - name: Restart MariaDB
      become: true
      service: name=mariadb state=restarted
    - name: Restore the backup DB
      shell: cat /var/tmp/test_bk_down/all-databases-*.sql | sudo mysql
    - name: Restart MariaDB to perms to refresh
      become: true
      service: name=mariadb state=restarted
    - name: Register root password
      become: true
      shell: cat /var/tmp/test_bk_down/root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'"
      register: oldpass
    - name: Clean root password from MariaDB to reinstall the UC
      shell: |
        mysqladmin -u root -p password ''
    - name: Clean users
      become: true
      mysql_user: name="" host_all="yes" state="absent"
        - ceilometer
        - glance
        - heat
        - ironic
        - keystone
        - neutron
        - nova
        - mistral
        - zaqar
    - name: Reinstall the undercloud
      shell: |
        openstack undercloud install

Running the Undercloud backup and restore tasks

To test the UC backup and restore procedure, run from the UC after creating the Ansible playbooks:

  # This playbook will create the UC backup
  ansible-playbook uc-backup.yaml
  # This playbook will download the UC backup to be used in the restore
  ansible-playbook uc-backup-download.yaml
  # This playbook will destroy the UC (remove DB server, remove DB files, remove config files)
  ansible-playbook uc-destroy.yaml
  # This playbook will reinstall the DB server, restore the DB backup, fix permissions and reinstall the UC
  ansible-playbook uc-restore.yaml

Checking the Undercloud state

After finishing the Undercloud restore playbook the user should be able to execute again any CLI command like:

  source ~/stackrc
  openstack stack list

Source code available in GitHub

by Carlos Camacho at May 18, 2018 12:00 AM

May 17, 2018


Introducing Mirantis Cloud Platform (MCP) Spring 2018 Release

We've added improvements to all three tiers of MCP, the DriveTrain CD layer, the cloud platform itself, which includes Kubernetes and OpenStack Pike, and the StackLight monitoring layer.

by Shaun OMeara at May 17, 2018 05:01 PM

OpenStack Superuser

Meeting the challenges of next-generation workloads: The OpenStack User Committee

The OpenStack User Committee helps increase operator involvement, collects feedback from the community, works with user groups around the globe and parses through user survey data, to name a few.

Here Superuser talks to one of the new members of the Committee, Yih Leong Sun, about multi-cloud, upcoming challenges for users and how you can get involved.  He’s been at Intel since 2015 and holds a PhD in multi-cloud orchestration.

You can catch him speaking at the Vancouver Summit with Henrik Blixt of VMware on “Lesson learned and best practices for large-scale enterprise private cloud” and connect with the User Committee at these events.

You’ve been involved with OpenStack since 2012,what is the biggest evolution you’ve seen?

OpenStack has been evolving as a de facto standard for open source cloud infrastructure platform. It started with a handful of services (also known as projects) and today counts more than 60 services [I may have lost count.. :-)].

The user base is also growing exponentially across industry in various countries and regions supporting different kinds of production workloads. Many organizations have already started to realize the benefits of using open-source architecture as part of the cloud strategy and OpenStack is definitely among the top choices!

What are some of the most pressing issues facing OpenStack users now? In the next year?

With the explosion of next-generation workloads (IoT, edge, AI, machine learning, NFV, etc.), the needs of dynamic composable and scalable infrastructure will bring new challenges to OpenStack architecture. Bridging the user gaps is always a challenge.

The User Committee and Technical Committee are collaborating closely to streamline the process of getting new user requirements into the project development pipeline.

We encourage users across the globe to actively participate in the various UC Teams and Working Groups, or the Special Interest Groups and help to enable OpenStack for various workload applications.

Multi-cloud is one of the most popular terms people search for on Superuser, what considerations should people make when they’re just getting started?

My PhD research is in the area of multi-cloud orchestration. There are many aspects that a user needs to be aware of when considering a multi-cloud solution: API, resource orchestration (compute, storage, network), security, authentication, federation, metadata, etc.

From a cloud operator’s perspective, a cloud management platform that provides a provider-agnostic model of operating multiple cloud providers can be useful. From a cloud app developer’s perspective, how to develop an application that can run across multiple cloud environments (e.g. through Kubernetes/container or other platform-as-a-service platform) could also of different interest.

Anything else you’d like to add?

I’m excited to be elected as one of the core members of User Committee. Special thanks to all of the community members who have supported me. With the variety of backgrounds and experience in 2018 User Committee members, I’m confident that the team will bring effective contribution and benefits to the OpenStack community, especially the users!

Get involved

The User Committee coordinates via the user-commitee mailing list  and you’re invited to post your ideas as well as via IRC on freenode in #openstack-uc channel. Consider joining the working groups and teams under UC governance found here.

They hold weekly IRC meetings on Monday at 1800 UTC in #openstack-meeting

Superuser wants to hear what you’re doing with open infrastructure, get in touch:

Photo // CC BY NC

The post Meeting the challenges of next-generation workloads: The OpenStack User Committee appeared first on Superuser.

by Superuser at May 17, 2018 04:29 PM

May 16, 2018

SUSE Conversations

OpenStack private cloud thriving – despite the skeptics

Every now and then I see a headline that I just find perplexing. Sometimes it’s for an article written by someone I highly regard and respect, which just makes the whole thing more baffling. One example was the bold headline “Wake up! The private cloud fantasy is over.” The same source followed up more recently […]

The post OpenStack private cloud thriving – despite the skeptics appeared first on SUSE Communities.

by Terri Schlosser at May 16, 2018 01:30 PM

OpenStack Superuser

OpenStack billing: Meet CloudKitty

Billing is subject that brings many cloud users back down to earth and is often top-of-mind for many OpenStack users. This is the second post kicked off by a thread started on the developer’s mailing list about current options and best practices.

Here Superuser talks to Christophe Sauthier, the project team lead (PTL) of CloudKitty, OSF’s rating-as-a-service project. CloudKitty is currently being developed by Sauthier’s company  Objectif Libre  and Catalyst IT and the hope is to get more developers on board.

You’ll find Sauthier at the Vancouver Summit, along with colleague Luka Peschke, for a CloudKitty onboarding session at the Forum. The pair are also leading a workshop titled “Chargeback and showback of your cloud services using CloudKitty and Gnocchi.”
You can check out all the sessions around billing at the Summit here.

What are your recommendations for older cloud users with home-grown solutions?

The first thing to do is to figure out if they’re happy with their solution in terms of mind coverage, maintenance cost, openness to the rest of their infrastructure and integration on OpenStack ecosystem, including the dashboard Horizon.

If they’re happy with it, then they should stick with what they’ve got. If not, one of the best ways to evaluate CloudKitty is to discuss it with us (over IRC, email, in real-life….) so that we can talk about coverage of their use cases…

Is CloudKitty suitable for use in large-scale production?

That depends on how large-scale you have in mind and the version of CloudKitty you want to use. Old releases were mainly just shipping some database back ends that clearly face some scalability issues.
We’ve already started to tackle that by:

    • Integrating Gnocchi storage in the Newton release to change that
    • Introducing a hybrid storage in Queens (Gnocchi+SQL) it’s a great improvement and the foundation for what we’re working on next

On the mailing list thread it was mentioned – maybe by you? – that CloudKitty’s biggest problem is that people don’t know about it – any thoughts on why and how to solve it?

Yes that was me! We know that we’re lacking some use cases, so sometimes we do what we think is the right thing but having some input would be so much better…We’re also trying to attend a lot of OpenStack-related events and improve our visibility that way…

What new features/enhancements/fixes are expected for the upcoming release?

We have a tons of new features coming in Rocky! The main features are:

  • Allowing CloudKitty to work as a standalone solution and thus to obtain non-OpenStack metrics. This work has already been integrated on the upstream branch! It’s clearly a step in the direction that encourages the Foundation with the Open Infrastructure Integration
  • Storage V2 which will be focused on scalablity and metric-oriented. It will be an evolution of the current hybrid storage, based on Gnocchi and SQL backends. It will be more efficient (speed, size,..). And finally it will ease the external contribution/evolutions.
  • API v2 which will also work on the scalability of CloudKitty (including some pagination by design). This new API paves to way for some externals collectors to use CloudKitty as a rating service — people will be able to write their own collectors to put whatever metrics they have in CloudKitty.

What contributions do you need most from the community?

ANY contribution is welcome (technical or not)!
We’re always looking for feedback and use cases from new people/companies…
Over the last cycle, we realized that having some help on the OpenStack community-wide goals would be great (since they are quite time consuming) and could be an interesting way to join the project. Oh and we would be really happy to discuss that at the onboarding session at the Summit.

Anything else you’d like people to know?

That we have a cute logo, so we definitely deserve a look!

Photo // CC BY NC

The post OpenStack billing: Meet CloudKitty appeared first on Superuser.

by Nicole Martinelli at May 16, 2018 01:07 PM

Chris Dent

Circumference 25 Beta

I've been working in the virtualisation and cloud space for a few years now, but for many of the experiments I want to do, having access to hardware that is local and visible to me makes thinking about the stuff I'm exploring easier. This is especially the case when wanting to experiment with dynamic hardware situations.

The Circumference 25 is a cluster-in-box designed to host up to 8 Raspberry Pi 3 Model B+ fronted by an UDOO X86. A crowd supply site is in progress to build C25 and C100 boxes that provide everything but the computers for a data centre in a box that you can put on a desk.

Courtesy of Ground Electronics, I have a beta build of the C25 to test its usefulness for OpenStack-related experiments. Mine came with the UDOO and 8 Raspberry PIs already in place. Looks like the one on the left:

circumference boxes

The UDOO uses an Intel Celeron N3160 2.24 Ghz, which has 4 cores, 4GB of RAM, a 250 GB SSD, and 3 gigabit ethernet ports.

The UDOO provides an Arduino-based interface to a backplane that allows software driven (command line and Python library) control and observation of the 8 Pis. Commands like compute 3 off.

Each Raspberry Pi is a 3 Model B+. They are connected to two ethernet switches, as well as the backplane.

The backplane provides access to a serial console, info about the individual devices and the "datacentre", and power control.

With the software control of power it is easy to model the addition and removal of nodes from the cluster.

This is a great way for me to experiment with real hardware failures and replacements in an OpenStack installation and to observe how those changes will impact the Placement service.

Each Raspberry Pi can operate as a small nova-compute, with everything else running on the UDOO. My initial experiments have been promising but I ran into some issues with quickly creating a working compute node.

Initially I started by installing a limited devstack on the UDOO. This resulted in a working but very small OpenStack cloud, with the UDOO (named C25-03) as the sole hypervisor.

Next I tried adding one of the Raspberry Pis as an additional hypervisor. While I did eventually get this to work (cc1 is one of the eight pis)

c25@C25-03:~/src/devstack$ openstack hypervisor list
| ID | Hypervisor Hostname | Hypervisor Type | Host IP       | State |
|  1 | C25-03              | QEMU            | | up    |
|  2 | cc1                 | QEMU            |      | up    |

there were issues that I will need to resolve for the next round:

  • Building nova-compute on the pi itself is terrifically slow and because of the way nova distributes itself, installation requests a lot of Python modules that aren't required for nova-compute itself. One way to improve this would be to create a single filesystem with nova-compute installed on it that each pi can net boot. In additiona, being able to describe solely the requirements of nova-compute, rather than all of nova, would be a helpful step with this.

  • Once running, the default video and logging modes that nova uses when using libvirt to build the VM did not work with the builds of qemu and libvirt that are available as debian ARMv7 packages. I was able to hack around this by changing the nova code. I would prefer a cleaner solution.

  • Getting networking to work effectively will require further work. For me, at least, networking always requires further work.

Discovering and resolving these kinds of issues is an excellent way to evolve OpenStack and makes the Circumference a useful tool, even before I'm able to do the "actual work" (whatever that may become).

When I return home from OpenStack summit I'm looking forward to working with the box more and will post additional reports on my experiments. What I hope to explore is controlling the backplane to add and remove compute nodes via code, faking network and power disruptions, and confirming or denying that the Placement service properly handles the changing resources. The Circumference provides a lot of hardware complexity (for example, multiple physical networks) and possibility in a small container that will make the experiments more useful. These experiments can be modeled in tests but experience has proven time and again that tests and real hardware are never quite the same and new learnings abound.

by Chris Dent at May 16, 2018 01:00 PM

May 15, 2018

OpenStack Superuser

OpenStack billing strategies: An overview

A recurring question for OpenStack operators is how to best follow the money. Specifically, what tools to use for chargeback, billing and whether the best strategy is to roll your own, use CloudKitty, Cielometer or another tool.

The question came up on the operator’s mailing list — full thread here — so we broadened the discussion by opening up an Etherpad for folks who had more to say.

Stay tuned tomorrow for a Q&A with Christophe Sauthier CEO of Objectif Libre about CloudKitty, OSF’s rating-as-a-service project. If you’re coming to the Vancouver Summit, you’ll want to check out these billing-related events.

Strategy: Using Ceilometer, OSF’s metering and data collection service

  • “We do not use events but audit data from Ceilometer.”
  • Ceilometer (now Panko) VM exists events coerced to look like jobs from an HPC batch system.”
  • OVH uses Ceilometer events/audit with a custom storage backend. “We store in MongoDB using the update operators of MongoDB ($inc, $max, …) to store only the final value and not all metrics/events.”
  • We use “the billing engine that’s part of the commercial software provided by Atomia. Using Ceilometer as of now, but they recently added support for Gnocchi which we’re going to use for our newer setups.”

Strategy: Roll your own

  • Catalyst Cloud wrote Distil which takes information from Ceilometer and creates aggregated samples more useful for billing. “An additional tool is triggered monthly to collect data from Distil and generate invoices in Odoo (potentially ERPNext in the future), which all the rating information comes from products in Odoo. A long-term goal is to move the invoicing logic into Distil itself as an automatic monthly (weekly, etc.) exporter into any external system like an ERP. The accounting system is module-based, so it should be easy enough to integrate with other systems. We’ve added a number of other pollsters to Ceilometer to collect various other items we want to bill, and we also have an flow traffic metering system which allows us to bill for different classes of network traffic.”
    They’ve also
    written an API for customers to be able to retrieve their invoices etc, example client is here: and for traffic billing:
    Catalyst Cloud’s Feilong Wang, Lingxian Kong and Adrian Turjak will be talking about this in a session titled “Lessons learned from public cloud billing” at the Vancouver Summit.
  • Cancer Genome Collaboratory developed in-house an app that pulls data from the Openstack DBs to generate simple cost and usage visualizations:
  • DataCentred uses  homegrown (Rails-based) application, but one which periodically polled Ceilometer for utilization information and subsequently performed a bunch of sanity checks It was open-sourced for posterity here: This was used to generate billing information with integration into Salesforce. “
  • “We use a homegrown billing system that periodically samples utilization of billable resources. (We turned off Ceilometer a few years ago because we weren’t really
    using it and found that it caused us trouble.)”
  • A combination of cASO and some low-level libvirt fabric monitoring. “The showback accounting reports are generated with merging with other compute/storage usage across various systems (HTCondor, SLURM, etc…)”
  • “We are running a cloud based on openstack-ansible and now are trying to
    integrate CloudKitty for billing. Up until now we used a self-written Python script to query Ceilometer for needed data, but that got more tedious than we are willing to handle. We hope it gets much easier once CloudKitty is set up.”

Stay tuned for tomorrow’s interview with CloudKitty.

The post OpenStack billing strategies: An overview appeared first on Superuser.

by Nicole Martinelli at May 15, 2018 04:06 PM

Chris Dent

TC Report 18-20

Trying to write a TC report after a gap of 3 weeks is hard enough, but when that gap involves some time off, the TC elections, and the run up to summit (next week in Vancouver) then it gets bewildering. Rather than trying to give anything like a full summary, I'll go for some highlights.

Be aware that since next week is summit and I'll be travelling the week after, there will be another gap in reports.


The elections were for seven positions. Of those, three are new to the TC: Graham Hayes, Mohammed Naser, Zane Bitter. Having new people is great. There's a growing sense that the TC needs to take a more active role in helping adapt the culture of OpenStack to its changing place in the world (see some of the comments below). Having new people helps with that greatly.

Doug Hellman has become the chair of the TC, taking the seat long held by Thierry. This is the first time (that I'm aware of) that a non-Foundation-staff individual has been the chair.

One of the most interesting parts of the election process were the email threads started by Doug. There's hope that existing TC members that were not elected in this cycle, those that have departed, and anyone else will provide their answers to them too. An email reminder exists.


Is next week, in Vancouver. The TC has several Forum sessions planned including:

Corporate Foundation Contributions

There's ongoing discussion about how to measure upstream contribution from corporate Foundation members and what to do if contribution seems lacking. Part of the reason this came up was because the mode of contribution from new platinum member, Tencent, is not clear. For a platinum member, it should be obvious.


There's been some concern expressed about the The Large Contributing OpenStack Operators (LCOO) group and the way they operate. They use an Atlassian Wiki and Slack, and have restricted membership. These things tend to not align with the norms for tool usage and collaboration in OpenStack. This topic came up in late April but is worth revisiting in Vancouver.


One of the things that came out in election campaigning is that OpenStack needs to be more clear about the many ways that OpenStack can be used, in part as a way of being more clear about what OpenStack is. Constellations are one way to do this and work has begun on one for Scientific Computing. There's some discussion there on what a constellation is supposed to accomplish. If you have an opinion, you should comment.

Board Meeting

The day before summit there is a "combined leadership" meeting with the Foundation Board, the User Committee and the Technical Committee. Doug has posted a review of the agenda. These meetings are open to any Foundation members and often involve a lot of insight into the future of OpenStack. And snacks.

Feedback, Leadership and Dictatorship of the Projects

Zane started an email thread about ways to replace or augment the once large and positive feedback loop that was present in earlier days of OpenStack. That now has the potential to trap us into what he describes as a "local maximum". The thread eventually evolved into concerns that the individual sub-projects in OpenStack can sometimes have too much power and identity compared to the overarching project, leading to isolation and difficulty getting overarching things done. There was a bit of discussion about this in IRC but the important parts are in the several messages in the thread.

Some people think that the community goals help to fill some of this void. Others thinks this is not quite enough and perhaps project teams as a point of emphasis is "no longer optimal".

But in all this talk of change, how do we do the work if we're already busy? What can we not do? That was a topic Monday morning.

API Version Bumps

Also on Monday, plans were made to have a session in Vancouver about how to do across-the-system minimum API version bumps. This started in response to a meandering thread on twitter about inconsistencies in the OpenStack's APIs "never" being resolved.

Where Now?

It's hard to make any conclusions from the election results. A relatively small number of people voted for a relatively small number of candidates. And there's always the sense that voting is primarily based on name recognition where platforms and policies have little bearing. However, if we are to take the results at face value then it appears that at least some of the electorate wants one or both of the following from the TC:

  • Increased communication and engagement.
  • Greater and more active exercising of whatever power they can dredge up to help lead and change the community more directly.

Do you think this is true? What direction do things need to go?

I'm currently in the state of mind where it is critical that we create and maintain the big picture information artifacts ("OpenStack is X, Y, and Z", "OpenStack is not A, B and C", "Next year OpenStack will start being E but will stop being Z") that allow contributors of any sort to pick amongst the (too) many opportunities for things to do. Especially making it easier—and socially and professionally safer—to say "no" to something. This makes it more clean and clear to get the right things done—rather than context switch—and to create the necessary headspace to consider improvements rather than doing the same thing over again.

by Chris Dent at May 15, 2018 03:30 PM

Edge computing and the importance of open infrastructure

Learn about edge computing and the importance of open infrastructure.

by ildikov at May 15, 2018 07:01 AM

Ghanshyam Mann

OpenStack Cloud DR PoC

With migration of application to cloud systems, cloud systems have become very important. Minor failure in the systems or human errors can cause major business impact ( like 1, 2)

To avoid business impact due to those failure, Disaster recovery solution is very much needed on critical Cloud. Disaster recovery solution provide business continuity in case of complete fail over of primary site. In case of a disaster situation, primary site workload would be switched to Replica site and would be made as active site so that businesses are not affected.

DR solution focus on switching to Replica site with defined RTO/RPO, so in case such a disaster occurs, the customer is notified of the maximum time in which his system will come back online.

NEC has done a PoC to try out the few DR solutions and checked their feasibility and stability. It was executed on single and miltinode env and collected the results of DR scenarios.

  Market Trend for DR solution:

Although the concept of DR in cloud is still nascent, a lot of SMBs are beginning to opt for DR to guard their business applications.

It is an attractive alternative for companies that are strapped for IT resources and who have a secondary infrastructure which is not effectively utilized.

Having DR sites in cloud reduces Data Centre costs/space, IT resources, IT Infra leading to significant cost reduction.

Each leading Cloud Service Provider ( AWS, Azure, Openstack ) now provide DR solutions via from their own end or via third-party software.

  Multiple options for DR mechanism:

       Option 1: Use Backup Images to recover at DR cloud

  • Primary cloud creates incremental backup images and saves them to backup storage at every predefined interval
  • Backup storage is expected to replicate backup images consistently between primary site and DR site
  • RPO is determined by how frequently incremental backups are performed

       Option 2: Use storage replication to replicate data continuously

  • Storage replication is established between two storage volumes upfront
  • Every write that happens at primary site is written to storage volume at DR site
  • Primary volume and secondary volumes are always in sync
  • During DR, secondary volume is made r/w and new VMs are created with secondary volumes and the VMs are restarted

        Option 3: This is kind of of the above two solutions leading to a hybrid solution.

  • Hybrid DR solution combines backup and replication to create a single solution.
  • Replication allows to copy data in volumes / data backend from Primary to DR cloud.
  • VM Metadata can be copied using Backup tooling
  • All other metadata can be copied directly from Primary to DR cloud.
  • Data Storage/Backend can be used to synchronize the Backup images from Primary to DR Cloud

  Comparison of different DR approaches:

CriteriaBackup DR
Replication DR
Hybrid DR
Dependency on Storage Backend
No dependency on Backend for Replication
Dependency on Backend for Replication
Dependency on Backend for Replication
Recovery Point Objective
RPO is large
RPO is small
RPO similar to Replication DR
Completely recoverable without manual intervention
Needs Manual intervention as VM metadata may not be recoverable.
Completely recoverable if supporting scripts exist.


Among all above 3 DR approach, this PoC tried the Backup based DR using Trilio data solution. Trilio is not open source but we found Trilio as one of the most stable and reliable for OpenStack Cloud DR Backup Approach.

We have tried it with single site as well as with multi site env:

                      Single Site






                      Multi Site





It has been tested with ~35 tests including functional as well as system testing. Below are the summary of outcome:

    Overall Test Summary:


    System Level stats:






For complete details of this PoC with test & results details can be found @OpenStack_Cloud_DR_PoC.

by Ghanshyam Mann at May 15, 2018 04:11 AM

May 14, 2018

OpenStack Superuser

How to start your journey to OpenStack deployment

For enterprise users, the journey to OpenStack deployment is fueled by the business opportunity of what lies ahead. IT systems were previously misunderstood as a crutch of sorts — something that plays a backseat role to the overlying business activities. However, with today’s emphasis on the cloud and particularly the opportunity with OpenStack, IT’s role has shifted into a driver of business outcomes — largely due to its ability to expedite business processes with an efficient foundation upon which business can thrive. Executives at the top understand that the future of IT is cloud-based and express the urgency to build a cloud that will scale to support growing customer demands.

With an improved underlying infrastructure layer, IT bottlenecks are thing of the past and positive ripple effects are felt across the organization. Suddenly, developers have self-service, on-demand access to compute, network and storage resources. They no longer have to rely on the central IT team to help facilitate their progress, allowing the organization to be faster to market and positioned more competitively.

OpenStack provides freedom from vendor lock-in and elasticity of clouds — both are imperatives when planning for long-term success. This innovative

oundation allows organizations to disrupt business as usual, shorten their time to market and ultimately create a unique cloud environment that addressees their specific business needs. What is exciting is how impactful and streamlined this shift to OpenStack can be. Whether it saves enterprises time or money, they are now able to rededicate resources towards core business functions.

As you start your OpenStack journey, it’s important to remember the bright future that lies ahead of this transformative foundation tailored to your business needs. Each OpenStack deployment takes different tracks to get there, but ultimately, organizations reach the same desired outcome.

Questions to start planning your OpenStack deployment

  1. Will you partner with a vendor-managed distributor (Red Hat, SUSE, Canonical, Mirantis, Upstream)?
  2. Will you deploy KVM or on a proprietary hypervisor?
  3. Which release and projects will you adopt?
  4. On what proportion of your infrastructure will you implement and what is your rollout strategy?

What type of organization should you partner with?

When deploying OpenStack, you can select a prepackaged distribution or you can take the DIY approach and build it from scratch. Although you may be quick to take the DIY approach to save costs, it is important to note that partnering with a distributor can ultimately save 45 percent of engineering costs. Getting support from a systems integrator or distributor is extremely valuable in setting up your journey for success. This partner will help you avoid bottlenecks and general confusion as you too are familiarizing yourself with OpenStack.

Deciding on a distribution partner for your OpenStack deployment is an important decision: you need specific skill requirements as well as emotional ones. It is worth taking the time to do thorough research to evaluate the technical capabilities of each OpenStack distribution. The systems integrator or distributor you choose will become an extension of your team, working closely with you to build and implement OpenStack in your organization. As you anticipate this journey, it’s important to highlight the areas that you can delegate work or streamline processes to make the adoption easier for everyone. This will proactively alleviate the challenges that may be ahead.

Top OpenStack purchasing factors

  • Price
  • Long-term support: you’ll want to work with companies that will be around for the foreseeable future to ensure that you have a steady partner in your journey
  • Existing relationship: consider partners you already have in place if you’re currently working with a systems integrator. Additionally, evaluate distributions that are offering OpenStack, as well as options for using proprietary software.
  • Reputation in community: many desire to use vendors that are involved in the OpenStack community and contributing code
  • One-stop shopping

Taking the first step

There’s no perfectly defined path to deploy OpenStack in your organization. It’s a creative process that’s different for each organization. There are many reasons why organizations adopt OpenStack and even more guiding principles that influence a company’s OpenStack deployment. With preparation, creativity and flexibility, the options with OpenStack are infinite.

As you start your journey, we’ve compiled a list of valuable resources to kick-start your research:

  • Anticipate the learning curve by getting familiar with the resources created by the strong community the OpenStack Foundation has fostered. Using videos from past OpenStack Summits, you’ll gain a wide understanding from many perspectives no matter where you are in the process.
  • Need a good introduction to OpenStack? Watch this 30 minute overview with Sandy Walsh from Rackspace.
  • Watch a short video to see all of the verticals currently running on OpenStack.
  • A great place to get started is by consulting distributions like SUSE, Canonical, Rackspace, Mirantis, VMware. These partners will not only guide you through the journey, but they also provide enhanced tools that the foundation doesn’t.
  • Consider subscribing to an OpenStack mailing list to learn more about topics that are relevant to you.
  • Attend the global summits hosted twice a year to engage with the vibrant community and learn about OpenStack updates.

As you start your journey, the strong community is an invaluable resource of OpenStack experts who are eager to help. At the global summits hosted twice a year, the vibrant, friendly community is a great resource and sounding board that acts like a one-stop-shop for all of your OpenStack needs.

This post first appeared on Trilio’s blog.
Superuser is always interested in community content. Get in touch at:

The post How to start your journey to OpenStack deployment appeared first on Superuser.

by Superuser at May 14, 2018 03:50 PM

May 11, 2018

OpenStack Superuser

Detangling Neutron and OVN database consistency

In this post I’ll talk about a problem that affects many (if not all) drivers in OpenStack Neutron and how it was solved for the OVN driver (AKA networking-ovn).

Problem description

In a common Neutron deployment model, multiple neutron-servers will handle the API requests concurrently. Requests that mutate the state of a resource (create, update and delete) will have their changes first committed to the Neutron database and then the loaded SDN driver is invoked to translate that information to its specific data model.
As the following diagram illustrates:

The model above can lead to two situations that can cause inconsistencies between the Neutron and the SDN databases:

Problem 1: Same resource updates race condition

When two or more updates to the same resource are issued at the same time and handled by different neutron-servers, the order in which these updates are written to the Neutron database is correct, however, the methods invoked in the driver to update the SDN database are not guaranteed to follow the same order as the Neutron database commits. That could lead to newer updates being overwritten by old ones on the SDN side resulting in both databases becoming inconsistent with each other.

The pseudo code for these updates looks something like this:

  In Neutron:

    with neutron_db_transaction:

  In the driver:

    def update_port_postcommit:
        port = neutron_db.get_port()

This problem has been reported at bug #1605089.

Problem two: Backend failures

The second situation is when changes are already fully persisted in the Neutron database but an error occurs upon trying to update the SDN database. Usually, what drivers do when that happens is to try to immediately rollback those changes in Neutron and then throw an error but, that rollback operation itself could also fail.

On top of that, rollbacks are not very straight forward when it comes to updates or deletes. For example, in the case where a VM is being teared down, part of the process includes deleting the ports associated with that VM. If the port deletion fails on the SDN side, re-creating that port in Neutron does not fix the problem. The decommission of a VM involves many other things, in fact, by recreating that port we could make things even worse because it will leave some dirty data around.

The networking-ovn solution

The solution used by the networking-ovn driver relies on the Neutron’s revision_number attribute. In short, for each resource in the Neutron database there’s an attribute called revision_number which gets incremented on every update, for example:

$ openstack port create --network nettest porttest
| revision_number | 2 |

$ openstack port set porttest --mac-address 11:22:33:44:55:66

$ mysql -e "use neutron; select standard_attr_id from ports \
where id=\"91c08021-ded3-4c5a-8d57-5b5c389f8e39\";"
| standard_attr_id |
|             1427 |

$ mysql -e "use neutron; SELECT revision_number FROM \
standardattributes WHERE id=1427;"
| revision_number |
|               3 |

The revision_number attribute is used by networking-ovn to solve the inconsistency problem in four situations:

1. Storing the revision_number in the OVN database

To be able to compare the version of the resource in Neutron against the version in OVN, we first need to know which version the OVN resource is present at.

Fortunately, each table in the OVN Northbound database1 contains a special column called external_ids which external systems (such as Neutron) can use to store information about its own resources that corresponds to the entries in the OVN database.

In our solution, every time a resource is created or updated by networking-ovn, the Neutron revision_number related to that change will be stored in the external_ids column of that resource in the OVN database. That allows networking-ovn to look at both databases and detect whether the version in OVN is up-to-date with Neutron or not.
Here’s how the revision_number is stored:

$ ovn-nbctl list Logical_Switch_Port
external_ids        : {"neutron:cidrs"="",
"neutron:device_id"="", "neutron:device_owner"="",

2. Performing a compare-and-swap operation based on the revision_number

To ensure accuracy when updating the OVN database, specifically when multiple updates are racing to change the same resource, we need to prevent older updates from overwriting newer ones.

The solution we found for this problem was to create a special OVSDB command that runs as part of the transaction that is updating the resource in the OVN database and prevents changes with a lower revision_number to be committed.
To achieve this, the OVSDB command does two things:

1 – Add a verify operation to the external_ids column in the OVN database so that if another client modifies that column mid-operation the transaction will be restarted.

A better explanation of what “verify” does is described in the doc string of the Transaction class in the OVS code itself, I quote:

“Because OVSDB handles multiple clients, it can happen that between the time that OVSDB client A reads a column and writes a new value, OVSDB client B has written that column. Client A’s write should not ordinarily overwrite client B’s, especially if the column in question is a “map” column that contains several more or less independent data items. If client A adds a “verify” operation before it writes the column, then the transaction fails in case client B modifies it first. Client A will then see the new value of the column and compose a new transaction based on the new contents written by client B.”

2 – Compare the revision_number from the Neutron update against what is presently stored in the OVN database. If the version in the OVN database is already higher than the version in the update, abort the transaction. Here’s a pseudo scenario where two concurrent updates are committed in the wrong order and how the solution above will deal with the problem:

Neutron worker 1 (NW-1): Updates port A with address X (revision_number: 2)

Neutron worker 2 (NW-2): Updates port A with address Y (revision_number: 3)

TRANSACTION 1: NW-2 transaction is committed first and the OVN resource
now has revision_number 3 in it's external_ids column

TRANSACTION 2: NW-1 transaction detects the change in the external_ids
column and is restarted

TRANSACTION 2: NW-1 the OVSDB command now sees that the OVN resource
is at revision_number 3, which is higher than the update version
(revision_number 2) and aborts the transaction.

3. Detecting inconsistent resources

When things are working as expected the two items above should ensure that the Neutron and OVN databases are in a consistent state but, what happens when things go bad ? For example, if the connectivity with the OVN database is temporarily lost, new updates will start to fail and as stated in problem 2 rolling back those changes in the Neutron database is not always a good idea.

Before this solution we used to maintain a script that would scan both databases and fix all the inconsistencies between them but, depending on the size of the deployment that operation can be very slow. We needed a better way.

The solution for the detection problem was to create an additional table in the Neutron database (called “ovn_revision_numbers”) with a schema that look like this:

Column name Type Description
standard_attr_id Integer Primary key. The reference ID from the standardattributes table in Neutron for that resource. ONDELETE SET NULL.
resource_type String Primary key. The type of the resource (e.g, ports, routers, …)
resource_uuid String The UUID of the resource
revision_number Integer The version of the object present in OVN
acquired_at DateTime The time that the entry was create. For troubleshooting purposes
updated_at DateTime The time that the entry was updated. For troubleshooting purposes

This table would serve as a “cache” for the revision numbers correspondent to the OVN resources in the Neutron database.

For the different operations: Create, update and delete; this table will be used as:

Create operation

Neutron has a concept of “precommit” and “postcommit” hooks for the drivers to implement when dealing with its resources. Basically, the precommit hook is invoked mid-operation when the Neutron’s database transaction hasn’t being committed yet. Also important, the context passed to the precommit hook will contain a session to the active transaction which drivers can use to make other changes to the database that will be part of the same commit. In the postcommit hook, the data is already fully persisted in Neutron’s database and this is where drivers are suppose to translate the changes to the SDN backend.

Now, to detect inconsistencies of a newly created resource, a new entry in the ovn_revision_numbers table will be created using the precommit hook for that resource and the same database transaction. The revision_number column for the new entry will have a placeholder value (we use -1) until the resource is successfully created in the OVN database (in the postcommit hook) and then the revision_number is bumped. If a resource fails to be created in OVN, the revision_number column in the ovn_revision_numbers table for that resource will still be set to -1 (the placeholder value) which is different than it’s correspondent entry in the standardattributes table (which is updated as part of the Neutron’s database transaction).

By looking at the differences in the revision_number’s on both tables is how inconsistencies are detected in the system.

The pseudo code for the create operation looks like this:

def create_port_precommit(context, port):
    create_initial_revision(port['id'], revision_number=-1,

def create_port_postcommit(port):
    bump_revision(port['id'], revision_number=port['revision_number'])

Update operation

Every time an update to a resource is successfully committed in the OVN database, the revision_number for that resource in the ovn_revision_numbers table will be bumped to match the revision_number in the standardattributes table. That way if something fails on the OVN side, the revision_number’s from these two tables will be different and inconsistencies will be detect in the same way as they are in the create operation.

The pseudo code for the delete operation looks like this:

def update_port_postcommit(port):
    bump_revision(port['id'], revision_number=port['revision_number'])

Delete operation

The standard_attr_id column in the ovn_revision_numbers table is a foreign key constraint with a ONDELETE=SET NULL set. Which means that, upon Neutron deleting a resource the standard_attr_id column in the ovn_revision_numbers table will be set to NULL.

If deleting a resource succeeds in Neutron but fails in OVN, the inconsistency can be detect by looking at all resources where the standard_attr_id column equals to NULL.

When the deletion succeeds in both databases the entry in the ovn_revision_numbers table can then be delete as well.

The pseudo code for the delete operation looks like this:

def delete_port_postcommit(ctx, port):

4. Fixing inconsistent resources

Now that the algorithm to detected inconsistencies has been optimized we were able to create a maintenance task that runs periodically (every 5 minutes) and is responsible for detecting and fixing any inconsistencies it may find in the system.

It’s important to note that only one instance of this maintenance task will be actively running in the whole cluster at a time, all other instances will be in standby mode (active-standby HA). To achieve that, each maintenance task instance will attempt to acquire an OVSDB named lock and only the instance that currently holds the lock will make active changes to the system.

The maintenance task operation is composed by two steps:

  1. Detect and fix resources that failed to be created or updated
  2. Detect and fix resources that failed to be deleted

Resources that failed to be created or updated will have their standard_attr_id column pointing to the numerical id of its counterpart in the standardattributes table but, their revision_number column will be different.

When inconsistent resources like that are found, the maintenance task will first try to fetch that resource from the OVN database and; if found, the resource is updated to what is latest in the Neutron database and then have their revision_number bumped in the ovn_revision_numbers table. If the resource is not found, it will be first created in OVN and upon success its revision_number will be bumped in the ovn_revision_numbers table.

Resources that failed to be deleted will have their standard_attr_id column set to NULL. To fix this type of inconsistency, the maintenance task will try to fetch the resource from the OVN database and; if found, the resource and its entry in the ovn_revision_numbers are deleted. If not found, only the entry in the ovn_revision_numbers is deleted.

Note that, the order in which the resources are fixed is important. When fixing creations and updates the maintenance task will start fixing the root resources (networks, security groups, etc…) and leave the leaf ones to the end (ports, floating ips, security group rules, etc…). That’s because, say a network and a port (that belongs to that network) failed to be created, if it tries to create the port prior to creating the network that operation will fail again and it will only be fixed on the next iteration of the maintenance task. The order also matters for deletion but in the opposite direction, it’s preferable to delete the leaf resources before the root ones to avoid any conflict.

Conclusion and other notes

This was a long post but I hope it will make it easier for people to understand with a certain level of detail how networking-ovn is dealing with this problem.

Also, in the OpenStack PTG for the Rocky cycle other Neutron driver developers were interested in implementing this approach for their own drivers. It was decided to attempt to make this approach more generic2 and migrate the code to the Neutron’s core repository so drivers can consume it and avoid duplication. The specification proposing this change in Neutron can be found here.

This post first appeared on Lucas Alvares Gomes’s blog. He’s a software engineer at Red Hat currently working on OVN (Open Virtual Network) and related OpenStack projects.


  1. Take a look at the OVN architecture for more information about the Northbound and Southbound databases in OVN ↩
  2. For example, by removing the dependency on things like the OVSDB named locks. Please check the proposed specification for Neutron. ↩

The post Detangling Neutron and OVN database consistency appeared first on Superuser.

by Lucas Alvares Gomes at May 11, 2018 03:45 PM

Chris Dent

Placement Update 18-19

This is placement update 18-19. 18 was skipped because I was on holiday. With this issue I'm going to start cross-posting to my blog to increase exposure, double up the archiving, and get the content showing up on the OpenStack Planet.

One upshot of this change is that the content will now be formatted more fully as Markdown.

I'll be travelling next week so there won't be one of these for weeks 20 or 21, unless someone else feels like it.

Most Important

We're continuing to hope that granular and nested resource providers will be fully merged by Summit (a bit more than a week from now). Not clear if this will happen as last I checked it seemed we have multiple threads of changes in progress, many of which will merge conflict with one another. But then again, I may be out of date, it's been difficult to find all those threads while trying to catch up this week.

If you're going to be at summit there are (at least) two placement-related forum sessions:

Please add to those etherpads if you have thoughts.

Also a summit, Ed and Eric will be presenting Placement, Present and Future, in Nova and Beyond.

What's Changed

Granular requests can now be made to GET /allocation_candidates (meaning resourcesN and requiredN are now accepted). A bug with the safe_connect handler masking real problems has been fixed. The spec for Network Bandwidth Resource Provider has finally merged after a lot of thinking and discussion. The spec for Return resources of entire trees in Placement has merged. This allows the inclusion of resource providers which are not providing inventory, but are part of the current tree, in the provider summaries of a /allocation_candidates response.

There are some new specs (see the end of the specs list, below) which extend required traits handling to be able to say "I need at least one of these traits".



Total two weeks ago: 11. Now: 13

Main Themes

Nested providers in allocation candidates

Unfortunately I'm really unclear on what the current state of this is. If someone else can give a quick overview that would be excellent. There's code in progress at both of the following topics, some of it is old and in merge conflict:

Mirror nova host aggregates to placement

This makes it so some kinds of aggregate filtering can be done "placement side" by mirroring nova host aggregates into placement aggregates.

This is still in progress but is still on its attention break.

Consumer Generations

This allows multiple agents to "safely" update allocations for a single consumer. The code is in progress:

There's also a related change for ensuring consumer records.


Ways and means of addressing granular requests when dealing with nested resource providers. Granular in this sense is grouping resource classes and traits together in their own lumps as required. Topic is:

The sole active change in that now is work in progress to get it working from the Nova side.


I've created patches that adjust devstack and zuul config to use the separate placement database connection.

All of these things could merge without requiring any action by users. Instead they allow people to use different connections, but don't require it.

Jay has made a first pass at an os-resource-classes which I thought was potentially more heavyweight than required, but other people should have a look too.

As mentioned above there will be a forum session about extraction.

In the meantime, some of the low hanging fruit on extraction is duplicating and extracting to their own files the various fixtures and base test classes that are required by both the functional and unit tests. And making them not import from the nova hierarchy.


17 entries two weeks ago. 19 now.

Some of the older items in this list are not getting much attention. That's a shame. The list is ordered the way it is on purpose.


Please followup with the many things I've missed.

by Chris Dent at May 11, 2018 11:45 AM

SUSE Conversations

An experiment with Gnocchi – (the database) – Part 3

In Part 2, I summarized the iterations applied to the kernel (the piece of CUDA code that executes on GPU) to remove the bottlenecks encountered during profiling, like using shared memory to avoid non-coalesced memory access. In this part, I will talk about the final version of the kernel and using the GPU in other […]

The post An experiment with Gnocchi – (the database) – Part 3 appeared first on SUSE Communities.

by Sumit Jamgade at May 11, 2018 10:00 AM

OpenStack in Production

Introducing GPUs to the CERN Cloud

High-energy physics workloads can benefit from massive parallelism -- and as a matter of fact, the domain faces an increasing adoption of deep learning solutions. Take for example the newly-announced TrackML challenge [7], already running in Kaggle! This context motivates CERN to consider GPU provisioning in our OpenStack cloud, as computation accelerators, promising access to powerful GPU computing resources to developers and batch processing alike.

What are the options?

Given the nature of our workloads, our focus is on discrete PCI-E Nvidia cards, like the GTX1080Ti and the Tesla P100. There are 2 ways to provision these GPUs to VMs: PCI passthrough and virtualized GPU. The first method is not specific to GPUs, but applies to any PCI device. The device is claimed by a generic driver, VFIO, on the hypervisor (which cannot use it anymore) and exclusive access to it is given to a single VM [1]. Essentially, from the host’s perspective the VM becomes a userspace driver [2], while the VM sees the physical PCI device and can use normal drivers, expecting no functionality limitation and no performance overhead.
Visualizing passthrough vs mdev vGPU [9]
In fact, perhaps some “limitation in functionality” is warranted, so that the untrusted VM can’t do low-level hardware configuration changes on the passed-through device, like changing power settings or even its firmware! In fact, security-wise PCI passthrough leaves a lot to be desired. Apart from allowing the VM to change the device’s configuration, it might leave a possibility for side-channel attacks on the hypervisor (although we have not observed this, and a hardware “IOMMU” protects against DMA attacks from the passed-through device). Perhaps more importantly, the device’s state won’t be automatically reset after deallocating from a VM. In the case of a GPU, data from a previous use may persist on the device’s global memory when it is allocated to a new VM. The first concern may be mitigated by improving VFIO, while the latter, the issue of device reset or “cleanup”, provides a use case for a more general accelerator management framework in OpenStack -- the nascent Cyborg project may fit the bill.
Virtualized GPUs are a vendor-specific option, promising better manageability and alleviating the previous issues. Instead of having to pass through entire physical devices, we can split physical devices into virtual pieces on demand (well, almost on demand; there needs to be no vGPU allocated in order to change the split) and hand out a piece of GPU to any VM. This solution is indeed more elegant. In Intel and Nvidia’s case, virtualization is implemented as a software layer in the hypervisor, which provides “mediated devices” (mdev [3]), virtual slices of GPU that appear like virtual PCI devices to the host and can be given to the VMs individually. This requires a special vendor-specific driver on the hypervisor (Nvidia GRID, Intel GVT-g), unfortunately not yet supporting KVM. AMD is following a different path, implementing SR-IOV at a hardware level.

CERN’s implementation

PCI passthrough has been supported in Nova for several releases, so it was the first solution we tried. There is a guide in the OpenStack docs [4], as well as previous summit talks on the subject [1]. Once everything is configured, the users will see special VM flavors (“g1.large”), whose extra_specs field includes passthrough of a particular kind of gpu. For example, to deploy a GTX 1080Ti, we use the following configuration:
add PciPassthroughFilter to enabled/default filters
flavor extra_specs
--property "pci_passthrough:alias"="nvP1080ti_VGA:1,nvP1080ti_SND:1"
A detail here is that most GPUs appear as 2 pci devices, the VGA and the sound device, both of which must be passed through at the same time (they are in the same IOMMU group; basically an IOMMU group [6] is the smallest passable unit).
Our cloud was in Ocata at the time, using CellsV1, and there were a few hiccups, such as the Puppet modules not parsing an option syntax correctly (MulitStrOpt) and CellsV1 dropping the pci requests. For Puppet, we were simply missing some upstream commits [15]. From Pike on and in CellsV2, these issues shouldn’t exist. As soon as we had worked around them and puppetized our hypervisor configuration, we started offering cloud GPUs with PCI passthrough and evaluating the solution. We created a few GPU flavors, following the AWS example of keeping the amount of vCPUs the same as the corresponding normal flavors.
From the user’s perspective, there proved to be no functionality issues. CUDA applications, like TensorFlow, run normally; the users are very happy that they finally have exclusive access to their GPUs (there is good tenant isolation). And there is no performance penalty in the VM, as measured by the SHOC benchmark [5] -- admittedly quite old, we preferred this benchmark because it also measures low-level details, apart from just application performance.
From the cloud provider’s perspective, there’s a few issues. Apart from the potential security problems identified before, since the hypervisor has no control over the passed-through device, we can’t monitor the GPU. We can’t measure its actual utilization, or get warnings in case of critical events, like overheating.
Normalized performance of VMs vs. hypervisor on some SHOC benchmarks. First 2: low-level gpu features, Last 2: gpu algorithms [8]. There are different test cases of VMs, to check if other parameters play a role. The “Small VM” has 2 vCPUs, “Large VM” has 4, “Pinned VM” has 2 pinned vCPUs (thread siblings), “2 pin diff N” and “2 pin same N” measure performance in 2 pinned VMs running simultaneously, in different vs the same NUMA nodes

Virtualized GPU experiments

The allure of vGPUs amounts largely to finer-grained distribution of resources, less security concerns (debatable) and monitoring. Nova support for provisioning vGPUs is offered in Queens as an experimental feature. However, our cloud is running on KVM hypervisors (on CERN CentOS 7.4 [14]), which Nvidia does not support as of May 2018 (Nvidia GRID v6.0). When it does, the hypervisor will be able to split the GPU into vGPUs according to one of many possible profiles, such as in 4 or in 16 pieces. Libvirt then assigns these mdevs to VMs in a similar way to hostdevs (passthrough). Details are in the OpenStack docs at [16].
Despite this promise, it remains to be seen if virtual GPUs will turn out to be an attractive offering for us. This depends on vendors’ licensing costs (such as per VM pricing), which, for the compute-compatible offering, can be significant. Added to that is the fact that only a subset of standard CUDA is supported (not supported are the unified memory and “CUDA tools” [11], probably referring to tools like the Nvidia profiler). vGPUs are also oversubscribing the GPU’s compute resources, which can be seen in either a positive or negative light. On the one hand, this guarantees higher resource utilization, especially for bursting workloads, like developers. On the other hand, we may expect a lower quality of service [12].

And the road goes on...

Our initial cloud GPU offering is very limited, and we intend to gradually increase it. Before that, it will be important to address (or at least be conscious about) the security repercussions of PCI passthrough. But even more significant is to address GPU accounting in a straightforward manner, by enforcing quotas on GPU resources. So far we haven’t tested the case of GPU P2P, with multi-GPU VMs, which is supposed to be problematic [13].
Another direction we’ll be researching is offering GPU-enabled container clusters, backed by pci-passthrough VMs. It may be that, with this approach, we can emulate a behavior similar to vGPUs and circumvent some of the bigger problems with pci passthrough.


[5]: SHOC benchmark suite:
[11]: CUDA Unified memory and tooling not supported on Nvidia vGPU:

by Konstantinos Samaras-Tsakiris ( at May 11, 2018 08:12 AM

May 10, 2018

OpenStack Superuser

OpenStack upgrades on a massive scale: Inside Yahoo

James Penick, currently architect director at Yahoo, knows a thing or two about upgrades. Starting out as a self-described “ticket monkey,” he has placed boots have been on the ground in nearly all aspects of large-scale production engineering. He and his team of 20 currently watch over about four million cores.

That’s why Superuser thought it’d be a good idea to get his thoughts on upgrades, one of the topics that’s always top-of-mind for OpenStack users, before the Vancouver Summit. For Summit-goers, here are featured sessions on upgrades and details on both of Penick’s talks.

In Vancouver, one of your talks is with Verizon’s Amrith Kumar about building and operating an enormous OpenStack cloud – can you give us details on the deployment and the size of your team?

The team is relatively small, less than 20 engineers at the moment, including dev ops and PE. In terms of scale we’re currently managing hundreds of thousands of bare metal instances and under 100,000 VM instances. In sum, we’re managing four million cores or so.

What open-source technologies are you using now?

Where to start? The open source culture has produced a rich pantheon of tools and platforms that have become such an integral part of our daily lives that I take them for granted. But off the top of my head: MySQL, RabbitMQ, Apache, KVM, Libvirt, Linux, Chef, Screwdriver, Athenz and of course OpenStack. Frankly there’s too many to name.

There’s no ideal/universal approach to guarantee a smooth upgrade — but what are a few key practices?

Read the release notes! Build a test cluster to test the upgrade process. Make sure to automate the fix to every problem you encounter. Whenever possible, file bugs or even send patches upstream.

In your Sydney talk, you compared moving to OpenStack “like a tug boat pushing at the front of an aircraft carrier to turn it around.” What’s your metaphor for upgrading from Juno to Ocata?

Upgrading from Juno to Ocata seems like climbing Everest, when in reality it’s more like walking up stairs. It’s strenuous, but there’s structure and support to help you get by.

What’s the most common misconception around OpenStack upgrades?

Everything is heresy until it’s common sense. The best way to dispel the FUD around OpenStack upgrades is to get the word out in talks, posts, and presentations, until people roll their eyes at how obvious of a truth it is. As deployers share their success they will change the perception of OpenStack upgrades.

That’s what happened with running OpenStack at scale; there was a time when running a 200-node cluster was considered impossible. Now multi-thousand node clusters are obvious.


Superuser wants to hear your open infrastructure story, get in touch at

Photo // CC BY NC

The post OpenStack upgrades on a massive scale: Inside Yahoo appeared first on Superuser.

by Nicole Martinelli at May 10, 2018 04:33 PM

SUSE Conversations

OpenStack and the Art of Fishing

Teach a man to fish and you feed him for a lifetime.  It’s an often quoted proverb when talking about the value of personal development.  Give someone the opportunity to gain the skills and develop the talents they need and you’ll equip that person to survive and thrive. If that person happens to work for […]

The post OpenStack and the Art of Fishing appeared first on SUSE Communities.

by Terri Schlosser at May 10, 2018 09:00 AM

Highlights from the OpenStack project teams gathering

A few weeks back in Dublin, Ireland, OpenStack engineers gathered from dozens of countries and companies to discuss the next release of OpenStack. This is always my favorite OpenStack event, because I get to do interviews with the various teams, to talk about what they did in the just-released version (Queens, in this case) and what they have planned for the next one (Rocky).

by rbowen at May 10, 2018 07:01 AM

May 09, 2018

Ed Leafe

Graph Database Follow-up

I just published my series on Graph Databases a few hours ago, and have already had lots of hits. I guess people found the topic as interesting as I did! One reader, Jay Pipes, is one of the core developers of the Placement service I mentioned in that series, and probably the most heavily opinionated … Continue reading "Graph Database Follow-up"

by ed at May 09, 2018 08:57 PM

Placement Graph Examples

In the previous two posts in this series, I wrote about graph databases in general, and why I think that they are much more suited to modeling the data in the Placement service than relational DBs such as MySQL. In this post I’ll show a few examples of common Placement issues and how they are … Continue reading "Placement Graph Examples"

by ed at May 09, 2018 04:24 PM

Graph Database Basics

It is not my intent here to create a full tutorial on using graph databases; I just want to convey enough understanding so that you might see why they are a better fit for placement than a relational DB. The most basic thing to understand is that your whiteboard design is your database design. Here’s … Continue reading "Graph Database Basics"

by ed at May 09, 2018 04:24 PM

Hammer and Nail

The work on the OpenStack Placement service has been getting more and more complex in the last few months, as the service has matured to handle more complex resources. At first, things were pretty basic: you had a ResourceProvider, and it had Inventory of various ResourceClasses. You told Placement how much you needed of one … Continue reading "Hammer and Nail"

by ed at May 09, 2018 04:24 PM

OpenStack Superuser

Inside telecom and NFV: Must-see sessions at the Vancouver Summit

Join the people building and operating open infrastructure at the OpenStack Summit Vancouver in May.  The Summit schedule features over 300 sessions organized by use cases including: artificial intelligence and machine learning, high performance computing, edge computing, network functions virtualization, container infrastructure and public, private and multi-cloud strategies.

Here we’re highlighting some of the sessions you’ll want to add to your schedule about telecom and network functions virtualization. Check out all the sessions, workshops and lightning talks focusing on these topics here.

High-performance Ceph for hyper-converged telco NFV infrastructure

The next wave of transformation for telco clouds is hyper-converged NFV infrastructure (NFVi) which is expected to bring cost efficiency and scale. Storage is a key piece of any hyper-converged platform and new solid-state storage technologies such as Intel Optane are bringing a big shift in compute vs storage budget balance in hyper-converged platforms. In this intermediate-level talk, Karl Vietmeier and Tushar Gohad from Intel with AT&T Labs Moo-Ryong Ra will talk about how they’re collaborating on developing new ways to leverage these technologies in Ceph to enable a low latency, high performance OpenStack platform for Telco NFV applications. You’ll hear about their contributions to Ceph and Openstack-HELM and take away the basics of tuning hyper-converged Ceph. Details here.

The practice of NFV pre-integration with OpenStack in ChinaMobile

ChinaMobile currently applies a distributed telecom cloud architecture. There are numerous edge telecom cloud nodes to be built as well as used in operation and maintained. The edge nodes are all based on OpenStack’s three-layer decoupling architecture. The ChinaMobile team built the pre-integrated model to solve the problem of OPNFV integrated tools and ONAP remote operation management. In this talk, they’ll also demo the pre-integrated delivery verification in the China Mobile NovoNet project, including the definition of pre-integrated templates (hardware and software planning and design), template instantiation (system deployment, integration testing, etc.) and integrated delivery on-line. Details here.

Achieving end-to-end NFV with OpenStack and open source MANO

In this intermediate-level talk, Whitestack’s Gianpietro Lavado and José Miguel Guzmán will show how both OpenStack and Open Source MANO projects can work tightly together to provide a robust, end-to-end ETSI NFV compliant deployment. The  architecture for both projects will be explained in relation to how they integrate and cover the complete network services life cycle management. The talk will also include some results from the latest ETSI NFV Plugtests, where Whitestack used OpenStack and open source MANO together and  inter-operating with other open and proprietary solutions to test an ETSI compliant NFV MANO. Details here.

Look Mom! No patches in our blazing fast, smart telco cloud

OpenStack for Telco NFV cloud deployment is rapidly gaining momentum, but Virtual Network Functions (VNFs) are pushing the limits on the network and demanding a feature rich, high-performance data path. The OpenStack, Open vSwitch and Linux communities have upstreamed open APIs for SmartNICs, enabling high-throughput low-latency data path with minimal performance tuning and the ability to manipulate inner packet headers providing services like firewall DDoS protection, Quality of Service (QoS) marking and Deep Packet Inspection (DPI). These upstream integrated changes are emerging in stable open source distros with no custom patches and no vendor lock-in, supported by multiple SmartNIC vendors and SDN controllers. This is opening new flow processing possibilities, while freeing up expensive CPU resources for running application workloads. This intermediate session lead by Mark Iskra (Nokia – Nuage Networks) Anita Tragler (Red Hat) and Ash Bhalgat (Cloud Marketing) will demo out-of-the-box fast data path functional progress and performance results achieved with OpenStack SmartNICs running VNF workloads. Details here.

Telco cloud next generation stack update: What’s changed with OpenStack-Helm and OpenContrail-Helm

Discuss the latest advancements and milestones reached within both the openstack-helm and opencontrail-helm projects and how they are being leveraged by telco clouds with Alan Meadows and Pete Birley from AT&T a Vinay Rao from Juniper Networks. They’ll review the current stable release, current functionality set, the community gate capabilities, the development tools for contributors and the timeline for additional OpenStack version support as well as the work occurring to support edge computing. They’ll also review the status of our undercloud deployment process and those open source tools as well as how it’s currently used at AT&T. Details here.

Steps to optimizing OpenStack resource utilization for VNFs

As the Verizon Wireless OpenStack Edge platform grows, making efficient use of resources also grows in importance, to deliver the cost benefits of OpenStack to Verizon customers. This beginner-level presentation by Susan Coombs explores how utilization at the highest level, Virtual Network Function (VNF) redundancy pairs, relates to utilizations at intermediate and lower levels. For example, an intermediate level Nova server corresponds to a lower level Libvirt instance, which in turn is composed of CPU cores which each have utilizations. In some cases, these CPU cores may be pinned by the VNF design, and used at 100 percent, in which case no optimization is possible. In other cases, some over-subscription may be possible without impacting services. In addition to tying a given Nova VM together with its foundational levels, each VNF frequently contains multiple nova instances, neutron networks, and cinder volumes, which also must be considered when over-subscribing. Details here.

See you at the OSF Summit in Vancouver, May 21-24, 2018! Register here.

Photo // CC BY NC

The post Inside telecom and NFV: Must-see sessions at the Vancouver Summit appeared first on Superuser.

by Superuser at May 09, 2018 04:04 PM

Why we use tests on OpenStack package builds in RDO

Unit tests are used to verify that individual units of source code work according to a defined specification (spec). While this may sound complicated to understand, in short it means that we try to verify that each part of our source code works as expected, without having to run the full program they belong to.

by jpena at May 09, 2018 07:01 AM

May 08, 2018

OpenStack Superuser

Get started with the Octavia Ingress Controller for Kubernetes

This guide explains how to deploy and configure the octavia-ingress-controller in Kubernetes cluster on top of an OpenStack cloud.

What’s an Ingress Controller, anyway?

In Kubernetes, Ingress allows external users and client applications access to HTTP services. Ingress consists of two components:

  • Ingress Resource is a collection of rules for the inbound traffic to reach Services. These are Layer 7 (L7) rules that allow hostnames (and optionally paths) to be directed to specific Services in Kubernetes.
  • Ingress Controller acts on the rules set by the Ingress Resource, typically via an HTTP or L7 load balancer.

It’s vital that both pieces are properly configured to route traffic from an outside client to a Kubernetes Service.

1. Benefits of using the octavia-ingress-controller

As an OpenStack-based public cloud provider, one of our goals at Catalyst Cloud is to continuously provide customers with the capability to innovate by delivering robust and comprehensive cloud services. After deploying Octavia and Magnum services in the public cloud, we’re now thinking about how to help customers to develop their applications running on the Kubernetes cluster and making their services accessible to the public in a high-performance way.

After creating a Kubernetes cluster in Magnum, the most common way to expose the application to the outside world is to use a LoadBalancer service. In the OpenStack cloud, Octavia (LBaaS v2) is the default implementation of LoadBalancer type service, as a result, for each LoadBalancer type service, there is a load balancer created in the cloud tenant account. There are a few drawbacks of this approach:

  • The cost of Kubernetes Service is a little bit high if it’s one-to-one mapping from the service to Octavia load balancer and the customers have to pay for a load balancer per exposed service, which can get expensive.
  • There is no filtering, no routing, etc. for the service. That means you can send almost any kind of traffic to it, like HTTP, TCP, UDP, WebSockets, gRPC, or whatever.
  • The traditional ingress controllers such as the NGINX ingress controller, HAProxy ingress controller, Træfik, etc. don’t make much sense in the cloud environment because the user still needs to expose the service for the ingress controller itself which may increase the network delay and decrease the performance of the application.

The octavia-ingress-controller can solve all of the above problems in an OpenStack environment by creating a single load balancer for multiple NodePort type services in an Ingress. In order to use the octavia-ingress-controller in Kubernetes cluster, use the value openstack for the annotation in the metadata section of the Ingress Resource as shown below:

annotations: openstack

2. How to deploy the octavia-ingress-controller

2.1 Prepare the kubeconfig file

The kubeconfig file is used to configure access to Kubernetes clusters. This is a generic way of referring to configuration files in Kubernetes. The following commands are performed in a Kubernetes cluster created using kubeadm.

  • Install cfssl tools, which are used for generating TLS certs
wget && chmod +x cfssl_linux-amd64 && mv cfssl_linux-amd64 /usr/local/bin/cfssl
wget && chmod +x cfssljson_linux-amd64 && mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
wget && chmod +x cfssl-certinfo_linux-amd64 && mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
  • Re-use the CA cert and key in the existing cluster
pushd /etc/kubernetes/pki
cat > ca-config.json <<EOF
  "signing": {
    "default": {
      "expiry": "87600h"
    "profiles": {
      "kubernetes": {
        "usages": [
            "key encipherment",
            "server auth",
            "client auth"
        "expiry": "87600h"
cat > ingress-openstack-csr.json <<EOF
  "CN": "octavia-ingress-controller",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  "names": [
      "C": "NZ",
      "ST": "Wellington",
      "L": "Wellington",
      "O": "Catalyst",
      "OU": "Lingxian"
cfssl gencert -ca=ca.crt -ca-key=ca.key -config=ca-config.json -profile=kubernetes ingress-openstack-csr.json | cfssljson -bare ingress-openstack
# You can take a look at the files generated by cfssl
ls -l | grep ingress-openstack
  • Create a kubeconfig file for the octavia-ingress-controller
ca_data=$(cat ca.crt | base64 | tr -d '\n')
client_cert_data=$(cat ingress-openstack.pem | base64 | tr -d '\n')
client_key_data=$(cat ingress-openstack-key.pem | base64 | tr -d '\n')
cat <<EOF > /etc/kubernetes/ingress-openstack.conf
apiVersion: v1
kind: Config
- cluster:
    certificate-authority-data: ${ca_data}
    server: https://${k8s_master_ip}:6443
  name: kubernetes
preferences: {}
- name: octavia-ingress-controller
    client-certificate-data: ${client_cert_data}
    client-key-data: ${client_key_data}
- context:
    cluster: kubernetes
    user: octavia-ingress-controller
  name: octavia-ingress-controller@kubernetes
current-context: octavia-ingress-controller@kubernetes

2.2 Configure role-based access control (RBAC) for the octavia-ingress-controller user

For testing purpose, grant the cluster-admin role for octavia-ingress-controller as user so that the user has full access to the Kubernetes cluster.

cat <<EOF | kc create -f -
kind: ClusterRoleBinding
  name: octavia-ingress-controller
  - kind: User
    name: octavia-ingress-controller
  kind: ClusterRole
  name: cluster-admin

2.3 Prepare the octavia-ingress-controller service configuration

We need credentials of admin user and a normal user (e.g. demo) in OpenStack.

source openrc_admin
project_id=$(openstack project show demo -c id -f value)
auth_url=$(export | grep OS_AUTH_URL | awk -F'"' '{print $2}')
subnet_id=$(openstack subnet show private-subnet -c id -f value)
public_net_id=$(openstack network show public -c id -f value)
cat <<EOF > /etc/kubernetes/ingress-openstack.yaml
    kubeconfig: /etc/kubernetes/ingress-openstack.conf
    username: demo
    password: password
    project_id: ${project_id}
    auth_url: ${auth_url}/v3
    region: RegionOne
    subnet_id: ${subnet_id}
    fip_network: ${public_net_id}

2.4 Setting up the octavia-ingress-controller service

cat <<EOF > /etc/kubernetes/manifests/octavia-ingress-controller.yaml
apiVersion: v1
kind: Pod
  annotations: ""
    component: octavia-ingress-controller
    tier: control-plane
  name: octavia-ingress-controller
  namespace: kube-system
    - name: octavia-ingress-controller
      image: lingxiankong/openstack-ingress-controller:0.0.2
      imagePullPolicy: Always
        - /bin/octavia-ingress-controller
        - --config=/etc/kubernetes/ingress-openstack.yaml
      - mountPath: /etc/kubernetes/ingress-openstack.yaml
        name: ingressconfig
        readOnly: true
      - mountPath: /etc/kubernetes/ingress-openstack.conf
        name: kubeconfig
        readOnly: true
          cpu: 200m
  hostNetwork: true
  - hostPath:
      path: /etc/kubernetes/ingress-openstack.yaml
      type: FileOrCreate
    name: ingressconfig
  - hostPath:
      path: /etc/kubernetes/ingress-openstack.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

Wait until the octavia-ingress-controller static pod is up and running.

$ kubectl get pod --all-namespaces | grep octavia-ingress-controller
kube-system   octavia-ingress-controller-lingxian-k8s-master   1/1       Running   0          1m

3. Setting up HTTP load balancing with Ingress

3.1 Create a backend service

Create a simple service (echo hostname) that listens on a HTTP server on port 8080.

$ cat <<EOF | kc create -f -
apiVersion: apps/v1
kind: Deployment
  name: hostname-echo-deployment
  replicas: 1
      app: hostname-echo
        app: hostname-echo
        - image: "lingxiankong/alpine-test"
          imagePullPolicy: Always
          name: hostname-echo-container
            - containerPort: 8080
apiVersion: v1
kind: Service
  name: hostname-echo-svc
     -  port: 8080
        protocol: TCP
        targetPort: 8080
    app: hostname-echo
  type: NodePort
$ kubectl get svc
NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hostname-echo-svc   NodePort   <none>        8080:32066/TCP   33s

When you create a Service of type NodePort, Kubernetes makes your Service available on a randomly selected high port number (e.g. 32066) on all the nodes in your cluster. Generally, the Kubernetes nodes are not externally accessible by default, so creating this Service does not make your application accessible from the Internet. However, we can verify the service using its CLUSTER-IP on Kubernetes master node:

$ curl

To make your HTTP web server application publicly accessible, you need to create an Ingress resource.

3.2 Create an Ingress resource

The following command defines an Ingress resource that directs traffic that requests to the hostname-echo Service:

cat <<EOF | kc create -f -
apiVersion: extensions/v1beta1
kind: Ingress
  name: test-octavia-ingress
  annotations: "openstack"
  - host:
      - path: /hostname
          serviceName: hostname-echo-svc
          servicePort: 8080

Kubernetes creates an Ingress resource on your cluster. The octavia-ingress-controller service running in your cluster is responsible for creating/maintaining the corresponding resources in Octavia to route all external HTTP traffic (on port 80) to the hostname-echo NodePort Service you exposed.

Verify that the Ingress Resource has been created. Please note that the IP address for the Ingress Resource will not be defined right away (wait a few moments for the ADDRESS field to get populated):

$ kubectl get ing
NAME                   HOSTS            ADDRESS   PORTS     AGE
test-octavia-ingress             80        12s
$ # Wait until the ingress gets an IP address
$ kubectl get ing
NAME                   HOSTS            ADDRESS      PORTS     AGE
test-octavia-ingress   80        9m

For testing purposes, log in to a host that has a network connection with the OpenStack cloud. To do that, you need to update /etc/hosts file in the host to resolve to the Ingress IP address, then you should be able to access the backend service by sending HTTP request to the domain name specified in the Ingress Resource:

$ echo "" >> /etc/hosts
$ curl

Live demo

You can also see a live demo here or below.

Author Lingxian Kong will be heading up four sessions at the Vancouver Summit, check them out here.

The post Get started with the Octavia Ingress Controller for Kubernetes appeared first on Superuser.

by Lingxian Kong at May 08, 2018 04:26 PM

Red Hat Stack

A modern hybrid cloud platform for innovation: Containers on Cloud with Openshift on OpenStack

Market trends show that due to long application life-cycles and the high cost of change, enterprises will be dealing with a mix of bare-metal, virtualized, and containerized applications for many years to come. This is true even as greenfield investment moves to a more container-focused approach.

Red Hat® OpenStack® Platform provides a solution to the problem of managing large scale infrastructure which is not immediately solved by containers or the systems that orchestrate them.

In the OpenStack world, everything can be automated. If you want to provision a VM, a storage volume, a new subnet or a firewall rule, all these tasks can be achieved using an easy to use UI or with a command line interface, leveraging Openstack API’s. All these infrastructure needs might require a ticket, some internal processing, and could take weeks. Now such provisioning could all be done with a script or a playbook, and could be completely automated. 

The applications and workloads can specify cloud resources to be provisioned and spun up from a definition file. This enables new levels of provision-as-you-need-it. As as demand increases, the infrastructure resources can be easily scaled! Operational data and meters can trigger and orchestrate new infrastructure provisioning automatically when needed.

On the consumption side, it is no longer a developer ssh’ing into a server and manually deploying an application server. Now, it’s simply run a few OpenShift commands, select from a list of predefined applications, language runtimes, databases, and then just have those resources provisioned, on top of the target infrastructure that was automatically provisioned and configured.

Red Hat OpenShift Container Platform gives you the ability to define an application from a single YAML file. This makes it convenient for a developer to share with other developers, allowing them to  launch an exact copy of that application, make code changes, and share it back. This capability is only possible when you have automation at this level.

Infrastructure and application platforms resources are now exposed differently in an easy and consumable way, and the days when you needed to buy a server, manually connect it to the network and install runtimes and applications manually are now very much a thing of the past.

With Red Hat OpenShift Container Platform on Red Hat OpenStack Platform you get:

A WORKLOAD DRIVEN I.T. PLATFORM: The underlying infrastructure doesn’t matter from a developer perspective. Container platforms exist to ensure the apps are the main focus. As a developer I only care about the apps and I want to have a consistent experience, regardless of the underlying infrastructure platform. Openstack provides this to Openshift.

DEEP PLATFORM INTEGRATION: Networks (kuryr), services (ironic, barbican, octavia), storage (cinder, ceph), installation (openshift-ansible) are all engineered to work together to provide the tightest integrations across the stack, right down to bare metal. All are based in Linux® and engineered in the open source community for exceptional performance

PROGRAMMATIC SCALE-OUT: OpenStack is 100% API driven across the infrastructure software stack. Storage, networking, compute VM’s or even bare metal all deliver the ability to scale out rapidly and programmatically. With scale under workloads, growth is easy.

ACROSS ANY TYPE OF INFRASTRUCTURE: OpenStack can utilise bare metal for virtualization or for direct consumption. It can interact with network switches and storage directly to ensure hardware is put to work for the workloads it supports.

FULLY MANAGED: Red Hat CloudForms and Red Hat Ansible Automation provide common tooling across multiple providers. Ansible is Red Hat’s automation engine for everything, and it’s present under the hood in Red Hat CloudForms. With Red Hat Openstack Platform, Red Hat CloudForms is deeply integrated into both the overcloud, the undercloud, and the container platform on top. Full stack awareness means total control. And our Red Hat Cloud Suite bundle of products provides access to OpenStack and OpenShift, as well as an array of supporting technologies. Red Hat Satellite, Red Hat Virtualization, Red Hat Insights, and even Red Hat CloudForms are included!

A SOLID FOUNDATION: All Red Hat products are co-engineered with Red Hat Enterprise Linux at their core. Fixes happen fast and accurately as all components of the stack are in unison and developmental harmony. Whether issues might lie at the PaaS, IaaS or underlying Linux layer, Red Hat will support you all the way!

Red Hat Services can help you accelerate your journey to Hybrid Cloud adoption, and realize the most value of best-of-breed open source technology platforms such as OpenShift on top of Openstack. Want to learn more about how we can help? Feel free to reach out to me directly for any questions, Or download our Containers on Cloud datasheet

by Stephane Lefrere at May 08, 2018 02:50 PM

How integrated, open infrastructure meets tomorrow's computing demands

Cloud infrastructure is quickly becoming an integral business component for nearly every company.

by Thierry Carrez at May 08, 2018 07:01 AM

May 07, 2018

Michael Still

Adding oslo privsep to a new project, a worked example


You’ve decided that using sudo to run command lines as root is lame and that it is time to step up and do things properly. How do you do that? Well, here’s a simple guide to adding oslo privsep to your project!

In a previous post I showed you how to add a new method that ran with escalated permissions. However, that’s only helpful if you already have privsep added to your project. This post shows you how to do that thing to your favourite python project. In this case we’ll use OpenStack Cinder as a worked example.

Note that Cinder already uses privsep because of its use of os-brick, so the instructions below skip adding oslo.privsep to requirements.txt. If your project has never ever used privsep at all, you’ll need to add a line like this to requirements.txt:


For reference, this post is based on OpenStack review 566,479, which I wrote as an example of how to add privsep to a new project. If you’re after a complete worked example in a more complete form than this post then the review might be useful to you.

As a first step, let’s add the code we’d want to write to actually call something with escalated permissions. In the Cinder case I chose the cgroups throttling code for this example. So first off we’d need to create the privsep directory with the relevant helper code:

diff --git a/cinder/privsep/ b/cinder/privsep/
new file mode 100644
index 0000000..7f826a8
--- /dev/null
+++ b/cinder/privsep/
@@ -0,0 +1,32 @@
+# Copyright 2016 Red Hat, Inc
+# Copyright 2017 Rackspace Australia
+# Copyright 2018 Michael Still and Aptira
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+"""Setup privsep decorator."""
+from oslo_privsep import capabilities
+from oslo_privsep import priv_context
+sys_admin_pctxt = priv_context.PrivContext(
+ 'cinder',
+ cfg_section='cinder_sys_admin',
+ pypath=__name__ + '.sys_admin_pctxt',
+ capabilities=[capabilities.CAP_CHOWN,
+ capabilities.CAP_DAC_OVERRIDE,
+ capabilities.CAP_DAC_READ_SEARCH,
+ capabilities.CAP_FOWNER,
+ capabilities.CAP_NET_ADMIN,
+ capabilities.CAP_SYS_ADMIN],

This code defines the permissions that our context (called cinder_sys_admin in this case) has. These specific permissions in the example above should correlate with those that you’d get if you ran a command with sudo. There was a bit of back and forth about what permissions to use and how many contexts to have while we were implementing privsep in OpenStack Nova, but we’ll discuss those in a later post.

Next we need the code that actually does the privileged thing:

diff --git a/cinder/privsep/ b/cinder/privsep/
new file mode 100644
index 0000000..15d47e0
--- /dev/null
+++ b/cinder/privsep/
@@ -0,0 +1,35 @@
+# Copyright 2016 Red Hat, Inc
+# Copyright 2017 Rackspace Australia
+# Copyright 2018 Michael Still and Aptira
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+Helpers for cgroup related routines.
+from oslo_concurrency import processutils
+import cinder.privsep
+def cgroup_create(name):
+    processutils.execute('cgcreate', '-g', 'blkio:%s' % name)
+def cgroup_limit(name, rw, dev, bps):
+    processutils.execute('cgset', '-r',
+                         'blkio.throttle.%s_bps_device=%s %d' % (rw, dev, bps),
+                         name)

Here we just provide two methods which manipulate cgroups. That allows us to make this change to the throttling implementation in Cinder:

diff --git a/cinder/volume/ b/cinder/volume/
index 39cbbeb..3c6ddaa 100644
--- a/cinder/volume/
+++ b/cinder/volume/
@@ -22,6 +22,7 @@ from oslo_concurrency import processutils
 from oslo_log import log as logging
 from cinder import exception
+import cinder.privsep.cgroup
 from cinder import utils
@@ -65,8 +66,7 @@ class BlkioCgroup(Throttle):
         self.dstdevs = {}
-            utils.execute('cgcreate', '-g', 'blkio:%s' % self.cgroup,
-                          run_as_root=True)
+            cinder.privsep.cgroup.cgroup_create(self.cgroup)
         except processutils.ProcessExecutionError:
             LOG.error('Failed to create blkio cgroup \'%(name)s\'.',
                       {'name': cgroup_name})
@@ -81,8 +81,7 @@ class BlkioCgroup(Throttle):
     def _limit_bps(self, rw, dev, bps):
-            utils.execute('cgset', '-r', 'blkio.throttle.%s_bps_device=%s %d'
-                          % (rw, dev, bps), self.cgroup, run_as_root=True)
+            cinder.privsep.cgroup.cgroup_limit(self.cgroup, rw, dev, bps)
         except processutils.ProcessExecutionError:
             LOG.warning('Failed to setup blkio cgroup to throttle the '
                         'device \'%(device)s\'.', {'device': dev})

These last two snippets should be familiar from the previous post about pivsep in this series. Finally for the actual implementation of privsep, we need to make sure that rootwrap has permissions to start the privsep helper daemon. You’ll get one daemon per unique security context, but in this case we only have one of those so we’ll only need one rootwrap entry. Note that I also remove the previous rootwrap entries for cgset and cglimit while I’m here.

diff --git a/etc/cinder/rootwrap.d/volume.filters b/etc/cinder/rootwrap.d/volume.filters
index abc1517..d2d1720 100644
--- a/etc/cinder/rootwrap.d/volume.filters
+++ b/etc/cinder/rootwrap.d/volume.filters
@@ -43,6 +43,10 @@ lvdisplay4: EnvFilter, env, root, LC_ALL=C, LVM_SYSTEM_DIR=, LVM_SUPPRESS_FD_WAR
 # This line ties the superuser privs with the config files, context name,
 # and (implicitly) the actual python code invoked.
 privsep-rootwrap: RegExpFilter, privsep-helper, root, privsep-helper, --config-file, /etc/(?!\.\.).*, --privsep_context, os_brick.privileged.default, --privsep_sock_path, /tmp/.*
+# Privsep calls within cinder iteself
+privsep-rootwrap-sys_admin: RegExpFilter, privsep-helper, root, privsep-helper, --config-file, /etc/(?!\.\.).*, --privsep_context, cinder.privsep.sys_admin_pctxt, --privsep_sock_path, /tmp/.*
 # The following and any cinder/brick/* entries should all be obsoleted
 # by privsep, and may be removed once the os-brick version requirement
 # is updated appropriately.
@@ -93,8 +97,6 @@ ionice_1: ChainingRegExpFilter, ionice, root, ionice, -c[0-3], -n[0-7]
 ionice_2: ChainingRegExpFilter, ionice, root, ionice, -c[0-3]
 # cinder/volume/ setup_blkio_cgroup()
-cgcreate: CommandFilter, cgcreate, root
-cgset: CommandFilter, cgset, root
 cgexec: ChainingRegExpFilter, cgexec, root, cgexec, -g, blkio:\S+
 # cinder/volume/

And because we’re not bad people we’d of course write a release note about the changes we’ve made…

diff --git a/releasenotes/notes/privsep-rocky-35bdfe70ed62a826.yaml b/releasenotes/notes/privsep-rocky-35bdfe70ed62a826.yaml
new file mode 100644
index 0000000..e78fb00
--- /dev/null
+++ b/releasenotes/notes/privsep-rocky-35bdfe70ed62a826.yaml
@@ -0,0 +1,13 @@
+    Privsep transitions. Cinder is transitioning from using the older style
+    rootwrap privilege escalation path to the new style Oslo privsep path.
+    This should improve performance and security of Nova in the long term.
+  - |
+    privsep daemons are now started by Cinder when required. These daemons can
+    be started via rootwrap if required. rootwrap configs therefore need to
+    be updated to include new privsep daemon invocations.
+  - |
+    The following commands are no longer required to be listed in your rootwrap
+    configuration: cgcreate; and cgset.

This code will now work. However, we’ve left out one critical piece of the puzzle — testing. If this code was uploaded like this, it would fail in the OpenStack gate, even though it probably passed on your desktop. This is because many of the gate jobs are setup in such a way that they can’t run rootwrapped commands, which in this case means that the rootwrap daemon won’t be able to start.

I found this quite confusing in Nova when I was implementing things and had missed a step. So I wrote a simple test fixture that warns me when I am being silly:

diff --git a/cinder/ b/cinder/
index c8c9e6c..a49cedb 100644
--- a/cinder/
+++ b/cinder/
@@ -302,6 +302,9 @@ class TestCase(testtools.TestCase):
         tpool._nthreads = 20
+        # NOTE(mikal): make sure we don't load a privsep helper accidentally
+        self.useFixture(cinder_fixtures.PrivsepNoHelperFixture())
     def _restore_obj_registry(self):
         objects_base.CinderObjectRegistry._registry._obj_classes = \
diff --git a/cinder/tests/ b/cinder/tests/
index 6e275a7..79e0b73 100644
--- a/cinder/tests/
+++ b/cinder/tests/
@@ -1,4 +1,6 @@
 # Copyright 2016 IBM Corp.
+# Copyright 2017 Rackspace Australia
+# Copyright 2018 Michael Still and Aptira
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
 #    not use this file except in compliance with the License. You may obtain
@@ -21,6 +23,7 @@ import os
 import warnings
 import fixtures
+from oslo_privsep import daemon as privsep_daemon
 _TRUE_VALUES = ('True', 'true', '1', 'yes')
@@ -131,3 +134,29 @@ class WarningsFixture(fixtures.Fixture):
                     ' This key is deprecated. Please update your policy '
                     'file to use the standard policy values.')
+class UnHelperfulClientChannel(privsep_daemon._ClientChannel):
+    def __init__(self, context):
+        raise Exception('You have attempted to start a privsep helper. '
+                        'This is not allowed in the gate, and '
+                        'indicates a failure to have mocked your tests.')
+class PrivsepNoHelperFixture(fixtures.Fixture):
+    """A fixture to catch failures to mock privsep's rootwrap helper.
+    If you fail to mock away a privsep'd method in a unit test, then
+    you may well end up accidentally running the privsep rootwrap
+    helper. This will fail in the gate, but it fails in a way which
+    doesn't identify which test is missing a mock. Instead, we
+    raise an exception so that you at least know where you've missed
+    something.
+    """
+    def setUp(self):
+        super(PrivsepNoHelperFixture, self).setUp()
+        self.useFixture(fixtures.MonkeyPatch(
+            'oslo_privsep.daemon.RootwrapClientChannel',
+            UnHelperfulClientChannel))

Now if you fail to mock a privsep’ed call, then you’ll get something like this:

Failed 1 tests - output below:


Captured traceback:
    Traceback (most recent call last):
      File "/srv/src/openstack/cinder/.tox/py27/local/lib/python2.7/site-packages/mock/", line 1305, in patched
        return func(*args, **keywargs)
      File "cinder/tests/unit/", line 66, in test_BlkioCgroup
        throttle = throttling.BlkioCgroup(1024, 'fake_group')
      File "cinder/volume/", line 69, in __init__
      File "/srv/src/openstack/cinder/.tox/py27/local/lib/python2.7/site-packages/oslo_privsep/", line 206, in _wrap
      File "/srv/src/openstack/cinder/.tox/py27/local/lib/python2.7/site-packages/oslo_privsep/", line 217, in start
        channel = daemon.RootwrapClientChannel(context=self)
      File "cinder/tests/", line 141, in __init__
        raise Exception('You have attempted to start a privsep helper. '
    Exception: You have attempted to start a privsep helper. This is not allowed in the gate, and indicates a failure to have mocked your tests.

The last bit is the most important. The fixture we installed has detected that you’ve failed to mock a privsep’ed call and has informed you. So, the last step of all is fixing our tests. This normally involves changing where we mock, as many unit tests just lazily mock the execute() call. I try to be more granular than that. Here’s what that looked like in this throttling case:

diff --git a/cinder/tests/unit/ b/cinder/tests/unit/
index 82e2645..edbc2d9 100644
--- a/cinder/tests/unit/
+++ b/cinder/tests/unit/
@@ -29,7 +29,9 @@ class ThrottleTestCase(test.TestCase):
             self.assertEqual([], cmd['prefix'])
     @mock.patch.object(utils, 'get_blkdev_major_minor')
-    def test_BlkioCgroup(self, mock_major_minor):
+    @mock.patch('cinder.privsep.cgroup.cgroup_create')
+    @mock.patch('cinder.privsep.cgroup.cgroup_limit')
+    def test_BlkioCgroup(self, mock_limit, mock_create, mock_major_minor):
         def fake_get_blkdev_major_minor(path):
             return {'src_volume1': "253:0", 'dst_volume1': "253:1",
@@ -37,38 +39,25 @@ class ThrottleTestCase(test.TestCase):
         mock_major_minor.side_effect = fake_get_blkdev_major_minor
-        self.exec_cnt = 0
+        throttle = throttling.BlkioCgroup(1024, 'fake_group')
+        with throttle.subcommand('src_volume1', 'dst_volume1') as cmd:
+            self.assertEqual(['cgexec', '-g', 'blkio:fake_group'],
+                             cmd['prefix'])
-        def fake_execute(*cmd, **kwargs):
-            cmd_set = ['cgset', '-r',
-                       'blkio.throttle.%s_bps_device=%s %d', 'fake_group']
-            set_order = [None,
-                         ('read', '253:0', 1024),
-                         ('write', '253:1', 1024),
-                         # a nested job starts; bps limit are set to the half
-                         ('read', '253:0', 512),
-                         ('read', '253:2', 512),
-                         ('write', '253:1', 512),
-                         ('write', '253:3', 512),
-                         # a nested job ends; bps limit is resumed
-                         ('read', '253:0', 1024),
-                         ('write', '253:1', 1024)]
-            if set_order[self.exec_cnt] is None:
-                self.assertEqual(('cgcreate', '-g', 'blkio:fake_group'), cmd)
-            else:
-                cmd_set[2] %= set_order[self.exec_cnt]
-                self.assertEqual(tuple(cmd_set), cmd)
-            self.exec_cnt += 1
-        with mock.patch.object(utils, 'execute', side_effect=fake_execute):
-            throttle = throttling.BlkioCgroup(1024, 'fake_group')
-            with throttle.subcommand('src_volume1', 'dst_volume1') as cmd:
+            # a nested job
+            with throttle.subcommand('src_volume2', 'dst_volume2') as cmd:
                 self.assertEqual(['cgexec', '-g', 'blkio:fake_group'],
-                # a nested job
-                with throttle.subcommand('src_volume2', 'dst_volume2') as cmd:
-                    self.assertEqual(['cgexec', '-g', 'blkio:fake_group'],
-                                     cmd['prefix'])
+        mock_create.assert_has_calls(['fake_group')])
+        mock_limit.assert_has_calls([
+  'fake_group', 'read', '253:0', 1024),
+  'fake_group', 'write', '253:1', 1024),
+            # a nested job starts; bps limit are set to the half
+  'fake_group', 'read', '253:0', 512),
+  'fake_group', 'read', '253:2', 512),
+  'fake_group', 'write', '253:1', 512),
+  'fake_group', 'write', '253:3', 512),
+            # a nested job ends; bps limit is resumed
+  'fake_group', 'read', '253:0', 1024),
+  'fake_group', 'write', '253:1', 1024)])

…and we’re done. This post has been pretty long so I am going to stop here for now. However, hopefully I’ve demonstrated that its actually not that hard to implement privsep in a project, even with some slight testing polish.


The post Adding oslo privsep to a new project, a worked example appeared first on Made by Mikal.

by mikal at May 07, 2018 11:27 PM

OpenStack Superuser

What to expect from your first OpenStack Summit

If you’re new to OpenStack and thinking about coming to the Vancouver Summit,  you’re in good company: around 50 – 60 percent of OpenStack summit attendees are first-timers.

Here, OpenStack community leaders share tips and need-to-knows for the Summit-curious.

The why and the what

“The OpenStack summit is really unique because the entire community comes together from all over the world, people from sixty countries people from hundreds of companies,” says Jonathan Bryce, OSF executive director. “They come together to collaborate about what to build on OpenStack how to deploy it and where we need to take the software and the community.”

The Forum is where we discuss the future of the OpenStack software in a cross-community manner,” says Thierry Carrez, VP of engineering at OSF. “We don’t do design behind closed doors. We’re really into including as many people as we can and the forum is where we achieve that promise.”

“If you work on a project just inside your company, everyone kind of thinks the same way,” says Major Hayden, principal software engineer at Red Hat. “But when you come to an OpenStack Summit you might find that you build for one use case and someone shows up and says, ‘Hey we use that for something completely different,’ it’s so awesome to be able to hear that not only your stuff is being used but that it’s being used for a use case you never even thought about.”

Who you’ll meet, what to bring

“There are very few women that are in technology in general there are even less black women, to know that we’re out there and that we’re into like seriously technical things is empowering,” says Treva N. Williams, author of  an OpenStack course at LinuxAcademy.

“Don’t take your computer, take some paper to take notes but don’t spend time on your emails, says Emilien Macchi of the OpenStack Technical Committee. Focus a few days on talking with people and working with them.

“Software is hard, but if you get to know the people then the software becomes easier,” says Davanum Srinivas, also a TC member. “This is a place where there are many people who are like you so get to know people, build your network and make friends.”

Join us May 21-24, 2018 in Vancouver for the next Summit.  More tips for newbies from Summit veterans here and here.

Check out all the interviews below.

The post What to expect from your first OpenStack Summit appeared first on Superuser.

by Superuser at May 07, 2018 03:23 PM

SUSE Conversations

An Experiment with Gnocchi – (the Database) – Part 2

In Part 1, I introduced the problem of metrics aggregations for Ceilometer in OpenStack and how Gnocchi (a time-series database) tries to solve it in a different way. I argued about possibility of offloading work to (Nvida Quadro K620) GPU. This part summarizes the iterations applied to the kernel (piece of CUDA code, that executes […]

The post An Experiment with Gnocchi – (the Database) – Part 2 appeared first on SUSE Communities.

by Sumit Jamgade at May 07, 2018 08:41 AM

May 06, 2018


Virtlet: run VMs as Kubernetes pods

Virtlet makes it possible to run VMs on Kubernetes clusters as if they were plain pods, enabling you to use standard kubectl commands to manage them.

by Guest Post at May 06, 2018 01:42 AM

May 05, 2018

Jian Li

openstack keystone api初始化流程(wsgi/PasteDeploy)

在N版本以前,keystone的api服务可以使用httpd方式进行部署,也可以使用python eventlet方式部署,从N版本开始,eventlet这种方式已经从keystone中移除。keystone使用PasteDeploy来实现WSGI服务,关于wsgi,可以参考理解 WSGI 框架,关于Paste+PasteDeploy+Routes+WebOb,可以参考通过demo学习OpenStack开发–API服务(2)

如下,是keystone v3 api初始化流程的时序图,v3版本中,已经不区分admin api和public api


简单解释下pipline, 如下paste文件,对于v3版本的api请求,根据[composite:admin], 请求被map到api_v3这个pipeline处理, 在api_v3这个pipeline中([pipeline:api_v3]),前面cors sizelimit ... s3_extension都是keystone实现的各种中间件,用于对client请求进行过滤封装等处理,根据名字你可以大致猜出来这些中间件都做了什么,最后service_v3一个是app,即为keystone真正处理client请求的api服务。也就是说,client发出的http的请求会穿过前面一系列的中间件,最后才会到达keystone代码真正处理请求

pipeline = cors sizelimit http_proxy_to_wsgi osprofiler url_normalize request_id build_auth_context token_auth json_body ec2_extension_v3 s3_extension service_v3  

/v3 = api_v3


#pipeline[-1]即为上面的app service_v3
context.app_context = self.get_context(APP, pipeline[-1], global_conf)
context.filter_contexts = [self.get_context(FILTER, name, global_conf) for name in pipeline[:-1]] 

    #初始化keystone app
    app = context.app_context.create()
    filters = [c.create() for c in context.filter_contexts]
    #所有中间件反序排列,即为:s3_extension, ec2_extension_v3, json_body, token_auth,...
    for filter in filters
        app = filter(app)
    return app

上面不好理解的话,我们用洋葱来举个例子,假如洋葱的心代表keystone服务(service_v3),心外面的每一层洋葱皮都代表一个中间件。这样,keystone api的初始化过程就是洋葱的生成过程,首先反序排列中间件filters.reverse(), 这样最靠近洋葱心(service_v3)的中间件最先被封装,一直封装到最外层中间件cors。那client的请求处理,就是剥洋葱的过程,最先从最外面中间件开始处理(cors),我们从返回的urlmap {(None, '/v3'): <oslo_middleware.cors.CORS object at 0x5531e10>中也可以看出,之后经过一层一层处理,一直到keystone服务。

时序图绘制使用的这个websequencediagrams, 本博客的时序图完整代码在这里

May 05, 2018 12:00 AM

May 04, 2018


The RDO Community Represents at RedHat Summit, May 8-10

Over the past few weeks we’ve been gearing up for Red Hat Summit and now it’s almost here! We hope to see you onsite — there are so many great talks, events, and networking opportunities to take advantage of. From panels to general sessions to hands-on labs, chances are you’re going to have a hard time choosing which sessions to attend!

We’re particularly excited about the below talks, but the full schedule of talks related to RDO, RHOSP, TripleO, and Ceph is over here.

Once you’re sessioned-out, come swing by the RDO booth, shared with ManageIQ and Ceph to see our newly updated hardware demo.

OpenStack use cases: How businesses succeed with OpenStack

Have you ever wondered just how the world’s largest companies are using Red Hat OpenStack Platform?

In this session, we’ll look at some of the most interesting use cases from Red Hat customers around the world, and give you some insight into how they achieved their technical and digital successes. You’ll learn how top organizations have used OpenStack to deliver unprecedented value.

Date:Tuesday, May 8
Time:3:30 PM – 4:15 PM
Location:Moscone West – 2007

Red Hat OpenStack Platform: The road ahead

OpenStack has reached a maturity level confirmed by wide industry adoption and the amazing number of active production deployments, with Red Hat a lead contributor to the project. In this session, we’ll share where we are investing for upcoming releases.

Date:Tuesday, May 8
Time:4:30 PM – 5:15 PM
Location:Moscone West – 2007

Production-ready NFV at Telecom Italia (TIM)

Telecom Italia (TIM) is the first large telco in Italy to deploy OpenStack into production. TIM chose to deploy a NFV solution based on Red Hat OpenStack Platform for its cloud datacenter and put critical VNFs—such as vIMS and vEPC—into production for its core business services. In this session, learn how TIM worked with Red Hat Consulting, as well as Ericsson as VNF vendor and Accenture as system integration, to set up an end-to-end NFV environment that matched its requirements with complex features like DPDK, completely automated.

Date:Wednesday, May 9
Time:11:45 AM – 12:30 PM
Location:Moscone West – 2002

Scalable application platform on Ceph, OpenStack and Ansible

How do you take a Ceph environment providing Cinder block storage to Openstack from a handful of nodes in a PoC environment all the way up to an 800+ node production environment, while serving live applications? In this session we will have two customers talking about how they did this and lessons learned! At Fidelity, they learned a lot about scaling hardware, tuning Ceph parameters, and handling version upgrades (using Ansible automation!). Solera Holdings Inc committed to modernizing the way Applications are developed and deployed with the need for highly performant, redundant and cost effective Object-Storage grew tremendously. After an successful PoC with Ceph Storage, Red Hat was chosen as a solution partner due to their excellence in customer experience and support as well as expertise in Ansible, as Solera iwill automate networking equipment (Fabric, Firewalls, Load-balancers) .

In addition to committing to reducing expensive enterprise SAN storage Solera also decided to commit to a new Virtualization strategy and building up a new IaaS to tackle challenges such as DBaaS and leveraging its newly built SDS backend for OpenStack while using the new SDS capabilities via iscsi to meet existing storage demands on VMware.

Solera will share why they chose RedHat as partner, how it has impacted and benefited Developers and DevOps Engineers alike and where the road will be taking us. Come to this session to hear about how both Fidelity and Solera Holdings Inc did it and what benefits were learned along the way!

Date:Thursday, May 10
Time:1:00 PM – 1:45 PM
Location:Moscone West – 2007

Building and maintaining open source communities

Being successful in creating an open source community requires planning, measurements, and clear goals. However, it’s an investment that can pay off tenfold when people come together around a shared vision. Who are we targeting, how can we achieve these goals, and why does it matter to the bigger business strategy?

In this panel you’ll hear from Amye Scarvada (Gluster), Jason Hibbets (, Greg DeKoenigsberg (Ansible), and Leslie Hawthorn (Open source and standards, Red Hat) as they share first-hand experiences about how open source communities have directly attributed to the success of a product, as well as best practices to build and maintain these communities. It will be moderated by Mary Thengvall (Persea Consulting), who after many years of building community programs is now working with companies who are building out a developer relations strategy.

Date:Thursday, May 10
Time:2:00 PM – 2:45 PM
Location:Moscone West – 2007

by Mary Thengvall at May 04, 2018 09:02 PM

Ben Nemec

OpenStack Virtual Baremetal on a Public Cloud


At long last, OVB has reached the goal we set for it way back when it was more idea than reality. The plan was to come up with a system where we could test baremetal deployment against OpenStack instances in an arbitrary cloud. That would allow something like TripleO CI to scale, for all practical purposes, infinitely. For years this wasn't possible because OVB required some configuration and/or hacks that were not appropriate for production clouds, which limited it to use in specific CI-oriented clouds. Theoretically it has been possible to use OVB on normal clouds for a couple of years now, but in practice public clouds were either running OpenStack versions that were too old or didn't expose the necessary features. Fortunately, this is no longer true.

Enter Vexxhost (I should note that I have no affiliation with Vexxhost). Generally speaking, the OVB requirements are:

  • OpenStack Mitaka or newer
  • Heat
  • Access to the Neutron port-security extension
  • The ability to upload images

That may not be an exhaustive list, but it covers the things that frequently blocked me from running OVB on a cloud. Vexxhost ticks all the boxes.


First, the OVB on Public Cloud demo video. As I mentioned in the video, there are some details that I glossed over in the interest of time (the video is nearly 20 minutes as it is). Here's a more complete explanation of those:

  • Quota. The default quota when you create an account on Vexxhost is somewhat restrictive for OVB use. You can fit an OVB environment into it, but you have to use the absolute minimum sizes for each instance. In some cases this may not work well. As I recall, the main restriction was CPU cores. To address this, just open a support ticket and explain what you're doing. I got my cpu core limit raised to 50, which enabled any nonha configuration I wanted. If you're doing an ha deployment you may need to have some other quota items raised, but you'll have to do the math on that.
  • Performance. Performance was a mixed bag, which is one of the downsides of running in a general purpose cloud. Some of my test runs were as fast as my dedicated local cloud, others were much slower.
  • Flavor disk sizes. As noted in the video, some of the Vexxhost flavors have a disk size of 0. Initially I thought this meant they could only be used with boot from volume, but fortunately it turns out you can still boot instances from ephemeral storage. This is important because on a public cloud you need to be able to nova rebuild the baremetal instances between deployments so they can PXE boot again. Nova can't currently rebuild instances booted from volume, so make sure to avoid doing that with OVB.

    Because of the 0 disk size, the virtual size of the images you boot will determine how much storage your instances get. By default, the CentOS 7 image only has 8 GB. That is not enough for a TripleO undercloud, although it's fine for the BMC. This can be worked around by either using qemu-img to resize a stock CentOS image and upload it to Glance, or by snapshotting an instance booted from a flavor that has a larger disk size. I did the latter because it allowed me to make some customizations (injecting ssh keys, setting a root password, etc.) to my base undercloud image. Either should work though.

    The original OVB ipxe-boot image has a similar problem. There is now an ipxe-boot-41 image with a virtual size of 41 GB that should provide enough storage for most deployment scenarios. If it doesn't, this image can also be resized with qemu-img.

    Another impact of this configuration is that the nodes.json created by build-nodes-json may have an incorrect disk size. It currently can't determine the size of a disk in an instance booted from a flavor with a 0 disk size. There are two options to deal with this. First, nodes.json can be hand-edited before importing to Ironic. Second, introspection can be run on node import, which is what I did in the video. It's a bit cleaner and demonstrates that introspection functionality works as expected. Introspection will look at the instance details from the inside so it can determine the correct disk size.

  • Cost. Obviously public cloud resources are not free. If you created the environment I used in my demo and left it running for a full month, at the current Vexxhost prices it would cost around $270 US. That's just for the instances, there will also be some small cost for image storage and bandwidth, although in my experience those would be negligible compared to the per-instance costs. That's not pocket change, but compared to the cost for purchasing and administering your own cloud it may be worth it.
  • [Edit 2018-5-7] Baremetal rebuilds. This is mentioned in the video, but it's important so I wanted to reiterate it here too. You must rebuild the baremetal instances between deployments in a public cloud environment. There is a rebuild-baremetal script provided that can automate this for you. In the case of the demo environment, it would have been called like this: rebuild-baremetal 2 baremetal demo. Note that this is run using the host cloud credentials, not the undercloud.

by bnemec at May 04, 2018 06:04 PM

OpenStack Superuser

Meet the newest members of the OpenStack Technical Committee

The OpenStack Technical Committee provides technical leadership for OpenStack as a whole. Responsibilities include enforcing OpenStack ideals (such as openness, transparency, commonality, integration and quality), deciding on issues that impact multiple programs, providing an ultimate appeals board for technical decisions and general oversight. It’s a fully-elected Committee that represents contributors to the project.

Made up of 13 directly elected members, the community hits refresh with partial elections every six months. There were 10 candidates for the seven spots open this round. While some have served on the TC before, we wanted to introduce them to the growing OpenStack community.

Here are the newly-elected members, in alphabetical order, with excerpts from their candidacy statements:

Zane Bitter, who works at Red Hat, has been working on OpenStack for six years, including as core contributor to and a former PTL for orchestration project Heat. “There are important discussions ahead — both within the technical community and between the TC and the Board — about where to draw the boundaries of OpenStack; the more user viewpoints that are represented, the better the result will be. We don’t get as much feedback from developers of cloud-aware applications as we do from other end users, because in many cases OpenStack doesn’t yet meet their minimum requirements. That is the gap I am hoping to bridge. If we succeed, OpenStack will not only gain a lot more users,but I expect more users will become contributors.”

Thierry Carrez is the vice-president of engineering at the OpenStack Foundation, helping ensure the long-term health of the OpenStack upstream open source project. A long-time elected member of the TC, he has been a Release Manager for the OpenStack project since its inception, coordinating the effort and facilitating collaboration between contributors. He previously served as elected chair of the OpenStack Technical Committee, which is in charge of the technical direction of the project.

Chris Dent, staff engineer at VMware who has been involved with OpenStack since 2014. (He also finds time to keep up a fantastic blog on the TC and the OSF community.) “The growth of projects under the OpenStack Foundation umbrella will present opportunities and challenges. We’ll be able to deal with those most effectively by having good communication hygiene: over communicating in a written and discoverable fashion. Changes in the shape of the community will impact the role of the TC and its members. The TC has been something of a high-level judiciary within the OpenStack technical community but increasingly will need to take on a role as a representative of the community that develops what has traditionally been known as “OpenStack” to the other nearby communities that are also now ‘OpenStack.’

Graham Hayes, who currently works at the Verizon Cloud Platform, has been contributing to OpenStack since the Havana cycle, mainly in Designate. “When it comes to pushing forward the TC vision, I think the community has made great steps forward to realizing it, on all but one section. We engage with groups like the CNCF, and specifically Kubernetes and help drive OpenStack adoption with first class support for the OpenStack Cloud Provider, the Cinder Container Storage Interface and other projects. We still need to find a way to show the world what a top tier private open source infrastructure of components like OpenStack, Kubernetes, Cloud Foundry or OpenShift looks like and helping companies understand why this is the way forward for their infrastructure.”

Sean McGinnis, a re-elected member of the TC, is a principal architect with Huawei and OSF community member since the Icehouse release. “Outside of specific governance proposals, I’ve been working on getting involved in the operators community by attending the last few Ops Meetups to be able to get face to face with more of the folks actually using OpenStack. I’ve found it very valuable to hear directly about what kinds of issues are being run into and what kinds of things we might be able to change on the development side to make things better.

Mohammed Naser, founder CEO at VEXXHOST, a current Superuser Award nominee, has been an OSF community member since 2013. “I strongly believe that we have the most solid fundamental infrastructure software at the moment. However, I think that the focus should increase in working with other external communities which depend on infrastructure to increase our implementations. For example, working with Kubernetes in order to strengthen the cloud providers and perhaps get even more integration features such as auto-scaling. This will solidify OpenStack as the best set of infrastructure services in the market.”

Davanum Srinivas, a principal architect with Huawei, and returning TC member will be a familiar face to many in the OSF community.”While I have been on the TC long enough to be consider turning over the leadership role to other folks, I do feel like I have some unfinished business that I’d like to concentrate on for the next year. I spent quite a bit of time with things that use OpenStack and have a new appreciation for the challenges in the field and in practice. Just for example the need for a “validator” by our partners in CF illustrates the kind of challenges our users face. I would like to spend some time working / tackling these kinds of issues.”

So who gets a voice in these elections? All OSF individual members who authored a change to the official project teams repositories during the release time frame (for this round, it was previous two release cycles.) More on the process here, including how to get involved for next time.

“Having a good group of candidates helps engage the community in our democratic process,” says Foundation staffer Kendall Nelson, one of four election officials. “Thank you to all who voted and who encouraged others to vote. We need to ensure your voices are heard!”

The post Meet the newest members of the OpenStack Technical Committee appeared first on Superuser.

by Superuser at May 04, 2018 02:53 PM


Machine Learning and AI in the Datacenter and How It Will Affect You

Managing a datacenter and software development is a mass of complications. One way of mitigating problems is to use machine learning and AI techniques to help with management, and optimization.

by Nick Chase at May 04, 2018 01:19 PM


Consuming Kolla Tempest container image for running Tempest tests

Kolla project provides a docker container image for Tempest.

The provided container image is available in two formats for centos: centos-binary-tempest and centos-binary-source.

The RDO community rebuilds the container image in centos-binary format and pushes it to and to

The Tempest container image contains openstack-tempest and all available Tempest plugins in it.

The benefit of running Tempest tests from Tempest container is that, we do not need to install any Tempest package or Tempest plugin on the deployed cloud and keep the environment safe from dependency mismatch and updates.

In TripleO CI, we run Tempest tests using Tempest container images in tripleo-ci-centos-7-undercloud-containers job using featureset027 set.

We can consume the same image for running Tempest tests locally in TripleO deployment:

  • Follow this link for installing containerized undercloud.

    Note: At step 5 in the above link, open undercloud.conf in an editor and set

    enable_tempest = true.

    It will pull the tempest container image on the undercloud.

  • If tempest container is not available on the undercloud, we pull the image from Dockerhub.

    $ sudo docker pull

  • Create two directories: container_tempest and tempest_workspace and copy stackrc, overcloudrc, tempest-deployer-input.conf, whitelist and blacklist related files to container_tempest. These files should be copied from undercloud to the container. Below commands do the same:

    $ mkdir container_tempest tempest_workspace

    $ cp stackrc overcloudrc tempest-deployer-input.conf whitelist.txt blacklist.txt container_tempest

  • Creating alias for running tempest within a container and with mounted container_tempest:/home/stack and tempest:/home/stack

    $ alias docker-tempest="sudo docker run -i \
    -v container_tempest:/home/stack \
    -v tempest:/home/stack \ \

  • Create tempest workspace using docker-tempest alias

    $ docker-tempest tempest init /home/stack/tempest

  • List tempest plugins installed within tempest container

    $ docker-tempest tempest list-plugins

  • Generate tempest.conf using discover-tempest-config

    Note: If tempest tests are running against undercloud then:

    $ source stackrc
    $ docker-tempest discover-tempest-config --create \
    --out /home/stack/tempest/etc/tempest.conf

    Note: If tempest tests are running against overcloud then:

    $ source overcloudrc
    $ docker-tempest discover-tempest-config --create \
    --out /home/stack/tempest/etc/tempest.conf \
    --deployer-input /home/stack/tempest-deployer-input.conf

  • Running tempest tests

    $ docker-tempest tempest run --workspace tempest \
    -c /home/stack/tempest/etc/tempest.conf \
    -r <tempest test regex> --subunit

    In the above command:

    • --workspace : To use tempest workspace
    • -c : Use the tempest.conf file
    • -r : To run tempest tests
    • --subunit: to generate tempest tests results subunit stream in v2 format

Once tests are finished, we can find the test output in /home/stack/tempest folder.

Thanks to Kolla team, Emilien, Wes, Arx, Martin, Luigi, Andrea, Ghanshyam, Alex, Sagi, Gabriel and RDO team for helping me in getting things in place.

Happy Hacking!

by chkumar246 at May 04, 2018 09:47 AM

Running Tempest tests against a TripleO Undercloud

Tempest is the integration test suite used to validate any deployed OpenStack cloud.

TripleO undercloud is the all-in-one OpenStack installation that includes components for provisioning and managing the OpenStack nodes that form your OpenStack environment (the overcloud).

For validating undercloud using Tempest, Follow the below steps:

  • Using tripleo-quickstart:

    • Follow this link to provision a libvirt guest environment through tripleo-quickstart
    • Deploy the undercloud and run Tempest tests on undercloud against undercloud

      $ bash -R master --no-clone --tags all \
      --nodes config/nodes/1ctlr_1comp.yml \
      -I --teardown none -p quickstart-extras-undercloud.yml \
      --extra-vars test_ping=False \
      --extra-vars tempest_undercloud=True \
      --extra-vars tempest_overcloud=False \
      --extra-vars run_tempest=True \
      --extra-vars test_white_regex='tempest.api.identity|tempest.api.compute' \

      The above command will:

      • Deploy an undercloud
      • Generate script in /home/stack folder
      • Run test_white_regex tempest tests
      • Store all the results in /home/stack/tempest folder.
  • Running Tempest tests manually on undercloud:

    • Deploy the undercloud manually by following this link and then ssh into undercloud.
    • Install openstack-tempest rpm on undercloud

      $ sudo yum -y install openstack-tempest

    • Source stackrc on undercloud

      $ source stackrc

    • Append Identity API version in $OS_AUTH_URL

      $OS_AUTH_URL defined in stackrc does not contain the Identity API version, what will lead to a failure while generating tempest.conf using python-tempestconf. In order to fix the above issue, we need to append the API version to the OS_AUTH_URL environment variable and export it.


    • Create the Tempest workspace

      $ tempest init <tempest_workspace>

    • Generate Tempest configuration using python-tempestconf

      $ cd <path to the tempest_workspace>

      $ discover-tempest-config --create --out etc/tempest.conf

      The above command will generate tempest.conf in /etc/ directory.

    • Now we are all set to run Tempest tests. Run the following command to run Tempest tests

      $ tempest run -r '(tempest.api.identity|tempest.api.compute)' --subunit

      The above command will:

      • Run tempest.api.identity and tempest.api.compute tests
      • All the Tempest test subunit results in v2 format will be stored in .stestr directory under Tempest workspace.
    • Use subunit2html command to generate results in html format

      $ sudo yum -y install python-subunit

      $ subunit2html <path to tempest workspace>/.stestr/0 tempest.html

And we are done with running Tempest on undercloud.

Currently tripleo-ci-centos-7-undercloud-oooq job is running Tempest tests on undercloud in TripleO CI using featureset003

Thanks to Emilien, Enrique, Wes, Arx, Martin, Luigi, Alex, Sagi, Gabriel and RDO team for helping me in getting things in place.

Happy Hacking!

by chkumar246 at May 04, 2018 08:22 AM

May 03, 2018

Michael Still

How to make a privileged call with oslo privsep


Once you’ve added oslo privsep to your project, how do you make a privileged call? Its actually really easy to do. In this post I will assume you already have privsep running for your project, which at the time of writing limits you to OpenStack Nova in the OpenStack universe.

The first step is to write the code that will run with escalated permissions. In Nova, we have chosen to only have one set of escalated permissions, so its easy to decide which set to use. I’ll document how we reached that decision and alternative approaches in another post.

In Nova, all code that runs with escalated permissions is in the nova/privsep directory, which is a pattern I’d like to see repeated in other projects. This is partially because privsep maintains a whitelist of methods that are allowed to be run this way, but its also because it makes it very obvious to callers that the code being called is special in some way.

Let’s assume that we’re going to add a simple method which manipulates the filesystem of a hypervisor node as root. We’d write a method like this in a file inside nova/privsep:

import nova.privsep


def update_motd(message):
    with open('/etc/motd', 'w') as f:

This method updates /etc/motd, which is the text which is displayed when a user interactively logs into the hypervisor node. “motd” stands for “message of the day” by the way. Here we just pass a new message of the day which clobbers the old value in the file.

The important thing is that entrypoint decorator at the start of the method. That’s how privsep decides to run this method with escalated permissions, and decides what permissions to use. In Nova at the moment we only have one set of escalated permissions, which we called sys_admin_pctxt because we’re artists. I’ll discuss in a later post how we came to that decision and what the other options were.

We can then call this method from anywhere else in Nova like this:

import nova.privsep.motd


nova.privsep.motd('This node is currently idle')

Note that we do imports for privsep code slightly differently. We always import the entire path, instead of creating a shortcut to just the module we’re using. In other words, we don’t do:

from nova.privsep import motd


motd('This node is a banana')

The above code would work, but is frowned on because it is less obvious here that the update_motd() method runs with escalated permissions — you’d have to go and read the imports to tell that.

That’s really all there is to it. The only other thing to mention is that there is a bit of a wart — code with escalated permissions can only use Nova code that is within the privsep directory. That’s been a problem when we’ve wanted to use a utility method from outside that path inside escalated code. The restriction happens for good reasons, so instead what we do in this case is move the utility into the privsep directory and fix up all the other callers to call the new location. Its not perfect, but its what we have for now.

There are some simple review criteria that should be used to assess a patch which implements new code that uses privsep in OpenStack Nova. They are:

  • Don’t use imports which create aliases. Use the “import nova.privsep.motd” form instead.
  • Keep methods with escalated permissions as simple as possible. Remember that these things are dangerous and should be as easy to understand as possible.
  • Calculate paths to manipulate inside the escalated method — so, don’t let someone pass in a full path and the contents to write to that file as root, instead let them pass in the name of the network interface or whatever that you are manipulating and then calculate the path from there. That will make it harder for callers to use your code to clobber random files on the system.

Adding new code with escalated permissions is really easy in Nova now, and much more secure and faster than it was when we only had sudo and root command lines to do these sorts of things. Let me know if you have any questions.


The post How to make a privileged call with oslo privsep appeared first on Made by Mikal.

by mikal at May 03, 2018 10:52 PM

OpenStack Superuser

Open source “critical” to UNICEF Innovation Labs

SAN FRANCISCO — When the tech and development communities get together, the acronyms can make your head spin.

At a recent round table breakfast at Autodesk headquarters called “2030Vision: Harnessing Technology to Advance the Sustainable Development Goals,” people from NGOs talked SDGs, LEED, RECs, ISO 260000, iOT, AI, VR and ROIs.

The two-hour panel was headed by Rob Cameron, CEO of SustainAbility along with Dominic Vergine, head of sustainability and corporate responsibility at ARM, Ben Thompson, senior manager for sustainability at Autodesk and Erica Kochi, co-founder, UNICEF Innovation.

The UN’s Sustainable Development Goals (SDGs) were at the heart of the discussion: they’re 17 global milestones, crisscrossing to hit 169 targets covering poverty, hunger, health, education, climate change, gender equality, water, sanitation, energy, urbanization, environment and social justice. Technology plays a crucial role in reaching those goals.

Vergine and Thompson both spoke from the corporate perspective, but what about the role of open source?

From left: UNICEF’s Erica Kochi, Ben Thompson, Autodesk, Dominic Vergine, ARM and Rob Cameron, CEO of SustainAbility.

UNICEF Innovation co-founder Erica Kochi says open source is “critically important,” to their projects adding that “you can work with closed-source products, but you have to have an understanding of what the goal is and if that’s going to get you there.” Also known as UNICEF Innovation labs, it’s an interdisciplinary team tasked with identifying, prototyping and scaling technologies and practices to strengthen the agency’s work for children.

Kochi says that after a decade of experience of working with governments, the agency only recommends open-source solutions. While she concedes that they sometimes use proprietary products in-house, they make a point not to procure or deploy them. “There are a couple of reasons for that. One is that we want to be sure that the government isn’t beholden to one vendor and the other is that we want to be able to take what we learn in one location and transpose it somewhere else.”

Some recent UNICEF Innovation projects include an app to help children displaced by conflict, a platform to help humanitarian and development workers manage protection-related data and a health line for fathers-to-be. Startups in program countries can also pitch their open-source projects to the UNICEF Innovation Fund.

For their work in Kosovo, Uganda and Zimbabwe, UNICEF Innovation Labs were honored in 2012 by Red Hat as one of the top four global open-source projects.

You can hear more from Kochi at the Red Hat Summit May 9 in San Francisco where she’ll be speaking about “Ideas worth exploring.”

Photo // CC BY NC

The post Open source “critical” to UNICEF Innovation Labs appeared first on Superuser.

by Nicole Martinelli at May 03, 2018 04:15 PM

SUSE Conversations

An Experiment with Gnocchi – (the Database) – Part 1

Background (Why): Openstack has component: Ceilometer (Telemetry). Ceilometer was huge and its main purpose was to help with the Monitoring and Metering of whole of th OpenStack. But because of the scale at which OpenStack operates, Ceilometer would often fall behind and become a major bottleneck for the same function it was designed to do. […]

The post An Experiment with Gnocchi – (the Database) – Part 1 appeared first on SUSE Communities.

by Sumit Jamgade at May 03, 2018 02:11 PM


Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


Last updated:
May 24, 2018 03:39 PM
All times are UTC.

Powered by: