February 25, 2021

OpenStack Blog

InMotion Hosting announces the general availability of its On-Demand Private Cloud powered by OpenStack

InMotion Hosting establishes a foothold in the IaaS and Private Cloud markets by announcing the general availability of Flex Metal Cloud IaaS. Flex Metal Cloud offers an On-Demand Private Cloud powered by OpenStack, an open source infrastructure platform. Built for enterprise infrastructure, traditionally it has been difficult for smaller teams to spin up small OpenStack... Read more »

by Sash Ghosh at February 25, 2021 01:00 PM

February 24, 2021

OpenStack Superuser

The Progress for Cloud Computing on Arm Architecture

In recent years, the Arm server has been widely used, and the upper software ecosystem has also been greatly developed. In particular, some public cloud vendors have begun to provide cloud computing products based on the Arm architecture. Companies and organizations like Huawei, EasyStack, and Linaro have invested quite a lot of resources to speed up the application of the Arm architecture in cloud computing.

Next, we’ll share the latest progress for the software ecosystem of Arm architecture in the cloud computing area, the challenges for landing Arm-based servers, and the experience of production readiness.

Full-Stack enablement of Open Source Software

As shown in the above picture, the infrastructure layer is only a small part of the whole software stack, in order to make this layer functional and perform better on Arm platform, developers have also done a lot in the lower layer projects like OS, drivers and libraries. On the other hand, endusers also care about the software running on top of the infrastructure, whether they can run on Arm platform or not is one of the key considerations for users thinking about using Arm platform as infrastructure. In the next part, let us look at the enablement of Arm platform from a bigger picture.

In the open source world, the development pipeline is particularly important for upstream developers. Currently, most of the open source projects only have x86 based dev pipeline, so the whole development process and outcome is not very friendly to Arm users, there might be some extra works before running them on Arm platform:

In order to make Arm platform the first citizen in the open source world, the first step is to provide a development pipeline to the open source projects, so that developers have resources to develop and test on the Arm platform, and the outcome product will also be Arm-native:

With this idea, Huawei has newly enabled Arm CI in over 50 top open source communities, covering 6 major fields including Cloud & SDS, Big Data, Database, Web, Libs & Middleware, and AI. And there are quite a lot of open source projects that already supported the Arm platform, users now have multiple choices in those fields, readers can refer to Arm CI landscape for more details (https://kunpengcompute.github.io/arm-landscape/).

Besides enabling, Huawei has also done a lot in improving the performance of open source projects on Arm platform and close the feature gaps between x86 platform. For example, we have enabled CPU info observation and host CPU comparison on Arm platform in Libvirt, and currently working on using them to have better migration experiences in OpenStack.

With these enablements and improvements, Arm platform became more competitive and attractive for both users and upstream developers.

Linaro’s work on ARM64 cloud computing ecosystem

Linaro is an open source organization on the ARM64 ecosystem, which mainly focuses on upstream development and maintenance such as Linaro Kernel, toolchains, Android, and some specified areas such as data center. 

Linaro has deeply participated in the Open Infrastructure community through contributing to ARM64 OpenStack enablement, deployment(Maintain ARM64 Kolla images), and maintaining the ARM64 OpenDev CI resources. At the same time, the Linaro Developer Cloud has been set up based on totally upstream work on OpenStack and Ceph,  which can be not only tested members’ hardware in cloud computing but also help developers to use ARM64 resources. 

As the chart above shows, Linaro Developer Cloud is completely based on OpenStack and Ceph. Now it can offer VM/BM services based on Nova/Ironic as well as ARM64 Kubernetes Service based on Magnum support. The production level OpenStack cluster is deployed by Kolla container images, which is more flexible for operation and upgrading. A lot of upstream work were included for supporting this:

  • Nova/Ironic/disk-image-builder enablement and bugfix on ARM64
  • Devstack enablement support for ARM64 OpenStack
  • Kolla image build, Kolla-ansible deployment support, and bugfix for OpenStack version upgrading.
  • Magnum multi-arch support and K8s cloud provider support on ARM64

Linaro Developer Cloud K8s service has now online for about one year and the K8s service version v1.17, v1.18 has been certified by the CNCF conformance test. It is a good example that ARM64 is eligible in the open-source cloud computing area.

Hardware Automation is another hot topic in recent years. Quite a lot of workloads need to run on bare metal(e.g cloud-native K8s on Bare metal, HPC) to achieve better performance of network and storage or virtualization limit(nested virtualization is not supported on ARM64), so that quick hardware provisioning, standard hardware management framework are essential and significant on ARM64. 

To meet these requests, Linaro has proposed the diskless boot solution on ARM64, which leverages OpenStack Ironic for BM management, and uses Ceph ISCSI to provision the volume boot support. Use Ceph volume as a disk could largely reduce BM provision time and improve the RootFS security, all the data reliability will be guarantee by Ceph.  

As the above picture shows, the control path of the workflow has relied on the Ironic PXE ISCSI boot from the volume, Cinder Ceph ISCSI driver, and Ceph ISCSI gateway. Linaro has contributed several features in order to make the diskless boot on ARM64 happen:

  • Ironic boot from PXE iSCSI support
  • Cinder Ceph ISCSI driver support
  • Ceph iSCSI client bugfix and stable enhancements

We believe that the hardware automation solution will be beneficial to ARM64 development and CI system. We will quickly online BM service support in Linaro Developer Cloud and offer more resources to external applications.

Best Practices of Running Cloud on Arm Servers

EasyStack, as a company focusing on cloud computing, has served more than 1000 customers since its establishment. The company started to support the Arm server in April 2019, released the technology preview version in February 2020, and released the formal GA version on January 31, 2021, which has reached productlevel support in function coverage and test intensity. So far, nearly 30 customers have deployed the Arm version of cloud computing products in the production environment.

In people’s minds, Arm ecology is not as extensive as x86. There are many unknown problems to be solved, including hardware, firmware, operating system, software, and so on. Actually, this impression lasted for quite a long time and things have changed rapidly during the last 5 years since great progress has been made in the development of the Arm ecosystem. The number of CPU cores is increasing, and the processing capacity of a single core is strengthening. We have seen that the performance of distributed storage in the Arm-based is even higher than that on x86 architecture.

Ecology has also been greatly developed. More and more operating systems support Arm. Open source software in cloud computing also has better support on Arm architecture, such as using Arm CI as a verification means in the normal development process. The Arm architecture has an advantage in energy efficiency, which is also one of the driving forces to attract public cloud vendors to design their own Arm CPUs to provide cloud computing services. In short, the difference between the application of Arm in the field of cloud computing and x86 architecture is greatly reduced.

During the whole adaptation stage, EasyStack‘s engineers adapted more than 10 types of servers. These servers had differences in BIOS/firmware, which led to the same operating system could run on some servers, but could not run some servers. The engineers analyzed the differences and made corresponding repairs. Finally, the operating system could run stably on all the Arm servers. The choice of the upper software version is also particularly important. EasyStack’s products use Kubernetes as the base, and other services run in the form of pod, including OpenStack components. This makes the Arm version product use the same software stack with x86 architecture, which is easy to maintain. In this process, libvirt’s CPU specific bugs, Openvswitch’s stability, and MariaDB’s stability are solved. In a word, Kubernetes, OpenStack, and other components can run perfectly on the Arm platform.

After the adaptation work, we started to support strict testing at the product level. All the functions of x86 have been fully tested. This testing work is integrated into the daily CI / CD process to ensure that any code change will not damage the functions of X86 and Arm. It is found that when doing network bandwidth test on Arm platform, the bandwidth value does not reach the expected value, which is solved by a binding specific interrupt to the corresponding CPU; it is found that VNC console has no output in some guest OS, which is mainly related to the setting of guest OS grub startup parameter tty0; Some Arm servers don’t support the hardware watchdog very well, sometimes they can’t work normally. We have completed complete tests, and compared with x86 architecture products, There are only four differences: 1. It doesn’t support Windows guest OS; 2. The GPU driver and SDK support on Arm architecture are not good enough, it can’t support GPU temporarily. 3. Because of the limitation of Arm architecture, each virtual machine can support fewer disks and network cards than x86; 4. Because of the limitation of IPMI, some monitoring indicators such as disk speed can not be obtained, That’s all of the difference between the Arm and x86 architecture.

There are more and more customers using Arm servers for production. Here, we need to complete the migration of x86 architecture applications to Arm architecture, the resource specification conversion from X86 platform to Arm platform, and so on.  For EasyStack, there are nearly 30 customers in total, with more than 500 cloud computing physical nodes based on Arm architecture are used in production and run well, among which the longestrunning customers are more than one year. Production readiness is the basis of large-scale promotion, including the strong support of various companies in the community. At present, we have seen the dawn. We believe that in the near future, more enterprises will adopt Arm architecture as the basis of cloud computing to provide cloud services.

Get Involved

This article is a summary of the Open Infrastructure Summit session, the Progress for Cloud Computing on Arm Architecture.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post The Progress for Cloud Computing on Arm Architecture appeared first on Superuser.

by Kevin Zhao, Zhenyu Zheng and ChangBo Guo at February 24, 2021 02:00 PM

February 22, 2021

Sean McGinnis

Cleaning up Gmail with App Script

After several years working in the open source community, I’ve ended up with a lot of emails from Gerrit code reviews, Google Group mailing lists, and various GitHub notifications. And when I say a lot, I mean my Google storage for Gmail, Photos, Drive, and others were getting closer and closer (78%) to the max free capacity.

To be fair, I do know quite a bit of that is from other things besides email. I have several large files in Drive, and a few longer videos from when my kids were younger sitting in Google Photos. But the weird thing that struck me was that even though I would delete emails from things like code reviews, they would still show up when I searched by inbox for various terms. I thought the way it was supposed to work was that I would delete an email, it would sit in my Trash for 30 days or so, then Gmail would clean things up for me. Based on what I was getting from my inbox searches, that apparently isn’t always the case.

So rather than allow things to keep up slowly creeping up to max out my storage quota, I decided to take a look at what I could do to clear out this old cruft. At first I would perform some searches, use the Select All to select all 50 messages on the page, and delete them. And the next page. And the next.

That was going to take a long time. It wouldn’t give me the message count from queries like from:review@openstack.org, but it looked like it was in the 1000’s. Even with no where to go, I didn’t have time to click through all that.

Automation (sorta) to the rescue

I’m probably late to the party - based on forum results, it appears several years late to the party - but it didn’t take me too long to stumble on the Google Apps Script site.

I started by looking to see if there was an API for interacting with Gmail. Turns out there is, for several different languages. But the Quickstart Guide included a link to how to get started with Apps Script. That has a nice walkthrough of getting a simple script set up to interact with Gmail.

From there, it was pretty easy to adapt the example they gave to work with messages instead. Looking through the API documentation, I found the message object has a moveToTrash() method. That led me to come up with this quick example:

function deleteMessages() {
  var count = 100
  while (count > 0) {
    var threads = GmailApp.search('from:review@openstack.org', 0, 10)
    count = threads.length
    for (var i = 0; i < count; i++) {
      var thread = threads[i];
      for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];

It’s fairly easy to work with the Gmail API. To break it down bit, here are some key parts.

Getting threads

var threads = GmailApp.search('from:review@openstack.org', 0, 10)

This queries the Gmail API, using a query just like you would use within the Gmail UI itself. In fact, I highly recommend first running your query in the UI and verifying it returns what you expect it to, then copying and pasting that query string into the script. We are deleting messages here, so you probably should be very cautious about what gets automatically deleted.

The two numbers, 0, 10, are the starting count and the max number of threads to retrieve. Starting off, this query would return a very, very large amount of messages. Pulling that all across the network and into memory would not be very nice. So chunking this out into smaller amounts is a lot more efficient. I would rather have to run the script repeatedly than try to load my entire inbox into the script.

And speaking of running repeatedly, there apparently is a hard coded 5 minute timeout on script execution. So although this speeds things up quite a bit over manually deleting, it still does take time. So one way or another I was going to have to keep rerunning the script to get everything cleared out anyway.

Bringing out the trash


Most of the rest of the code is just looping through results. The next code of interest is the actual deletion. This is accomplished by calling the moveToTrash method on the message object. Just as it sounds like, it moves the message from wherever it is into your Trash folder.

Slight optimization

This was great progress. I ran it a few times, and I could see from my search result that things were getting cleared out. It was taking a while to execute, so I looked through the API docs some more. It turns out, I was wasting some time iterating through all those messages within the thread.

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {
    var thread = threads[i];
    for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];

That’s looping through each thread that is returned from the query, and in turn, retrieving and looping through each message within the thread. For things like OpenStack code reviews, some patches with a lot of feedback, updates, and CI comments, that can mean quite a few messages per thread.

Luckily, the thread object itself also has a moveToTrash method. So knowing that, I was able to clean that up quite a bit:

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {

This was definitely more efficient.

Space, the final frontier

I still had a problem with the script though. Or maybe, more of a problem with my patience and my trust in the Trash actually being emptied after 30 days. What I really wanted was to immediately and completely delete these messages and get them out of there.

None of the examples I read in the official docs mentioned it, though that could very well just be my impatience and not wanting to read up on more than I really needed to. But I found an old forum posting somewhere that mentioned a remove call.

DANGER: Only do this if you are very careful!

Seriously, don’t blame me if you mess things up. I warned ya!

This call will actually do exactly what I wanted. It gets the message completely out of there right away. Of course, with scripting and automation, there is no “Are you really sure?” dialogs to click and slow you down. If you get the query wrong, too late, it’s gone.

So again being very cautious about the query string being used to retrieve the threads, I tried one more approach.

Stepping back

Now the only bad thing here (OK, probably not the only bad thing) is that the remove call only works on message IDs. Not threads.

I just had to roll back some of the optimization I had done before and go back to iterating through the messages in the thread again. It slowed things down, but ultimately it was still faster, at least to me, because it would permanently get rid of the messages. The final version I ended up with was then:

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {
    var thread = threads[i];
    for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];
        Gmail.Users.Messages.remove('me', message.getId());

That last line takes a user ID, but luckily it has a special value of me that will recognize the current user. And the ID of the message to remove, retrieved by calling the getId() method on the message. After running this version over and over after the 5 minute timeouts, I started to see my storage usage number start to drop down.

More cleanup

One more time - it’s very important to make sure you query to get threads is correct.

The query is very flexible though. So I was able to use wildcards and change the to/from to also clean up noreply.github.com, googlegroups.com, and several others where I have accumulated mass amounts of junk over time.

As I said, I’m also impatient, so once I had these cleaned up, I made another pass through some of the worse offenders and added in:trash to the query to immediately clean out some messages from the trash.

All tidied up

In the end, my total Google account storage went down from about 12.78G to 11.8G. Not bad.

I’m guessing I can’t automate Google Photos clean up as well. At least not without learning some image recognition APIs and adding a lot of risk. So that’s a clean up for another day.

But with a few quick hacky scripts, I’m pretty happy with how much space I was able to free up.

by Sean McGinnis at February 22, 2021 12:00 AM

February 21, 2021

Doug Hellmann


I started contributing to OpenStack in 2012, just before the Folsom summit. I was a core reviewer and one of the founding members of the Ceilometer project, and a core reviewer for the Oslo, requirements, and unified command line interface projects. I was the PTL for the Oslo project for the Icehouse, Juno, and Kilo release cycles. I was the PTL for the Release Management team, and helped build the tools to allow self-service releases.

February 21, 2021 08:00 PM

February 18, 2021

Alessandro Pilotti

OpenStack on ARM64 – LBaaS

In part 2 of this series about OpenStack on ARM64, we got to the point where our cloud is fully deployed with all the Compute (VMs), Software Defined Networking (SDN) and Software Defined Storage (SDS) up and running. One additional component that we want to add is a Load Balancer as a Service (LBaaS), which is a key requirement for pretty much any high available type of workload and a must-have feature in any cloud.

OpenStack’s current official LBaaS component is called Octavia, which replaced the older Neutron LBaaS v1 project, starting with the Liberty release. Deploying and configuring requires a few steps, which explains the need for a dedicated blog post.

Octavia’s reference implementation uses VM instances called Amphorae to perform the actual load balancing. The octavia-worker service takes care of communicating with the amphorae and to do that we need to generate a few X509 CAs and certificates used to secure the communications. The good news is that starting with the Victoria release, kolla-ansible simplifies a lot this task. Here’s how to:

# Change the following according to your organization
echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml

# This is the kolla-ansible virtual env created in the previous blog post
cd kolla
source venv/bin/activate

sudo chown $USER:$USER /etc/kolla
kolla-ansible octavia-certificates

The communication between Octavia and the Amphorae needs an isolated network, as we don’t want to share it with the tenant network for security reasons. A simple way to accomplish that is to create a provider network with a dedicated VLAN ID, which is why we enabled Neutron provider networks and OVS VLAN segmentation in the previous post. Again, starting with Victoria, this got much easier with kolla-ansible.

# This is a dedicated network, outside your management LAN address space, change as needed

sudo tee -a /etc/kolla/globals.yml << EOT
  name: lb-mgmt-net
  provider_network_type: vlan
  provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID
  provider_physical_network: physnet1
  external: false
  shared: false
    name: lb-mgmt-subnet
    allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"
    allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"
    gateway_ip: "$OCTAVIA_MGMT_HOST_IP"
    enable_dhcp: yes

Unless there is a dedicated network adapter, a virtual ethernet one can be used. This needs to be configured at boot and added to the OVS br-ex switch.

# This sets up the VLAN veth interface
# Netplan doesn't have support for veth interfaces yet
sudo tee /usr/local/bin/veth-lbaas.sh << EOT
sudo ip link add v-lbaas-vlan type veth peer name v-lbaas
sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas
sudo ip link set v-lbaas-vlan up
sudo ip link set v-lbaas up
sudo chmod 744 /usr/local/bin/veth-lbaas.sh

sudo tee /etc/systemd/system/veth-lbaas.service << EOT


sudo chmod 644 /etc/systemd/system/veth-lbaas.service

sudo systemctl daemon-reload
sudo systemctl enable veth-lbaas.service
sudo systemctl start veth-lbaas.service

docker exec openvswitch_vswitchd ovs-vsctl add-port \
  br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID

A few more Octavia kolla-ansible configurations…

echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml

# Flavor used when booting an amphora, change as needed
sudo tee -a /etc/kolla/globals.yml << EOT
  name: "amphora"
  is_public: no
  vcpus: 1
  ram: 1024
  disk: 5

sudo mkdir /etc/kolla/config/octavia
# Use a config drive in the Amphorae for cloud-init
sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT
user_data_config_drive = true

…and we can finally tell kolla-ansible to deploy Octavia:

kolla-ansible -i all-in-one deploy --tags common,horizon,octavia

Octavia uses a special VM image for the Amphorae, which needs to be built for ARM64. We prepared Dockerfiles for building either an Ubuntu or a CentOS image, you can choose either one in the following snippets. We use containers to perform the build in order to isolate the requirements and be independent from the host OS.

git clone https://github.com/cloudbase/openstack-kolla-arm64-scripts
cd openstack-kolla-arm64-scripts/victoria

# Choose either Ubuntu or CentOS (not both!)

# Ubuntu
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Ubuntu \
  -t amphora-image-build-arm64-ubuntu

# Centos
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Centos \
  -t amphora-image-build-arm64-centos

ARM64 needs a trivial patch in the diskimage-create.sh build script (we also submitted it upstream):

git clone https://opendev.org/openstack/octavia -b stable/victoria
# Use latest branch Octavia to create Ubuntu image
cd octavia
# diskimage-create.sh includes armhf but not arm64
git apply ../0001-Add-arm64-in-diskimage-create.sh.patch
cd ..

Build the image (this will take a bit):

# Again, choose either Ubuntu or CentOS (not both!)

# Note the mount of /mnt and /proc in the docker container
# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,
# making the host unusable
docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \
  -v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu

# Create CentOS 8 Amphora image
docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \
  -ti amphora-image-build-arm64-centos

Add the image to Glance, using the octavia user in the service project. The amphora tag is used by Octavia to find the image.

. /etc/kolla/admin-openrc.sh

# Switch to the octavia user and service project
export OS_USERNAME=octavia
export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service

openstack image create amphora-x64-haproxy.qcow2 \
  --container-format bare \
  --disk-format qcow2 \
  --private \
  --tag amphora \
  --file octavia/diskimage-create/amphora-x64-haproxy.qcow2

# We can now delete the image file
rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2

Currently, we need a small patch in Octavia to properly render the userdata for the Amphorae:

# Patch the user_data_config_drive_template
cd octavia
git apply  ../0001-Fix-userdata-template.patch
# For now just update the octavia-worker container, no need to restart it
docker cp octavia/common/jinja/templates/user_data_config_drive.template \

Finally, let’s create a load balancer to make sure everything works fine:

# To create the loadbalancer
. /etc/kolla/admin-openrc.sh

openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet

# Check the status until it's marked as ONLINE
openstack loadbalancer list

Congratulations! You have a working LBaaS in your private cloud!!


In case something goes wrong, finding the root cause might be tricky. Here are a few suggestions to ease up the process.

# Check for errors
sudo tail -f /var/log/kolla/octavia/octavia-worker.log

# SSH into amphora
# Get amphora VM IP either from the octavia-worker.log or from:
openstack server list --all-projects

ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu
ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos

# Instances stuck in pending create cannot be deleted
# Password: grep octavia_database_password /etc/kolla/passwords.yml
docker exec -ti mariadb mysql -u octavia -p octavia
update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';

The post OpenStack on ARM64 – LBaaS appeared first on Cloudbase Solutions.

by Alessandro Pilotti at February 18, 2021 01:00 PM

February 17, 2021

OpenStack Superuser

2020 Annual Report: OpenInfra Foundation Working Groups

2020 redefined collaboration. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the Open Infrastructure (OpenInfra) Foundation’s mission: building communities who write software that runs in production. In 2020, the Diversity & Inclusion Working Group and the Interop Working Group continued their efforts of collaboration virtually to grow the OpenInfra mission as well as the Working Groups’ missions.

Diversity & Inclusion Working Group

The Diversity and Inclusion Working Group continued reaching out to the other OIF projects and inviting them to join our efforts as we are not OpenStack specific and hope to continue offering opportunities for diversity at events and in the community.

Our main focus for the year was the creation of the OpenInfra Foundation’s stance on divisive language. All OpenInfra projects were contacted and over the course of the work all projects gave input except for Kata. The stance was approved by the Board and the community with several recommendations being relayed to the Board for continuing efforts from them as well as the creation of a Wiki page with suggestions to be used when addressing the replacement of divisive words.

The D&I WG also helped to lead OpenStack’s presence during Grace Hopper’s Open Source Day. We had about 13 mentors from the community and 45 attendees throughout the day.

In 2021, we plan on continuing to aid the OpenInfra projects in their efforts to remove divisive language and to use more inclusive words. We also hope to be able to resume efforts with the community during virtual and in person events.

Interop Working Group

The OpenInfra Interop Working Group has been issuing guidelines for OpenStack Logos and Branding programs as part of Marketplace development every release as listed below:

  • OpenStack Powered Platform
  • OpenStack Powered Compute
  • OpenStack Powered Object Storage

During Ussuri we had two add-ons:

  • OpenStack Powered DNS
  • OpenStack Powered Orchestration

In Victoria we added one more add-on: the OpenStack Powered Shared File System.

The Interop repos were moved under the OpenInfra Foundation in Ussuri and are consolidated in Victoria to enable QA and test team to plan for ongoing efforts in OpenInfra projects Kata, Airship and StarlingX for container conformance.

The forward looking statement from the OpenInfra Interop Working Group is to start the ground up efforts to set new guidelines for Integrated Wallaby release and define the OpenInfra Container Conformance Program. The current work in the technical team has been efforts around containerizing Refstack to move to Ansible and Go besides python 3.7 support for test tools and libraries used by all OpenInfra Projects. The Baremetal logo program was deferred to future releases after new guidelines are in place.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Reportjoin the OpenInfra community to build the next decade of OpenInfra, and learn about the different Working Groups.

The post 2020 Annual Report: OpenInfra Foundation Working Groups appeared first on Superuser.

by Helena Spease at February 17, 2021 02:00 PM

February 16, 2021

OpenStack Superuser

You Know Nothing, Jon Snow! OpenStack Troubleshooting From a Beginner’s Perspective

Jon Snow is a prominent character in the Game of Thrones TV series. He is a skilled swordsman, defined by a strong sense of justice and honor. Jon Snow proves his leadership skills and strategic thinking in many situations.

But you might remember Jon meeting the Wildings, a new culture that he did not fully understand the first time he met them. It took time until Jon could resonate with them, and he looked like a newb in the process. This triggered Ygritte to call out, “You know nothing, Jon Snow” several times. Soon enough, Jon learned how to deal with the Wildlings, he earned their respect and trust, and they followed him as a leader.

I find this analogy to fit people starting to work with OpenStack. It feels very overwhelming in the beginning, but you can build on your skills and experience once you get familiar with what project is responsible for what functionality.

I had the unforeseen-except-perhaps-in-my-dreams chance to get a new job in 2015, to work with OpenStack. One way I prepared for the new job was by listening to OSPod, I’ve listened to many episodes several times. I found it interesting to listen to OpenStack’s initial cast talking about the projects they contributed to. Several of these podcast episodes are quite inspiring. Take this one for example.

OpenStack breaks down to services running on Linux and troubleshooting OpenStack goes down to troubleshooting these services and checking their log files. The trick is to know where to look first without wasting time checking services that are not the cause of the problem you are investigating. It might sound simple, but it’s not, and sometimes one needs to employ diligent sleuthing to find what the problem is and fix it.

I started working with OpenStack from its Mitaka release. My first role was as a support engineer and, I helped telecom operators around the globe fix problems in their OpenStack deployments. Looking back, I can draw the line, and I can tell you that once in a blue moon I stumbled over bugs in OpenStack. Maybe the reason for that was that the Mitaka release reached an acceptable maturity level. Another reason could be the fine-tooth combing of OpenStack services through all sorts of testing like functional testing, system testing, and end-to-end testing. Extensive testing prevents going to production and discovering significant bugs. Most of the support cases that came in condensed to networking issues:

  • Cannot access a VM on a port – networking problem
  • Cannot access storage media from VM – networking problem
  • VM does not boot anymore – networking problem
  • OpenStack controllers froze – networking problem

Most of the time, the networking problems did not result in bugs filed under the Neutron project. Many cases were resolved by restarting services, patching network devices, fixing Linux configuration, properly configuring network devices like routers and switches.

This article builds on experiences troubleshooting OpenStack. You can use this approach:

  • Define what the problem is
  • Narrow down what the cause is
  • Solve the problem
  • Pass on the experience through proper documentation

Define the problem

A customer sent in a ticket saying that they cannot connect from VM A to VM B on port 1521 over TCP, but ping between the two hosts is working. They mentioned the problem must be in Neutron, but they are not sure what Neutron logs they should check.

As a first step, I must understand the problem better. It’s easy to fall for the confirmation bias and ask the customer for Neutron configuration printouts and logs. The customer might be sitting in a different time zone, and email communication can be sparse because of the time difference. Assume I get the printouts and logs I asked for, and I can’t find anything obviously wrong. Then I need at least one more day to ask for more information.

So to avoid sending the customer scurrying grumpily all over the place it’s good to ask for printouts that confirm the problem they raised.

We cannot access VM B on port 1521? I would assume the service running on port 1521 on VM B is up and running.

Ask for printouts to confirm it:

VM_A $ telnet 1521
VM_B $ telnet 1521

where is the IP address on VM B received from the customer in the problem description.

There might be a problem with the commands above. On most production servers, telnet is not installed, but cat should be there. We can instead ask for:

cat < /dev/tcp/<hostname_or_ip>/<port_number>
VM_A $ cat < /dev/tcp/
VM_B $ cat < /dev/tcp/

In this example, bash will attempt to open a TCP connection to the corresponding socket.

The customer mentioned that ping between the hosts works. We want to run a command that confirms this fact but that also would check for things like MTU size, which could be a cause for the problem they raised:

Ask for a printout of ping that tests a bigger packet size, prohibiting fragmentation, with packets being sent at shorter intervals:

VM_A $ ping -s 1472 -M do -i 0.2 -c 50

What I got back was this:

VM_A $ cat < /dev/tcp/
bash: connect: Connection refused
bash: /dev/tcp/ Connection refused

The service running on port 1521 was not up, which explains the symptoms. It often happens that the cause of a problem is something simple that was overlooked, like a service not running, a service that keeps restarting, or the VM where the service should be running is down. In her talk about troubleshooting NeutronRossella Sblendido mentions just that.

It’s worth to ask for printouts that confirm the problem description. It might result in solving a problem in the definition phase.

One might assume there’s not much to document when solving the problem described above. Add the questions you asked together with the customer replies and printouts to the customer ticket. If someone later finds your case, it will serve as a good example of asking questions and defining the problem.

Solve the problem

We will now look into a support ticket where a customer says they rebooted one out of three OpenStack controllers and lost access to it. In this case, the OpenStack services are running inside three different VMs, and thus, we have three virtual OpenStack controllers.

We should avoid asking questions like “What did you do, what did you change?” which renders an interlocutor to defense mode. Such questions might read as accusations when the situation is tensed.

When defining the problem, we could ask:

  • “How do you connect to the controller VM?”
  • “Please send me the console log, /var/log/libvirt/qemu/controller-1
  • “Please send me a printout of $ script -a VM_log ; sudo virsh console controller-1

The console logs show the following:

The disk drive for /mnt/backups is not ready yet or not present.
Continue to wait, or Press S to skip mounting or M for manual recovery

It seems that a block device is no longer attached to the controller VM or there is some misconfiguration in /etc/fstab.

We need to access the VM, possibly by booting in single mode, and correct /etc/fstab by removing the offending line. Some months ago, someone added a line to /etc/fstab to mount a network shared device and back up files to a remote destination, which does not exist anymore.

Documentation is again very important. Besides properly documenting the information exchanged with the customer, you can write a knowledge object that explains how to connect to an OpenStack controller VM when the VM won’t boot, and it is stuck looking for block devices that do not exist anymore.

Document the problem

The third support case is one where the customer observed slow access to block devices hosted on the backend storage from some VMs. Other VMs cannot access their block devices at all, and a third part of VMs experience no problems whatsoever.

Start with the VMs that cannot access the block devices at all. How are they spread across computes? Here is an example of how to filter for the computes where the VMs are running, assuming all VMs belong to the same Heat stack. List resources filtered on Nova::Server, pipe the output to xargs and extract the hypervisor_hostname to get the compute names. Is there a pattern here? Do the problematic VMs belong to the same computes?

If the VMs cannot access their block devices on the storage backend, it might be a networking problem. Look for log entries concerning the storage network interfaces on the computes hosting the VMs in trouble. The dmesg log show several entries like this one

NETDEV WATCHDOG: eth4 (i40e): transmit queue 2 timed out
i40e 0000:83:00.2 eth4: NIC Link is Down

The storage interface seems to be down, so try to bring it up. It worked, and the VM can access the block device again.

ip link set up eth4

Apply the same fix for the rest of the computes. On the computes where we have slow access to storage, reset the problematic interface, based on findings in the dmesg log.

We fixed the customer’s problem, but we do not know yet what caused it. It is important we continue the investigation otherwise, the problem might surface again.

If you can’t find anything obvious in the logs to help you conclude what cause the interfaces to go rogue, it might be a good time to turn to an expert and ask for help.

When you start working with OpenStack, I can highly recommend you find yourself an expert. Everyone knows who they are: a person with some years of experience, that can help with almost any technical problem. It’s also the person that everybody wants a piece of. In most cases, it is the person that writes excellent documentation too. Ask for help if you get stuck, but prepare your questions thoroughly. Make sure you can coherently define the problem. Have the logs ready in case someone needs to check them. By doing this, you don’t waste someone else’s time; it’s a matter of showing respect.

The problem happens only on 12 out of 168 computes Let’s look at the network interfaces on these computes. They look like this:

root@compute-40-3 # lshw  -C network -businfo
Bus info          Device          Class          Description
pci@0000:01:00.0  eth0            network        I350 Gigabit Network Connection
pci@0000:01:00.1  eth1            network        I350 Gigabit Network Connection
pci@0000:83:00.0  eth2            network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.1                  network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.2  eth4            network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.3                  network        Ethernet Controller X710 for 10GbE SFP+

So we have two interfaces for the control network, eth0 and eth1. We have two interfaces for the storage network, eth2 and eth4. And we have two interfaces for the traffic network. The storage network interfaces use the kernel driver; you can see their device name in this output. The traffic network interfaces, the ones that don’t have a device name. use the DPDK driver, ( the technology to offload TCP packet processing from the operating system kernel to processes running in user space, you want this for higher performance.) All 4 of them share the same bus, look at the PCI addresses.

The network card looks like this. So we have network interfaces controlled by the kernel driver and network interfaces controlled by the DPDK driver, on the same physical card. The problem showed up on the computes having these NICs installed. We assume it is related to these NICs.

We can run iperf on the NICs with ports controlled by multiple drivers and observe:

  • packet drops on the ports controlled by the kernel driver, and lower TCP throughput, or
  • complete traffic loss due to “tx hang” on the ports controlled by the kernel driver.
$ iperf -c 
Client connecting to, TCP port 5001
TCP window size:  586 KByte (default)
[  3] local port 59454 connected with port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.43 GBytes  9.4 Gbits/sec

We raised bug reports towards the manufacturer of the network card. Later, the manufacturer provided patches that solved the problem.

Proper documentation is very important! You will document the problem and its solution, but it can be useful to write down the steps you took to figure it out, the tools and commands you used.

Besides properly recording this problem with its solution in your company’s (most probably internal) tools, it would be nice to share the problem on the internet because it might help someone else. Write about it on your blog, tweet about it, or record a video and upload it to youtube.

There are many problem-solving strategies and methods. I wish I could state that I found the few-step approach that solves any OpenStack problem, but there is no silver bullet. What can help troubleshooting OpenStack is to ask the right questions to help you understand and define the problem, to establish and test a few hypotheses of a possible cause and narrow down where the problem is located. Then, once a fix is available, to prioritize properly documenting the problem with its solution.

Get Involved

This article is a summary of the Open Infrastructure Summit session, You know nothing, Jon Snow! OpenStack troubleshooting from a beginner’s perspective.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post You Know Nothing, Jon Snow! OpenStack Troubleshooting From a Beginner’s Perspective appeared first on Superuser.

by Elena Lindqvist at February 16, 2021 02:00 PM

February 15, 2021

Galera Cluster by Codership

Database Administration for Galera Cluster online training course March 16-17 and 23-24

Codership, the developers of Galera Cluster, have set new days for online training for Galera Cluster. There are two sets of times for our EMEA and American attendees, with the former happening on March 2021, 16th and 17th, starting 10 AM CET and the latter happening on March 2021, 23rd and 24th, starting 9 AM EST.

This is a hands on course that spans two days, and you will have about six contact hours per day in this instructor-led course. If you are a DBA or been given a Galera Cluster to manage, this is the course for you, as you’ll learn how to run your Galera Cluster in an optimal fashion, from setup, performance tuning, monitoring, as well as backups and more. We will cover not just Codership’s Galera Cluster distribution but also MariaDB Galera Cluster and Percona XtraDB Cluster in this course. For more information, please read the full content breakdown.

As an added bonus, we will also be covering database monitoring with Galera Manager our new GUI tool.

What are you waiting for? Sign up now, as the early bird rate ends 28 February. There is also a volume discount available.

by Sakari Keskitalo at February 15, 2021 05:42 PM

OpenStack Superuser

2020 Annual Report: OpenInfra Event Highlights

As many organizations experienced in 2020, the Open Infrastructure (OpenInfra) Foundation had several challenging decisions to make as the pandemic began to spread around the world. Having the safety of the community as the top priority, the decision was made to move the entire 2020 programs, including the PTG (Project Teams Gathering), OpenDev series, and OpenInfra Summit to a virtual format. With the help of the Foundation and event sponsors, partners, and the support of the entire community, we were able to continue our mission to collaborate, share knowledge, network, and continue improving the software with the broadest global reach ever despite the inherent challenges of the year.

OpenInfra Summit

The first virtual OpenInfra Summit was held in October 2020, gathering 10,000 attendees (largest Summit attendance to date) from 120+ different countries to discuss 30+ open source projects. In addition to use cases from users like Volvo Cars Corporation, Ant Group, GE Digital, Société Générale, China Mobile, European Weather Cloud, Workday, and China Tower- plus a big announcement about the future of the Foundation:

  • The Summit kicked off with the Keynote announcement that the OpenStack Foundation was officially becoming the OpenInfra Foundation
  • Volvo Cars Corporation discussed how it successfully uses Zuul for continuous integration in a range of different software components, and how Zuul is a game changer that helps them avoid merging broken code.
  • Ant Group, the leading peer-to-peer payments processor in China, presented why isolation methods such as scheduling isolation and LLC isolation are used in combination with Kata Containers.
  • GE Digital presented the tools, migration procedure, and highlighted the biggest challenges in upgrading from OpenStack Newton to Queens with minimal downtime.
  • SK Telecom 5GX Labs won the 12th Superuser Award for developing a containerized OpenStack on Kubernetes solution called SKT All Container Orchestrator (TACO), based on OpenStack-helm and Airship, all while contributing upstream to both the OpenStack and Airship projects.

Summit videos are available on the Summit videos page. Thank you to our Summit sponsors for supporting the event!

OpenInfra TV: Bringing the Summit to Asia

At the annual OpenInfra events, the global community convenes to discuss the challenges and success of integrating different open source projects. Averaging attendees from over 60 countries at face-to-face events and attracting attendees from over 120 countries for the virtual OpenInfra Summit, the Foundation events team concentrates on making sure all events cater to the diversity of the audience.

For the virtual event, time zones were one of the biggest challenges as one block of time is never perfect for every single person in the world. The team selected four communities to pilot OpenInfra TV—China, India, Japan, and Korea—to provide local coverage of Summit highlights presented by community leaders in their local language.


The OpenDev event, held in-person in previous years, was a three part virtual series that focused on three topics: Large-scale Usage of OpenInfra Software, Hardware Automation, and Containers in Production.

OpenDev events bring together the developers and users of the open source software powering today’s infrastructure, to share best practices, identify gaps, and advance the state of the art in OpenInfra. For each topic, participants joined discussion oriented, collaborative sessions where they explored challenges, shared common architectures, and collaborated around potential solutions. Each topic’s agenda was set by a programming committee made up of community members who met for weeks leading up to the event to make sure it was tailored to the broader community.

The three events gathered over 2,000 registrants from over 70 countries.

Here are some of the highlights:

Large-scale Usage of OpenInfra Software – June 2020

The event kicked off with user stories from Blizzard Entertainment, OpenInfra Labs, and Verizon who shared their own scaling challenges in short presentations followed by Q&A. After that, the discussion focused on a variety of topics including upgrades, high availability (HA) networking, bare metal, and more. At the conclusion of the event, participants looking to get further involved were encouraged to join the OpenStack Large Scale SIG to continue sharing challenges and solutions around scaling.

Thank you to the programming committee members: Beth Cohen, Roman Gorshunov, Belmiro Moreira, Masahito Muroi, and Allison Price!

Read the full recap and catch up on the discussion recordings.

Hardware Automation – July 2020

This OpenDev event focused on topics around hardware provisioning lifecycle for bare metal, bare metal infrastructure, networking and network security. Participants representing over 200 companies spent three days sharing their knowledge and discussing their experience of building and operating software that automates their hardware in cloud environments. One of the ways mentioned to keep the momentum going after this event was to join the OpenStack Multi Arch SIG and OpenStack Bare Metal SIG to continue sharing challenges and solutions around Hardware Automation. Julia Kreger also highlighted the latest white paper from the OpenStack Bare Metal SIG: “Building the Future on Bare Metal, How Ironic Delivers Abstraction and Automation using Open Source Infrastructure.

Thank you to the programming committee members: Keith Berger, Mark Collier, Julia Kreger, Mohammed Naser, and James Penick!

PTGs / Forums

In June of 2020, we hosted our very first virtual Project Teams Gathering. We were pleased to have over 700 people register for the event – more than we’ve had for any in person PTG. By taking it virtual instead of in-person, we had new people participate that have otherwise never been able to attend the PTG. Over the course of the week, 44 teams across Airship, Kata Containers, OpenStack, StarlingX, and Zuul met to discuss their upcoming releases and make progress on standing items. The format of the PTG was different than any event we’ve hosted before because it was all virtual and we needed to accommodate for all timezones. We settled on 4 hour blocks of working hours with 4 hours inbetween and allowed teams to schedule themselves to better fit their day to day schedules and time zone restrictions. The PTG again made use of the community developed PTGbot with some tweaks to help keep people aware of active discussions and find where and when teams were meeting.

The community had two options to use for their video meetings: the OpenInfra Foundation subsidized Zoom rooms, or the OpenDev team hosted meetpad rooms. Many teams wrote up summaries of their discussions and accomplishments throughout the week:

We continued to meet virtually throughout the year and the week after our first Virtual OpenInfra summit, we decided to again, keep people safe, and run it like we did in June. In October of 2020 the teams met on Zoom and in Meetpad to talk to each other and collaborate. Over 500 people registered for the event spanning 45 countries. The event hosted community members across 46 teams and had their usual technical discussions which you can find summarized here:

OpenStack Days / OpenInfra Days / Cloud Operator Days

Overcoming the local challenges of the pandemic, the global OpenInfra community organized four OpenStack/OpenInfra Day events (a combination of virtual and face-to-face in Turkey, China, Indonesia and Korea – educating over 10,000 attendees around the world.

The typical attendee reached at these local events are cloud and IT architects, software developers, platform and solution engineers and product management.

The Tokyo community introduced a new local event, Cloud Operator Days, a two day online event that gathered 1,400 attendees to discuss challenges of operating cloud infrastructure.

These one or two day regional events are organized by the local community and supported by the OpenInfra Foundation to include project workshops, upstream training, and booths from the local ecosystem. These events continue to be an excellent touchstone for the community to engage with our ecosystem, and to gather local open source developers and users to collaborate, share use cases, and support for the OpenInfra projects.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Reportjoin the OpenInfra community to build the next decade of OpenInfra, and learn about what events to expect for 2021.

The post 2020 Annual Report: OpenInfra Event Highlights appeared first on Superuser.

by Helena Spease at February 15, 2021 02:00 PM

February 12, 2021


How the OpenStack community is collaborating during the pandemic

The OpenStack community is BIG. From Argentina to Morocco to Israel to Vietnam, we literally span the globe, so it's not surprising that we largely knew what to do to accommodate COVID-19's circumstances. But it still has been a struggle to keep moving forward and adapt while still delivering Ussuri and Victoria, the 21st and 22nd releases of OpenStack.

by knelson at February 12, 2021 08:01 AM

February 11, 2021

OpenStack Superuser

2020 Annual Report: OpenInfra Community Highlights

The Open Infrastructure (OpenInfra) community had a strong year in spite of the unprecedented global challenges in 2020. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the OpenInfra mission: building communities who write software that runs in production.

OpenInfra User Groups Growth

In 2019, OpenInfra User Groups migrated to the Meetup Pro platform, which allowed the groups more visibility on the web, as well as a streamlined way to organize and communicate.

In 2020, we saw the number of groups grow past 40, adding communities in places like the Congo and Tijuana.

Amidst challenging circumstances, User Groups showed their commitment to community, organizing in new and unique ways. When in-person events were put on hold, several groups took on the task of hosting virtual meetups covering a variety of topics including hybrid cloud, storage, containers, and edge. Some groups even organized watch parties for the first virtual OpenInfra Summit that took place in October.

While we know User Groups are looking forward to meeting in person again, we’re sure that virtual meetups are still going to be around because of the valuable accessibility they provide so many community members around the world.

Outreachy Program

For the current round, December 2020 to March 2021, 5 interns were selected:

  • Kuryr: T.I. Fasoyin from Nigeria
  • Manila: Dina Saparbaeva from Karakol, Kyrgyzstan
  • Ironic / OpenStack SDK: Anuradha from India
  • Ironic: Oluwatosin Farai from Nigeria
  • Manila: Paulali from Kenya

Students in OpenStack

Students are the future- the future of the workforce, the future of the tech industry and the future of open source. The OpenStack community has been slowly building relationships with more and more universities, getting them involved in work upstream in the community. These relationships are mutually beneficial; students get practical experience and build skills in communication and working collaboratively and the OpenStack community gets fresh perspective and the excitement that comes with new contributors.

Boston University Senior Design Program

Three students- Nicole Chen, Ashley Rodriguez, and Mark Tony- at Boston University selected OpenStack Manila as their senior design project. With the help of community mentors- Goutham Pacha Ravi, Victoria Martinez De La Cruz, Jeremy Freudberg, Maari Tamm and Kendall Nelson- the students have begun working on implementing the Manila APIs in the OpenStackSDK. There is a community effort to reach feature parity between the individual python clients for OpenStack services and the OpenStackSDK. The students were onboarded into the community, they have begun attending community meetings, and they also have been working on the specification outlining the work they are beginning and the details of implementation. If they are able to complete all the work in the OpenStackSDK, they can then begin implementing those features in the OpenStack Client. They will continue their work for the remainder of their school year, wrapping up in May of 2021.

North Dakota State University Capstone Program

For several years now, we have participated in North Dakota State University’s Capstone program that gives students a chance to select a project- either open source or industry- to work on throughout the spring semester. At the start of 2020, the OpenStack community had four students- Dawson Coleman, Luke Tollefson, Noah Mickus, and Steven Glasford- helping to add support for configuring TLS ciphers and protocols on Octavia load balancers. Michael Johnson, Adam Harwell, and myself mentored students through their semester helping them learn not only about Octavia, but about OpenStack, and open source. With their hard work, they landed TLS cipher and protocol support before the end of the Ussuri release and it became a large part of the promotion and marketing around that release. The rest of their work landed shorty after feature freeze and was included in the Victoria release that happened in October of 2020.

Oregon State University Internship

This year the Foundation started a partnership with Oregon State University funding one student to work in the OpenStack community on the Glance project to give them real world experience working on an open source project. Kendall Nelson and Erno Kuvaja mentored Khuong Luu. He worked on a variety of tasks in Glance. Khuong made several documentation improvments including adding missing information and reording topics to make them more readable and the information more discoverable. He cleaned up config options that were causing orphan processes in Glance when a user sent a SIGHUP signal. Khuong also cleaned up a bug being caused my a missing image property. While he didn’t spend time on one large feature, his efforts cleaned up a lot ot technical debt. He finished up his internship when the school year concluded and we thank him for all his hard work!

We have a new student intern getting started with his onboarding process. Ryan Zimmerman, once up to speed on OpenStack and our community, wil begin work on the OpenStackSDK and helping OpenStack services reach feature pairity with their own project specific python clients.

OpenStack Upstream Institute

The OpenStack Upstream Institute (OUI) training is an interactive course that teaches attendees about how the OpenStack community operates including the community structure and processes. The training also includes hands-on practice to teach participants how to use the tools that the community depends on. These tools -such as Git, Gerrit, and more – are maintained by the OpenDev (https://opendev.org) community. As in person events were not possible during 2020, we had to re-invent OUI to make it suitable for being hosted in a virtual format.

The virtual training had a slightly different structure compared to the in person classes. The training was split into two 3-hour long blocks: OpenStack specifics and OpenDev tools. The OpenStack module covered items such as governance, OpenStack release cycles, ways to communicate within the community, and more. The attendees learned how the community operates as we went through the topics during the first block. The class utilized active synchronous conversation between students and mentors, where attendees are able to ask questions from our mentors, who are all active community members.

The OpenDev tools module was also a hands-on class that focused on teaching attendees to setup and use tools that are used for software and documentation development processes as well as task tracking. Many of these tools are used by multiple OpenInfra communities and adjacent projects. By organizing this way, we were able to welcome and educate students who might not have an interest in OpenStack specifically which gave us a broader audience. During the class the attendees worked through exercises with guidance from the mentors to give them practice using the tools. During the training we also had breakout rooms students and mentors could hop into to ensure any technical issues students had were solved before the end of the class.

OUI, with the new structure, was held during the OpenInfra Summit in October. The class material was also utilized at other events to hold the training in different formats. Ghanshyam Mann hosted a virtual OUI session before the Summit and Amy Marrich led a group of over 15 mentors to educate participants of the Grace Hopper Conference’s Open Source Day on how to get started in OpenStack.

Press & Analyst Highlights

Public relations efforts are led by a distributed team of professionals in the United States, Europe, and Asia Pacific. This team engaged analysts and journalists worldwide in dialogue, proactively delivering news, commentary, and contributed byline content for both the Foundation and for its individual projects. Community news highlights are available at https://openstack.org/news.

Key media relations initiatives included the following:

  • Launch of the OpenInfra Foundation
  • Project software releases including OpenStack Ussuri and Victoria, Kata Containers 2.0, and StarlingX 4.0
  • Virtual OpenInfra Summit
  • Virtual OpenDev event
  • Celebration of the OpenStack project’s 10th anniversary
  • Momentum of OpenInfra projects in the broad OpenInfra and cloud IT landscape

Analysis shows that OpenStack/OpenInfra Foundation news coverage comprised more than 24,100 total placements in 2020. Media coverage was distributed globally (31% APAC, 30% North America, 22% Europe, by volume), and extended into 94 countries.

As evidenced by the highlights shared below, media attention has shifted from questions about OpenStack’s future to an active interest in use cases and the progress of the other projects supported by the OpenInfra Foundation.

In 2020, the OpenInfra Foundation adopted a virtual briefing format, shifting from in-person analyst day sessions at the physical events. During the course of the year, Foundation staff leadership conducted briefing sessions with more than two dozen analysts in Asia, North America, and Europe, providing updates on both the evolution of the Foundation’s business model and the status of individual projects in the organization’s portfolio. Firms with strong ongoing interest in our community’s work include IDC, Forrester, Redmonk, 451 Research, and others.

Analyst interest in the community’s work has shifted steadily since the early days of the Foundation. While early coverage necessarily focused on the OpenStack project, the focus has broadened substantially to include other projects under the OpenInfra Foundation. Additionally, coverage has shifted to include OpenInfra Foundation projects as components of broad-based solutions that leverage OpenInfra technologies such as Kubernetes.


The OpenInfra Foundation continued to support the popular Superuser publication, covering the OpenInfra ecosystem news, case studies, event recaps, product updates and announcements, project release and more. In 2020, superuser has reached more than 150,000 unique visitors from 193 countries. Editors and contributors are members of the OpenInfra Foundation, the OpenStack and adjacent open source communities, and professional journalists.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report and join the OpenInfra community to build the next decade of OpenInfra.

The post 2020 Annual Report: OpenInfra Community Highlights appeared first on Superuser.

by Helena Spease at February 11, 2021 02:00 PM

February 10, 2021

OpenStack Superuser

Enhancement of Kubernetes Controller Pattern in Python with CRD

Kuryr is an OpenStack project that relies on Neutron and Octavia to provide connectivity for Kubernetes Pods and Services. This improves the network performance especially when running Kubernetes clusters on top of OpenStack Virtual Machines with Kuryr. It avoids double encapsulation of network packets as only the Neutron overlay is used and consequently simplifies the network stack and facilitates debugging. The following diagram shows the gain of using Kuryr when compared to using Openshift SDN with TCP stream traffic between Pods on nodes across different hypervisors.

Kuryr gain over another sdn https://www.openshift.com/blog/accelerate-your-openshift-network-performance-on-openstack-with-kuryr

Kuryr-Kuberntes has two components that are deployed as K8s Pods: the controller and the CNI. It differs from other OpenStack projects that react to events resulting from user interaction with OpenStack APIs since the CNI and controller actions are based on events that happen at the Kubernetes API level. Kuryr controller is responsible for mapping those Kubernetes API calls into Neutron or Octavia objects and saving the information about the OpenStack resources generated, while the CNI provides the proper network interface binding to the Pods. Figure 02 shows the interactions of Kuryr components, Neutron, Octavia and Kubernetes components.

Kuryr architecture

A controller pattern in Kubernetes works like a control loop that aims to make the system state match the system specification, which is embedded in the Kuberntes API. To achieve this the controller needs to react to Kubernetes events, change the system state by interacting with external systems and saving data when necessary. There are golang libraries to facilitate the implementation of the controller pattern, but Kuryr needed its own version as it’s mainly written in Python.

Controller workflow

Kuryr controller can be considered a collection of controllers where each controller handles a Kubernetes resource that is represented by an OpenStack resource, e.g. Pods are mapped into Neutron ports, Services into Octavia load balancers and Network Policies into Neutron Security groups. Previously, the references to the OpenStack resource used to be saved directly on the corresponding Native Kubernetes resources with key-value pairs of string to ensure the proper OpenStack resource was updated or cleaned up when needed. However, that proved to not facilitate troubleshooting and synchronization of the current state with the specification.

Users can now rely on Kubernetes API extensions with custom resource definitions to introspect the state of the system and view Kubernetes resources easily mapped to OpenStack resources and being handled like any other native Kubernetes object. The custom resource not only allows having small specialized controllers, but also follows the Kubernetes API conventions and patterns, improves readability and debugging. Each resource definition contains the specification and status fields facilitating the synchronization and finalizers to avoid the hard deletion of the Kubernetes resource while the OpenStack resource deletion is being handled. Previously, if the Kubernetes resource was deleted prior to the corresponding OpenStack one being removed a conflict could happen when a new object was created. Examples of Kuryr CRD and more can be found on the Kuryr documentation.

All these new improvements are already available Upstream on Kuryr-Kubernetes since Victoria release.

Get Involved

This article is a summary of the Open Infrastructure Summit session, K8s controller and CRD patterns in Python – kuryr-kubernetes case study.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post Enhancement of Kubernetes Controller Pattern in Python with CRD appeared first on Superuser.

by Maysa Macedo at February 10, 2021 02:00 PM

February 09, 2021

OpenStack Superuser

2020 Annual Report: OpenInfra Collaboration

2020 redefined collaboration. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the Open Infrastructure (OpenInfra) Foundation’s mission: building communities who write software that runs in production. In 2020, Magma’s code and documentation were contributed to a new, independent software project magmacore.org and the need for the services provided by OpenDev has become more apparent; OpenDev has stepped to the plate and made a number of changes to help make 2020 as productive as possible all while minimizing disruption to existing services and users.


Magma is an open-source software platform that delivers an open, flexible and extendable mobile network core solution. Magma’s mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. It is designed to be access network agnostic (4G, 5G or WiFi), flexibly supporting a radio access network with minimal development and deployment effort.

Magma was developed as an open-source project as part of Facebook Connectivity’s incubator program, with significant contributions from the OpenAirInterface Software Alliance. In July of 2020 Magma’s code and documentation was contributed to a new, independent software project magmacore.org.Magma’s architecture consists of three major subsystems:

  • The Access Gateway (AGW) provides network services and policy enforcement. In an LTE network, the AGW implements an evolved packet core (EPC), and a combination of an AAA and a PGW. It works with existing, unmodified commercial radio hardware. Support of 5G radios is under active development.
  • Orchestrator is a service that provides a simple and consistent way to configure and monitor the wireless network securely. The orchestrator can be is normally hosted on a public or private cloud. The metrics acquired through the platform allow you to see the analytics and traffic flows of the wireless users through the Magma web UI. Orchestrator also provides REST APIs allowing simple integration with monitoring and management systems.
  • The federation gateway allows Magma to integrate with other Mobile Network Operator core network functions using standard 3GPP interfaces. It acts as a proxy between the Magma AGW and the operator’s network and facilitates core functions, such as authentication, data plans, policy enforcement, and charging to stay uniform between an existing MNO network and the expanded network with Magma.


At the OpenInfra Summit in October 2020, the OpenInfra Foundation announced its support of the Magma project. With help from the Foundation, magmacore.org has been structured to make it easy to contribute to the project. It is centered around a developer-led governance with Maintainers providing code review and approval for contributions and a small Technical Committee selected from among the project maintainers providing overall architecture and roadmap oversight. In addition, the OpenInfra Foundation also helped launch the Magma Slack workspace back where many community and technical conversations take place. After the Summit announcement, the Slack workspace grew by over 100 people.The OpenInfra Foundation also assists with the monthly Magma community meetings where project updates and use cases are presented. Community members also get the opportunity to ask questions directly to the Magma engineers. The first community meeting took place in November 2020 and had over 20 attendees. View the meeting recordings here.


OpenDev believes that free software needs free tools. OpenDev is a collaboratory for open source software development at a huge scale. Its focus is on code review, continuous integration, and project hosting provided exclusively through open source solutions like Git, Gerrit, Zuul, and Gitea.

As the world shifted to remote work and online collaboration in 2020, the need for the services provided by OpenDev has become more apparent. OpenDev has stepped to the plate and made a number of changes to help make 2020 as productive as possible all while minimizing disruption to existing services and users. In particular we would like to call out the new formalized governance for the project, new services like meetpad, and service upgrades (like Gerrit).

In 2020 OpenDev formalized its governance with two main bodies: the Service Coordinator and the Advisory Board. The Service Coordinator is an individual elected from OpenDev’s contributors whose primary role is to communicate planned changes to users and incorporate their feedback. The Advisory Board is made up of liasons from the various projects using OpenDev and organizations that contribute compute resources. They serve as the primary communcation point between users and the service. If you represent an OpenDev using project or a contributing organizationand would like to get more involved feel free to join the Advisory Board.

One of the big changes we have had to deal with as a community is the transition to virtual events. The OpenDev team has deployed a Jitsi Meet instance which is paired with our Etherpad service to act as a video conferencing with built in note taking tool. This service has been used in multiple Project Team Gatherings at this point and helped us keep an event format that is familiar to those who have attended in person events. Outside of events we have seen this service used for team meetings and other places where video conferencing is helpful. We expect this service to replace the existing Asterisk based phone conference tools that we previously deployed.

Several services have also seen major upgrades during 2020. The most notable (and likely noticeable) is Gerrit, but Gitea and Etherpad have also been upgraded. For Gerrit this upgrade was long overdue and gets us much closer to a modern upstream Gerrit experience. We are hoping that this change will help us work with upstream Gerrit more closely going forward, bringing our needs into upstream development. A number of features come with this upgrade including use of NoteDB for data storage and single page diffs. We continuously upgrade our Zuul deployment, but over the course of 2020 it got some major new features and a new look to its web dashboard as well.

Our long running effort to migrate away from Puppet for service deployments and configuration management continues. Several more services have been migrated onto Ansible and Docker including Hound, Grafana, and our IRC accessbot. This effort enables us to use tools that more developers are comfortable with to manage our services and gives us better control over what is being deployed.One of the major tools that has enabled our transition to well tested Docker based deployments is Zuul’s image build pipeline job library. Zuul provides a set of building blocks to run Docker image builds for changes using local image repositories to share the resulting images between test jobs and changes. This allows any OpenDev user to test sequences of changes to different services and know that each change in the sequence is a landable production state.

On the continuous integration side of the house our CI jobs helped 1,800 users merge more than 54,000 changes. We continue to keep our test environments up to date, pulling in new distro releases and removing old EOL versions. Keep this in mind as CentOS’s announcement that version 8 will be EOL at the end of 2021 means that we will be removing it at the end of this year. CentOS Stream 8 test nodes are available as are alternative distros with long term support and stable releases.

OpenDev continues to be a community run project. These services exist because we all help work together to make it happen. 2020 was a difficult year in many ways, but for OpenDev one of the most noticeable difficulties has been losing individuals who have long been a part of the community providing tremendous amounts of help to OpenDev. We would like to thank Andreas Jaeger and Monty Taylor for all the work they have done over the years. This also serves as a reminder that we can always use more help. If you are interested in getting involved do not hesitate to reach out.Find us at:

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report, and join the OpenInfra community to build the next decade of OpenInfra.

The post 2020 Annual Report: OpenInfra Collaboration appeared first on Superuser.

by Helena Spease at February 09, 2021 02:00 PM

Fleio Blog

2021.02.1: New fleio commands, url uploads in memory stream, performance improvements

Fleio 2021.02.1 was released today, 9th of February. The latest version is marked as stable and can be used in production environment (read more about our release model here). New fleio commands In the latest release we have worked on improving the new fleio bash command (which is available only when using Fleio deployed with […]

by Marian Chelmus at February 09, 2021 09:43 AM

February 08, 2021

Alessandro Pilotti

OpenStack on ARM64 – Deployment

In the previous blog post we created the ARM OpenStack Kolla container images and we can now proceed with deploying OpenStack. The host is a Lenovo server with an Ampere Computing eMAG 32 cores Armv8 64-bit CPU, running Ubuntu Server 20.04. For simplicity, this will be an “All-in-One” deployment, where all OpenStack components run on the same host, but it can be easily adapted to a multi-node setup.

Let’s start with installing the host package dependencies, in case those are not already there, including Docker.

sudo apt update
sudo apt install -y qemu-kvm docker-ce
sudo apt install -y python3-dev libffi-dev gcc libssl-dev python3-venv
sudo apt install -y nfs-kernel-server

sudo usermod -aG docker $USER
newgrp docker

We can now create a local directory with a Python virtual environment and all the kolla-ansible components:

mkdir kolla
cd kolla

python3 -m venv venv
source venv/bin/activate

pip install -U pip
pip install wheel
pip install 'ansible<2.10'
pip install 'kolla-ansible>=11,<12'

The kolla-ansible configuration is stored in /etc/kolla:

sudo mkdir -p /etc/kolla/config
sudo cp -r venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla
sudo chown -R $USER:$USER /etc/kolla
cp venv/share/kolla-ansible/ansible/inventory/* .

Let’s check if everything is ok (nothing gets deployed yet):

ansible -i all-in-one all -m ping

kolla-genpwd is used to generate random passwords for every service, stored in /etc/kolla/passwords.yml, quite useful:


Log in to the remote Docker registry, using the registry name and credentials created in the previous post:

ACR_NAME=# Value from ACR creation
SP_APP_ID_PULL_ONLY=# Value from ACR SP creation
SP_PASSWD_PULL_ONLY=# Value from ACR SP creation

docker login $REGISTRY --username $SP_APP_ID --password $SP_PASSWD

Now, there are a few variables that we need to set, specific to the host environment. The external interface is what is used for tenant traffic.

VIP_ADDR=# An unallocated IP address in your management network
MGMT_IFACE=# Your management interface
EXT_IFACE=# Your external interface
# This must match the container images tag

Time to write the main configuration file in /etc/kolla/globals.yml:

sudo tee -a /etc/kolla/globals.yml << EOT
kolla_base_distro: "ubuntu"
openstack_tag: "$OPENSTACK_TAG"
kolla_internal_vip_address: "$VIP_ADDR"
network_interface: "$MGMT_IFACE"
neutron_external_interface: "$EXT_IFACE"
enable_cinder: "yes"
enable_cinder_backend_nfs: "yes"
enable_barbican: "yes"
enable_neutron_provider_networks: "yes"
docker_registry: "$REGISTRY"
docker_registry_username: "$SP_APP_ID_PULL_ONLY"

The registry password goes in /etc/kolla/passwords.yml:

sed -i "s/^docker_registry_password: .*\$/docker_registry_password: $SP_PASSWD_PULL_ONLY/g" /etc/kolla/passwords.yml

Cinder, the OpenStack block storage component, supports a lot of backends. The easiest way to get started is by using NFS, but LVM would be a great choice as well if you have unused disks.

# Cinder NFS setup
# Replace with your local network CIDR if you plan to add more nodes
sudo mkdir /kolla_nfs
echo "/kolla_nfs $CINDER_NFS_ACCESS(rw,sync,no_root_squash)" | sudo tee -a /etc/exports
echo "$CINDER_NFS_HOST:/kolla_nfs" | sudo tee -a /etc/kolla/config/nfs_shares
sudo systemctl restart nfs-kernel-server

The following settings are mostly needed for Octavia, during the next blog post in this series:

# Increase the PCIe ports to avoid this error when creating Octavia pool members:
# libvirt.libvirtError: internal error: No more available PCI slots
sudo mkdir /etc/kolla/config/nova
sudo tee /etc/kolla/config/nova/nova-compute.conf << EOT
resume_guests_state_on_host_boot = true


# This is needed for Octavia
sudo mkdir /etc/kolla/config/neutron
sudo tee /etc/kolla/config/neutron/ml2_conf.ini << EOT
network_vlan_ranges = physnet1:100:200

Time to do some final checks, bootstrap the host and deploy OpenStack! The Deployment will take some time, this is a good moment for a coffee.

kolla-ansible -i ./all-in-one prechecks
kolla-ansible -i ./all-in-one bootstrap-servers
kolla-ansible -i ./all-in-one deploy

Congratulations, you have an ARM OpenStack cloud! Now we can get the CLI tools to access it:

pip3 install python-openstackclient python-barbicanclient python-heatclient python-octaviaclient
kolla-ansible post-deploy
# Load the vars to access the OpenStack environment
. /etc/kolla/admin-openrc.sh

The next steps are optional, but highly recommended in order to get the basic functionalities, including basic networking, standard flavors and a basic Linux image (Cirros):

# Set you external netwrork CIDR, range and gateway, matching your environment, e.g.:
export EXT_NET_CIDR=''
export EXT_NET_RANGE='start=,end='

All done! We can now create a basic VM from the command line:

# Create a demo VM
openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1

You can also head to “http://<VIP_ADDR>” and access Horizon, OpenStack’s web ui. The username is admin and the password is in /etc/kolla/passwords.yml:

grep keystone_admin_password /etc/kolla/passwords.yml

In the next post we will add to the deployment Octavia, the load balancer as a service (LBaaS) component, Enjoy your ARM OpenStack cloud in the meantime!

P.S.: In case you would like to delete your whole deployment and start over:

#kolla-ansible -i ./all-in-one destroy --yes-i-really-really-mean-it

The post OpenStack on ARM64 – Deployment appeared first on Cloudbase Solutions.

by Alessandro Pilotti at February 08, 2021 01:00 PM

February 04, 2021

CERN Tech Blog

10 Years of Openstack at CERN - Part 2

In this blog post we will go back 10 years to understand how OpenStack started at CERN. In Part 1 we explored the motivation, the first installation, the prototypes, the deployment challenges and the initial production architecture. Now, we will dive into the service offering evolution of the CERN cloud infrastructure over the last years and how we scaled from 2 cells and few hundred compute nodes to 75 cells with more than 8000 compute nodes.

by CERN (techblog-contact@cern.ch) at February 04, 2021 08:00 AM

February 02, 2021

Fleio Blog

2021.02.0 beta: New fleio commands, url uploads in memory stream, performance improvements

Fleio version 2021.02.0 is now available in beta and you can test it in your lab environment, since this release is not recommended for production. You can read more about our release model here. New fleio commands In the latest development cycle we have added some quality of life improvements to Fleio (when installed with […]

by Marian Chelmus at February 02, 2021 09:17 AM

February 01, 2021


Bare Basics of DNS Service with OpenStack Designate

Bare Basics of DNS Service with OpenStack Designate

DNS as a Service (DNSaaS) is a topic that pops up quite frequently, with many companies outsourcing their IT infrastructure to cloud providers and such. Designate is the DNSaaS component for OpenStack-based clouds, and here is a basic overview of the service. First, let us start by explaining what DNS means. 

What is a DNS Service?

DNS, short for Domain Name System, is a ranked and distributed naming system for computers and similar resources connected to the internet or private networks. The DNS system has been around since 1985 and is still a major component of the way the internet works. DNS associates different information with assigned domain names and translates them to IP addresses for easier location across networks. 

To say it in more simple terms, there are three elements you need to make up a website – 

Registered domain name – for example, www.vexxhost.com.

DNS – configuration file for your domain.

Web host – manages all the files that constitute the website.

DNS simplifies the communication between computers and servers through a network and provides a friendly method for users to interact with and get the desired information. 

Now that we have a basic idea of what DNS is let us move on to OpenStack’s DNS Service, Designate.

OpenStack Designate as a DNS Service

Designate is an open source project providing DNS as a Service and comes under the umbrella of OpenStack services. Like every other OpenStack service, it has a standard RESTful API and allows multi-tenancy.

For Designate, this API helps in record and domain management. The DNS can be configured to create records based on VMs and network triggers automatically. The service also boasts an integrated identity authentication with OpenStack Keystone. Here is a look at the various advantages of Designate as a DNS Service.

Advantages of OpenStack Designate

OpenStack Designate is a fully managed DNS service that helps in automating the mapping and management of IP addresses to domains. With Open APIs, it provides a simplified management structure and, as mentioned before, automatically generates DNS records for servers. It is also beneficial in maintaining consistent registration records for the servers. Users often employ the same DNS backend within projects without collision issues. 

Furthermore, seamless integration with other OpenStack projects is another advantage of Designate. The prior mentioned ease of integration with Keystone identity service provides increased security. Its integration with Nova and Neutron notifications is helpful in the auto-generation of records. Designate also allows for a multi-tenant atmosphere, hosting different projects with ease and keeping the resources secure and ready to use. 

VEXXHOST Cloud with DNS Service

At VEXXHOST, we offer DNS Service using Designate to our customers in creating their OpenStack-based cloud environments. Various OpenStack services such as Designate are deployed according to specific use cases. They are also often customized in our cloud solutions, such as the case with our highly scalable private cloud offerings. 

Speaking of private clouds, you can now run on a fully agile and customized cloud from VEXXHOST, with no licensing fees and smooth 2-week migration. In fact, we’re ready to put our money where our mouth is. We’re so confident in being able to save you at least 20% or more on your current cloud infrastructure expenditure that if proven wrong- we’ll give you $1,000 credit to our public cloud.   

Excited? Find out more

Like what you’re reading?
Deep dive into a hands-on ebook about how you can build a successful infrastructure from the ground up!

The post Bare Basics of DNS Service with OpenStack Designate appeared first on VEXXHOST.

by Athul Domichen at February 01, 2021 08:58 PM

January 29, 2021

Galera Cluster by Codership

Video: Installing Galera Cluster with MySQL

Codership is pleased to announce the launch of our training video library. The aims are quite clear as we want you to be helped in learning, installing, configuring and running Galera Cluster and related software in a successful fashion. Naturally, this also serves as a good teaser for our training courses which we are going to deliver online. This is part of an upcoming series of blogs covering the video contents, and why you might want to watch them.

Today we are focusing on Installing Galera Cluster with MySQL. Naturally it is worth clicking that link because you will see how well organised it is, including the fact that there are training exercises, slides in case you choose to take this offline, example configuration, as well as a sample company database (note that it is about 64MB in size).

It starts with an introduction of Codership. There is an introduction to Galera Cluster, focusing on nodes, databases, replication, load balancing and High Availability, around the master-slave (primary-secondary, leader-follower) model. It then covers important concepts and wins around Galera Cluster: virtually synchronous replication, multi-master, conflict detection & resolution on commit (when there are conflicting transactions), easy maintenance (automatic node provisioning, node isolation, rolling upgrades).

The basics that you need to be on Linux (CentOS 7 in this video) or FreeBSD, the fact that you want to have odd-numbers of nodes (this training course focuses on 3 nodes), and to ensure that all nodes have equal hardware configuration. The rsync software as well as the firewall software (firewalld) is also something you need. We talk about the Primary Component, and how you may get an Incremental State Transfer (IST) or a State Snapshot Transfer (SST) (it also is a good idea to read: node provisioning). We  then move on to show you the terminal so you can follow along, and how to get the repository URLs for a Galera Cluster repository.

This is based on MySQL 5.7 + Galera 3 which you get from the Codership repositories, so remember that when you execute system start mysqld you will need to grep 'temporary password' /var/log/mysqld.log to get the initial password. Good practice of running mysql_secure_installation is also something Russell covers; and the fact that you need to do this on each of the three nodes.

Configuring Galera Cluster is not just now in your /etc/my.cnf but /etc/my.cnf.d/server.cnf. I can highly recommend you follow the configuration, including turning on the error log. It is worth spending time learning how to configure Galera Cluster, including understanding the basic wsrep_provider_options, the wsrep_slave_threads, and what the wsrep_sst_method should be (it does not go in deep, here you use rsync).

We mention about the ports that you need to open up as well, and to ensure that SELinux understands this. The ports in question are: 3306, 4567 (TCP and UDP), 4444, 4568 (when not specified, just TCP is required). You will also need to set these inbound rules on AWS — this is not pictured in the video, so remember to turn this on in the AWS firewall. Then configuring SELinux (it is good, do not disable it) and the firewall via firewall-cmd is shown.

The video goes on to talk about bootstrapping the cluster, e.g. via mysqld_bootstrap. Within the terminal, and also shows you what it is like in /var/lib/mysql/grastate.dat and the important option of safe_to_bootstrap.

Overall, once you are done with the video, you will be able to have Galera Cluster running. Next up, how to do this with MariaDB Server.


by Sakari Keskitalo at January 29, 2021 09:15 AM

January 28, 2021

OpenStack Superuser

Inside Open Infrastructure: The latest from the OpenInfra Foundation

Spotlight on:

Open Infrastructure Summit schedule is live!

Despite being a very different year than most, the OpenInfra community, which has over 110,000 community members, made it a productive and successful year. 2020 was the start of the next decade of OpenInfra tackling challenges including hardware diversification, deployment diversification, government regulations like data sovereignty, and the availability of more open source than ever.

Each community member, working group, SIG, and contributor was instrumental in continuing to support the OpenInfra Foundation’s mission: building open source communities that write software that runs in production. The 2020 Annual Report was published today highlighting the achievements across the community and the goals for the year ahead.

Let’s break down some of the highlights of last year:

  • The OpenStack Foundation has taken the next step in its ongoing evolution by becoming the OpenInfra Foundation to support open infrastructure for a market estimated at $20 billion USD.
  • Over 60 global organizations joined in establishing the OpenInfra Foundation to build the next decade of infrastructure for AI, 5G and edge computing, including Platinum Members Ant Group, AT&T, Ericsson, FiberHome, Huawei, Red Hat, Wind River, Tencent, and Facebook.
  • OpenStack, one of the top three most active open source projects with 15 million cores in production, marked its 10th anniversary in July 2020
  • The OpenInfra Foundation shared its support for the Magma project, announced OpenInfra Labs as its newest pilot project and confirmed StarlingX as a top-level Open Infrastructure project.
  • The OpenInfra community became more accessible internationally through the first virtual OpenInfra Summit that gathered 10,000 attendees from 120+ different countries to discuss more than 30 open source projects

The OpenInfra Foundation would like to extend a huge thanks to the global community for all of the work that went into 2020 and is continuing in 2021 to help people build and operate OpenInfra. Check out the full OpenInfra Foundation 2020 Annual Report and join us to build the next decade of OpenInfra!

OpenInfra Foundation news

  • This week, the 2021 OpenInfra Board of Directors met for the first time, onboarding new members and thanking the members who served last year. Allison Randal was elected Board Chair, and you can learn here about the new board as they start the next decade of OpenInfra. 
  • The OpenInfra Foundation is helping the Magma community organize its Magma Developers Conference on February 3. The virtual event is free to attend and participants can expect to hear about the Magma development roadmap, including the upcoming 1.4 release, as well as use cases in production.

Airship: Elevate your infrastructure

  • Airship 2.0 is Kubernetes certified! 
    • Airship 2.0 has been designated as a Certified Kubernetes Distribution through the Cloud Native Computing Foundation’s Software Conformance Program, guaranteeing that Airship provides a consistent installation of Kubernetes, supports the latest Kubernetes versions, and provides a portable cloud-native environment with other Certified Platforms.
  • Airship Container as a Service
    • Two operators will manage the lifecycle of virtual machines and their relationship to the cluster: vNode-Operator (ViNO) and the Support Infra Provider (SIP). These operators are new projects to the Airship ecosystem and were migrated to OpenDev earlier this month. Read more about ViNO and SIP on the blog.
  • General Availability of Airship 2.0 is on track for release in Q1.

Kata Containers: The speed of containers, the security of VMs

  • Kata Containers was represented at the Cloud Native Infrastructures Meetup in Beijing.
    • Learn about why China Mobile Cloud selected Kata Containers as its security containers, and some performance tuning exploration and optimization ideas for Kata on Kata blogs.
  • Are you a Kata Containers user? Please take a few moments to fill out the User Survey to provide feedback and information around your deployment. All information is confidential to the OpenInfra Foundation unless you designate that it can be public.

OpenStack: Open source software for creating private and public clouds

StarlingX: A fully featured cloud for the distributed edge

  • The community is currently working on the 5.0 release with focus on security related features like secret management and image signature validation. The release is planned to come out by the summer.
  • The next round of the Technical Steering Committee election will also happen during the second quarter of the year. If you are interested in running for a seat start contributing now to be eligible to participate!

Zuul: Stop merging broken code

  • The release of Nodepool 3.14.0 brings improvements in Kubernetes, OpenShift, and containers.
  • James E. Blair is reconfirmed as the Zuul Project Lead for 2021, continuing to serve as a central point of contact for the project, and to guide the consensus-driven decision making process on which the community relies.
  • The Zuul Maintainers reached agreement on a new specification for a solution to locally reproduce job builds.
  • Planning is currently underway for a possible build log and artifact storage access proxy complimenting Zuul, feel free to join the discussion.

Upcoming Virtual OpenInfra and Community Events

For more information about OpenInfra community events, please contact community@openinfra.dev

Questions / feedback / contribute

This newsletter is written and edited by the OSF staff to highlight open infrastructure communities. We want to hear from you! If you have feedback, news or stories that you want to share, reach us through community@openinfra.dev . To receive the newsletter, sign up here.

The post Inside Open Infrastructure: The latest from the OpenInfra Foundation appeared first on Superuser.

by Helena Spease at January 28, 2021 06:32 PM

OpenStack Blog

OpenStack Project Update 2020

Despite being a very different year than most, the Open Infrastructure community, which has over 110,000 community members, made it a productive and successful year. One of the biggest milestones that happened in the global community last year was that OpenStack, one of the top three most active open source projects with 15 million cores... Read more »

by Sunny at January 28, 2021 05:38 PM

January 27, 2021

OpenStack Superuser

Magma Developers Conference Agenda Highlights Community Development Momentum and Production Use Cases

The Magma open source community is joining the world of virtual conferences next week with their Magma Developer Conference on February 3. Hundreds of developers, system integrators, engineers, and end users are expected to attend and discuss the technical roadmap as well as how Magma is already connecting people in production.

Magma is an open-source software platform that delivers an open, flexible and extendable mobile network core solution. Magma’s mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. It is designed to be access network agnostic (4G, 5G or WiFi), flexibly supporting a radio access network with minimal development and deployment effort. The 1.4 release is scheduled for later this week, so at the Developer Conference, you can learn more about the newest features and what you can expect from upcoming releases on the project roadmap.

To highlight the potential of running Magma in production, case studies will be presented by Baicells and MuralNet—whose user story may sound familiar if you attended the OpenInfra Summit keynotes in October 2020. I’ve also heard that the 5G fixed wireless access demo will be something worth tuning in for. Below is a sneak peek of what you can expect.

To participate, you can register for free, and I encourage you to join the Magma community Slack and the #magma-dev-conf channel in particular so you can chat with other attendees. The conversation will also occur on Twitter around #MagmaDevConf.

If you want to learn more about the open source community, how you can get involved, and if it is a good network solution for your organization, join us!

The post Magma Developers Conference Agenda Highlights Community Development Momentum and Production Use Cases appeared first on Superuser.

by Allison Price at January 27, 2021 04:00 PM

January 25, 2021

OpenStack Superuser

Auto-scaling in Viettel Networks Without OpenStack Telemetry

Auto-scaling is a must ability when it comes to the functionality of a cloud. OpenStack already provides multiple methods to auto-scale virtual machines with Heat, Telemetry stack. However, Telemetry the main stack, which involves in the evaluation process whether an auto-scale is needed, either lacks the contribution or malfunctions. In this article, we will introduce a new tool Faythe, an open-sourced software that we built ourselves, works as a bridge between the cloud platform (OpenStack) and monitoring system (Prometheus), to solve the auto-scale problem in Viettel, which is the largest telco in Vietnam.

Auto-scaling with the traditional approach with Heat & Telemetry

The fundamental idea

Auto-scaling is the combination of three steps:

  • Metering
  • Alarming
  • Scaling

Metering service collects data about service/instance/host load. Alarm service evaluates the data to find out whether you need to add scale-out/scale-in, which triggers the scale-out/scale-in action on Scaling service. The idea is quite simple.

Auto-scaling with the traditional approach with Heat & Telemetry


This approach consists of two main components which are packaged and distributed together with OpenStack infrastructure.

  • Orchestration: The core component providing auto-scaling is Orchestration (heat). Use Orchestration to define rules using human-readable YAML templates. These rules are applied to evaluate system load based on Telemetry data to find out whether you need to add more instances into the stack. When the load drops, Orchestration can automatically remove the unused instances again.
  • Telemetry: Telemetry does OpenStack environment performance monitoring, collecting data on CPU, storage, and memory utilization for instances and physical hosts. Telemetry also involves three smaller services:
  • Ceilometer – Monitoring Service: a data collection service that provides the ability to normalize and transform data across all OpenStack core components.
  • Gnocchi – Storage Service: provides a time-series resource indexing, metrics storage.
  • Aodh – Alarming Service: triggers actions based on defined rules against sample or event data collected by Ceilometer and stored in Gnocchi.

The Telemetry issues

  • Telemetry projects (in the Rocky release, at the time we took a survey) have both lack of contribution from the opensource community, which is one of the factors defining open-sourced project longevity.
  • As of the Queens release, the Telemetry project had lost most developers. Development on Aodh and Panko stopped.
  • Gnocchi was moved out of OpenStack in June 2017. Gnocchi is unmaintained.
  • The three projects are dependent on each other, however, which are developed separately. Leading to a small change in one project can make the whole system collapses. Moreover, since Gnocchi is no longer a part of OpenStack, unsynchronized development issues between these are unavoidable.
  • In our experiences, RabbitMQ was under heavy load due to Ceilometer workload. If Ceilometer gets stuck, its queues overflow.
  • Telemetry only collects information about virtual machines operating systems and OpenStack physical hosts. It is impossible for Telemetry to monitor services and applications running on virtual machines. Additionally, changing and implementing more features for Telemetry are not simple tasks. That includes writing codes, compiling codes, packaging, testing, and deploying to real system.
  • Telemetry doesn’t fit our ecosystem. We chose Prometheus to monitor the whole infrastructure system as well as applications. Switching/Setup other systems like Telemetry alongside Prometheus is not ideal.

Auto-scaling with Faythe and Prometheus

How it works

Actually, this is not a completely new approach, it leverages Orchestration to do scaling action with new Alarming Service – Faythe and Monitoring Service – Prometheus. As mentioned before, auto-scaling is the combination of three steps: Metering, Monitoring, and Scaling. OpenStack Orchestration is already providing superb scaling service, Heat. Instead of using OpenStack Telemetry for Metering and Alarming, the new approach introduces Faythe, an open-sourced software that we built ourselves, working as a bridge between Cloud platform (OpenStack) to the Monitoring system (Prometheus),  and Prometheus.

Why Faythe?

Faythe is one of our proudly created projects, which is designed, developed and implemented while we are working at Viettlet Network. Faythe emerges to fill in the gap between Prometheus and OpenStack.

  • Faythe supports Prometheus to collect metrics and PromQL to evaluate and decide whether it is necessary for an alarm.
  • PromQL is a very nice query language.
  • A wide range of exporters: this is the most valuable point of Prometheus. The open-source community is creating more and more new exporters, which means there are fresh metrics that can be collected by Prometheus.
  • Flexible: besides the system factor like CPU or Memory usage, users can evaluate any metrics that can be collected by any of the exporters, for example, JVM metrics, HTTP requests related metrics. For example, you can define a scaling rule depend on the number of Django HTTP requests.

Faythe provides not only a user-friendly UI but also complete APIs.

For more details, you can take a look at Faythe repository.

The post Auto-scaling in Viettel Networks Without OpenStack Telemetry appeared first on Superuser.

by Kien Nguyen-Tuan and Dat Vu Tuan at January 25, 2021 02:00 PM

Alessandro Pilotti

OpenStack on ARM64 – Kolla container images

This is the beginning of a short series detailing how to deploy OpenStack on ARM64, using Docker containers with Kolla and Kolla-ansible.

The objective of this first post is to create the ARM64 container images and push them to a remote registry in order to be used later on, when deploying our OpenStack cloud. We are going to use Azure Container Registry to store the images, but any other OCI compliant registry will do.

Create a container registry

Let’s start by creating the container registry and related access credentials. This can be done anywhere, e.g. from a laptop, doesn’t need to be ARM. All we need is to have the Azure CLI installed.

az login
# If you have more than one Azure subscription, choose one:
az account list --output table
az account set --subscription "Your subscription"

Next, let’s create a resource group and a container registry with a unique name. Choose also an Azure region based on your location.


az group create --name $RG --location $LOCATION
az acr create --resource-group $RG --name $ACR_NAME --sku Basic

We’re now creating two sets of credentials, one with push and pull access to be used when creating the images and one with pull only access to be used later on during the OpenStack deployment.

ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)
SP_PASSWD=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpush --query password --output tsv)
SP_APP_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)

SP_PASSWD_PULL_ONLY=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpull --query password --output tsv)
SP_APP_ID_PULL_ONLY=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)

Create and push the OpenStack Kolla container images

It’s now time to switch to an ARM server where the Kolla container images will be built. We are going to use a Lenovo server with an eMAG 32 cores Armv8 64-bit CPU provided by Ampere Computing. The host operating system is Ubuntu 20.04, but the following instructions can be easily adapted to other Linux distros.

Let’s start with installing the dependencies and add your current user to the docker group (or create a separate user).

sudo apt update
sudo apt install -y docker-ce python3-venv git
sudo usermod -aG docker $USER
newgrp docker

Let’s get Docker to login into the remote registry that we just created. Set ACR_NAME, SP_APP_ID and SP_PASSWD as obtained in the previous steps.

docker login $REGISTRY --username $SP_APP_ID --password $SP_PASSWD

Now we can install Kolla in a Python virtual environment and get ready to start building our container images. The OpenStack version is the recently released Victoria but a previous version can be used if needed (e.g. Ussuri).

mkdir kolla-build
cd kolla-build
python3 -m venv venv
source venv/bin/activate
pip install wheel
# Install Kolla, Victoria version
pip install "kolla>=11,<12"

Edit: the following step can be skipped on Victoria since it defaults to Ubuntu 20.04 where pmdk-tools is available. Additionally, thanks to a recent patch, it can be skipped on Ussuri and Train.

The pmdk-tools Ubuntu package is not available on ARM, so we need to remove it from the nova-compute docker image build. This is done by creating a “template override” that we are going to pass to the build process.

tee template-overrides.j2 << EOT
{% extends parent_template %}

# nova-compute
{% set nova_compute_packages_remove = ['pmdk-tools'] %}

We can now build the container images and push them to the registry. This will take a while since it’s building and pushing container images for all OpenStack projects and services. Alternatively, it is possible to reduce the number of containers to a subset by creating a profile in kolla-build.conf as explained here.

kolla-build -b ubuntu --registry $REGISTRY --push
# If you created a template override run:
# kolla-build -b ubuntu --registry $REGISTRY --template-override template-overrides.j2 --push

We are finally ready for our OpenStack AMR64 deployment with Kolla-ansible in the next post!

The post OpenStack on ARM64 – Kolla container images appeared first on Cloudbase Solutions.

by Alessandro Pilotti at January 25, 2021 09:00 AM

January 20, 2021

Lee Yarwood

OpenStack Nova Block Device Mapping Data Structures

Block Device Mapping? Block Device Mapping(s)(BDMs) define how block devices are exposed to an instance by Nova. At present Nova accepts and stores data relating to these mappings in various ways across the codebase often leading to the five stages of grief being experienced by anyone unlucky enough to come into contact with them. The user facing format while awkward and unforgiving is pretty well documented both in the API reference guide and project user documentation.

January 20, 2021 07:35 PM

OpenStack Superuser

New in OpenInfra: VEXXHOST’s Public Cloud Revamp Includes a Lot More Than the New Data Center in Amsterdam

VEXXHOST is a Canadian cloud computing provider with an impressive global presence and record. As a former Superuser Awards winner and a major contributor to the open source community, VEXXHOST’s growth trajectory is of keen interest to any member of the community. 

From small businesses to governments, VEXXHOST’s operations are spread over 150 countries. The company started as a web hosting provider in 2006 and later transitioned into cloud solutions. They adopted OpenStack for their cloud operations in 2011, coinciding with the platform’s second release, Bexar. VEXXHOST now offers OpenStack-based public cloud, private cloud, consulting, and other enterprise-grade cloud solutions. 

VEXXHOST recently announced a massive revamp in its public cloud offerings during its keynote address at the Open Infrastructure Summit 2020. There are some new and exciting developments, including a brand new Data Center in Amsterdam. The company says that the revamp is part of their new growth strategy, powered by their association with various open source foundations and communities. They are a reputed member of the Open Infrastructure Foundation, Linux Foundation, CNCF, and the Ceph Foundation, to name a few. 

With this revamp, VEXXHOST joins the range of community-driven companies evolving and growing strong even during these testing times (i.e., pandemic time). Without further delay, here’s what they have included in the cloud revamp. 

Amsterdam – New Region

The Amsterdam data center is VEXXHOST’s third region to date, the other two being Montreal and Santa Clara. VEXXHOST also manages several private cloud operations for different enterprises and organizations in various other parts of the world, in association with local data centers. The launch of the new region not only puts them physically in a different continent but also cements their global footprint.

The state-of-the-art data center used in Amsterdam provides unparalleled service in terms of connectivity, efficiency, server protection mechanisms,  security, and reliability. The facility is built to Uptime Institute Tier 3 standards, and has  PCI/SOC, and AMS-IX, and multiple ISO certifications. The Amsterdam data center also boasts of a multi-tier security system, 24×7 on-site support, high energy efficiency with green standards, advanced global connectivity via being linked to AMS-IX, among other features. 

Modified Public Cloud Pricing 

Another change that is part of the revamp is VEXXHOST’s new pricing strategy. The modified pricing is available to all new users as well as existing customers who make the switch to their latest solutions. 

According to VEXXHOST’s new introductory and limited-time pricing model, the hourly rate for a standard 2 core, 8 GB offering is just $0.055, compared to the market rate, which stands at $0.086. The difference between the current market rate and the new VEXXHOST rate is 36%. Higher core/memory offerings such as 4 core – 16 GB, 8 core – 32 GB, and 16 core – GB, will also carry proportionate price reduction.

This new pricing strategy couldn’t have come at a better time. Many businesses are facing a lot of challenges due to the pandemic situation and there is also a need for companies to find remote and cloud solutions to adapt to the rapid changes around. Getting users enterprise-grade cloud solutions at great pricing is beneficial not only for the parties involved but also for the overall community.

Montreal Region – 2nd Gen AMD EPYC™-powered Servers 

Next in VEXXHOST’s revamp is the addition of new servers equipped with 2nd Gen AMD EPYC™ processors for their Montreal region. Driven by AMD Infinity Architecture, these processors are the first x86 processor of their kind for servers based on 7nm process technology. Also boasting of a hybrid, multi-die architecture, they offer up to 64 high-performance cores per SOC and are equipped with AMD Infinity Guard and PCIe® Gen 4 I/O security features. 

The new AMD processors bring about significant change to VEXXHOST’s servers in better performance and secure computing, improving workload acceleration, data protection, and overall infrastructure to the users.

Montreal Region – Upgraded Storage from SSD to NVMe

VEXXHOST has upgraded storage from SSD to NVMe in their Montreal region, offering its public cloud users the fastest hard drive in the market. A major benefit available with this storage type is faster parallel read and write capabilities.  

NVMe drives can work at speeds of more than 2,000MB/s compared to the typical SATA III SSD running under 600MB/s. This is possible because of the interaction with flash memory, made through a PCIe interface. This interface is bi-directional and runs at a stellar speed. NVMe storage is also better in power-efficiency, reducing consumption by significant percentages in standby mode. NVMe is also a scalable alternative going beyond the four lanes found in most conventional PCIe SSDs. 

Moving forward, VEXXHOST has a lot planned in terms of cloud offerings to customers. If you would like to know more about the company and its extensive cloud solutions, check out VEXXHOST’s website.

The post New in OpenInfra: VEXXHOST’s Public Cloud Revamp Includes a Lot More Than the New Data Center in Amsterdam appeared first on Superuser.

by Allison Price at January 20, 2021 02:00 PM

January 14, 2021


RDO plans to move to CentOS Stream

What changed with CentOS?
CentOS announced recently that they will focus on CentOS Stream and CentOS Linux 8 will be EOL at the end of 2021.

While CentOS Linux 8 (C8) is a pure rebuild of Red Hat Enterprise Linux (RHEL), CentOS Stream 8 (C8S) tracks just ahead of the current RHEL release. This means that we will have a continuous flow of new packages available before they are included in the next RHEL minor release.

What’s the current situation in RDO?
RDO has been using the latest CentOS Linux 8 to build both the OpenStack packages and the required dependencies since the Train release for both for the official CloudSIG repos and the RDO Trunk (aka DLRN) repos.

In the last
few months, we have been running periodic CI jobs to validate RDO Trunk repos built on CentOS Linux 8 along with CentOS Stream 8 to find any potential issues created by OS package updates before they are shipped in CentOS Linux 8. As expected, during these tests we have not found any issue related to the buildroot environment, packages can be used for both C8 and C8S. We did find a few issues related to package updates which allowed us to propose the required fixes upstream.

What’s our plan for RDO roadmap?
  • RDO Wallaby (ETA is end of April 2021) will be built, tested and released only on CentOS 8 Stream.
  • RDO CloudSIG repos for Victoria and Ussuri will be updated and tested for both CentOS Stream and CentOS Linux 8 until end of 2021 and then continue on CentOS Stream.
  • We will create and test new RDO CloudSIG repos for Victoria and Ussuri on CentOS Stream 8.
  • The RDO Trunk repositories (aka DLRN repos) will be built and tested using CentOS 8 Stream buildroot for all releases currently using CentOS Linux 8 (since Train on)

How do we plan to implement these changes?
Some implementation details that may be of interest:
  • We will keep building packages just once. We will move buildroots for both DLRN and CloudSIG to use CentOS Stream 8 in the near future.
  • For Ussuri and Victoria CloudSIG repos, while we are supporting both C8 and C8S, we will be utilizing separated CBS Tags. This will allow us to have separated repositories, promotion jobs and package versions for each OS.
  • In order to reduce the impact of potential issues and discover issues related to C8S as soon as possible, we will put more focus on periodic jobs on C8S.
  • At a later stage, we will move the CI jobs used to gate changes in distgits to use C8S instead of C8 for all RDO releases where we use CentOS 8.
  • The CentOS/RHEL team has made public their interest in applying Continuous Delivery approach to CentOS Stream to provide a stable CentOS Stream using gating integration jobs. Our intent is to collaborate with the CentOS team on any initiatives that will help to validate RDO as early in the delivery pipeline as possible and reduce the impact on potential issues.

What’s next?
We plan to start the activities needed to carry out this plan in the next weeks.

We will continue discussing and sharing the progress
during the RDO weekly meetings, feel free to join us if you are interested.

Also, If you have any question or suggestion related to these changes, don’t hesitate to contact us
in the #rdo freenode channel or using the RDO mailing lists.

by amoralej at January 14, 2021 03:17 PM

Galera Cluster by Codership

Galera Cluster for MySQL 5.6.50, 5.7.32, and 8.0.22 released

Codership is pleased to announce a new Generally Available (GA) release of the multi-master Galera Cluster for MySQL 5.6, 5.7 and 8.0, consisting of MySQL-wsrep 5.6.50 (release notes, download), 5.7.32 (release notes, download) and 8.0.22 (release notes, download) with Galera replication library 3.32 (release notes, download) implementing wsrep API version 25, and Galera replication library 4.7 (release notes, download) implementing wsrep API version 26 for 8.0. This release incorporates all changes to MySQL 5.6.50, 5.7.32 and 8.0.22 respectively, adding a synchronous option for your MySQL High Availability solutions.

This release is notable because there have been improvements with CRC32 hardware support compatibility on ARM based architectures for the Galera replication library 3.32 and 4.7. GCache locking has also been improved.

From a lower level, the build system has changed from using SCons to CMake, so while SCons scripts are still included they will not be actively maintained any longer.

In MySQL 5.6.50, we improved the XtraBackup v2 SST script for FreeBSD users.

In MySQL 5.7.32, we added a new variable wsrep_mode. The first application of the variable is to allow ignoring native replication filter rules if configured with replicate-do-db. We also improved LOAD DATA splitting to generate Xid events for binary log events with intermediate commits.

In MySQL 8.0.22, we now have a new SST method based on the CLONE plugin. Similar improvements around wsrep_mode exist too. Improvements around foreign key handling are also present in this release: load data with foreign keys had intermittent failures that have been fixed, as well as BF-BF conflicts between OPTIMIZE/REPAIR/ALTER TABLE and DML in presence of foreign key constraints were suppressed. We would also like to note that Percona XtraBackup version 8.0.22-15.0 and greater is required to perform XtraBackup based SSTs.

For now, MySQL 5.6.50 and 5.7.32 are the last official builds for CentOS 6 and Red Hat Enterprise Linux 6. It is also important to remember that MySQL 5.6 is nearing End Of Life (EOL) so we do recommend that you upgrade to MySQL-wsrep 5.7.

by Sakari Keskitalo at January 14, 2021 12:11 PM

Fleio Blog

2021.01.1: File uploads in memory streams, improved user feedback, ussuri support, new angular pages for enduser

Fleio 2021.01.1 was released today, 14th of January. The latest version is marked as stable and can be used in production environment (read more about our release model here). File uploads in memory stream As we have mentioned in the beta release blogpost, we have changed the way that file uploads (image uploads and object […]

by Marian Chelmus at January 14, 2021 07:42 AM

January 13, 2021

OpenStack Superuser

Enhancement of New Heterogeneous Accelerators Based on Cyborg

Cyborg is an accelerator resource (GPU, vGPU, FPGA, NVMe SSD, QAT, DPDK, SmartNIC etc.) management project, and it uses a micro-service architecture to support distributing deployment, which contains cyborg-api, cyborg-conductor and cyborg-agent services. Cyborg-agent collects accelerator resources information and reports it to the cyborg-conductor. Cyborg-conductor stores accelerator resources information to the database, and reports it to Placement resource management service. Placement stores the resources information and provides available resources for Nova scheduling when the server is creating. Cyborg-api service provides interfaces to request accelerator resources information. Diagram 0 is the architecture of Cyborg.

Diagram 0: The architecture of Cyborg

With Cyborg, we can boot server with accelerators by interacting with Nova and Placement, and support batch boot servers with scheduling accelerators (this is an enhancement in Inspur InCloud OpenStack Enterprise edition that we plan to contribute to the Nova and Cyborg community.). Then users can use these devices in server to program with FPGA, progress images via GPU and so on. We can also bind and unbind accelerators to exist server by hot-plug and non-hot plug devices, which guarantees the convenient usage of accelerators. Diagram 1 is the interaction flow between Nova and Cyborg when booting a server.

Diagram 1: Nova and Cyborg interaction process

On the side of enhancing server operations with accelerators, we have supported most operations for servers, such as creation and deletion, reboots (soft and hard), pause and unpause, stop and start, take a snapshot, backup, rescue and unrescue, rebuild, evacuate. And there are other operations going on, such as shelve and unshelve, suspend and resume that are close to merge. We plan to support migration (live and cold) and resize soon. With these enhancements, operators can use accelerators more flexibly. Diagram 2 is the sequence flow about booting a server with vGPU devices.

Diagram 2: Sequence for virtual GPU implements

And in Inspur InCloud OpenStack Enterprise edition we have made enhancements to some features, such as batch boot servers, bind and unbind accelerator devices based on hot-plug which are mentioned above. Cyborg was used to managing virtual GPU and the utilization rate was improved by 80%. Data synchronization strategy makes Cyborg and Placement data transportation increased by 30% on efficiency.

Diagram 3: Arria® 10 FPGA architecture

From diagram 3, the main components of N3000 include the Intel Arria® 10 FPGA, Dual Intel Ethernet Converged Network Adapter XL710, Intel MAX® 10 FPGA Baseboard Management Controller (BMC), 9 GB DDR4, 144 Mb QDR-IV. It can support High-speed network with 10Gbps/25Gbps interface and High-speed host interface with PCIe* Gen 3×16.

Intel® FPGA Programmable Acceleration Card N3000 (Intel FPGA PAC N3000) is a highly customized platform that enables high-throughput, lower latency, and high-bandwidth applications. It allows the optimization of data plane performance to achieve lower costs while maintaining a high degree of flexibility. End-to-end industry-standard and open-source tool support allow users to quickly adapt to evolving workloads and industry standards. Intel is accelerating 5G and network functions virtualization (NFV) adoption for ecosystem partners, such as telecommunications equipment manufacturers (TEMs), virtual network functions (VNF) vendors, system integrators, and telcos, to bring scalable and high-performance solutions to market. Diagram 4 is the sequence flow for program in Cyborg.

Diagram 4: Sequence for the program in Cyborg

For SmartNIC, we can program it with OVS image as NFVI function in OpenStack, e.g. N3000, Mellanox CX5 and BF2. Diagram 5 is the sequence for enabling accelerators for SmartNIC.

Diagram 5: Sequence for enabling accelerators for SmartNIC

In Cyborg, we support several new features:

  • Asynchronous program API.
  • Program process API.
  • Resource report dynamically.

Users can start a new program process request for N3000. It is an async API, and it is safe to detect intelligently whether there is a conflict in resource usage and decide on accepting or rejecting the request. We also support a friendly program process query API. Users can use it to check the stage of the process and what percentage is completed at any time. When the program is finished, the resource typed and quantified, then Cyborg can discover and report the change dynamically.

In the whole OpenStack, we have also made some new improvements.

  • Cyborg supports new drivers for new SRIOV NIC. 
  • Neutron supports a new option to create a port with accelerator device profile.
  • Improvements on Nova, to retrieve the new VF NIC and pass-through to server as an interface.

After these improvements, the OpenStack can support SmartNIC more flexibly and conveniently, and the improvements include SmartNIC cards (as mentioned above), even other SRIOV card. 

All the new features and improvements on OpenStack will be upstream.

On November 25, Inspur InCloud OpenStack (ICOS) completed the 1000-nodes “The world’s largest single-cluster practice of Cloud, Bigdata and AI”, a practice for convergence of Cloud, Bigdata and Artificial Intelligence. It is the largest SPEC Cloud test and the first large-scale multi-dimensional fusion test in the industry. It has achieved a comprehensive breakthrough in scale, scene and performance, and completed the upgrade from 500 nodes to 1000 nodes, and achieved the sublimation from quantitative change to qualitative change. Inspur is working on the white paper for this large-scale test, which will be released soon. This will certainly be a reference for products in large-scale environments.

Get Involved

This article is a summary of the Open Infrastructure Summit session, Enhancement of new heterogeneous accelerators based on Cyborg.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

Thanks to the 2020 Open Infrastructure Summit sponsors for making the event possible:

Headline: Canonical (ubuntu), Huawei, VEXXHOST

Premier: Cisco, Tencent Cloud

Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post Enhancement of New Heterogeneous Accelerators Based on Cyborg appeared first on Superuser.

by Brin Zhang, Shaohe Feng and Wenping Song at January 13, 2021 02:00 PM

January 11, 2021


An Overview of Cloud File Sharing with OpenStack Manila

An Overview of Cloud File Sharing with OpenStack Manila

File sharing and storage is a thriving entity within cloud-based environments. As OpenStack’s cloud file sharing tool, Manila is a reliable solution providing high-performance, highly scalable storage to users. Here is an overview of the project and how it benefits OpenStack cloud users. 

Basics of OpenStack Manila

Manila was originally derived from OpenStack’s block storage tool, Cinder. Similar to Cinder offering canonical control plane for block storage, Manila provides canonical storage provisioning control plane for shared or distributed file systems in OpenStack. Like Cinder, users in Manila can configure multiple backends. With Manila, a user can store content repositories, web server farms, development environments, big data apps, home directories, and more. The service also has a share-server assigned to every tenant.

As an open source cloud file sharing service, Manila has certain objectives:

  • To quickly add new behaviors through a component-based architecture
  • Scaling serious workloads with high availability
  • Be fault-tolerant by isolating processes and thus avoiding cascading failures
  • Be recoverable by easily diagnosing, debugging, and rectifying failures
  • Have open standards by being a reference implementation for community-driven API

Benefits of Cloud File Sharing with Manila

Cloud file sharing with OpenStack Manila gives users several benefits. Secure file sharing with user access control is one of the major advantages. This means that the service allows you to control access by various users. This access control is done by limiting permissions on mounting and operating files. The file-sharing can only be accessed through servers with appropriate credentials. 

Another advantage of file sharing with Manila is its ability to provide a simplified integration with Cinder, OpenStack’s block storage tool. Manila integrates with Cinder by installing the necessary software, which allows any designated system to access the file sharing with ease. 

Finally, OpenStack Manila also allows your environment to have a performance conscious and efficient processing. Users have the ability to download the files that they specifically need instead of downloading the files as a whole. This capability will save a lot of processing power and time for the user. For big data use cases, this becomes particularly useful as the system can read part of the file and distribute the workload. 

VEXXHOST and Cloud File Sharing

At VEXXHOST, we offer cloud file sharing among the various OpenStack services we provide to our customers. The pay as you go structure allows the user to scale storage as they fit the application needs, and the user only pays for the hours the service is used. Services such as Manila are deployed according to specific use cases through our cloud solutions, including highly scalable private clouds

Speaking of private clouds, you can now run on a fully agile and customized cloud from VEXXHOST, with no licensing fees and smooth 2-week migration. In fact, we’re ready to put our money where our mouth is. We’re so confident in being able to save you at least 20% or more on your current cloud infrastructure expenditure that if proven wrong- we’ll give you $1,000 credit to our public cloud.   

Excited? Find out more

The post An Overview of Cloud File Sharing with OpenStack Manila appeared first on VEXXHOST.

by Athul Domichen at January 11, 2021 08:46 PM

OpenStack Superuser

Scaling Bare Metal Provisioning with Nova and Ironic at CERN: Challenges & Solutions

For people who have attended the Open Infrastructure Summits in recent years, you have probably heard of the first winner of the Superuser Awards, CERN. CERN is the European Organization for Nuclear Research. It has a laboratory that is located at the border between France and Switzerland, and the main mission of CERN is to uncover what the universe is made of and how it works. 

For a few years, all physical servers deployed in the CERN IT data centers have been provisioned as part of the OpenStack private cloud offering, leveraging Nova and Ironic. What started with a few test nodes has now grown to more than 5,000 physical machines, with the aim to move to around 10,000 nodes.

CERN’s Ironic Deployment: Node Growth

CERN’s Nova and Ironic Deployment

CERN and CERN IT rely heavily on OpenStack, and we have deployed OpenStack in production since 2013 with around 8,500 compute nodes with 300k cores and 35k instances in around 80 cells. In addition, we deploy three regions mainly for scalability and to ease the rollout of the new features.

CERN’s Nova deployment: 200 compute nodes per cell, which are managed by one child cell controller, and 30 cells in three regions
CERN’s Ironic Deployment: a user requesting physical instances will contact Nova and, by means of the Ironic driver, will contact the API. The Conductor will talk to Glance and Neutron in order to provide physical nodes

Issues and Solutions when Scaling Bare Metal with Nova and Ironic

When we scaled our bare metal deployment with Nova and Ironic to several thousand nodes, we encountered three main issues so far:

  • Controller Crashes
  • API Responsiveness
  • Resource Discovery

Scaling Issue 1: Controller Crashes

The CERN Cloud Infrastructure Team uses the iscsi deploy interface for Ironic. It means that a node upon deployment is exporting an iscsi device to the controller, and the controller is then dumping the image downloaded from Glance onto it. (Note that this deploy interface will be deprecated in the future, so direct deploy interface should be used instead.) 

For deployment, since the images are tunneled through the controller, many parallel deployments will drive the conductor into out-of-memory (OOM) situations. In consequence, the controller would crash and leave the nodes in an error state. 


To address this issue we horizontally scaled the controllers and introduced the “wing” controllers to have more controllers to handle the requests. Another solution would be to use a scalable deploy interface, such as direct or ansible which makes the node download the image directly.

Scaling Issue 2: API Responsiveness

When scaling up the infrastructure, we noticed that all requests involving the database were slow. Looking at the request logs, we realized that the inspector as the other component running on the controller gets a list of all nodes to clean up its database every 60 seconds. In addition, we had disabled pagination when we reached 1,000 nodes, which means that every request that went to the API assembled all the nodes in one giant request and tried to give that back to the requester. 


To solve this issue, we re-enabled pagination and changed the sync interval from 60 seconds to one hour. You can see how this solution affected the response time from the graph below. For a more scalable solution, we introduced “Inspector Leader Election”, which we developed together with upstream and deployed now in production. Inspector Leader Election will be available in the Victoria release, and you can see more details about how it works here.

The effect on the response time when pagination was re-enabled and the sync interval was changed.

Scaling Issue 3: Resource Discovery

Another issue that we faced when increasing the number of physical nodes managed by Ironic is the time that the Resource Tracker (RT), which runs in each nova-compute, takes to report all the available resources to Placement. From the graph below, you can see the original OpenStack Nova and Ironic setup for the dedicated bare metal cell at CERN. 

Initial Nova/Ironic Setup at CERN

There is a dedicated Nova cell for Ironic which is the standard way that we partition and manage the infrastructure. In the dedicated Nova cell, we only have the cell control plane, nova-conductor, RabbitMQ, and nova-compute. This single nova-compute is responsible for all the communication between Nova and Ironic using the Ironic API. It is possible to run several nova-computes in parallel based on a hash ring to manage the nodes, but when we were testing this functionality, several issues were observed. 

Since there is only one nova-compute that interacts with Ironic and the RT needs to report all the Ironic resources, the RT cycle takes a long time when the number of resources is increased in Ironic. In fact, it took more than three hours to complete our deployment with about 5,000 nodes. During the RT cycle, all the users’ actions are queued until all the resources are updated. This created a bad user experience with the new resources creation taking a few hours. 

Conductor Groups

In order to have failure domains in Ironic and to allow to split the infrastructure, a new feature was introduced in the Stein release of Ironic and Nova. It’s called “conductor groups”. A conductor group is an association between a set of physical nodes and a set of (one or more) Ironic conductors which manage these physical nodes. This association reduces the number of nodes a conductor is looking after, which is key in scaling the deployment. 

Conductor Groups Configuration

The conductor group an Ironic conductor is taking care of is configured in Ironic’s configuration file “ironic.conf” on each Ironic conductor:

conductor_group = MY_CONDUCTOR_GROUP

Next, each Ironic resource needs to be mapped to the conductor group that is selected, and this can be done in the Ironic API. 

openstack baremetal node set --conductor-group "MY_CONDUCTOR_GROUP" 

Finally, each group of nova-compute nodes needs to be configured to manage only a conductor group. This is done in Nova’s configuration file “nova.conf” for each “nova-compute” node:

partition_key = MY_CONDUCTOR_GROUP

Now, there is one nova-compute per conductor group, and you can see how the deployment looks like in the graph below.

Nova/Ironic setup at CERN with conductor groups

The Transition Steps

How did we deploy the conductor groups? The transition steps below are summarized from a CERN Tech Blog, Scaling Ironic with Conductor Groups

  • Nova-compute and Ironic services stopped since the databases will be manually updated
  • Updated the Ironic and Nova configuration files with the conductor groups’ names 
  • Updated Ironic “nodes” tables to map conductor groups to nodes
  • Updated Nova “compute_nodes” and “instances” tables to set the new nova-compute dedicated to the conductor groups
  • Finally, start again all the services 
The nodes per Ironic controller during the intervention

Impact on the Resource Tacker (RT) Cycle Time

In the graph below, you can see the number of Placement requests per conductor group. As a result, the RT cycle time now only takes about 15 minutes to complete instead of three hours previously since it’s now divided through the number of conductor groups.

RT cycle of after enabling conductor groups

Number of Resources Per Conductor Group

How did we decide the number of resources per conductor groups? The RT cycle time increases linearly with the number of resources. The compromise between management and RT cycle time is around 500 nodes per conductor group in our deployment. The deployment scales horizontally as we add conductor groups

Conclusion and Outlook

Scaling infrastructure is a constant challenge! By introducing conductor groups, the CERN Cloud Infrastructure Team was able to split this locking and reduce the effective lock time from three hours to 15 minutes (and even shorter in the “leading” group)

Although we have addressed various issues, some issues are still open and new ones will arise. Good monitoring is key to see and understand issues.

Get Involved

This article is a summary of the Open Infrastructure Summit session, Scaling Bare Metal Provisioning with Nova and Ironic at CERN, and the CERN Tech Blog, Scaling Ironic with Conductor Groups

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post Scaling Bare Metal Provisioning with Nova and Ironic at CERN: Challenges & Solutions appeared first on Superuser.

by Arne Wiebalck, Belmiro Moreira and Sunny Cai at January 11, 2021 02:00 PM

January 07, 2021


Adding Linux Images to OpenStack Using IaC

Adding Linux Images to OpenStack Using IaC 

Infrastructure as Code (IaC)

Nowadays, Infrastructure can be created and managed using code – you can create 100 servers with prebaked software using a for loop to manage and tag the servers. All of this is created by a CI/CD server triggered by a single commit on a given repository. With this in mind, all the infrastructure for the new application can be created and managed using IaC tools such as Terraform or Pulumi, which are cloud agnostic, or cloud vendor’s proprietary solutions. Cloud providers are now enabling SDKs for a more developer experience/oriented development with more compatibility and capabilities than a given “Provider” on Terraform or even their main IaC solutions. 

Choosing the right IaC tool/product will fully depend on the application logic and the level of automation that is needed (which should be the entire stack), but in the end, having a complete pipeline for the Infrastructure should be one of the main goals of having our applications running on the cloud as it allows to have complete control of our systems. Soon after this, we will end up using GitOps methodologies, which will increase our agility to deploy not just our applications but also the entire infrastructure. 

As soon as you have developed your entire infrastructure on any IaC, you are ready to deploy it “n” times with the same precision, without any need for human intervention on any of the configurations that your application needs in terms of inventory management or infrastructure requirements. You will be creating all the environments on where the application will live, which normally tends to be development, staging/UAT, and production. Sometimes you will also need other environments for testing, experimenting, or even innovating. This will be just as easy as running the scripts to repeatedly create the same infrastructure without worrying about drifts in the configurations. 

Terraform Modules 

When using Terraform, you can input the code to a Terraform module and reuse that module in multiple places throughout your code. Instead of having the same code in the staging and production environments, you will have both environments reuse code from the same module and then spin up “n” environments with the same set of resources and configurations. 

This is a game-changer. Modules are the foundation of writing reusable, maintainable, and testable Terraform code and, as a result, writing infrastructure. With this in mind, teams can now develop their own modules that can be published via the Terraform registry or GitHub. Furthermore, anyone in the world can use the module and create the infrastructure or components needed by an application. 

VEXXHOST’s Terraform Project

At VEXXHOST, we are advocates of sharing technology. Hence, most of our offerings and solutions are open source, and we have done the same with this project as well. Please refer to the public repository to access the Terraform code.  

This module allows you to bootstrap an OpenStack cloud by adding images to some of the most common Linux distributions from their official release sites. You can also use this module to build a collection of the most recent and popular images into your OpenStack cloud, either public or private. 

Additionally, this the same tooling used to deploy OS images for the VEXXHOST’s public cloud and private cloud offerings. Therefore, if you want to add an image to our public catalog, you can submit a pull request to our repository, and once it is approved and merged, the image should appear in our clouds. 

Use Case/Examples 

To summarize the offering, you have a module that will allow you and your team to add different Linux images into your existing OpenStack clouds, be it public or private. This will increase your ability to use and install custom solutions that are needed or restricted in some cases by the Linux distribution and version that your servers run on.  

Anyone with an OpenStack cloud can take advantage of this module. As mentioned earlier, it allows users to make PRs for changes, and then if approved, the new images will appear on their OpenStack clouds. At the end of the day, it truly serves as a testament to VEXXHOST’s commitment as an open source cloud provider and contributor.  

The Takeaway – IaC

Infrastructure as Code is here to stay. With Terraform, modules are now the main foundation of all resources for public and private cloud providers as they contribute to a better methodology when writing Infrastructure. 

VEXXHOST provides a variety of OpenStack-based cloud solutions, including highly secure private clouds. The advanced level of customization possibilities make private cloud a favorite among enterprises, without the burden of licensing fees or vendor lock-ins. Learn more about VEXXHOST’s cloud offerings by contacting our team. Check out our private cloud resource page to improve your knowledge on the topic. 

The post Adding Linux Images to OpenStack Using IaC  appeared first on VEXXHOST.

by Athul Domichen at January 07, 2021 07:47 PM

Fleio Blog

2021.01.0 beta: File uploads in memory streams, improved user feedback, new angular pages

Fleio version 2021.01.0 is now available in beta and you can test it in your lab environment, since this release is not recommended for production. You can read more about our release model here. With the latest release we would like to announce that we have added the following changes and improvements to Fleio. File […]

by Marian Chelmus at January 07, 2021 12:54 PM

January 05, 2021


How Resource Pooling Works in an OpenStack Private Cloud

How Resource Pooling Works in an OpenStack Private Cloud  

In a cloud computing context, resource pooling is a collection of resources such as cores, RAM, storage, etc., treated as a single, shared pool to achieve a sort of fail-proof flexibility. It is a way to spread the load across multiple paths, increasing the utilization rate of servers. It is seen as one of the many benefits of an OpenStack Cloud. In this blog, we examine how resource pooling works in a private cloud environment and along with various OpenStack services.  

Resource Pooling in the Cloud 

The major focuses of resource pooling are on a low cost, unused delivery of services, and a user-preference based resource division. For additional context, let us take an example outside a cloud scenario. Resource pooling is often used in wireless technology such as radio communication, where individual channels are pooled together to form a more robust channel, without any interference or conflict.  

For clouds, pooling is most used in a multi-tenant environment, according to demands from the users. This model is particularly beneficial for Software as a Service (SaaS), which runs in a centralized manner. When different users or tenants share resources within the same cloud, the overall operational costs can drastically decrease.  

In a private cloud deployment such as OpenStack, the pool is created, and computing resources are transferred over a private network. The OpenStack provider adds a range of IP addresses into the interface, and when the VM boots up, it recognizes the IPs and collects the resources in a pool.  

Pooling in Conjunction with OpenStack Services 

Various OpenStack Services are deployed and used when the resources are pooled in a server-cluster, and there is a need to launch a VM. These services enable the cloud to function more efficiently. When more services are needed, the provider adds another feature to the setup. OpenStack services have the advantage of reducing complexity in managing hardware requirements according to user needs. From OpenStack’s identity service Keystone to its network service Neutron, everything can be enabled to aid the pooling of resources and ensure the running of an efficient private cloud. 

VEXXHOST has been offering OpenStack-powered cloud solutions since 2011. With a safe and efficient deployment, our customers enjoy the various benefits of an OpenStack private cloud, including resource pooling, scalability, load balancing, and more. Contact our team to know more and check out our resource page to improve your knowledge of OpenStack private clouds. 

Like what you’re reading?
Deep dive into a hands-on ebook about how you can build a successful infrastructure from the ground up!

The post How Resource Pooling Works in an OpenStack Private Cloud   appeared first on VEXXHOST.

by Athul Domichen at January 05, 2021 09:37 PM

January 03, 2021

Michael Still

Deciding when to filter out large scale refactorings from code analysis

I want to be able to see the level of change between OpenStack releases. However, there are a relatively small number of changes with simply huge amounts of delta in them — they’re generally large refactors or the delete which happens when part of a repository is spun out into its own project.

I therefore wanted to explore what was a reasonable size for a change in OpenStack so that I could decide what maximum size to filter away as likely to be a refactor. After playing with a couple of approaches, including just randomly picking a number, it seems the logical way to decide is to simply plot a histogram of the various sizes, and then pick a reasonable place on the curve as the cutoff. Due to the large range of values (from zero lines of change to over a million!), I ended up deciding a logarithmic axis was the way to go.

For the projects listed in the OpenStack compute starter kit reference set, that produces the following histogram:A histogram of the sizes of various OpenStack commitsI feel that filtering out commits over 10,000 lines of delta feels justified based on that graph. For reference, the raw histogram buckets are:

Commit sizeCount
< 225747
< 11237436
< 101326314
< 1001148865
< 1000116928
< 1000013277
< 1000001522
< 1000000113

by mikal at January 03, 2021 08:47 AM

A quick summary of OpenStack release tags

I wanted a quick summary of OpenStack git release tags for a talk I am working on, and it turned out to be way more complicated than I expected. I ended up having to compile a table, and then turn that into a code snippet. In case its useful to anyone else, here it is:

ReleaseRelease dateFinal release tag
AustinOctober 20102010.1
BexarFebruary 20112011.1
CactusApril 20112011.2
DiabloSeptember 20112011.3
EssexApril 20122012.1.3
FolsomSeptember 20122012.2.4
GrizzlyApril 20132013.1.5
HavanaOctober 20132013.2.4
IcehouseApril 20142014.1.5
JunoOctober 20142014.2.4
KiloApril 20152015.1.4
LibertyOctober 2015Glance: 11.0.2
Keystone: 8.1.2
Neutron: 7.2.0
Nova: 12.0.6
MitakaApril 2016Glance: 12.0.0
Keystone: 9.3.0
Neutron: 8.4.0
Nova: 13.1.4
NewtonOctober 2016Glance: 13.0.0
Keystone: 10.0.3
Neutron: 9.4.1
Nova: 14.1.0
OcataFebruary 2017Glance: 14.0.1
Keystone: 11.0.4
Neutron: 10.0.7
Nova: 15.1.5
PikeAugust 2017Glance: 15.0.2
Keystone: 12.0.3
Neutron: 11.0.8
Nova: 16.1.8
QueensFebruary 2018Glance: 16.0.1
Keystone: 13.0.4
Neutron: 12.1.1
Nova: 17.0.13
RockyAugust 2018Glance: 17.0.1
Keystone: 14.2.0
Neutron: 13.0.7
Nova: 18.3.0
SteinApril 2019Glance: 18.0.1
Keystone: 15.0.1
Neutron: 14.4.2
Nova: 19.3.2
TrainOctober 2019Glance: 19.0.4
Keystone: 16.0.1
Neutron: 15.3.0
Nova: 20.4.1
UssuriMay 2020Glance: 20.0.1
Keystone: 17.0.0
Neutron: 16.2.0
Nova: 21.1.1
VictoriaOctober 2020Glance: 21.0.0
Keystone: 18.0.0
Neutron: 17.0.0
Nova: 22.0.1

Or in python form for those so inclined:

    'austin': {'all': '2010.1'},
    'bexar': {'all': '2011.1'},
    'cactus': {'all': '2011.2'},
    'diablo': {'all': '2011.3'},
    'essex': {'all': '2012.1.3'},
    'folsom': {'all': '2012.2.4'},
    'grizzly': {'all': '2013.1.5'},
    'havana': {'all': '2013.2.4'},
    'icehouse': {'all': '2014.1.5'},
    'juno': {'all': '2014.2.4'},
    'kilo': {'all': '2015.1.4'},
    'liberty': {
        'glance': '11.0.2',
        'keystone': '8.1.2',
        'neutron': '7.2.0',
        'nova': '12.0.6'
    'mitaka': {
        'glance': '12.0.0',
        'keystone': '9.3.0',
        'neutron': '8.4.0',
        'nova': '13.1.4'
    'newton': {
        'glance': '13.0.0',
        'keystone': '10.0.3',
        'neutron': '9.4.1',
        'nova': '14.1.0'
    'ocata': {
        'glance': '14.0.1',
        'keystone': '11.0.4',
        'neutron': '10.0.7',
        'nova': '15.1.5'
    'pike': {
        'glance': '15.0.2',
        'keystone': '12.0.3',
        'neutron': '11.0.8',
        'nova': '16.1.8'
    'queens': {
        'glance': '16.0.1',
        'keystone': '13.0.4',
        'neutron': '12.1.1',
        'nova': '17.0.13'
    'rocky': {
        'glance': '17.0.1',
        'keystone': '14.2.0',
        'neutron': '13.0.7',
        'nova': '18.3.0'
    'stein': {
        'glance': '18.0.1',
        'keystone': '15.0.1',
        'neutron': '14.4.2',
        'nova': '19.3.2'
    'train': {
        'glance': '19.0.4',
        'keystone': '16.0.1',
        'neutron': '15.3.0',
        'nova': '20.4.1'
    'ussuri': {
        'glance': '20.0.1',
        'keystone': '17.0.0',
        'neutron': '16.2.0',
        'nova': '21.1.1'
    'victoria': {
        'glance': '21.0.0',
        'keystone': '18.0.0',
        'neutron': '17.0.0',
        'nova': '22.0.1'

by mikal at January 03, 2021 04:49 AM

December 31, 2020

Michael Still

Rejected talk proposal: Shaken Fist, thought experiments in simpler IaaS clouds

This proposal was submitted for FOSDEM 2021. Given that acceptances were meant to be sent out on 25 December and its basically a week later I think we can assume that its been rejected. I’ve recently been writing up my rejected proposals, partially because I’ve put in the effort to write them and they might be useful elsewhere, but also because I think its important to demonstrate that its not unusual for experienced speakers to be rejected from these events.

OpenStack today is a complicated beast — not only does it try to perform well for large clusters, but it also embraces a diverse set of possible implementations from hypervisors, storage, networking, and more. This was a deliberate tactical choice made by the OpenStack community years ago, forming a so called “Big Tent” for vendors to collaborate in to build Open Source cloud options. It made a lot of sense at the time to be honest. However, OpenStack today finds itself constrained by the large number of permutations it must support, ten years of software and backwards compatability legacy, and a decreasing investment from those same vendors that OpenStack courted so actively.

Shaken Fist makes a series of simplifying assumptions that allow it to achieve a surprisingly large amount in not a lot of code. For example, it supports only one hypervisor, one hypervisor OS, one networking implementation, and lacks an image service. It tries hard to be respectful of compute resources while idle, and as fast as possible to deploy resources when requested — its entirely possible to deploy a new VM and start it booting in less than a second for example (if the boot image is already held in cache). Shaken Fist is likely a good choice for small deployments such as home labs and telco edge applications. It is unlikely to be a good choice for large scale compute however.

by mikal at December 31, 2020 11:57 PM

December 28, 2020


An Overview of OpenStack Barbican for Secure Key Management

An Overview of OpenStack Barbican for Secure Key Management

You can’t talk about open source cloud-based key management without mentioning OpenStack Barbican. 

Data security and encryption are always priorities for cloud users. At VEXXHOST, many of our clients have asked and keep asking about tools that will help them secure their data and enable safe access to authorized users. We believe that key management plays a big part in it.  

The Need for Key Management

It is generally an easy task for the end-user to encrypt data before saving it into the cloud. For tenant objects such as database archives and media files, this kind of encryption is a viable option. A key management service is used in some instances, such as presenting keys to encrypt and decrypt data, providing seamless security and accessibility to the data, and not burden the clients with managing all the keys. Barbican is a tool that enables the creation and secure storage of such keys. 

Basics of OpenStack Barbican

In simple terms, OpenStack Barbican is an open source Key Management service that provides safe storage, provisioning, and management of sensitive or secret data. This data could be of various content types – anything from symmetric or asymmetric keys, certificates, or raw binary data.

With Barbican, users can secure their data seamlessly and maintain its accessibility without personally managing their keys. Barbican also addresses concerns about privacy or misuse of data among users. 

Benefits of Key Management with Barbican

As mentioned above, users can utilize OpenStack Barbican for safe and secure storage, provisioning, and management of sensitive data. Barbican boasts of a plug-in based architecture that allows users to store their data in multiple secret stores. These stores can be software-based, like a software token, or hardware-device-based, like a Hardware Security Module (HSM). The tool allows users to securely store and manage anything from passwords to encryption keys to X.509 certificates. 

Another advantage of OpenStack Barbican is its ability to integrate seamlessly with all other enterprise-grade cloud services from OpenStack. With a simplified integration with block storage, the key management service stores keys for encrypted volumes. With object storage, Barbican enables encryption of data at rest. Similarly, seamless integration happens with Keystone, providing identity authentication and complete role-based access control. With OpenStack’s image storage, the integration allows users to verify signed images, to check whether an uploaded image is altered or not. 

VEXXHOST and OpenStack Barbican

Among other OpenStack-based services, VEXXHOST offers Barbican as our open source solution for key management. Barbican works in tandem with other OpenStack projects in creating highly secure private cloud environments for our customers. Contact our team regarding any queries you might have on key management or to know how VEXXHOST can help build your OpenStack cloud. Check out our dedicated resource page to improve your knowledge of private clouds.

Like what you’re reading?
Deep dive into a hands-on ebook about how you can build a successful infrastructure from the ground up!

The post An Overview of OpenStack Barbican for Secure Key Management appeared first on VEXXHOST.

by Athul Domichen at December 28, 2020 09:31 PM

December 21, 2020


Santa’s Scalable Sack Storage & Delivery Automation

Aptira previously built a scalable sack storage solution for Santa. While the sack storage has been reliable, Santa wanted to offer a virtual chimney-drop user experience for families worldwide on top of this on-sack-premises storage.

The Challenge

Aptira worked to improve access to the sack storage in a number of ways:

Improve ease-of-use and features for Santa’s Helpers:
the traditional sack storage protocols offered by the original design were not readily accessible by less technical elves. Santa wanted to make storage available to a broader cohort of Santa’s helpers by adding a user-friendly virtual interface with sharing and collaboration features.

Reduce the maintenance required of their Elf Services Department to manage it:
At the time Santa was using a variety of methods to provision access to sack storage – all of them requiring manual steps before the storage could be delivered to a family household. Keeping track of the current storage allocations had also become a burden for Santa helpers.

Integrate authentication with their existing Sack management system:
Allow users to login via SSO (Santa sign-on).

Integrate storage account requests into their existing SSM (Sack service management) system to enable self-service provisioning for Elves.

Santa had identified a few candidate products that might fulfill their requirements, but had not looked at each in any great depth due to excessive eggnog and internal resourcing constraints.

The Aptira Solution

Aptira first undertook an evaluation of four candidate Sack storage applications. We rapidly deployed each application in an environment within the North Pole so the features and functionality of each could be compared. We produced a detailed evaluation report that allowed Santa to make an informed decision about which application to move forward with for his Sack. Two leading candidates were put forward by Aptira and those deployments were converted into a larger-scale proof-of-concept that included integration with the actual Sack storage so Santa’s helpers and ELF services team could get a feel for using each application.

The SuchSack application was eventually chosen as it met the majority of Santa’s user and business requirements. From here Aptira developed a comprehensive solution architecture, paying particular concern to high demand and the ability to scale as the world’s population increased.

According to the solution architecture, Aptira deployed:

  • A Ho, Ho, Ho cluster
  • An Elfernetes cluster to host the SuchSack platform
  • SuchSack, Red-baubles and a Ho, Ho, Ho proxy as gift-containers
  • Aptira SuchSack Architecture Diagram
  • Elfernetes was selected as the container orchestration platform due to its self-healing and scaling capabilities, and its ability to simplify application deployment and configuration. While the SuchSack community provides a pre-built sack image, it was not suitable for a multi-node production deployment, so we developed a custom image using the existing image as a base.

Maintainability was a significant concern for Santa so we ensured that all components of the architecture were deployed using Presentible to eliminate any manual steps in the deployment. We integrated our Presentible work into Santa’s existing Presentible Tower deployment, creating job templates so that deployments could be triggered from the Tower server. Since all of our work was being stored in Git on Santa’s GitLab server, we also created CICD pipelines to both build the SuperSack sack image and to trigger deployment to their test and production environments via Presentible Tower. During handover, Santa ELF staff were able to deploy changes to the test environment by simply committing gifts to the repository.

Finally, we worked with ELFSM staff to integrate the new service into Santa’s self-service portal, so users can request access to sack and make changes to their allocated quota.

The Result

Santa’s Helpers now have a stable and performant virtual sack storage service where they can upload, manage and share on-premises giftage.

As the uptake of the service increases, ELF staff also have the confidence that the service can be scaled out to handle the increasing world population.

By recommending applications with an external API, Aptira made sure that Santa’s ELFSM system would easily integrate with SuperSack and satisfy Santa’s requirement to have a single snowglobe screen for all household service requests. With ELFSM integration, Santa’s IT have also gained a charge-back capability to recover gift labour & production costs from other departments.

The solution was built with 100% open source components, reducing vendor lock-in.

While Aptira is happy to recommend and deploy snowfield DevOps infrastructure to support a company’s CICD needs, this project showed that we can also customise our solutions to fit in with our customers’ existing infrastructure, configuring a complete deployment pipeline for provisioning the entire solution.

Merry Christmas!

Team Aptira

The post Santa’s Scalable Sack Storage & Delivery Automation appeared first on Aptira.

by Aptira at December 21, 2020 04:51 AM

December 18, 2020


Why a Managed OpenStack Cloud Is Better For You Than Self-Managed


Managed OpenStack cloud or a self-managed cloud? Many organizations and enterprises out there considering private clouds debate between these choices. It is good to have these deliberations since the choice one makes impacts the company’s future operations. 

For most companies, the answer is pretty simple—a Managed OpenStack Cloud. Let us see why it is so.

Managing a Self-Managed Cloud 

While many companies are successful in deploying a self-managed cloud, the struggle begins with the daily management of and operation. Unless the company has an excellent IT team and the budgetary capability to handle growing infrastructural needs, sustaining a private cloud environment could be difficult. If you’re one of the bigwigs, an ultra-large company, this could be a cakewalk. You can hire a big team of personnel and get the necessary infrastructure, and there you have it – a self-managed cloud.

On the other hand, for most small to medium to almost-largish organizations, self-managing a cloud means a large financial and infrastructural burden. An entity that needs to be taken care of always. Also, something that should also be adapting according to the operational needs and wants of the company. This is not considering the team you have to hire, their salaries, payments for any support services you might need, etc. 

There also remains the fact that platforms like OpenStack are complex and constantly evolving, and you would need a team that can really stay on top of things to self-manage a solid cloud environment. 

A fully managed OpenStack cloud is the right solution for enterprises facing this situation. By taking care of infrastructural and operational needs and upgrades, a managed private cloud is a much cheaper and effortless alternative. 

Managed OpenStack Cloud As A Solution

A Managed OpenStack environment offloads most of the tasks and infrastructural necessities to an outside provider and their expert and dedicated team of cloud experts. The process couldn’t be any simpler – you let the provider know what you want in your cloud, they build your OpenStack cloud the way you want it, and you start using the cloud. There. That’s it. 

All the operational needs are taken care of by the provider in such a setting. If you need to scale vertically or horizontally, you let the provider know, and they do it for you. As mentioned earlier, OpenStack is a complex and dynamic platform. It keeps evolving with contributions from open source developers from around the world. These upgrades can feel like a hassle for self-managed clouds and could come with a need to change infrastructure or hire additional personnel. According to your business structure, with a managed private cloud, the provider will implement all the upgrades as they come. 

Overall, opting for a managed OpenStack rather than a self-managed one will save your enterprise a lot of time, money, infrastructural requirements, and human resources necessary to deploy and operate a cloud environment.

Fully Managed OpenStack Cloud With VEXXHOST

VEXXHOST has been in the cloud game since 2006, and we have been OpenStack experts for close to a decade. Fully managed OpenStack private cloud is one of our signature offerings, and we have helped numerous clients across industries and regions by setting their desired cloud environments. Improve your knowledge of private clouds from our dedicated resource page. Contact us today for more details.

The post Why a Managed OpenStack Cloud Is Better For You Than Self-Managed appeared first on VEXXHOST.

by Athul Domichen at December 18, 2020 09:35 PM

OpenStack Superuser

Digital Sovereignty – Why Open Infrastructure Matters

In 2019, the term “digital sovereignty” gained momentum in Europe. For the first time in history, several ministries such as the German and French ministries for Economic Affairs pushed forward initiatives to establish digital sovereignty.

At the Open Infrastructure Summit keynotes, Johan Christenson, CEO of City Network, discussed the importance of digital sovereignty in the infrastructure world. Christenson mentioned that on July 16th 2020, the Court of Justice of the European Union invalidated “Privacy Shield”, which means companies seeking to transfer personal data from the European Economic Area to the United States must now use other mechanisms recognized by General Data Protection Regulation (GDPR) to appropriately safeguard personal data.

“As Europe grapples with an ever more centralized cloud-world and lack of true choice in providers, new initiatives are surging like GAIA-X. Europe is looking to have vendors whom can support both its values and laws in the infrastructure layer.” Christenson said. Therefore, digital sovereignty is becoming critically important from both the aspects of achieving digital independence and the long-term innovation for Europe as a whole.

But, what exactly is digital sovereignty? Why is open infrastructure important to achieve digital sovereignty? What is the European initiative GAIA-X project and how is it leveraging open infrastructure for its use case? In this article, we will focus on what digital sovereignty is and why open infrastructure matters, from the talk that Marius Feldmann, Chief Operation Officer of Cloud&Heat Technologies, and Kurt Garloff, leading the Sovereign Cloud Stack (SCS) initiative, delivered—Digital Sovereignty Why Open Infrastructure Matters—at the Open Infrastructure Summit.

“Digital sovereignty is about sovereignty, not about autarky.

It’s easy to confuse digital sovereignty with digital autarky, and they are completely different concepts. Digital sovereignty is about integrating the rising and existing technologies and innovative ideas into an overall system and adding its own solutions in order to have the freedom of choice among different options that exist. 

“Sovereignty matters – for sovereignty you need control and choice.”

Just like Feldmann discussed in his Summit session, “digital sovereignty is built on freedom of choice.” Consumers should have particularly effective freedom of choice when they become the “active managers” of their own data (Digital Sovereignty Report by the Advisory Council for Consumer Affairs, 2017).

But what are the essential preconditions to achieve having the freedom of choice in digital sovereignty? One of the most important preconditions is that alternatives should be available to achieve the freedom of choice. In addition, the alternatives must differ regarding their functional and non-function properties. It’s also fundamentally important that discovery based on these properties must be rendered possible, which means there should be possibilities and means to discover the digital services that are offered based on the properties.

European Initiative GAIA-X

With the aim to develop common requirements for a European data infrastructure, GAIA-X is a project that has been initiated by the German ministry for economy and energy. Currently, hundreds of companies worldwide work together on GAIA-X to push forward the overall vision of digital sovereignty. 

High-level overview of the GAIA-X architecture showing the major architecture elements and functions accompanied by the Federation Services.

“A strong open infrastructure ecosystem is the foundation of digital sovereignty” (GAIA-X: Technical Architecture, June 2020).

GAIA-X aims to develop common requirements for a European data infrastructure. Therefore openness, transparency and the ability to connect to other European countries are central to GAIA-X. This project is the cradle of an open, transparent digital ecosystem, where data and services can be made available, collated and shared in an environment of trust. Read more about GAIA-X on GAIA-X: Technical Architecture document

Towards Decentralization

To further explain what it needs to achieve digital sovereignty, let’s assume a heterogeneous / distributed ecosystem as the foundation for a digital sovereign future. 

Provider properties have a huge impact on the properties of the offered services.

Without having heterogeneous and decentralized infrastructure, there is no freedom of choice. Therefore, Having various providers and decentralized infrastructures is key to digital sovereignty. Feldmann summarized it well that 

“Future digital infrastructures are not monolithic and centralized but highly distributed and heterogeneous regarding their properties.”

However, to avoid a potential drawback of this future digital infrastructure, which is having various APIs to achieve mainly the same things on the infrastructure side, there should be a standardized API for all the cloud providers. Therefore, on one hand, the future digital infrastructures should be heterogeneous in order to have alternatives available due to the different properties. On the other hand, there should be standards for the API and for the platform itself. 

To clarify, when talking about standardized APIs as well as standardized tools to make an infrastructure platform operational, Feldmann was only talking about the software layer for commodity (compute, storage, networking). There is no need for many alternatives on this layer. The commodity layer should leverage open infrastructure projects to render possible collaborative development and to avoid API fragmentation. 

“Open Infrastructure is bridging the chasm on the commodity layer.”

Reinventing the wheel on the commodity layer may be an interesting exercise; however, it slows down innovative and technical progress.

On the path towards digital sovereignty, on one hand, it is crucial to provide a modular open infrastructure solution that renders it possible to enable various actors to set up an operational platform for hosting services quickly. On the other hand, open infrastructure projects should reduce development overhead and should ensure API interoperability in order to avoid vendor-lock-in contradicting the freedom of choice.

Just like what Christenson said, “open source is the only realistic way to a solid choice in vendors. Open source is also the foundation for Europe’s long term innovation and sovereignty in anything data.” 

Get Involved

This article is a summary of the Open Infrastructure Summit session, Digital Sovereignty – Why Open Infrastructure Matters.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

Thanks to the 2020 Open Infrastructure Summit sponsors for making the event possible:=

Headline: Canonical (ubuntu), Huawei, VEXXHOST

Premier: Cisco, Tencent Cloud

Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post Digital Sovereignty – Why Open Infrastructure Matters appeared first on Superuser.

by Superuser at December 18, 2020 08:24 PM

December 16, 2020

CERN Tech Blog

Adoption of in-production physical nodes into Ironic and Nova

When Ironic is chosen to manage bare metal nodes, it is not unusual that there is already an existing in-production fleet of physical servers. This may leave the operator in the sub-optimal situation where tools and workflows need to handle Ironic and pre-Ironic systems. To address this, Ironic supports an “adoption” feature: adoption allows to add nodes which Ironic should regard as in-use, and they can therefore take a slightly different path through the Ironic state machine.

by CERN (techblog-contact@cern.ch) at December 16, 2020 10:44 AM

Fleio Blog

2020.12.1: Swift object store, automated Let’s Encrypt SSL in docker install, easier debugging

Fleio v2020.12.1 was released today, 16th of December. The latest version is marked as stable and can be used in production environment (read more about our release model here). Swift object store In the last blog post, we have announced that we have added Swift object store to our support Openstack projects. With the latest […]

by Marian Chelmus at December 16, 2020 08:51 AM

Adam Young

Moving things around in OpenStack

While reviewing  the comments on the Ironic spec, for Secure RBAC. I had to ask myself if the “project” construct makes sense for Ironic.  I still think it does, but I’ll write this down to see if I can clarify it for me, and maybe for you, too.

Baremetal servers change.  The whole point of Ironic is to control the change of Baremetal servers from inanimate pieces of metal to “really useful engines.”  This needs to happen in a controlled and unsurprising way.

Ironic the server does what it is told. If a new piece of metal starts sending out DHCP requests, Ironic is going to PXE boot it.  This is the start of this new piece of metals journey of self discovery.  At least as far as Ironic is concerned.

But really, someone had to rack and wire said piece of metal.  Likely the person that did this is not the person that is going to run workloads on it in the end.  They might not even work for the same company;  they might be a delivery person from Dell or Supermicro.  So, once they are done with it, they don’t own it any more.

Who does?  Who owns a piece of metal before it is enrolled in the OpenStack baremetal service?

No one.  It does not exist.

Ok, so lets go back to someone pushing the button, booting our server for the first time, and it doing its PXE boot thing.

Or, we get the MAC address and enter that into the ironic database, so that when it does boot, we know about it.

Either way, Ironic is really the playground monitor, just making sure it plays nice.

What if Ironic is a multi-tenant system?  Someone needs to be able to transfer the baremetal server from where ever it lands up front to the people that need to use it.

I suspect that ransferring metal from project to project is going to be one of the main use cases after the sun has set on day one.

So, who should be allowed to say what project a piece of baremetal can go to?

Well, in Keystone, we have the idea of hierarchy.  A Project is owned by a domain, and a project can be nested inside another project.

But this information is not passed down to Ironic.  There is no way to get a token for a project that shows its parent information.  But a remote service could query the project hierarchy from Keystone. 


Say I want to transfer a piece of metal from one project to another.  Should I have a token for the source project or the remote project.  Ok, dump question, I should definitely have a token for the source project.  The smart question is whether I should also have a token for the destination project.

Sure, why not.  Two tokens. One has the “delete” role and one that has the “create” role.

The only problem is that nothing like this exists in Open Stack.  But it should.

We could fake it with hierarchy;  I can pass things up and down the project tree.  But that really does not one bit of good.  People don’t really use the tree like that.  They should.  We built a perfectly nice tree and they ignore it.  Poor, ignored, sad, lonely tree.

Actually, it has no feelings.  Please stop anthropomorphising the tree.

What you could do is create the destination object, kind of a potential piece-of-metal or metal-receiver.  This receiver object gets  a UUID.  You pass this UUID to the “move” API. But you call the MOVE api with a token for the source project.   The move is done atomically. Lets call this thing identified by a UUID a move-request. 

The order of operations could be done in reverse.  The operator could create the move request on the source, and then pass that to the receiver.  This might actually make mores sense, as you need to know about the object before you can even think to move it.

Both workflows seem to have merit.

And…this concept seems to be something that OpenStack needs in general. 

Infact, why should the API not be a generic API. I mean, it would have to be per service, but the same API could be used to transfer VMs between projects in Nova nad between Volumes in Cinder. The API would have two verbs one for creating a new move request, and one for accepting it.

POST /thingy/v3.14/resource?resource_id=abcd&destination=project_id

If this is called with a token, it needs to be scoped. If it is scoped to the project_id in the API, it creates a receiving type request. If it is scoped to the project_id that owns the resource, it is a sending type request. Either way, it returns an URL. Call GET on that URL and you get information about the transfer. Call PATCH on it with the appropriately scoped token, and the resource is transferred. And maybe enough information to prove that you know what you are doing: maybe you have to specify the source and target projects in that patch request.

A foolish consistency is the hobgoblin of little minds.

Edit: OK, this is not a new idea. Cinder went through the same thought process according to Duncan Thomas. The result is this API: https://docs.openstack.org/api-ref/block-storage/v3/index.html#volume-transfer

Which looks like it then morphed to this one:


by Adam Young at December 16, 2020 12:49 AM

December 15, 2020


How OpenStack Sahara Works As a High-Performance Big Data Platform


Big Data platforms and technologies are exceeding expectations all over. Initially considered a means of generating a large amount of data, they are now pushing limits with high volume, performance, variety, and processing speed.

For OpenStack users, the choice platform to build Big Data applications is the high-performance and versatile Sahara. Let us find out what makes OpenStack Sahara special and how it functions along with other OpenStack projects in creating solid Big Data platforms. 

General Objectives and Key Features

OpenStack Sahara provides users with simplified ways to provision clusters such as Hadoop and Stark. This provisioning is done by specifying various parameters like framework version, hardware node details, cluster topology, and more. Sahara can do this process in minutes and add or remove nodes on demand to scale already provisioned clusters. 

As a Big Data Platform, Sahara is designed to address the following:

  • OpenStack data processing clusters are provisioned fast for QA and development
  • Using up unused compute power from general-purpose IaaS clouds
  • Analytics as a Service for certain ad hoc or bursty workloads

Here are some of the key features of the platform:

  • It works as a stable OpenStack component
  • The UI is available on the OpenStack dashboard and is managed through a RESTful API
  • Provides support to several data processing frameworks
  • It features predefined configuration templates that can modify parameters

VEXXHOST and the Big Data Platform, Sahara

At VEXXHOST, we use Sahara combined with other OpenStack projects to deploy clouds with Big Data requirements seamlessly without any upfront costs. We can even deploy big data applications (Cloudera CDH, Hortonworks HDP, etc.) 300% more efficiently. This efficiency results in much faster data access and better cost-efficiency. 

Quick Deployment – With the backing of the range of OpenStack services, Sahara can deploy big data applications quickly. By quick, we mean hundreds of servers in a matter of minutes! Incredible, right? This quick deployment means that your team experiences no delays in building big data applications.    

Cost Savings – Using a VEXXHOST cloud means that your organization doesn’t have to worry about procuring expensive hardware. We are ready for your Big Data project with enterprise-grade hardware solutions. 

Scalability for Easy Resource Management – Business needs keep changing. Our OpenStack-based cloud allows you to scale servers up and down as needed quickly, and this comes in highly beneficial for Big Data applications. You don’t have to waste time and resources, adding and removing extra space and hardware.

Optimized, High-Performance Cloud – Users are ensured high-performance clouds consistently, avoiding ‘noisy neighbor’ issues.

No Vendor Lock-Ins, Constant Monitoring, and Support – We ensure full monitoring, support, and incident management for your Big Data applications are ensured with our platform. The open source nature of the cloud makes the entire stack devoid of vendor lock-ins. 

As mentioned earlier, VEXXHOST uses the Big Data platform Sahara along with OpenStack projects to create secure private cloud environments. They are highly flexible and secure. You can get to know more about Sahara and the whole cloud offerings by contacting our team or checking our dedicated resource page on private clouds.

The post How OpenStack Sahara Works As a High-Performance Big Data Platform appeared first on VEXXHOST.

by Athul Domichen at December 15, 2020 09:31 PM

John Likes OpenStack

Running tripleo-ansible molecule locally for dummies

I've had to re-teach myself how to do this so I'm writing my own notes.


  1. Get a working undercloud (perhaps from tripleo-lab)
  2. git clone https://git.openstack.org/openstack/tripleo-ansible.git ; cd tripleo-ansible
  3. Determine the test name: ls roles

Once you have your environment ready run a test with the name from step 3.

./scripts/run-local-test tripleo_derived_parameters
Some tests in CI are configured to use `--skip-tags`. You can do this for your local tests too by setting the appropriate environment variables. For example:

export TRIPLEO_JOB_ANSIBLE_ARGS="--skip-tags run_ceph_ansible,run_uuid_ansible,ceph_client_rsync,clean_fetch_dir"
./scripts/run-local-test tripleo_ceph_run_ansible

This last tip should get added to the docs.

by Unknown (noreply@blogger.com) at December 15, 2020 03:46 PM

December 14, 2020

Galera Cluster by Codership

Database Administration for Galera Cluster online training course

Codership is pleased to announce our first ever online training for Galera Cluster, with the launch of the Database Administration for Galera Cluster course. There are two sets of times for our EMEA and American attendees, with the former happening on January 2021, 13th and 14th, starting 10 AM CET and the latter happening on January 2021, 20th and 21st, starting 9 AM PST.

This is a hands on course that spans two days, and you will have about six contact hours per day in this instructor-led course. If you are a DBA or been given a Galera Cluster to manage, this is the course for you, as you’ll learn how to run your Galera Cluster in an optimal fashion, from setup, performance tuning, monitoring, as well as backups and more. We will cover not just Codership’s Galera Cluster distribution but also MariaDB Galera Cluster and Percona XtraDB Cluster in this course. For more information, please read the full content breakdown.

As an added bonus, we will also be covering database administration with Galera Manager our new GUI tool.

What are you waiting for? Sign up now, as the early bird rate ends 31 December 2020. There is also a volume discount available.

by Sakari Keskitalo at December 14, 2020 09:38 AM

December 11, 2020


Ceph Storage Basics and How It Benefits Your OpenStack Private Cloud



First things first. We love Ceph storage. It has been a part of VEXXHOST’s OpenStack private cloud offering for a while now. Since storage is one of the prime requirements for most enterprises approaching us for OpenStack solutions, here we are, giving you the basics of Ceph and how it will benefit your private cloud.

With its first stable release in 2012, Ceph is the most popular distributed storage solution for OpenStack. What really is it? How does it work?

What is Ceph Storage?

In simple terms, Ceph is a free and open source storage solution that is designed to allow object, block, and file storages from a unified system. Ceph is designed to be self-managed and self-healing. It can deal with outages on its own and constantly works to reduces costs in administration.

Ceph is highly scalable, runs on commodity hardware, and is specifically designed to handle enterprise workloads aiming for completely distributed operations sans any failure points. Ceph storage is also fault-tolerant and becomes so by replicating data. This means that there really are no bottlenecks in the process while Ceph is operating.

The first major stable release of Ceph was Argonaut, which was released in July 2012. Since then, there have been 15 releases within 8 years, the latest in line being Nautilus and Octopus. The next release is titled Pacific, with the date of release yet to be announced.

How Ceph Works

Ceph nodes work by employing five fully distributed and distinct daemons allowing direct user interaction. Here is a look at each of them and what they do.

Ceph Monitors – (ceph-mon) – These cluster monitors help in keeping track of both active and failed nodes.

Ceph Managers – (ceph-mgr) – They work in tandem with Ceph monitors and support external systems in monitoring and management.

Object storage devices – (ceph-osd) – They work in storing the content files.

Metadata servers – (ceph-mds) – They help in the storage of metadata from inodes and directories.

Representational state transfer – (ceph-rgw) – These gateways bring out the object storage layer make the interface compatible with relevant APIs

When one or more monitors and two or more object storage are deployed, it is known as a Ceph Storage Cluster. The file system, object storage, and block devices read and write data to and from the storage cluster. A Ceph cluster can have thousands of storage nodes since the object storage devices store data in such nodes.

Ceph uses an architectural system of distributed object storage within the storage system where data as objects, as opposed to other architectures where data is managed in a file hierarchy. Another aspect worth mentioning is that Ceph’s libraries give direct access for users to RADOS (Reliable Autonomic Distributed Object Store) storage system. This feature also lays the foundation for Ceph Filesystem and RADOS Block Device.

Benefits from Ceph Storage for OpenStack Private Cloud

Ceph brings in many great advantages to OpenStack-based private clouds. Here is a look at some of them.

Easy adoption – A shift into software-defined storage platforms can sometimes be complicated. Ceph solves this problem by allowing block and object storage in the same cluster. There is no worry about administering separate storage services using other APIs or tech.

High availability & improved performance – The coding erasure feature improves data availability by adding resiliency and durability. Sometimes, the writing speeds can almost be double the previous backend.

Cost control – Since Ceph runs on commodity hardware, there is no need for expensive and extra hardware. With an OpenStack private cloud from a reliable and reputed provider such as VEXXHOST, the pay-as-you-go structure also contributes to the overall cost control.

Better security – LDAP, Active Directory Integration, encryption features, etc., in place with Ceph can limit unnecessary access into the system.

Interested in knowing more about Ceph storage and secure and scalable OpenStack Private clouds? VEXXHOST has been using Ceph for storage for a long while now, and since 2019, we are a member of the Ceph Foundation. Reach out to the expert team at VEXXHOST, and we can guide you through the process easily. Looking forward to hearing from you!

The post Ceph Storage Basics and How It Benefits Your OpenStack Private Cloud appeared first on VEXXHOST.

by Athul Domichen at December 11, 2020 01:09 PM

December 10, 2020

Fleio Blog

2020.12.0 beta: SWIFT object store, automated Let’s Encrypt SSL in docker install

Fleio version 2020.12.0 is now available in beta and you can test it in your lab environment, since this release is not recommended for production. You can read more about our release model here. With the latest release we would like to announce that we have implemented Swift object store user interface. With the 2020.12 […]

by Marian Chelmus at December 10, 2020 11:33 AM

December 07, 2020

OpenStack Blog

10 Years of OpenStack – Genghang at China Railway

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at December 07, 2020 02:00 PM

December 03, 2020

OpenStack Blog

10 Years of OpenStack – Thomas Goirand at Infomaniak

Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make OpenStack and the global Open Infrastructure community successful.  Here, we’re talking to Thomas Goirand from... Read more »

by Sunny at December 03, 2020 02:00 PM

December 02, 2020

OpenStack Superuser

OpenStack in Production and Integration with Ceph: A European Weather Cloud User Story

European Centre for Medium range Weather Forecasts (ECMWF), an intergovernmental organisation, was established in 1975. Based in Reading, UK and the data center soon moving to Bologna Italy, ECMWF spans 34 States in Europe. It operates one of the largest supercomputer complexes in Europe and the world’s largest archive of numerical weather prediction data. In terms of its IT infrastructure, ECMWF’s HPC (High-performance computing) facility is one of the largest weather sites globally. With cloud infrastructure for Copernicus Climate Change Service (C3S), Copernicus Atmosphere Monitoring Service (CAMS) and WEkEO, which is a Data and Information Access Service (DIAS) platform, and the European Weather Cloud, teams at ECMWF maintain an archive of climatological data with a size of 250 PB with a daily growth of 250TB.

ECMWF’s production workflow: from data acquisition to data dissemination, distributed via internet leased lines Regional Meteorological Data Communication Network (RMDCN) to the end users.

European Weather Cloud:

The European Weather Cloud started three years ago in a collaboration between ECMWF and The European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) aiming to make it easier to work on weather and climate big data in a cloud-based infrastructure. With the goal of bringing the computation resources (Cloud) closer to their Big data (meteorological archive and satellite data), ECMWF’s pilot infrastructure was with open source software – Ceph and OpenStack using TripleO.

The graph below shows the current state of the European Weather Cloud overall infrastructure comprising two OpenStack clusters: one built with OpenStack Rocky and another one with OpenStack Ussuri. The total hardware of the current configuration comprises around 3,000 vCPUs, 21 TB RAM for both clusters, 1PB of storage and 2×5 NVIDIA Tesla V100 GPUs.

Integration with Ceph:

The graph below shows the cloud infrastructure of the European Weather Cloud. As you can see, Ceph is built and maintained separately from OpenStack which gives the teams at the European Weather Cloud a lot of flexibility in building different clusters on the same Ceph storage. Both of its OpenStack clusters use the same Ceph infrastructure and the same rbd pools. Besides some usual HDD failures, Ceph performs very well, and the teams at the European Weather Cloud are planning to gradually move to CentOS8 (due to partial support of CentOS7) and upgrade to Octopus and cephadm on a live cluster after a lot of testing to their development environment.

European Weather Cloud infrastructure

OpenStack with Rocky version:

The first OpenStack cluster in the European Weather Cloud, which was built in September 2019, is based on Rocky with TripleO installer. In the meantime, engineers at the European Weather Cloud also created another development environment with OpenStack and Ceph clusters that are similarly configured for testing-experimentation.

Experience and Problems:

Their deployment has about 2,600 VCPUs with 11TB RAM which doesn’t have any significant problem. The external Ceph cluster integration worked with minimum effort by simply configuring the ceph-config.yaml with little modifications. The two external networks (one public facing and another for fast access to their 300PB data archive) were straightforward. 

Most of their VMs are attached to both external networks with no floating IPs, which was a challenging VM routing issue without dynamic routing on the switches. To solve this issue, they used dhcp hooks and configured VM routing before they made the images available to the users.

There were some problems that they encountered with the NIC bond interface configuration and in conjunction with the configuration of their switches at the beginning. Therefore, the engineers decided to not use a Link Aggregation Control Protocol (LACP) configuration, and now they have a single network interface card (NIC) deployment for OpenStack. They also encountered some problems with Load-Balancing-as-a-Service (LBaas) due to Octavia problems with certificates overridden on each deployment. The problem is presented here

As soon as they found the solutions to these challenges, the engineers updated their live system and moved the whole cluster from a Single NIC to a Multiple NIC deployment which is transparent to the users with zero downtime. The whole first cluster was redeployed, and the network was re-configured with Distributed Virtual Routing (DVR) configuration for better network performance. 

OpenStack update efforts from Stein-Train-Ussuri:

In March 2020, engineers at the European Weather Cloud added more hardware for their OpenStack and Ceph cluster, and they decided to investigate upgrading to the newest versions of OpenStack. 

Experience and Problems:

First, they converted their Rockey undercloud to a VM for better management and as a safety net for backups and recovery. From March to May 2020, they investigated and tested upgrading to Stein (first undercloud and then overcloud upgrade to a test environment). Updating was possible from Rocky to Stein to Train and finally to Ussuri, but due to the CentOS7 to CentOS8 transition since Ussuri was based on CentOS8, it was considered impractical. Therefore, they made a big jump from Rocky to Ussuri by skipping three updates and decided to directly deploy their new systems on OpenStack Ussuri.

OpenStack Ussuri Cluster:

The second OpenStack cluster, based on Ussuri, was first built in May 2020, 17 days after the release of Ussuri on May 13. This cluster was a plain vanilla configuration which means that although the network was properly configured with OVN and provider networks with 25 nodes they haven’t had any integration with Ceph storage. 

Experience and Problems:

The new building method that was based on Ansible instead of Mistral had some hiccups, such as the switch from stack to heat-admin which is not what the users are used to deploy. In addition, they were trying to quickly understand and master the CentOS8 base operating system for both the host systems and service containers. Engineers at the European Weather Cloud also continued with OVS instead of OVN because of the implications in assigning floating IP addresses. With the help from the OpenStack community, the problems were solved, and the cluster was built again in mid-June 2020.

Regarding GPUs, the configuration of Nvidia GPUs was straightforward. However, since they haven’t implemented IPv6 to their Ussuri cluster when they installed and configured the GPUs drivers to a node, OVS was trying to bind to IPv6 addresses during the booting time which results in a considerable increase in booting time. A workaround was to explicitly remove PIv6 configuration to their GPU nodes. All nodes with a GPU also resolved as normal compute nodes, and they have configured nova.conf with their Ansible playbooks.  

GPU profiles assignment to VMs based on specific VM flavors for each profile

Future Next Steps:

In terms of the European Weather Cloud’s infrastructure, the engineers are planning to integrate the infrastructure with other internal systems for better monitoring and logging. They are also planning to phase out the Rocky cluster and move all the nodes to Ussuri. Trying to follow the latest versions of OpenStack and Ceph, they will continue to operate, maintain and upgrade the Cloud’s infrastructure.

For the federation, the goal is to federate their Cloud infrastructure with infrastructures of their Member states. They have identified and will continue to explore potential good use cases to federate. 

Regarding the integration with other projects, the European Weather Cloud will be interfacing with the Digital Twin Earth which is a part of the Destination Earth Program of the EU. 

Teams at the European Weather Cloud are also planning to contribute code and help other users that are facing the same problems while deploying clusters in the OpenStack community.

Get Involved:

Watch this session video and more on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source story using #WeAreOpenInfra on social media.

Special thanks to our 2020 Open Infrastructure Summit sponsors for making the event possible:

  • Headline: Canonical (ubuntu), Huawei, VEXXHOST
  • Premier: Cisco, Tencent Cloud
  • Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post OpenStack in Production and Integration with Ceph: A European Weather Cloud User Story appeared first on Superuser.

by Superuser at December 02, 2020 02:00 PM


Set up OpenStack on a Raspberry Pi cluster

In the year since the Raspberry Pi 4 was released, I've seen many tutorials (like this and this) and articles on how well the 4GB model works with container platforms such as Kubernetes (K8s), Lightweight Kubernetes (K3s), and Docker Swarm. As I was doing research, I read that Arm processors are "first-class citizens" in OpenStack.

by ajscanlas at December 02, 2020 08:01 AM

December 01, 2020

OpenStack Superuser

Embarking on a New Venture: Creating a Private Cloud with OpenStack for Under $1700

I have two business ideas to explore, and I decided that now is a good time to take the plunge and create a prototype. My hesitation throughout the last year was due to the time and financial investment required. After some inspiration, detailed thought, and self-evaluation, I am ready to go for it. Worst case scenario, this is going to eat up a lot of my time. Even if I lose time, I will learn a lot about cloud infrastructure, cloud networking, and cloud instance provisioning. My first business idea is in the realm of home and small business network cyber security. The second utilizes a private cloud platform to provision labs for IT and cyber security training. A small virtual lab isn’t going to cut it for these ventures.

My Current Setup

Before I can pursue these builds, I need to upgrade my home network and lab and select a platform. I currently have 3 old used servers (2 Dell PowerEdge R510s and an HP Proliant DL360) for the cloud. For networking, I have an ancient Cisco switch. I think I can get by with the old switch for now, but my small private cloud requires more servers. I can use the private cloud to provision networks to test out capabilities, learn, and design. These can also hold prototypes and proof of concepts for demonstrations. For the private cloud, I selected OpenStack as my platform. This will allow me to provision instances using Terraform, and have more flexibility with networking configuration. I can also avoid a large AWS and Azure bill while I experiment with different configurations. The only thing that will suffer is my power bill.

These are my Dell R510s and Cisco 3560, forgive the mess, straightening this out is part of the project.

Project Goals

Based on the OpenStack documentation I will need at least 4-5 servers to support my configuration which is a small compute cloud. To use Juju and Metal as a Service (MAAS) to deploy the cloud, I will need 2 more servers, but I could probably use one of my servers and host 2 VMs instead of purchasing another server. I haven’t yet decided whether I am going to use Juju and MAAS to deploy OpenStack, but I do know that I need at least 2 more servers for my project. I also want to separate my private cloud from the rest of my network and still maintain network performance with the added security, so I will need a firewall / IPS appliance. Once complete, my home network will look something like this:

The private cloud will be located on a DMZ allowing me to apply different security standards.

My Private Cloud Budget

I am trying to stay under $2,000 total for this project (including what I already spent). Below is the price I paid for everything I already have.

Device Qty Unit Cost Shipping Total Cost
HP ProLiant DL360 1 $149.99 $112.89 $262.88
Dell PowerEdge R510 2 $238.99 $75.00 $552.98
Cisco Catalyst 3560 1 $69.00 $17.95 $86.95
Total Cost $902.81

Existing devices with costs at the time of purchase

So, based on that I have about $1100 to spend. Although I have plenty of room, I am sticking with used equipment. The only exception I am making is my firewall appliance.

Purchasing New Equipment

I was able to find 2 Dell PowerEdge R610s for $157 each, well within budget. My shipping costs to my location are really high, so I have to keep that in mind. Even with the shipping costs, I still consider these a bargain and they meet my needs. These servers also come from the same vendor as my previous purchases (PC Server and Parts), so I know they will arrive in good condition and operate well.

Dell PowerEdge R610 server

Next I need a firewall appliance, for this I am going straight to a vendor because their site is a lot cheaper than Amazon. This appliance from Protectli has 4 NICs, a quad core processor, and a small SSD. This is more than enough to run pfsense (and it was already tested for it), so it will easily meet my needs and be a step up from my current options for under $300.

Protectli Firewall Appliance

Total Costs

With those 2 purchases, I have all the equipment I will need, and significantly under my max budget! The only other purchase I might make is a rack to store the equipment and a PDU. For now, I just have to wait for them to arrive. I plan to start sometime in December. While I wait, I am going to work on my remote access solutions, determine what IDS/IPS I am going to use (Suricata, Snort, or Bro), and finalize my design of how this will all fit together.

Device Qty Unit Cost Shipping Total Cost
HP ProLiant DL360 1 $149.99 $112.89 $262.88
Dell PowerEdge R510 2 $238.99 $75.00 $552.98
Cisco Catalyst 3560 1 $69.00 $17.95 $86.95
Protectli FW4B 1 $282.00 $7.00 $289.00
Dell PowerEdge R610 2 $156.99 $111.00 $424.98
Total Cost $1616.79

Existing devices with costs at time of purchase

This article was originally posted on mattglass-it.com. See the original article here.

The post Embarking on a New Venture: Creating a Private Cloud with OpenStack for Under $1700 appeared first on Superuser.

by Matt Glass at December 01, 2020 02:00 PM

November 30, 2020


Why OpenStack Private Cloud for DevOps is the Way Forward


OpenStack private cloud for DevOps is gaining much traction even among fierce competition. The flexible nature of the open source platform allows DevOps engineers to innovate from time to time. OpenStack also maximizes existing infrastructure and helps engineers tackle untoward incidents with ease.

Agile Development with OpenStack

OpenStack emerged and established itself as a gold standard in building private clouds, among other Infrastructure as a Service (IaaS) platforms. The open source elements of the platform allow engineers to act autonomously to provision and de-provision cloud environments. OpenStack works as a self-service mechanism with all the flexibility cloud builders need. Another advantage is that engineers being able to provision things reduces downstream bottlenecks for the operations team.

OpenStack is not just open source but also vendor-agnostic. This enables the end-user to take full advantage of competitive pricing. There is no vendor lock-in with OpenStack. The availability of a private cloud at prices comparable to public clouds works great for organizations with large-scale data needs.

Another significant feature of OpenStack private cloud, compared to a public cloud, is its ability to have more control in optimizing application performance and security. Companies with sensitive data to handle prefer OpenStack private clouds for DevOps and further use for the same reason.

Beginning the Journey with an OpenStack Private Cloud 

In the initial phases, the integration of OpenStack clouds might seem like a challenge to enterprises used to traditional IT infrastructures. But, an experienced cloud provider can make this process a breeze. Once the company makes it clear what they want in their cloud for DevOps and later use, the provider takes care of the rest. The flexibility of OpenStack really comes in handy here as it allows tailoring the platform according to individual needs.

Moreover, OpenStack also comes with regular updates and releases across the board frequently. The cloud provider ensures that enterprises get these upgrades promptly so that operations run smoothly with the latest technology.

For compute, storage, and network, OpenStack is clearly one of the leaders in the game, with its flexibility and vendor-agnostic nature. The fact that developers are able to create cloud environments with high agility is invaluable for DevOps.

VEXXHOST Private Cloud for DevOps

VEXXHOST’s is an established cloud provider with a decade-worth of experience in OpenStack. We build and deploy private clouds for DevOps according to varying specifications and requirements from clients worldwide. We also provide Managed Zuul, a specialized tool that can accompany DevOps cycles. Talk to our team for further assistance. Check our private cloud resources page to learn more about highly secure cloud environments. 

You’ve got Big Data?
We’ve got your Big Data Infrastructure Solution in a free ebook!

The post Why OpenStack Private Cloud for DevOps is the Way Forward appeared first on VEXXHOST.

by Athul Domichen at November 30, 2020 09:14 PM

November 26, 2020


A Brief History of the Open Infrastructure Foundation


A Brief History of the Open Infrastructure Foundation

Open Infrastructure Foundation is an entity supporting open source development in IT infrastructure globally.

For almost a decade, the governing body of OpenStack and related projects was the OpenStack Foundation. Recently, at the Open Infrastructure Summit 2020, the foundation announced that it has evolved into the Open Infrastructure Foundation.

This move is part of the multi-year drive of community evolution and expansion into including newer projects under the foundation’s wing. In this context, let’s take a look at the timeline that led to this evolution.

The Beginning

The origin of OpenStack, and later the foundation, can be traced back to something that happened in a true open source fashion – a collaboration.

Rackspace was rewriting the infrastructure code (what was later known as Swift) to its cloud offerings. They decided to make the existing code open source. Simultaneously, through its contractor Anso Labs, NASA did the same with the Python-based cloud fabric controller Nova. The teams realized that both the projects are complementary and decided to collaborate. This shared program marked the beginning of OpenStack.

Tech professionals from 25+ companies attended the first OpenStack Design Summit, held in July 2010 in Austin, Texas. Team VEXXHOST joined the OpenStack community by the time the second OpenStack project, Bexar, was released.

OpenStack and the community were growing, and there was a need to promote and develop projects in a more sustainable and organized manner. This thought resulted in the creation of the OpenStack Foundation in September 2012.

The Foundation

The creation of the OpenStack Foundation was a defining moment for cloud computing users across the globe. The foundation launched with over 5,600 members representing numerous companies. We are proud to say that VEXXHOST was also part of it all from the very beginning.

To govern OpenStack and other open source projects better, the foundation set up three bodies under its wing – the Board of Directors, the Technical Committee, and the User Committee. Over the years, the foundation grew with the growth of the projects. Recently there arose a need to build a larger umbrella to adopt and develop more open source projects. Hence, the OpenStack Foundation evolved into the Open Infrastructure Foundation, with OpenStack still being in the heart of it all.

The Summits

The first OpenStack Summit was held in Paris in 2014. The event changed its name to the Open Infrastructure Summit with its Denver edition in 2019. Held almost bi-annually, the summits have always given timely boosts to the development of open sources. The global community of OpenStack developers, contributors, and users come together during the summits and share their ideas collectively. VEXXHOST is a regular presence at the summits and won the SuperUser Award at the Denver Summit, 2019.

The Open Infrastructure Summit was held virtually from 19th to 23rd October 2020, owing to the pandemic. The foundation announced its evolution and name change at the Summit and was greeted with much fanfare.

VEXXHOST and Open Infrastructure Foundation

VEXXHOST was a Corporate Member of the OpenStack Foundation for many years. Our association with the OpenStack community began in 2011, and we’ve been a part of the journey so far as an avid contributor and user. With the latest evolution, we are proud to be a Founding Silver Member of the Open Infrastructure Foundation and accompany it to new heights of open source development.

VEXXHOST has a wide range of cloud solutions powered by OpenStack and other open source projects, including a fully customizable private cloud. If you have further queries on our services, contact us, and we’ll get back to you.

The post A Brief History of the Open Infrastructure Foundation appeared first on VEXXHOST.

by Athul Domichen at November 26, 2020 09:43 PM

November 24, 2020

OpenStack Superuser

Running Percona Kubernetes Operator for Percona XtraDB Cluster with Kata Containers

Kata containers are containers that use hardware virtualization technologies for workload isolation almost without performance penalties. Top use cases are untrusted workloads and tenant isolation (for example in a shared Kubernetes cluster). This blog post describes how to run Percona Kubernetes Operator for Percona XtraDB Cluster (PXC Operator) using Kata containers.

Prepare Your Kubernetes Cluster

Setting up Kata containers and Kubernetes is well documented in the official github repo (cri-ocontainerdKubernetes DaemonSet). We will just cover the most important steps and pitfalls.

Virtualization Support

First of all, remember that Kata containers require hardware virtualization support from the CPU on the nodes. To check if your linux system supports it run on the node.

$ egrep ‘(vmx|svm)’ /proc/cpuinfo

VMX (Virtual Machine Extension) and SVM (Secure Virtual Machine) are Intel and AMD features that add various instructions to allow running a guest OS with full privileges, but still keeping host OS protected.

For example, on AWS only i3.metal and r5.metal instances provide VMX capability.


Kata containers are OCI (Open Container Interface) compliant, which means that they work pretty well with CRI (Container Runtime Interface) and hence well supported by Kubernetes. To use Kata containers please make sure your Kubernetes nodes run using CRI-O or containerd runtimes.

The image below describes pretty well how Kubernetes works with Kata.

Hint: GKE or kops allows you to start your cluster with containerd out of the box and skip manual steps.

Setting Up Nodes

To run Kata containers, k8s nodes need to have kata-runtime installed and runtime configured properly. The easiest way is to use DaemonSet which installs required packages on every node and reconfigures containerd. As a first step apply the following yamls to create the DaemonSet:

$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-deploy/base/kata-deploy.yaml

DaemonSet reconfigures containerd to support multiple runtimes. It does that by changing /etc/containerd/config.toml. Please note that some tools (ex. kops) keep containerd in a separate configuration file config-kops.toml. You need to copy the configuration created by DaemonSet to the corresponding file and restart containerd.

Create runtimeClasses for Kata. RuntimeClass is a feature that allows you to pick runtime for the container during its creation. It has been available since Kubernetes 1.14 as Beta.

$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/k8s-1.14/kata-qemu-runtimeClass.yaml

Everything is set. Deploy test nginx pod and set the runtime:

$ cat nginx-kata.yaml
apiVersion: v1
kind: Pod
  name: nginx-kata
  runtimeClassName: kata-qemu
    - name: nginx
      image: nginx

$ kubectl apply -f nginx-kata.yaml
$ kubectl describe pod nginx-kata | grep “Container ID”
    Container ID:   containerd://3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111

On the node verify if Kata is used for this container through ctr tool:

# ctr --namespace k8s.io containers list | grep 3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111
3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111    sha256:f35646e83998b844c3f067e5a2cff84cdf0967627031aeda3042d78996b68d35 io.containerd.kata-qemu.v2cat 

Runtime is showing kata-qemu.v2 as requested.

The current latest stable PXC Operator version (1.6) does not support runtimeClassName. It is still possible to run Kata containers by specifying io.kubernetes.cri.untrusted-workload annotation. To ensure containerd supports this annotation add the following into the configuration toml file on the node:

# cat <> /etc/containerd/config.toml
  runtime_type = "io.containerd.kata-qemu.v2"

# systemctl restart containerd

Install the Operator

We will install the operator with regular runtime but will put the PXC cluster into Kata containers.

Create the namespace and switch the context:

$ kubectl create namespace pxc-operator
$ kubectl config set-context $(kubectl config current-context) --namespace=pxc-operator

Get the operator from github:

$ git clone -b v1.6.0 https://github.com/percona/percona-xtradb-cluster-operator

Deploy the operator into your Kubernetes cluster:

$ cd percona-xtradb-cluster-operator
$ kubectl apply -f deploy/bundle.yaml

Now let’s deploy the cluster, but before that, we need to explicitly add an annotation to PXC pods and mark them untrusted to enforce Kubernetes to use Kata containers runtime. Edit deploy/cr.yaml:

  size: 3
  image: percona/percona-xtradb-cluster:8.0.20-11.1
      io.kubernetes.cri.untrusted-workload: "true"

Now, let’s deploy the PXC cluster:

$ kubectl apply -f deploy/cr.yaml

The cluster is up and running (using 1 node for the sake of experiment):

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
pxc-kata-haproxy-0                                 2/2     Running   0          5m32s
pxc-kata-pxc-0                                     1/1     Running   0          8m16s
percona-xtradb-cluster-operator-749b86b678-zcnsp   1/1     Running   0          44m

In crt output you should see percona-xtradb cluster running using Kata runtime:

# ctr --namespace k8s.io containers list | grep percona-xtradb-cluster | grep kata
448a985c82ae45effd678515f6cf8e11a6dfca159c9abf05a906c7090d297cba    docker.io/percona/percona-xtradb-cluster:8.0.20-11.2 io.containerd.kata-qemu.v2

We are working on adding the support for runtimeClassName option for our operators. The support of this feature enables users to freely choose any container runtime.


Running databases in containers is an ongoing trend and keeping data safe is always the top priority for a business. Kata containers provide security isolation through mature and extensively tested qemu virtualization with little-to-none changes to the existing environment.

Deploy Percona XtraDB Cluster with ease in your Kubernetes cluster with our Operator and Kata containers for better isolation without performance penalties.

This article was originally posted on percona.com/blog. See the original article here.

The post Running Percona Kubernetes Operator for Percona XtraDB Cluster with Kata Containers appeared first on Superuser.

by Sergey Pronin at November 24, 2020 02:00 PM

November 22, 2020

Ghanshyam Mann

OpenStack Victoria CI/CD migration from Ubuntu Bionic (LTS 18.04)-> Focal (LTS 20.04)

OpenStack upstream CI/CD tests the things on defined  LTS or stable distribution versions. OpenStack Technical Committee defines each cycle testing runtime. As per OpenStack Victoria testing runtime, defined versions are:

  • Ubuntu 20.04
  • CentOS 8
  • openSUSE Leap 15

Ubuntu Focal (Ubuntu LTS 20.04) was released on April 23, 2020, and in OpenStack Victoria (released on 14th Oct 2020), we migrated the upstream CI/CD testing on the above-defined testing runtime. This work is done as one of the community-wide goals “Migrate CI/CD jobs to new Ubuntu LTS Focal“.

    What is this migration:

OpenStack CI/CD is implemented with Zuul jobs prepare the node to deploy the OpenStack using Devstack and run tests (Tempest or its plugins, project in-tree tests, rally tests etc). Base OS installed on the node is where OpenStack will be deployed by DevStack.

Till OpenStack Ussuri release, the base OS on the majority of the job’s node was Ubuntu Bionic (18.04). So DevStack used to deploy OpenStack on Ubuntu Bionic and then run tests.

With the new version of Ubuntu Focal (20.04), the node’s base OS has been moved from Ubuntu Bionic -> Ubuntu Focal. On every code change, it will make sure OpenStack work properly on Ubuntu Focal.

NOTE: This migration target only zuulv3 native jobs. Legacy jobs are left to keep running on Bionic and plan to be migrated on Focal while they migrate to zuulv3 native jobs . We had another community-goal to migrate all the legacy jobs to zuulv3 native.


    Perform migration testing in advance:

We started the work in June and prepared the devstack, tempest, and tox-based base jobs on Focal so that all projects gate can be tested and fixed in advance before devstack and Tempest base jobs merge. The idea behind the advance testing is to avoid or minimize the gate failure in any of the repo under any projects. This advance testing includes integration as well as tox based unit, functional, doc, pep8, and lower-constraint testing.

    Bugs & fixes:

This migration had more things to fix compared to the previous migration from Ubuntu Xenial to Bionic. One reason for that was Ubuntu Focal and python dependencies dropping python2.7 support and MySQL 8.0 compatibility. OpenStack already dropped the Python2.7 in the Ussuri release but OpenStack dependencies lower constraints were not updated to their python-3 only version because many of them were not python3-only at that time.  So in Ubuntu Focal, those dependencies versions are python3-only which caused many failures in our lower constraints jobs.

A few of the key issues we had to fix for this migrations are:

  1. Failing device detachments on Focal: Bug#1882521
  2. Few of the lower constraints are not compatible with python3.8: Bug#1886298.
  3. Migrations broken on MySQL 8.x: story/2007732
  4. Duplicate check constraint name ‘ck_started_before_ended’: story/2008121
  5. No implicit user creation with GRANT syntax in MySQL 8.0: Bug#1885825
  6.  pyflakes till 2.1.0 not compatible with py3.8: Bug#1886296.

    Actual Migration:

Fixing these bugs took a lot of time for us and that is the reason this migration was late and missed the initial deadlines.

  • Tox based job migration happened on Sept 9th.
  • devstack and Tempest base jobs migration happened on Sept 24th.

All the work for this migration are tracked on: https://storyboard.openstack.org/#!/story/2007865

All changes are: https://review.opendev.org/q/topic:%2522migrate-to-focal%2522+(status:open+OR+status:merged)

    How to migrate the third-party CI:

If your 3rd party CI jobs are still not migrated to zuulv3 then you need to first migrate legacy jobs to zuulv3 native. Refer to this community-goal for details.

For zuulv3 native jobs, like upstream jobs you need to switch the jobs nodeset from ubuntu Bionic to ubuntu Focal.

Below Diagram gives a quick glance of changing the nodeset to Ubuntu Focal:

If you want to verify the nodeset used in your zuul jobs, you can see the hostname and label in job-output.txt

In the same way, you can migrate your third-party CI also to Focal. If third-party job is using the base job without overriding the ‘nodeset’ then the job is automatically switched to Focal. If the job overrides the ‘nodeset’ then, you need to switch to Focal nodeset like shown above. All the Ubuntu Focal nodeset for a single node to multinode jobs are defined in devstack.

We encourage all the 3rd party jobs to migrate to Focal asap as devstack will not support the Bionic related fixes from Victoria onwards.

    Dependencies compatibility versions for upgrades to OpenStack Victoria:

There are many dependencies constraints that need to be bumped to upgrade the OpenStack cloud to the Victoria release. To know all those compatible versions, check the project-specific patches merged from here or from this Bug#1886298. This can help to have prepared the smooth upgrades.

    Completion Summary Report:

  • A lot of work was involved in this. There are many changes required for this migration as compare to previous migration from Xenial to Bionic.
    • Many dependencies with lower constraints compatible with py3.8 and Focal distro had to be updated. Almost in all the repos.
    • Mysql 8.0 caused many incompatible DB issues.
  • I missed the original deadline of m-2 to complete this goal due to falling gates and to avoid any gate block for any projects.
  • More than 300 repo had been fixed or tested in advance before migration happened. This helped a lot to keep the gate green during this migration.


I would like to convey special Thanks to everyone who helped in this goal and made it possible to complete it in the Victoria cycle itself.

by Ghanshyam Mann at November 22, 2020 02:32 AM

November 20, 2020


Why Cloud Networking with OpenStack Neutron Works Great for Enterprises


Cloud networking is an important element within all types of cloud building – public, private, or hybrid. For private clouds from VEXXHOST, our open source networking choice is OpenStack Neutron. We believe that Neutron brings in great value for enterprises in building the ‘central nervous system’ of their cloud. Let us see why.

Overview of OpenStack Neutron

Neutron is an extremely powerful networking project of OpenStack. It is considered complex by many users but let me reassure you that its capabilities make it a virtual powerhouse like nothing else out there. We have a previous post that lays out the basics of OpenStack Neutron.

OpenStack Neutron can help you create virtual networks, firewalls, routers, and more. It is flexible and secure. With it, OpenStack can offer network-connectivity-as-a-service. Neutron can help other OpenStack projects manage interface devices through the implementation of API.

Here is a breakdown of a few points mentioned above and how they benefit enterprises.

Cloud Networking with Neutron and Benefits for Enterprises

OpenStack Neutron provides cloud tenants with a flexible API, which helps them build strong networking topologies while also allowing them to configure advanced network policies. There is no unnecessary vendor lock-in as well. A use-case scenario of this capability for enterprises is that they can create multi-tier topologies of web applications.

Neutron allows organizations to have peace of mind regarding security and segmentation as it enables single-tenant networks. These fully isolated networks work in a way that’s almost like having your own secure control switch to the servers, with no possibility of someone else accessing. Moreover, segmentation is possible for these connections. This segmentation enables each VM that comes within a given hypervisor the capability to be private to the respective network.

With Neutron for Cloud Networking, enterprises can leverage automatic IP Address Management and ensure consistency. This means that you don’t have to manually manage IP addresses, and it allows for consistency between the system and the documentation. Another advantage of these dynamic IP addresses is that the possibility of manipulating IP via blocking the layer above is eliminated.

Did you know that enterprises can use OpenStack Neutron for bare metal scaling? Yes, it is true. Each rack of the system works as a network of its own. The vast scheduling network enables these racks to be interconnected with each other. This capability also allows the system to assign appropriate IP addresses.

Overall, Neutron works as a safe, reliable, and flexible cloud networking option for businesses.

VEXXHOST and OpenStack Neutron

VEXXHOST provides Neutron as our open source solution for networking with private cloud. We also provide various other OpenStack-based services for our clients across the globe. If you want to know more about our services and solutions, contact our team. Improve your knowledge of private clouds from our ever-evolving and dedicated resource page.

Would you like to know about OpenStack Cloud? So download our white paper and get reading!


Conquer The Competition With OpenStack Cloud

The post Why Cloud Networking with OpenStack Neutron Works Great for Enterprises appeared first on VEXXHOST.

by Athul Domichen at November 20, 2020 09:44 PM


Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.


Last updated:
February 26, 2021 01:23 AM
All times are UTC.

Powered by: