In recent years, the Arm server has been widely used, and the upper software ecosystem has also been greatly developed. In particular, some public cloud vendors have begun to provide cloud computing products based on the Arm architecture. Companies and organizations like Huawei, EasyStack, and Linaro have invested quite a lot of resources to speed up the application of the Arm architecture in cloud computing.

Next, we’ll share the latest progress for the software ecosystem of Arm architecture in the cloud computing area, the challenges for landing Arm-based servers, and the experience of production readiness.

Full-Stack enablement of Open Source Software

As shown in the above picture, the infrastructure layer is only a small part of the whole software stack, in order to make this layer functional and perform better on Arm platform, developers have also done a lot in the lower layer projects like OS, drivers and libraries. On the other hand, end–users also care about the software running on top of the infrastructure, whether they can run on Arm platform or not is one of the key considerations for users thinking about using Arm platform as infrastructure. In the next part, let us look at the enablement of Arm platform from a bigger picture.

In the open source world, the development pipeline is particularly important for upstream developers. Currently, most of the open source projects only have x86 based dev pipeline, so the whole development process and outcome is not very friendly to Arm users, there might be some extra works before running them on Arm platform:

In order to make Arm platform the first citizen in the open source world, the first step is to provide a development pipeline to the open source projects, so that developers have resources to develop and test on the Arm platform, and the outcome product will also be Arm-native:

With this idea, Huawei has newly enabled Arm CI in over 50 top open source communities, covering 6 major fields including Cloud & SDS, Big Data, Database, Web, Libs & Middleware, and AI. And there are quite a lot of open source projects that already supported the Arm platform, users now have multiple choices in those fields, readers can refer to Arm CI landscape for more details (https://kunpengcompute.github.io/arm-landscape/).

Besides enabling, Huawei has also done a lot in improving the performance of open source projects on Arm platform and close the feature gaps between x86 platform. For example, we have enabled CPU info observation and host CPU comparison on Arm platform in Libvirt, and currently working on using them to have better migration experiences in OpenStack.

With these enablements and improvements, Arm platform became more competitive and attractive for both users and upstream developers.

Linaro’s work on ARM64 cloud computing ecosystem

Linaro is an open source organization on the ARM64 ecosystem, which mainly focuses on upstream development and maintenance such as Linaro Kernel, toolchains, Android, and some specified areas such as data center. 

Linaro has deeply participated in the Open Infrastructure community through contributing to ARM64 OpenStack enablement, deployment(Maintain ARM64 Kolla images), and maintaining the ARM64 OpenDev CI resources. At the same time, the Linaro Developer Cloud has been set up based on totally upstream work on OpenStack and Ceph,  which can be not only tested members’ hardware in cloud computing but also help developers to use ARM64 resources. 

As the chart above shows, Linaro Developer Cloud is completely based on OpenStack and Ceph. Now it can offer VM/BM services based on Nova/Ironic as well as ARM64 Kubernetes Service based on Magnum support. The production level OpenStack cluster is deployed by Kolla container images, which is more flexible for operation and upgrading. A lot of upstream work were included for supporting this:

• Nova/Ironic/disk-image-builder enablement and bugfix on ARM64
• Devstack enablement support for ARM64 OpenStack
• Kolla image build, Kolla-ansible deployment support, and bugfix for OpenStack version upgrading.
• Magnum multi-arch support and K8s cloud provider support on ARM64

Linaro Developer Cloud K8s service has now online for about one year and the K8s service version v1.17, v1.18 has been certified by the CNCF conformance test. It is a good example that ARM64 is eligible in the open-source cloud computing area.

Hardware Automation is another hot topic in recent years. Quite a lot of workloads need to run on bare metal(e.g cloud-native K8s on Bare metal, HPC) to achieve better performance of network and storage or virtualization limit(nested virtualization is not supported on ARM64), so that quick hardware provisioning, standard hardware management framework are essential and significant on ARM64. 

To meet these requests, Linaro has proposed the diskless boot solution on ARM64, which leverages OpenStack Ironic for BM management, and uses Ceph ISCSI to provision the volume boot support. Use Ceph volume as a disk could largely reduce BM provision time and improve the RootFS security, all the data reliability will be guarantee by Ceph.  

As the above picture shows, the control path of the workflow has relied on the Ironic PXE ISCSI boot from the volume, Cinder Ceph ISCSI driver, and Ceph ISCSI gateway. Linaro has contributed several features in order to make the diskless boot on ARM64 happen:

• Ironic boot from PXE iSCSI support
• Cinder Ceph ISCSI driver support
• Ceph iSCSI client bugfix and stable enhancements

We believe that the hardware automation solution will be beneficial to ARM64 development and CI system. We will quickly online BM service support in Linaro Developer Cloud and offer more resources to external applications.

Best Practices of Running Cloud on Arm Servers

EasyStack, as a company focusing on cloud computing, has served more than 1000 customers since its establishment. The company started to support the Arm server in April 2019, released the technology preview version in February 2020, and released the formal GA version on January 31, 2021, which has reached product–level support in function coverage and test intensity. So far, nearly 30 customers have deployed the Arm version of cloud computing products in the production environment.

In people’s minds, Arm ecology is not as extensive as x86. There are many unknown problems to be solved, including hardware, firmware, operating system, software, and so on. Actually, this impression lasted for quite a long time and things have changed rapidly during the last 5 years since great progress has been made in the development of the Arm ecosystem. The number of CPU cores is increasing, and the processing capacity of a single core is strengthening. We have seen that the performance of distributed storage in the Arm-based is even higher than that on x86 architecture.

Ecology has also been greatly developed. More and more operating systems support Arm. Open source software in cloud computing also has better support on Arm architecture, such as using Arm CI as a verification means in the normal development process. The Arm architecture has an advantage in energy efficiency, which is also one of the driving forces to attract public cloud vendors to design their own Arm CPUs to provide cloud computing services. In short, the difference between the application of Arm in the field of cloud computing and x86 architecture is greatly reduced.

During the whole adaptation stage, EasyStack‘s engineers adapted more than 10 types of servers. These servers had differences in BIOS/firmware, which led to the same operating system could run on some servers, but could not run some servers. The engineers analyzed the differences and made corresponding repairs. Finally, the operating system could run stably on all the Arm servers. The choice of the upper software version is also particularly important. EasyStack’s products use Kubernetes as the base, and other services run in the form of pod, including OpenStack components. This makes the Arm version product use the same software stack with x86 architecture, which is easy to maintain. In this process, libvirt’s CPU specific bugs, Openvswitch’s stability, and MariaDB’s stability are solved. In a word, Kubernetes, OpenStack, and other components can run perfectly on the Arm platform.

After the adaptation work, we started to support strict testing at the product level. All the functions of x86 have been fully tested. This testing work is integrated into the daily CI / CD process to ensure that any code change will not damage the functions of X86 and Arm. It is found that when doing network bandwidth test on Arm platform, the bandwidth value does not reach the expected value, which is solved by a binding specific interrupt to the corresponding CPU; it is found that VNC console has no output in some guest OS, which is mainly related to the setting of guest OS grub startup parameter tty0; Some Arm servers don’t support the hardware watchdog very well, sometimes they can’t work normally. We have completed complete tests, and compared with x86 architecture products, There are only four differences: 1. It doesn’t support Windows guest OS; 2. The GPU driver and SDK support on Arm architecture are not good enough, it can’t support GPU temporarily. 3. Because of the limitation of Arm architecture, each virtual machine can support fewer disks and network cards than x86; 4. Because of the limitation of IPMI, some monitoring indicators such as disk speed can not be obtained, That’s all of the difference between the Arm and x86 architecture.

There are more and more customers using Arm servers for production. Here, we need to complete the migration of x86 architecture applications to Arm architecture, the resource specification conversion from X86 platform to Arm platform, and so on.  For EasyStack, there are nearly 30 customers in total, with more than 500 cloud computing physical nodes based on Arm architecture are used in production and run well, among which the longest–running customers are more than one year. Production readiness is the basis of large-scale promotion, including the strong support of various companies in the community. At present, we have seen the dawn. We believe that in the near future, more enterprises will adopt Arm architecture as the basis of cloud computing to provide cloud services.

Get Involved

This article is a summary of the Open Infrastructure Summit session, the Progress for Cloud Computing on Arm Architecture.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post The Progress for Cloud Computing on Arm Architecture appeared first on Superuser.

After several years working in the open source community, I’ve ended up with a lot of emails from Gerrit code reviews, Google Group mailing lists, and various GitHub notifications. And when I say a lot, I mean my Google storage for Gmail, Photos, Drive, and others were getting closer and closer (78%) to the max free capacity.

To be fair, I do know quite a bit of that is from other things besides email. I have several large files in Drive, and a few longer videos from when my kids were younger sitting in Google Photos. But the weird thing that struck me was that even though I would delete emails from things like code reviews, they would still show up when I searched by inbox for various terms. I thought the way it was supposed to work was that I would delete an email, it would sit in my Trash for 30 days or so, then Gmail would clean things up for me. Based on what I was getting from my inbox searches, that apparently isn’t always the case.

So rather than allow things to keep up slowly creeping up to max out my storage quota, I decided to take a look at what I could do to clear out this old cruft. At first I would perform some searches, use the Select All to select all 50 messages on the page, and delete them. And the next page. And the next.

That was going to take a long time. It wouldn’t give me the message count from queries like from:review@openstack.org, but it looked like it was in the 1000’s. Even with no where to go, I didn’t have time to click through all that.

Automation (sorta) to the rescue

I’m probably late to the party - based on forum results, it appears several years late to the party - but it didn’t take me too long to stumble on the Google Apps Script site.

I started by looking to see if there was an API for interacting with Gmail. Turns out there is, for several different languages. But the Quickstart Guide included a link to how to get started with Apps Script. That has a nice walkthrough of getting a simple script set up to interact with Gmail.

From there, it was pretty easy to adapt the example they gave to work with messages instead. Looking through the API documentation, I found the message object has a moveToTrash() method. That led me to come up with this quick example:

function deleteMessages() {
  var count = 100
  while (count > 0) {
    var threads = GmailApp.search('from:review@openstack.org', 0, 10)
    count = threads.length
    for (var i = 0; i < count; i++) {
      var thread = threads[i];
      for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];
        message.moveToTrash()
      }
    }
  }
}

It’s fairly easy to work with the Gmail API. To break it down bit, here are some key parts.

Getting threads

var threads = GmailApp.search('from:review@openstack.org', 0, 10)

This queries the Gmail API, using a query just like you would use within the Gmail UI itself. In fact, I highly recommend first running your query in the UI and verifying it returns what you expect it to, then copying and pasting that query string into the script. We are deleting messages here, so you probably should be very cautious about what gets automatically deleted.

The two numbers, 0, 10, are the starting count and the max number of threads to retrieve. Starting off, this query would return a very, very large amount of messages. Pulling that all across the network and into memory would not be very nice. So chunking this out into smaller amounts is a lot more efficient. I would rather have to run the script repeatedly than try to load my entire inbox into the script.

And speaking of running repeatedly, there apparently is a hard coded 5 minute timeout on script execution. So although this speeds things up quite a bit over manually deleting, it still does take time. So one way or another I was going to have to keep rerunning the script to get everything cleared out anyway.

Bringing out the trash

message.moveToTrash()

Most of the rest of the code is just looping through results. The next code of interest is the actual deletion. This is accomplished by calling the moveToTrash method on the message object. Just as it sounds like, it moves the message from wherever it is into your Trash folder.

Slight optimization

This was great progress. I ran it a few times, and I could see from my search result that things were getting cleared out. It was taking a while to execute, so I looked through the API docs some more. It turns out, I was wasting some time iterating through all those messages within the thread.

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {
    var thread = threads[i];
    for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];

That’s looping through each thread that is returned from the query, and in turn, retrieving and looping through each message within the thread. For things like OpenStack code reviews, some patches with a lot of feedback, updates, and CI comments, that can mean quite a few messages per thread.

Luckily, the thread object itself also has a moveToTrash method. So knowing that, I was able to clean that up quite a bit:

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {
    thread.moveToTrash()

This was definitely more efficient.

Space, the final frontier

I still had a problem with the script though. Or maybe, more of a problem with my patience and my trust in the Trash actually being emptied after 30 days. What I really wanted was to immediately and completely delete these messages and get them out of there.

None of the examples I read in the official docs mentioned it, though that could very well just be my impatience and not wanting to read up on more than I really needed to. But I found an old forum posting somewhere that mentioned a remove call.

DANGER: Only do this if you are very careful!

Seriously, don’t blame me if you mess things up. I warned ya!

This call will actually do exactly what I wanted. It gets the message completely out of there right away. Of course, with scripting and automation, there is no “Are you really sure?” dialogs to click and slow you down. If you get the query wrong, too late, it’s gone.

So again being very cautious about the query string being used to retrieve the threads, I tried one more approach.

Stepping back

Now the only bad thing here (OK, probably not the only bad thing) is that the remove call only works on message IDs. Not threads.

I just had to roll back some of the optimization I had done before and go back to iterating through the messages in the thread again. It slowed things down, but ultimately it was still faster, at least to me, because it would permanently get rid of the messages. The final version I ended up with was then:

var threads = GmailApp.search('from:review@openstack.org', 0, 10)
count = threads.length
for (var i = 0; i < count; i++) {
    var thread = threads[i];
    for (j = 0; j < thread.getMessageCount(); j++) {
        var message = threads.getMessages()[j];
        Gmail.Users.Messages.remove('me', message.getId());

That last line takes a user ID, but luckily it has a special value of me that will recognize the current user. And the ID of the message to remove, retrieved by calling the getId() method on the message. After running this version over and over after the 5 minute timeouts, I started to see my storage usage number start to drop down.

More cleanup

One more time - it’s very important to make sure you query to get threads is correct.

The query is very flexible though. So I was able to use wildcards and change the to/from to also clean up noreply.github.com, googlegroups.com, and several others where I have accumulated mass amounts of junk over time.

As I said, I’m also impatient, so once I had these cleaned up, I made another pass through some of the worse offenders and added in:trash to the query to immediately clean out some messages from the trash.

All tidied up

In the end, my total Google account storage went down from about 12.78G to 11.8G. Not bad.

I’m guessing I can’t automate Google Photos clean up as well. At least not without learning some image recognition APIs and adding a lot of risk. So that’s a clean up for another day.

But with a few quick hacky scripts, I’m pretty happy with how much space I was able to free up.

In part 2 of this series about OpenStack on ARM64, we got to the point where our cloud is fully deployed with all the Compute (VMs), Software Defined Networking (SDN) and Software Defined Storage (SDS) up and running. One additional component that we want to add is a Load Balancer as a Service (LBaaS), which is a key requirement for pretty much any high available type of workload and a must-have feature in any cloud.

OpenStack’s current official LBaaS component is called Octavia, which replaced the older Neutron LBaaS v1 project, starting with the Liberty release. Deploying and configuring requires a few steps, which explains the need for a dedicated blog post.

Octavia’s reference implementation uses VM instances called Amphorae to perform the actual load balancing. The octavia-worker service takes care of communicating with the amphorae and to do that we need to generate a few X509 CAs and certificates used to secure the communications. The good news is that starting with the Victoria release, kolla-ansible simplifies a lot this task. Here’s how to:

# Change the following according to your organization
echo "octavia_certs_country: US" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_state: Oregon" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organization: OpenStack" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_certs_organizational_unit: Octavia" | sudo tee -a /etc/kolla/globals.yml

# This is the kolla-ansible virtual env created in the previous blog post
cd kolla
source venv/bin/activate

sudo chown $USER:$USER /etc/kolla
kolla-ansible octavia-certificates

The communication between Octavia and the Amphorae needs an isolated network, as we don’t want to share it with the tenant network for security reasons. A simple way to accomplish that is to create a provider network with a dedicated VLAN ID, which is why we enabled Neutron provider networks and OVS VLAN segmentation in the previous post. Again, starting with Victoria, this got much easier with kolla-ansible.

# This is a dedicated network, outside your management LAN address space, change as needed
OCTAVIA_MGMT_SUBNET=192.168.43.0/24
OCTAVIA_MGMT_SUBNET_START=192.168.43.10
OCTAVIA_MGMT_SUBNET_END=192.168.43.254
OCTAVIA_MGMT_HOST_IP=192.168.43.1/24
OCTAVIA_MGMT_VLAN_ID=107

sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_network:
  name: lb-mgmt-net
  provider_network_type: vlan
  provider_segmentation_id: $OCTAVIA_MGMT_VLAN_ID
  provider_physical_network: physnet1
  external: false
  shared: false
  subnet:
    name: lb-mgmt-subnet
    cidr: "$OCTAVIA_MGMT_SUBNET"
    allocation_pool_start: "$OCTAVIA_MGMT_SUBNET_START"
    allocation_pool_end: "$OCTAVIA_MGMT_SUBNET_END"
    gateway_ip: "$OCTAVIA_MGMT_HOST_IP"
    enable_dhcp: yes
EOT

Unless there is a dedicated network adapter, a virtual ethernet one can be used. This needs to be configured at boot and added to the OVS br-ex switch.

# This sets up the VLAN veth interface
# Netplan doesn't have support for veth interfaces yet
sudo tee /usr/local/bin/veth-lbaas.sh << EOT
#!/bin/bash
sudo ip link add v-lbaas-vlan type veth peer name v-lbaas
sudo ip addr add $OCTAVIA_MGMT_HOST_IP dev v-lbaas
sudo ip link set v-lbaas-vlan up
sudo ip link set v-lbaas up
EOT
sudo chmod 744 /usr/local/bin/veth-lbaas.sh

sudo tee /etc/systemd/system/veth-lbaas.service << EOT
[Unit]
After=network.service

[Service]
ExecStart=/usr/local/bin/veth-lbaas.sh

[Install]
WantedBy=default.target
EOT
sudo chmod 644 /etc/systemd/system/veth-lbaas.service

sudo systemctl daemon-reload
sudo systemctl enable veth-lbaas.service
sudo systemctl start veth-lbaas.service

docker exec openvswitch_vswitchd ovs-vsctl add-port \
  br-ex v-lbaas-vlan tag=$OCTAVIA_MGMT_VLAN_ID

A few more Octavia kolla-ansible configurations…

echo "enable_octavia: \"yes\"" | sudo tee -a /etc/kolla/globals.yml
echo "octavia_network_interface: v-lbaas" | sudo tee -a /etc/kolla/globals.yml

# Flavor used when booting an amphora, change as needed
sudo tee -a /etc/kolla/globals.yml << EOT
octavia_amp_flavor:
  name: "amphora"
  is_public: no
  vcpus: 1
  ram: 1024
  disk: 5
EOT

sudo mkdir /etc/kolla/config/octavia
# Use a config drive in the Amphorae for cloud-init
sudo tee /etc/kolla/config/octavia/octavia-worker.conf << EOT
[controller_worker]
user_data_config_drive = true
EOT

…and we can finally tell kolla-ansible to deploy Octavia:

kolla-ansible -i all-in-one deploy --tags common,horizon,octavia

Octavia uses a special VM image for the Amphorae, which needs to be built for ARM64. We prepared Dockerfiles for building either an Ubuntu or a CentOS image, you can choose either one in the following snippets. We use containers to perform the build in order to isolate the requirements and be independent from the host OS.

git clone https://github.com/cloudbase/openstack-kolla-arm64-scripts
cd openstack-kolla-arm64-scripts/victoria

# Choose either Ubuntu or CentOS (not both!)

# Ubuntu
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Ubuntu \
  -t amphora-image-build-arm64-ubuntu

# Centos
docker build amphora-image-arm64-docker -f amphora-image-arm64-docker/Dockerfile.Centos \
  -t amphora-image-build-arm64-centos

ARM64 needs a trivial patch in the diskimage-create.sh build script (we also submitted it upstream):

git clone https://opendev.org/openstack/octavia -b stable/victoria
# Use latest branch Octavia to create Ubuntu image
cd octavia
# diskimage-create.sh includes armhf but not arm64
git apply ../0001-Add-arm64-in-diskimage-create.sh.patch
cd ..

Build the image (this will take a bit):

# Again, choose either Ubuntu or CentOS (not both!)

# Note the mount of /mnt and /proc in the docker container
# BEWARE!!!!! Without mounting /proc, the diskimage-builder fails to find mount points and deletes the host's /dev,
# making the host unusable
docker run --privileged -v /dev:/dev -v /proc:/proc -v /mnt:/mnt \
  -v $(pwd)/octavia/:/octavia -ti amphora-image-build-arm64-ubuntu

# Create CentOS 8 Amphora image
docker run --privileged -v /dev:/dev -v $(pwd)/octavia/:/octavia \
  -ti amphora-image-build-arm64-centos

Add the image to Glance, using the octavia user in the service project. The amphora tag is used by Octavia to find the image.

. /etc/kolla/admin-openrc.sh

# Switch to the octavia user and service project
export OS_USERNAME=octavia
export OS_PASSWORD=$(grep octavia_keystone_password /etc/kolla/passwords.yml | awk '{ print $2}')
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service

openstack image create amphora-x64-haproxy.qcow2 \
  --container-format bare \
  --disk-format qcow2 \
  --private \
  --tag amphora \
  --file octavia/diskimage-create/amphora-x64-haproxy.qcow2

# We can now delete the image file
rm -f octavia/diskimage-create/amphora-x64-haproxy.qcow2

Currently, we need a small patch in Octavia to properly render the userdata for the Amphorae:

# Patch the user_data_config_drive_template
cd octavia
git apply  ../0001-Fix-userdata-template.patch
# For now just update the octavia-worker container, no need to restart it
docker cp octavia/common/jinja/templates/user_data_config_drive.template \
  octavia_worker:/usr/lib/python3/dist-packages/octavia/common/jinja/templates/user_data_config_drive.template

Finally, let’s create a load balancer to make sure everything works fine:

# To create the loadbalancer
. /etc/kolla/admin-openrc.sh

openstack loadbalancer create --name loadbalancer1 --vip-subnet-id public1-subnet

# Check the status until it's marked as ONLINE
openstack loadbalancer list

Congratulations! You have a working LBaaS in your private cloud!!

Troubleshooting

In case something goes wrong, finding the root cause might be tricky. Here are a few suggestions to ease up the process.

# Check for errors
sudo tail -f /var/log/kolla/octavia/octavia-worker.log

# SSH into amphora
# Get amphora VM IP either from the octavia-worker.log or from:
openstack server list --all-projects

ssh ubuntu@<amphora_ip> -i octavia_ssh_key #ubuntu
ssh cloud-user@<amphora_ip> -i octavia_ssh_key #centos

# Instances stuck in pending create cannot be deleted
# Password: grep octavia_database_password /etc/kolla/passwords.yml
docker exec -ti mariadb mysql -u octavia -p octavia
update load_balancer set provisioning_status = 'ERROR' where provisioning_status = 'PENDING_CREATE';
exit;

The post OpenStack on ARM64 – LBaaS appeared first on Cloudbase Solutions.

2020 redefined collaboration. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the Open Infrastructure (OpenInfra) Foundation’s mission: building communities who write software that runs in production. In 2020, the Diversity & Inclusion Working Group and the Interop Working Group continued their efforts of collaboration virtually to grow the OpenInfra mission as well as the Working Groups’ missions.

Diversity & Inclusion Working Group

The Diversity and Inclusion Working Group continued reaching out to the other OIF projects and inviting them to join our efforts as we are not OpenStack specific and hope to continue offering opportunities for diversity at events and in the community.

Our main focus for the year was the creation of the OpenInfra Foundation’s stance on divisive language. All OpenInfra projects were contacted and over the course of the work all projects gave input except for Kata. The stance was approved by the Board and the community with several recommendations being relayed to the Board for continuing efforts from them as well as the creation of a Wiki page with suggestions to be used when addressing the replacement of divisive words.

The D&I WG also helped to lead OpenStack’s presence during Grace Hopper’s Open Source Day. We had about 13 mentors from the community and 45 attendees throughout the day.

In 2021, we plan on continuing to aid the OpenInfra projects in their efforts to remove divisive language and to use more inclusive words. We also hope to be able to resume efforts with the community during virtual and in person events.

Interop Working Group

The OpenInfra Interop Working Group has been issuing guidelines for OpenStack Logos and Branding programs as part of Marketplace development every release as listed below:

• OpenStack Powered Platform
• OpenStack Powered Compute
• OpenStack Powered Object Storage

During Ussuri we had two add-ons:

• OpenStack Powered DNS
• OpenStack Powered Orchestration

In Victoria we added one more add-on: the OpenStack Powered Shared File System.

The Interop repos were moved under the OpenInfra Foundation in Ussuri and are consolidated in Victoria to enable QA and test team to plan for ongoing efforts in OpenInfra projects Kata, Airship and StarlingX for container conformance.

The forward looking statement from the OpenInfra Interop Working Group is to start the ground up efforts to set new guidelines for Integrated Wallaby release and define the OpenInfra Container Conformance Program. The current work in the technical team has been efforts around containerizing Refstack to move to Ansible and Go besides python 3.7 support for test tools and libraries used by all OpenInfra Projects. The Baremetal logo program was deferred to future releases after new guidelines are in place.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report, join the OpenInfra community to build the next decade of OpenInfra, and learn about the different Working Groups.

The post 2020 Annual Report: OpenInfra Foundation Working Groups appeared first on Superuser.

Jon Snow is a prominent character in the Game of Thrones TV series. He is a skilled swordsman, defined by a strong sense of justice and honor. Jon Snow proves his leadership skills and strategic thinking in many situations.

But you might remember Jon meeting the Wildings, a new culture that he did not fully understand the first time he met them. It took time until Jon could resonate with them, and he looked like a newb in the process. This triggered Ygritte to call out, “You know nothing, Jon Snow” several times. Soon enough, Jon learned how to deal with the Wildlings, he earned their respect and trust, and they followed him as a leader.

I find this analogy to fit people starting to work with OpenStack. It feels very overwhelming in the beginning, but you can build on your skills and experience once you get familiar with what project is responsible for what functionality.

I had the unforeseen-except-perhaps-in-my-dreams chance to get a new job in 2015, to work with OpenStack. One way I prepared for the new job was by listening to OSPod, I’ve listened to many episodes several times. I found it interesting to listen to OpenStack’s initial cast talking about the projects they contributed to. Several of these podcast episodes are quite inspiring. Take this one for example.

OpenStack breaks down to services running on Linux and troubleshooting OpenStack goes down to troubleshooting these services and checking their log files. The trick is to know where to look first without wasting time checking services that are not the cause of the problem you are investigating. It might sound simple, but it’s not, and sometimes one needs to employ diligent sleuthing to find what the problem is and fix it.

I started working with OpenStack from its Mitaka release. My first role was as a support engineer and, I helped telecom operators around the globe fix problems in their OpenStack deployments. Looking back, I can draw the line, and I can tell you that once in a blue moon I stumbled over bugs in OpenStack. Maybe the reason for that was that the Mitaka release reached an acceptable maturity level. Another reason could be the fine-tooth combing of OpenStack services through all sorts of testing like functional testing, system testing, and end-to-end testing. Extensive testing prevents going to production and discovering significant bugs. Most of the support cases that came in condensed to networking issues:

• Cannot access a VM on a port – networking problem
• Cannot access storage media from VM – networking problem
• VM does not boot anymore – networking problem
• OpenStack controllers froze – networking problem

Most of the time, the networking problems did not result in bugs filed under the Neutron project. Many cases were resolved by restarting services, patching network devices, fixing Linux configuration, properly configuring network devices like routers and switches.

This article builds on experiences troubleshooting OpenStack. You can use this approach:

• Define what the problem is
• Narrow down what the cause is
• Solve the problem
• Pass on the experience through proper documentation

Define the problem

A customer sent in a ticket saying that they cannot connect from VM A to VM B on port 1521 over TCP, but ping between the two hosts is working. They mentioned the problem must be in Neutron, but they are not sure what Neutron logs they should check.

As a first step, I must understand the problem better. It’s easy to fall for the confirmation bias and ask the customer for Neutron configuration printouts and logs. The customer might be sitting in a different time zone, and email communication can be sparse because of the time difference. Assume I get the printouts and logs I asked for, and I can’t find anything obviously wrong. Then I need at least one more day to ask for more information.

So to avoid sending the customer scurrying grumpily all over the place it’s good to ask for printouts that confirm the problem they raised.

We cannot access VM B on port 1521? I would assume the service running on port 1521 on VM B is up and running.

Ask for printouts to confirm it:

VM_A $ telnet 192.168.102.155 1521
VM_B $ telnet 192.168.102.155 1521

where 192.168.102.155 is the IP address on VM B received from the customer in the problem description.

There might be a problem with the commands above. On most production servers, telnet is not installed, but cat should be there. We can instead ask for:

cat < /dev/tcp/<hostname_or_ip>/<port_number>
VM_A $ cat < /dev/tcp/192.168.102.155/1521
VM_B $ cat < /dev/tcp/127.0.0.1/1521

In this example, bash will attempt to open a TCP connection to the corresponding socket.

The customer mentioned that ping between the hosts works. We want to run a command that confirms this fact but that also would check for things like MTU size, which could be a cause for the problem they raised:

Ask for a printout of ping that tests a bigger packet size, prohibiting fragmentation, with packets being sent at shorter intervals:

VM_A $ ping -s 1472 -M do -i 0.2 -c 50 192.168.102.155

What I got back was this:

VM_A $ cat < /dev/tcp/193.226.122.155/1521
bash: connect: Connection refused
bash: /dev/tcp/193.226.122.155/1521: Connection refused

The service running on port 1521 was not up, which explains the symptoms. It often happens that the cause of a problem is something simple that was overlooked, like a service not running, a service that keeps restarting, or the VM where the service should be running is down. In her talk about troubleshooting Neutron, Rossella Sblendido mentions just that.

It’s worth to ask for printouts that confirm the problem description. It might result in solving a problem in the definition phase.

One might assume there’s not much to document when solving the problem described above. Add the questions you asked together with the customer replies and printouts to the customer ticket. If someone later finds your case, it will serve as a good example of asking questions and defining the problem.

Solve the problem

We will now look into a support ticket where a customer says they rebooted one out of three OpenStack controllers and lost access to it. In this case, the OpenStack services are running inside three different VMs, and thus, we have three virtual OpenStack controllers.

We should avoid asking questions like “What did you do, what did you change?” which renders an interlocutor to defense mode. Such questions might read as accusations when the situation is tensed.

When defining the problem, we could ask:

• “How do you connect to the controller VM?”
• “Please send me the console log, /var/log/libvirt/qemu/controller-1“
• “Please send me a printout of $ script -a VM_log ; sudo virsh console controller-1“

The console logs show the following:

The disk drive for /mnt/backups is not ready yet or not present.
Continue to wait, or Press S to skip mounting or M for manual recovery

It seems that a block device is no longer attached to the controller VM or there is some misconfiguration in /etc/fstab.

We need to access the VM, possibly by booting in single mode, and correct /etc/fstab by removing the offending line. Some months ago, someone added a line to /etc/fstab to mount a network shared device and back up files to a remote destination, which does not exist anymore.

Documentation is again very important. Besides properly documenting the information exchanged with the customer, you can write a knowledge object that explains how to connect to an OpenStack controller VM when the VM won’t boot, and it is stuck looking for block devices that do not exist anymore.

Document the problem

The third support case is one where the customer observed slow access to block devices hosted on the backend storage from some VMs. Other VMs cannot access their block devices at all, and a third part of VMs experience no problems whatsoever.

Start with the VMs that cannot access the block devices at all. How are they spread across computes? Here is an example of how to filter for the computes where the VMs are running, assuming all VMs belong to the same Heat stack. List resources filtered on Nova::Server, pipe the output to xargs and extract the hypervisor_hostname to get the compute names. Is there a pattern here? Do the problematic VMs belong to the same computes?

If the VMs cannot access their block devices on the storage backend, it might be a networking problem. Look for log entries concerning the storage network interfaces on the computes hosting the VMs in trouble. The dmesg log show several entries like this one

NETDEV WATCHDOG: eth4 (i40e): transmit queue 2 timed out
i40e 0000:83:00.2 eth4: NIC Link is Down

The storage interface seems to be down, so try to bring it up. It worked, and the VM can access the block device again.

ip link set up eth4

Apply the same fix for the rest of the computes. On the computes where we have slow access to storage, reset the problematic interface, based on findings in the dmesg log.

We fixed the customer’s problem, but we do not know yet what caused it. It is important we continue the investigation otherwise, the problem might surface again.

If you can’t find anything obvious in the logs to help you conclude what cause the interfaces to go rogue, it might be a good time to turn to an expert and ask for help.

When you start working with OpenStack, I can highly recommend you find yourself an expert. Everyone knows who they are: a person with some years of experience, that can help with almost any technical problem. It’s also the person that everybody wants a piece of. In most cases, it is the person that writes excellent documentation too. Ask for help if you get stuck, but prepare your questions thoroughly. Make sure you can coherently define the problem. Have the logs ready in case someone needs to check them. By doing this, you don’t waste someone else’s time; it’s a matter of showing respect.

The problem happens only on 12 out of 168 computes Let’s look at the network interfaces on these computes. They look like this:

root@compute-40-3 # lshw  -C network -businfo
Bus info          Device          Class          Description
============================================================
pci@0000:01:00.0  eth0            network        I350 Gigabit Network Connection
pci@0000:01:00.1  eth1            network        I350 Gigabit Network Connection
pci@0000:83:00.0  eth2            network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.1                  network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.2  eth4            network        Ethernet Controller X710 for 10GbE SFP+
pci@0000:83:00.3                  network        Ethernet Controller X710 for 10GbE SFP+

So we have two interfaces for the control network, eth0 and eth1. We have two interfaces for the storage network, eth2 and eth4. And we have two interfaces for the traffic network. The storage network interfaces use the kernel driver; you can see their device name in this output. The traffic network interfaces, the ones that don’t have a device name. use the DPDK driver, ( the technology to offload TCP packet processing from the operating system kernel to processes running in user space, you want this for higher performance.) All 4 of them share the same bus, look at the PCI addresses.

The network card looks like this. So we have network interfaces controlled by the kernel driver and network interfaces controlled by the DPDK driver, on the same physical card. The problem showed up on the computes having these NICs installed. We assume it is related to these NICs.

We can run iperf on the NICs with ports controlled by multiple drivers and observe:

• packet drops on the ports controlled by the kernel driver, and lower TCP throughput, or
• complete traffic loss due to “tx hang” on the ports controlled by the kernel driver.

$ iperf -c 192.168.122.111 
------------------------------------------------------------
Client connecting to 192.168.122.111, TCP port 5001
TCP window size:  586 KByte (default)
------------------------------------------------------------
[  3] local 192.168.122.100 port 59454 connected with 192.168.122.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.43 GBytes  9.4 Gbits/sec

We raised bug reports towards the manufacturer of the network card. Later, the manufacturer provided patches that solved the problem.

• https://dpdk.org/dev/patchwork/patch/34947/
• https://dpdk.org/dev/patchwork/patch/34948/

Proper documentation is very important! You will document the problem and its solution, but it can be useful to write down the steps you took to figure it out, the tools and commands you used.

Besides properly recording this problem with its solution in your company’s (most probably internal) tools, it would be nice to share the problem on the internet because it might help someone else. Write about it on your blog, tweet about it, or record a video and upload it to youtube.

There are many problem-solving strategies and methods. I wish I could state that I found the few-step approach that solves any OpenStack problem, but there is no silver bullet. What can help troubleshooting OpenStack is to ask the right questions to help you understand and define the problem, to establish and test a few hypotheses of a possible cause and narrow down where the problem is located. Then, once a fix is available, to prioritize properly documenting the problem with its solution.

Get Involved

This article is a summary of the Open Infrastructure Summit session, You know nothing, Jon Snow! OpenStack troubleshooting from a beginner’s perspective.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post You Know Nothing, Jon Snow! OpenStack Troubleshooting From a Beginner’s Perspective appeared first on Superuser.

Codership, the developers of Galera Cluster, have set new days for online training for Galera Cluster. There are two sets of times for our EMEA and American attendees, with the former happening on March 2021, 16th and 17th, starting 10 AM CET and the latter happening on March 2021, 23rd and 24th, starting 9 AM EST.

This is a hands on course that spans two days, and you will have about six contact hours per day in this instructor-led course. If you are a DBA or been given a Galera Cluster to manage, this is the course for you, as you’ll learn how to run your Galera Cluster in an optimal fashion, from setup, performance tuning, monitoring, as well as backups and more. We will cover not just Codership’s Galera Cluster distribution but also MariaDB Galera Cluster and Percona XtraDB Cluster in this course. For more information, please read the full content breakdown.

As an added bonus, we will also be covering database monitoring with Galera Manager our new GUI tool.

What are you waiting for? Sign up now, as the early bird rate ends 28 February. There is also a volume discount available.

As many organizations experienced in 2020, the Open Infrastructure (OpenInfra) Foundation had several challenging decisions to make as the pandemic began to spread around the world. Having the safety of the community as the top priority, the decision was made to move the entire 2020 programs, including the PTG (Project Teams Gathering), OpenDev series, and OpenInfra Summit to a virtual format. With the help of the Foundation and event sponsors, partners, and the support of the entire community, we were able to continue our mission to collaborate, share knowledge, network, and continue improving the software with the broadest global reach ever despite the inherent challenges of the year.

OpenInfra Summit

The first virtual OpenInfra Summit was held in October 2020, gathering 10,000 attendees (largest Summit attendance to date) from 120+ different countries to discuss 30+ open source projects. In addition to use cases from users like Volvo Cars Corporation, Ant Group, GE Digital, Société Générale, China Mobile, European Weather Cloud, Workday, and China Tower- plus a big announcement about the future of the Foundation:

• The Summit kicked off with the Keynote announcement that the OpenStack Foundation was officially becoming the OpenInfra Foundation
• Volvo Cars Corporation discussed how it successfully uses Zuul for continuous integration in a range of different software components, and how Zuul is a game changer that helps them avoid merging broken code.
• Ant Group, the leading peer-to-peer payments processor in China, presented why isolation methods such as scheduling isolation and LLC isolation are used in combination with Kata Containers.
• GE Digital presented the tools, migration procedure, and highlighted the biggest challenges in upgrading from OpenStack Newton to Queens with minimal downtime.
• SK Telecom 5GX Labs won the 12th Superuser Award for developing a containerized OpenStack on Kubernetes solution called SKT All Container Orchestrator (TACO), based on OpenStack-helm and Airship, all while contributing upstream to both the OpenStack and Airship projects.

Summit videos are available on the Summit videos page. Thank you to our Summit sponsors for supporting the event!

OpenInfra TV: Bringing the Summit to Asia

At the annual OpenInfra events, the global community convenes to discuss the challenges and success of integrating different open source projects. Averaging attendees from over 60 countries at face-to-face events and attracting attendees from over 120 countries for the virtual OpenInfra Summit, the Foundation events team concentrates on making sure all events cater to the diversity of the audience.

For the virtual event, time zones were one of the biggest challenges as one block of time is never perfect for every single person in the world. The team selected four communities to pilot OpenInfra TV—China, India, Japan, and Korea—to provide local coverage of Summit highlights presented by community leaders in their local language.

OpenDev

The OpenDev event, held in-person in previous years, was a three part virtual series that focused on three topics: Large-scale Usage of OpenInfra Software, Hardware Automation, and Containers in Production.

OpenDev events bring together the developers and users of the open source software powering today’s infrastructure, to share best practices, identify gaps, and advance the state of the art in OpenInfra. For each topic, participants joined discussion oriented, collaborative sessions where they explored challenges, shared common architectures, and collaborated around potential solutions. Each topic’s agenda was set by a programming committee made up of community members who met for weeks leading up to the event to make sure it was tailored to the broader community.

The three events gathered over 2,000 registrants from over 70 countries.

Here are some of the highlights:

Large-scale Usage of OpenInfra Software – June 2020

The event kicked off with user stories from Blizzard Entertainment, OpenInfra Labs, and Verizon who shared their own scaling challenges in short presentations followed by Q&A. After that, the discussion focused on a variety of topics including upgrades, high availability (HA) networking, bare metal, and more. At the conclusion of the event, participants looking to get further involved were encouraged to join the OpenStack Large Scale SIG to continue sharing challenges and solutions around scaling.

Thank you to the programming committee members: Beth Cohen, Roman Gorshunov, Belmiro Moreira, Masahito Muroi, and Allison Price!

Read the full recap and catch up on the discussion recordings.

Hardware Automation – July 2020

This OpenDev event focused on topics around hardware provisioning lifecycle for bare metal, bare metal infrastructure, networking and network security. Participants representing over 200 companies spent three days sharing their knowledge and discussing their experience of building and operating software that automates their hardware in cloud environments. One of the ways mentioned to keep the momentum going after this event was to join the OpenStack Multi Arch SIG and OpenStack Bare Metal SIG to continue sharing challenges and solutions around Hardware Automation. Julia Kreger also highlighted the latest white paper from the OpenStack Bare Metal SIG: “Building the Future on Bare Metal, How Ironic Delivers Abstraction and Automation using Open Source Infrastructure.”

Thank you to the programming committee members: Keith Berger, Mark Collier, Julia Kreger, Mohammed Naser, and James Penick!

PTGs / Forums

In June of 2020, we hosted our very first virtual Project Teams Gathering. We were pleased to have over 700 people register for the event – more than we’ve had for any in person PTG. By taking it virtual instead of in-person, we had new people participate that have otherwise never been able to attend the PTG. Over the course of the week, 44 teams across Airship, Kata Containers, OpenStack, StarlingX, and Zuul met to discuss their upcoming releases and make progress on standing items. The format of the PTG was different than any event we’ve hosted before because it was all virtual and we needed to accommodate for all timezones. We settled on 4 hour blocks of working hours with 4 hours inbetween and allowed teams to schedule themselves to better fit their day to day schedules and time zone restrictions. The PTG again made use of the community developed PTGbot with some tweaks to help keep people aware of active discussions and find where and when teams were meeting.

The community had two options to use for their video meetings: the OpenInfra Foundation subsidized Zoom rooms, or the OpenDev team hosted meetpad rooms. Many teams wrote up summaries of their discussions and accomplishments throughout the week:

• Airship
• Kata Containers
• OpenStack
• StarlingX

We continued to meet virtually throughout the year and the week after our first Virtual OpenInfra summit, we decided to again, keep people safe, and run it like we did in June. In October of 2020 the teams met on Zoom and in Meetpad to talk to each other and collaborate. Over 500 people registered for the event spanning 45 countries. The event hosted community members across 46 teams and had their usual technical discussions which you can find summarized here:

• Airship: http://lists.airshipit.org/pipermail/airship-discuss/2020-November/001135.html
• OpenStack: https://www.openstack.org/blog/wallaby-vptg-summaries/
• StarlingX: http://lists.starlingx.io/pipermail/starlingx-discuss/2020-November/009958.html

OpenStack Days / OpenInfra Days / Cloud Operator Days

Overcoming the local challenges of the pandemic, the global OpenInfra community organized four OpenStack/OpenInfra Day events (a combination of virtual and face-to-face in Turkey, China, Indonesia and Korea – educating over 10,000 attendees around the world.

The typical attendee reached at these local events are cloud and IT architects, software developers, platform and solution engineers and product management.

The Tokyo community introduced a new local event, Cloud Operator Days, a two day online event that gathered 1,400 attendees to discuss challenges of operating cloud infrastructure.

These one or two day regional events are organized by the local community and supported by the OpenInfra Foundation to include project workshops, upstream training, and booths from the local ecosystem. These events continue to be an excellent touchstone for the community to engage with our ecosystem, and to gather local open source developers and users to collaborate, share use cases, and support for the OpenInfra projects.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report, join the OpenInfra community to build the next decade of OpenInfra, and learn about what events to expect for 2021.

The post 2020 Annual Report: OpenInfra Event Highlights appeared first on Superuser.

The Open Infrastructure (OpenInfra) community had a strong year in spite of the unprecedented global challenges in 2020. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the OpenInfra mission: building communities who write software that runs in production.

OpenInfra User Groups Growth

In 2019, OpenInfra User Groups migrated to the Meetup Pro platform, which allowed the groups more visibility on the web, as well as a streamlined way to organize and communicate.

In 2020, we saw the number of groups grow past 40, adding communities in places like the Congo and Tijuana.

Amidst challenging circumstances, User Groups showed their commitment to community, organizing in new and unique ways. When in-person events were put on hold, several groups took on the task of hosting virtual meetups covering a variety of topics including hybrid cloud, storage, containers, and edge. Some groups even organized watch parties for the first virtual OpenInfra Summit that took place in October.

While we know User Groups are looking forward to meeting in person again, we’re sure that virtual meetups are still going to be around because of the valuable accessibility they provide so many community members around the world.

Outreachy Program

For the current round, December 2020 to March 2021, 5 interns were selected:

• Kuryr: T.I. Fasoyin from Nigeria
• Manila: Dina Saparbaeva from Karakol, Kyrgyzstan
• Ironic / OpenStack SDK: Anuradha from India
• Ironic: Oluwatosin Farai from Nigeria
• Manila: Paulali from Kenya

Students in OpenStack

Students are the future- the future of the workforce, the future of the tech industry and the future of open source. The OpenStack community has been slowly building relationships with more and more universities, getting them involved in work upstream in the community. These relationships are mutually beneficial; students get practical experience and build skills in communication and working collaboratively and the OpenStack community gets fresh perspective and the excitement that comes with new contributors.

Boston University Senior Design Program

Three students- Nicole Chen, Ashley Rodriguez, and Mark Tony- at Boston University selected OpenStack Manila as their senior design project. With the help of community mentors- Goutham Pacha Ravi, Victoria Martinez De La Cruz, Jeremy Freudberg, Maari Tamm and Kendall Nelson- the students have begun working on implementing the Manila APIs in the OpenStackSDK. There is a community effort to reach feature parity between the individual python clients for OpenStack services and the OpenStackSDK. The students were onboarded into the community, they have begun attending community meetings, and they also have been working on the specification outlining the work they are beginning and the details of implementation. If they are able to complete all the work in the OpenStackSDK, they can then begin implementing those features in the OpenStack Client. They will continue their work for the remainder of their school year, wrapping up in May of 2021.

North Dakota State University Capstone Program

For several years now, we have participated in North Dakota State University’s Capstone program that gives students a chance to select a project- either open source or industry- to work on throughout the spring semester. At the start of 2020, the OpenStack community had four students- Dawson Coleman, Luke Tollefson, Noah Mickus, and Steven Glasford- helping to add support for configuring TLS ciphers and protocols on Octavia load balancers. Michael Johnson, Adam Harwell, and myself mentored students through their semester helping them learn not only about Octavia, but about OpenStack, and open source. With their hard work, they landed TLS cipher and protocol support before the end of the Ussuri release and it became a large part of the promotion and marketing around that release. The rest of their work landed shorty after feature freeze and was included in the Victoria release that happened in October of 2020.

Oregon State University Internship

This year the Foundation started a partnership with Oregon State University funding one student to work in the OpenStack community on the Glance project to give them real world experience working on an open source project. Kendall Nelson and Erno Kuvaja mentored Khuong Luu. He worked on a variety of tasks in Glance. Khuong made several documentation improvments including adding missing information and reording topics to make them more readable and the information more discoverable. He cleaned up config options that were causing orphan processes in Glance when a user sent a SIGHUP signal. Khuong also cleaned up a bug being caused my a missing image property. While he didn’t spend time on one large feature, his efforts cleaned up a lot ot technical debt. He finished up his internship when the school year concluded and we thank him for all his hard work!

We have a new student intern getting started with his onboarding process. Ryan Zimmerman, once up to speed on OpenStack and our community, wil begin work on the OpenStackSDK and helping OpenStack services reach feature pairity with their own project specific python clients.

OpenStack Upstream Institute

The OpenStack Upstream Institute (OUI) training is an interactive course that teaches attendees about how the OpenStack community operates including the community structure and processes. The training also includes hands-on practice to teach participants how to use the tools that the community depends on. These tools -such as Git, Gerrit, and more – are maintained by the OpenDev (https://opendev.org) community. As in person events were not possible during 2020, we had to re-invent OUI to make it suitable for being hosted in a virtual format.

The virtual training had a slightly different structure compared to the in person classes. The training was split into two 3-hour long blocks: OpenStack specifics and OpenDev tools. The OpenStack module covered items such as governance, OpenStack release cycles, ways to communicate within the community, and more. The attendees learned how the community operates as we went through the topics during the first block. The class utilized active synchronous conversation between students and mentors, where attendees are able to ask questions from our mentors, who are all active community members.

The OpenDev tools module was also a hands-on class that focused on teaching attendees to setup and use tools that are used for software and documentation development processes as well as task tracking. Many of these tools are used by multiple OpenInfra communities and adjacent projects. By organizing this way, we were able to welcome and educate students who might not have an interest in OpenStack specifically which gave us a broader audience. During the class the attendees worked through exercises with guidance from the mentors to give them practice using the tools. During the training we also had breakout rooms students and mentors could hop into to ensure any technical issues students had were solved before the end of the class.

OUI, with the new structure, was held during the OpenInfra Summit in October. The class material was also utilized at other events to hold the training in different formats. Ghanshyam Mann hosted a virtual OUI session before the Summit and Amy Marrich led a group of over 15 mentors to educate participants of the Grace Hopper Conference’s Open Source Day on how to get started in OpenStack.

Press & Analyst Highlights

Public relations efforts are led by a distributed team of professionals in the United States, Europe, and Asia Pacific. This team engaged analysts and journalists worldwide in dialogue, proactively delivering news, commentary, and contributed byline content for both the Foundation and for its individual projects. Community news highlights are available at https://openstack.org/news.

Key media relations initiatives included the following:

• Launch of the OpenInfra Foundation
• Project software releases including OpenStack Ussuri and Victoria, Kata Containers 2.0, and StarlingX 4.0
• Virtual OpenInfra Summit
• Virtual OpenDev event
• Celebration of the OpenStack project’s 10th anniversary
• Momentum of OpenInfra projects in the broad OpenInfra and cloud IT landscape

Analysis shows that OpenStack/OpenInfra Foundation news coverage comprised more than 24,100 total placements in 2020. Media coverage was distributed globally (31% APAC, 30% North America, 22% Europe, by volume), and extended into 94 countries.

As evidenced by the highlights shared below, media attention has shifted from questions about OpenStack’s future to an active interest in use cases and the progress of the other projects supported by the OpenInfra Foundation.

In 2020, the OpenInfra Foundation adopted a virtual briefing format, shifting from in-person analyst day sessions at the physical events. During the course of the year, Foundation staff leadership conducted briefing sessions with more than two dozen analysts in Asia, North America, and Europe, providing updates on both the evolution of the Foundation’s business model and the status of individual projects in the organization’s portfolio. Firms with strong ongoing interest in our community’s work include IDC, Forrester, Redmonk, 451 Research, and others.

Analyst interest in the community’s work has shifted steadily since the early days of the Foundation. While early coverage necessarily focused on the OpenStack project, the focus has broadened substantially to include other projects under the OpenInfra Foundation. Additionally, coverage has shifted to include OpenInfra Foundation projects as components of broad-based solutions that leverage OpenInfra technologies such as Kubernetes.

Superuser

The OpenInfra Foundation continued to support the popular Superuser publication, covering the OpenInfra ecosystem news, case studies, event recaps, product updates and announcements, project release and more. In 2020, superuser has reached more than 150,000 unique visitors from 193 countries. Editors and contributors are members of the OpenInfra Foundation, the OpenStack and adjacent open source communities, and professional journalists.

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report and join the OpenInfra community to build the next decade of OpenInfra.

The post 2020 Annual Report: OpenInfra Community Highlights appeared first on Superuser.

Kuryr is an OpenStack project that relies on Neutron and Octavia to provide connectivity for Kubernetes Pods and Services. This improves the network performance especially when running Kubernetes clusters on top of OpenStack Virtual Machines with Kuryr. It avoids double encapsulation of network packets as only the Neutron overlay is used and consequently simplifies the network stack and facilitates debugging. The following diagram shows the gain of using Kuryr when compared to using Openshift SDN with TCP stream traffic between Pods on nodes across different hypervisors.

Kuryr-Kuberntes has two components that are deployed as K8s Pods: the controller and the CNI. It differs from other OpenStack projects that react to events resulting from user interaction with OpenStack APIs since the CNI and controller actions are based on events that happen at the Kubernetes API level. Kuryr controller is responsible for mapping those Kubernetes API calls into Neutron or Octavia objects and saving the information about the OpenStack resources generated, while the CNI provides the proper network interface binding to the Pods. Figure 02 shows the interactions of Kuryr components, Neutron, Octavia and Kubernetes components.

A controller pattern in Kubernetes works like a control loop that aims to make the system state match the system specification, which is embedded in the Kuberntes API. To achieve this the controller needs to react to Kubernetes events, change the system state by interacting with external systems and saving data when necessary. There are golang libraries to facilitate the implementation of the controller pattern, but Kuryr needed its own version as it’s mainly written in Python.

Kuryr controller can be considered a collection of controllers where each controller handles a Kubernetes resource that is represented by an OpenStack resource, e.g. Pods are mapped into Neutron ports, Services into Octavia load balancers and Network Policies into Neutron Security groups. Previously, the references to the OpenStack resource used to be saved directly on the corresponding Native Kubernetes resources with key-value pairs of string to ensure the proper OpenStack resource was updated or cleaned up when needed. However, that proved to not facilitate troubleshooting and synchronization of the current state with the specification.

Users can now rely on Kubernetes API extensions with custom resource definitions to introspect the state of the system and view Kubernetes resources easily mapped to OpenStack resources and being handled like any other native Kubernetes object. The custom resource not only allows having small specialized controllers, but also follows the Kubernetes API conventions and patterns, improves readability and debugging. Each resource definition contains the specification and status fields facilitating the synchronization and finalizers to avoid the hard deletion of the Kubernetes resource while the OpenStack resource deletion is being handled. Previously, if the Kubernetes resource was deleted prior to the corresponding OpenStack one being removed a conflict could happen when a new object was created. Examples of Kuryr CRD and more can be found on the Kuryr documentation.

All these new improvements are already available Upstream on Kuryr-Kubernetes since Victoria release.

Get Involved

This article is a summary of the Open Infrastructure Summit session, K8s controller and CRD patterns in Python – kuryr-kubernetes case study.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post Enhancement of Kubernetes Controller Pattern in Python with CRD appeared first on Superuser.

2020 redefined collaboration. While the global community was not able to meet in person, everyone still convened to discuss how we collaborate to deliver the Open Infrastructure (OpenInfra) Foundation’s mission: building communities who write software that runs in production. In 2020, Magma’s code and documentation were contributed to a new, independent software project magmacore.org and the need for the services provided by OpenDev has become more apparent; OpenDev has stepped to the plate and made a number of changes to help make 2020 as productive as possible all while minimizing disruption to existing services and users.

Magma

Magma is an open-source software platform that delivers an open, flexible and extendable mobile network core solution. Magma’s mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. It is designed to be access network agnostic (4G, 5G or WiFi), flexibly supporting a radio access network with minimal development and deployment effort.

Magma was developed as an open-source project as part of Facebook Connectivity’s incubator program, with significant contributions from the OpenAirInterface Software Alliance. In July of 2020 Magma’s code and documentation was contributed to a new, independent software project magmacore.org.Magma’s architecture consists of three major subsystems:

• The Access Gateway (AGW) provides network services and policy enforcement. In an LTE network, the AGW implements an evolved packet core (EPC), and a combination of an AAA and a PGW. It works with existing, unmodified commercial radio hardware. Support of 5G radios is under active development.
• Orchestrator is a service that provides a simple and consistent way to configure and monitor the wireless network securely. The orchestrator can be is normally hosted on a public or private cloud. The metrics acquired through the platform allow you to see the analytics and traffic flows of the wireless users through the Magma web UI. Orchestrator also provides REST APIs allowing simple integration with monitoring and management systems.
• The federation gateway allows Magma to integrate with other Mobile Network Operator core network functions using standard 3GPP interfaces. It acts as a proxy between the Magma AGW and the operator’s network and facilitates core functions, such as authentication, data plans, policy enforcement, and charging to stay uniform between an existing MNO network and the expanded network with Magma.

Community

At the OpenInfra Summit in October 2020, the OpenInfra Foundation announced its support of the Magma project. With help from the Foundation, magmacore.org has been structured to make it easy to contribute to the project. It is centered around a developer-led governance with Maintainers providing code review and approval for contributions and a small Technical Committee selected from among the project maintainers providing overall architecture and roadmap oversight. In addition, the OpenInfra Foundation also helped launch the Magma Slack workspace back where many community and technical conversations take place. After the Summit announcement, the Slack workspace grew by over 100 people.The OpenInfra Foundation also assists with the monthly Magma community meetings where project updates and use cases are presented. Community members also get the opportunity to ask questions directly to the Magma engineers. The first community meeting took place in November 2020 and had over 20 attendees. View the meeting recordings here.

OpenDev

OpenDev believes that free software needs free tools. OpenDev is a collaboratory for open source software development at a huge scale. Its focus is on code review, continuous integration, and project hosting provided exclusively through open source solutions like Git, Gerrit, Zuul, and Gitea.

As the world shifted to remote work and online collaboration in 2020, the need for the services provided by OpenDev has become more apparent. OpenDev has stepped to the plate and made a number of changes to help make 2020 as productive as possible all while minimizing disruption to existing services and users. In particular we would like to call out the new formalized governance for the project, new services like meetpad, and service upgrades (like Gerrit).

In 2020 OpenDev formalized its governance with two main bodies: the Service Coordinator and the Advisory Board. The Service Coordinator is an individual elected from OpenDev’s contributors whose primary role is to communicate planned changes to users and incorporate their feedback. The Advisory Board is made up of liasons from the various projects using OpenDev and organizations that contribute compute resources. They serve as the primary communcation point between users and the service. If you represent an OpenDev using project or a contributing organizationand would like to get more involved feel free to join the Advisory Board.

One of the big changes we have had to deal with as a community is the transition to virtual events. The OpenDev team has deployed a Jitsi Meet instance which is paired with our Etherpad service to act as a video conferencing with built in note taking tool. This service has been used in multiple Project Team Gatherings at this point and helped us keep an event format that is familiar to those who have attended in person events. Outside of events we have seen this service used for team meetings and other places where video conferencing is helpful. We expect this service to replace the existing Asterisk based phone conference tools that we previously deployed.

Several services have also seen major upgrades during 2020. The most notable (and likely noticeable) is Gerrit, but Gitea and Etherpad have also been upgraded. For Gerrit this upgrade was long overdue and gets us much closer to a modern upstream Gerrit experience. We are hoping that this change will help us work with upstream Gerrit more closely going forward, bringing our needs into upstream development. A number of features come with this upgrade including use of NoteDB for data storage and single page diffs. We continuously upgrade our Zuul deployment, but over the course of 2020 it got some major new features and a new look to its web dashboard as well.

Our long running effort to migrate away from Puppet for service deployments and configuration management continues. Several more services have been migrated onto Ansible and Docker including Hound, Grafana, and our IRC accessbot. This effort enables us to use tools that more developers are comfortable with to manage our services and gives us better control over what is being deployed.One of the major tools that has enabled our transition to well tested Docker based deployments is Zuul’s image build pipeline job library. Zuul provides a set of building blocks to run Docker image builds for changes using local image repositories to share the resulting images between test jobs and changes. This allows any OpenDev user to test sequences of changes to different services and know that each change in the sequence is a landable production state.

On the continuous integration side of the house our CI jobs helped 1,800 users merge more than 54,000 changes. We continue to keep our test environments up to date, pulling in new distro releases and removing old EOL versions. Keep this in mind as CentOS’s announcement that version 8 will be EOL at the end of 2021 means that we will be removing it at the end of this year. CentOS Stream 8 test nodes are available as are alternative distros with long term support and stable releases.

OpenDev continues to be a community run project. These services exist because we all help work together to make it happen. 2020 was a difficult year in many ways, but for OpenDev one of the most noticeable difficulties has been losing individuals who have long been a part of the community providing tremendous amounts of help to OpenDev. We would like to thank Andreas Jaeger and Monty Taylor for all the work they have done over the years. This also serves as a reminder that we can always use more help. If you are interested in getting involved do not hesitate to reach out.Find us at:

• On the mailing list: service-discuss@lists.opendev.org
• On IRC: #opendev on Freenode
• Code review

Looking forward to 2021…

Thank you to everyone in the OpenInfra community for all of the work that went into 2020 and is continuing in 2021 to contribute and support the community! Check out the full 2020 Annual Report, and join the OpenInfra community to build the next decade of OpenInfra.

The post 2020 Annual Report: OpenInfra Collaboration appeared first on Superuser.

In the previous blog post we created the ARM OpenStack Kolla container images and we can now proceed with deploying OpenStack. The host is a Lenovo server with an Ampere Computing eMAG 32 cores Armv8 64-bit CPU, running Ubuntu Server 20.04. For simplicity, this will be an “All-in-One” deployment, where all OpenStack components run on the same host, but it can be easily adapted to a multi-node setup.

Let’s start with installing the host package dependencies, in case those are not already there, including Docker.

sudo apt update
sudo apt install -y qemu-kvm docker-ce
sudo apt install -y python3-dev libffi-dev gcc libssl-dev python3-venv
sudo apt install -y nfs-kernel-server

sudo usermod -aG docker $USER
newgrp docker

We can now create a local directory with a Python virtual environment and all the kolla-ansible components:

mkdir kolla
cd kolla

python3 -m venv venv
source venv/bin/activate

pip install -U pip
pip install wheel
pip install 'ansible<2.10'
pip install 'kolla-ansible>=11,<12'

The kolla-ansible configuration is stored in /etc/kolla:

sudo mkdir -p /etc/kolla/config
sudo cp -r venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla
sudo chown -R $USER:$USER /etc/kolla
cp venv/share/kolla-ansible/ansible/inventory/* .

Let’s check if everything is ok (nothing gets deployed yet):

ansible -i all-in-one all -m ping

kolla-genpwd is used to generate random passwords for every service, stored in /etc/kolla/passwords.yml, quite useful:

kolla-genpwd

Log in to the remote Docker registry, using the registry name and credentials created in the previous post:

ACR_NAME=# Value from ACR creation
SP_APP_ID_PULL_ONLY=# Value from ACR SP creation
SP_PASSWD_PULL_ONLY=# Value from ACR SP creation
REGISTRY=$ACR_NAME.azurecr.io

docker login $REGISTRY --username $SP_APP_ID --password $SP_PASSWD

Now, there are a few variables that we need to set, specific to the host environment. The external interface is what is used for tenant traffic.

VIP_ADDR=# An unallocated IP address in your management network
MGMT_IFACE=# Your management interface
EXT_IFACE=# Your external interface
# This must match the container images tag
OPENSTACK_TAG=11.0.0

Time to write the main configuration file in /etc/kolla/globals.yml:

sudo tee -a /etc/kolla/globals.yml << EOT
kolla_base_distro: "ubuntu"
openstack_tag: "$OPENSTACK_TAG"
kolla_internal_vip_address: "$VIP_ADDR"
network_interface: "$MGMT_IFACE"
neutron_external_interface: "$EXT_IFACE"
enable_cinder: "yes"
enable_cinder_backend_nfs: "yes"
enable_barbican: "yes"
enable_neutron_provider_networks: "yes"
docker_registry: "$REGISTRY"
docker_registry_username: "$SP_APP_ID_PULL_ONLY"
EOT

The registry password goes in /etc/kolla/passwords.yml:

sed -i "s/^docker_registry_password: .*\$/docker_registry_password: $SP_PASSWD_PULL_ONLY/g" /etc/kolla/passwords.yml

Cinder, the OpenStack block storage component, supports a lot of backends. The easiest way to get started is by using NFS, but LVM would be a great choice as well if you have unused disks.

# Cinder NFS setup
CINDER_NFS_HOST=# Your local IP
# Replace with your local network CIDR if you plan to add more nodes
CINDER_NFS_ACCESS=$CINDER_NFS_HOST
sudo mkdir /kolla_nfs
echo "/kolla_nfs $CINDER_NFS_ACCESS(rw,sync,no_root_squash)" | sudo tee -a /etc/exports
echo "$CINDER_NFS_HOST:/kolla_nfs" | sudo tee -a /etc/kolla/config/nfs_shares
sudo systemctl restart nfs-kernel-server

The following settings are mostly needed for Octavia, during the next blog post in this series:

# Increase the PCIe ports to avoid this error when creating Octavia pool members:
# libvirt.libvirtError: internal error: No more available PCI slots
sudo mkdir /etc/kolla/config/nova
sudo tee /etc/kolla/config/nova/nova-compute.conf << EOT
[DEFAULT]
resume_guests_state_on_host_boot = true

[libvirt]
num_pcie_ports=28
EOT

# This is needed for Octavia
sudo mkdir /etc/kolla/config/neutron
sudo tee /etc/kolla/config/neutron/ml2_conf.ini << EOT
[ml2_type_vlan]
network_vlan_ranges = physnet1:100:200
EOT

Time to do some final checks, bootstrap the host and deploy OpenStack! The Deployment will take some time, this is a good moment for a coffee.

kolla-ansible -i ./all-in-one prechecks
kolla-ansible -i ./all-in-one bootstrap-servers
kolla-ansible -i ./all-in-one deploy

Congratulations, you have an ARM OpenStack cloud! Now we can get the CLI tools to access it:

pip3 install python-openstackclient python-barbicanclient python-heatclient python-octaviaclient
kolla-ansible post-deploy
# Load the vars to access the OpenStack environment
. /etc/kolla/admin-openrc.sh

The next steps are optional, but highly recommended in order to get the basic functionalities, including basic networking, standard flavors and a basic Linux image (Cirros):

# Set you external netwrork CIDR, range and gateway, matching your environment, e.g.:
export EXT_NET_CIDR='10.0.2.0/24'
export EXT_NET_RANGE='start=10.0.2.150,end=10.0.2.199'
export EXT_NET_GATEWAY='10.0.2.1'
./venv/share/kolla-ansible/init-runonce

All done! We can now create a basic VM from the command line:

# Create a demo VM
openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1

You can also head to “http://<VIP_ADDR>” and access Horizon, OpenStack’s web ui. The username is admin and the password is in /etc/kolla/passwords.yml:

grep keystone_admin_password /etc/kolla/passwords.yml

In the next post we will add to the deployment Octavia, the load balancer as a service (LBaaS) component, Enjoy your ARM OpenStack cloud in the meantime!

P.S.: In case you would like to delete your whole deployment and start over:

#kolla-ansible -i ./all-in-one destroy --yes-i-really-really-mean-it

The post OpenStack on ARM64 – Deployment appeared first on Cloudbase Solutions.

The post Bare Basics of DNS Service with OpenStack Designate appeared first on VEXXHOST.

Codership is pleased to announce the launch of our training video library. The aims are quite clear as we want you to be helped in learning, installing, configuring and running Galera Cluster and related software in a successful fashion. Naturally, this also serves as a good teaser for our training courses which we are going to deliver online. This is part of an upcoming series of blogs covering the video contents, and why you might want to watch them.

Today we are focusing on Installing Galera Cluster with MySQL. Naturally it is worth clicking that link because you will see how well organised it is, including the fact that there are training exercises, slides in case you choose to take this offline, example configuration, as well as a sample company database (note that it is about 64MB in size).

It starts with an introduction of Codership. There is an introduction to Galera Cluster, focusing on nodes, databases, replication, load balancing and High Availability, around the master-slave (primary-secondary, leader-follower) model. It then covers important concepts and wins around Galera Cluster: virtually synchronous replication, multi-master, conflict detection & resolution on commit (when there are conflicting transactions), easy maintenance (automatic node provisioning, node isolation, rolling upgrades).

The basics that you need to be on Linux (CentOS 7 in this video) or FreeBSD, the fact that you want to have odd-numbers of nodes (this training course focuses on 3 nodes), and to ensure that all nodes have equal hardware configuration. The rsync software as well as the firewall software (firewalld) is also something you need. We talk about the Primary Component, and how you may get an Incremental State Transfer (IST) or a State Snapshot Transfer (SST) (it also is a good idea to read: node provisioning). We  then move on to show you the terminal so you can follow along, and how to get the repository URLs for a Galera Cluster repository.

This is based on MySQL 5.7 + Galera 3 which you get from the Codership repositories, so remember that when you execute system start mysqld you will need to grep 'temporary password' /var/log/mysqld.log to get the initial password. Good practice of running mysql_secure_installation is also something Russell covers; and the fact that you need to do this on each of the three nodes.

Configuring Galera Cluster is not just now in your /etc/my.cnf but /etc/my.cnf.d/server.cnf. I can highly recommend you follow the configuration, including turning on the error log. It is worth spending time learning how to configure Galera Cluster, including understanding the basic wsrep_provider_options, the wsrep_slave_threads, and what the wsrep_sst_method should be (it does not go in deep, here you use rsync).

We mention about the ports that you need to open up as well, and to ensure that SELinux understands this. The ports in question are: 3306, 4567 (TCP and UDP), 4444, 4568 (when not specified, just TCP is required). You will also need to set these inbound rules on AWS — this is not pictured in the video, so remember to turn this on in the AWS firewall. Then configuring SELinux (it is good, do not disable it) and the firewall via firewall-cmd is shown.

The video goes on to talk about bootstrapping the cluster, e.g. via mysqld_bootstrap. Within the terminal, and also shows you what it is like in /var/lib/mysql/grastate.dat and the important option of safe_to_bootstrap.

Overall, once you are done with the video, you will be able to have Galera Cluster running. Next up, how to do this with MariaDB Server.

Spotlight on:

Open Infrastructure Summit schedule is live!

Despite being a very different year than most, the OpenInfra community, which has over 110,000 community members, made it a productive and successful year. 2020 was the start of the next decade of OpenInfra tackling challenges including hardware diversification, deployment diversification, government regulations like data sovereignty, and the availability of more open source than ever.

Each community member, working group, SIG, and contributor was instrumental in continuing to support the OpenInfra Foundation’s mission: building open source communities that write software that runs in production. The 2020 Annual Report was published today highlighting the achievements across the community and the goals for the year ahead.

Let’s break down some of the highlights of last year:

• The OpenStack Foundation has taken the next step in its ongoing evolution by becoming the OpenInfra Foundation to support open infrastructure for a market estimated at $20 billion USD.
• Over 60 global organizations joined in establishing the OpenInfra Foundation to build the next decade of infrastructure for AI, 5G and edge computing, including Platinum Members Ant Group, AT&T, Ericsson, FiberHome, Huawei, Red Hat, Wind River, Tencent, and Facebook.
• OpenStack, one of the top three most active open source projects with 15 million cores in production, marked its 10th anniversary in July 2020
• The OpenInfra Foundation shared its support for the Magma project, announced OpenInfra Labs as its newest pilot project and confirmed StarlingX as a top-level Open Infrastructure project.
• The OpenInfra community became more accessible internationally through the first virtual OpenInfra Summit that gathered 10,000 attendees from 120+ different countries to discuss more than 30 open source projects

The OpenInfra Foundation would like to extend a huge thanks to the global community for all of the work that went into 2020 and is continuing in 2021 to help people build and operate OpenInfra. Check out the full OpenInfra Foundation 2020 Annual Report and join us to build the next decade of OpenInfra!

OpenInfra Foundation news

• This week, the 2021 OpenInfra Board of Directors met for the first time, onboarding new members and thanking the members who served last year. Allison Randal was elected Board Chair, and you can learn here about the new board as they start the next decade of OpenInfra. 
• The OpenInfra Foundation is helping the Magma community organize its Magma Developers Conference on February 3. The virtual event is free to attend and participants can expect to hear about the Magma development roadmap, including the upcoming 1.4 release, as well as use cases in production.

Airship: Elevate your infrastructure

• Airship 2.0 is Kubernetes certified! 
  • Airship 2.0 has been designated as a Certified Kubernetes Distribution through the Cloud Native Computing Foundation’s Software Conformance Program, guaranteeing that Airship provides a consistent installation of Kubernetes, supports the latest Kubernetes versions, and provides a portable cloud-native environment with other Certified Platforms.
• Airship Container as a Service
  • Two operators will manage the lifecycle of virtual machines and their relationship to the cluster: vNode-Operator (ViNO) and the Support Infra Provider (SIP). These operators are new projects to the Airship ecosystem and were migrated to OpenDev earlier this month. Read more about ViNO and SIP on the blog.
• General Availability of Airship 2.0 is on track for release in Q1.

Kata Containers: The speed of containers, the security of VMs

• Kata Containers was represented at the Cloud Native Infrastructures Meetup in Beijing.
  • Learn about why China Mobile Cloud selected Kata Containers as its security containers, and some performance tuning exploration and optimization ideas for Kata on Kata blogs.
• Are you a Kata Containers user? Please take a few moments to fill out the User Survey to provide feedback and information around your deployment. All information is confidential to the OpenInfra Foundation unless you designate that it can be public.

OpenStack: Open source software for creating private and public clouds

• The Wallaby cycle community goals are underway! Our focuses this cycle are on migration from oslo.rootwrap to oslo.privsep for a faster more secure command privilege model and on migrating RBAC policy formats from JSON to YAML helping to reduce policy management issues for OpenStack deployers.
• Three services have been removed from the upcoming Wallaby release meaning their last release was Victoria. The three services that have been retired are Karbor, Qinling, and Searchlight. We thank those that contributed to these retired projects for all their hard work and dedication to the OpenStack community!   
• Placement has been moved back under Nova’s governance. The Placement project had tried to use the new distributed project leadership model that the TC put in place last release, but due to the small team size and overlap with those involved with Nova, it was concluded that it would be better to move Placement back under Nova’s governance.

StarlingX: A fully featured cloud for the distributed edge

• The community is currently working on the 5.0 release with focus on security related features like secret management and image signature validation. The release is planned to come out by the summer.
• The next round of the Technical Steering Committee election will also happen during the second quarter of the year. If you are interested in running for a seat start contributing now to be eligible to participate!

Zuul: Stop merging broken code

• The release of Nodepool 3.14.0 brings improvements in Kubernetes, OpenShift, and containers.
• James E. Blair is reconfirmed as the Zuul Project Lead for 2021, continuing to serve as a central point of contact for the project, and to guide the consensus-driven decision making process on which the community relies.
• The Zuul Maintainers reached agreement on a new specification for a solution to locally reproduce job builds.
• Planning is currently underway for a possible build log and artifact storage access proxy complimenting Zuul, feel free to join the discussion.

Upcoming Virtual OpenInfra and Community Events

• February 3, Magma Dev Conference

For more information about OpenInfra community events, please contact community@openinfra.dev

Questions / feedback / contribute

This newsletter is written and edited by the OSF staff to highlight open infrastructure communities. We want to hear from you! If you have feedback, news or stories that you want to share, reach us through community@openinfra.dev . To receive the newsletter, sign up here.

The post Inside Open Infrastructure: The latest from the OpenInfra Foundation appeared first on Superuser.

The Magma open source community is joining the world of virtual conferences next week with their Magma Developer Conference on February 3. Hundreds of developers, system integrators, engineers, and end users are expected to attend and discuss the technical roadmap as well as how Magma is already connecting people in production.

Magma is an open-source software platform that delivers an open, flexible and extendable mobile network core solution. Magma’s mission is to connect the world to a faster network by enabling service providers to build cost-effective and extensible carrier-grade networks. It is designed to be access network agnostic (4G, 5G or WiFi), flexibly supporting a radio access network with minimal development and deployment effort. The 1.4 release is scheduled for later this week, so at the Developer Conference, you can learn more about the newest features and what you can expect from upcoming releases on the project roadmap.

To highlight the potential of running Magma in production, case studies will be presented by Baicells and MuralNet—whose user story may sound familiar if you attended the OpenInfra Summit keynotes in October 2020. I’ve also heard that the 5G fixed wireless access demo will be something worth tuning in for. Below is a sneak peek of what you can expect.

To participate, you can register for free, and I encourage you to join the Magma community Slack and the #magma-dev-conf channel in particular so you can chat with other attendees. The conversation will also occur on Twitter around #MagmaDevConf.

If you want to learn more about the open source community, how you can get involved, and if it is a good network solution for your organization, join us!

The post Magma Developers Conference Agenda Highlights Community Development Momentum and Production Use Cases appeared first on Superuser.

Auto-scaling is a must ability when it comes to the functionality of a cloud. OpenStack already provides multiple methods to auto-scale virtual machines with Heat, Telemetry stack. However, Telemetry the main stack, which involves in the evaluation process whether an auto-scale is needed, either lacks the contribution or malfunctions. In this article, we will introduce a new tool Faythe, an open-sourced software that we built ourselves, works as a bridge between the cloud platform (OpenStack) and monitoring system (Prometheus), to solve the auto-scale problem in Viettel, which is the largest telco in Vietnam.

Auto-scaling with the traditional approach with Heat & Telemetry

The fundamental idea

Auto-scaling is the combination of three steps:

• Metering
• Alarming
• Scaling

Metering service collects data about service/instance/host load. Alarm service evaluates the data to find out whether you need to add scale-out/scale-in, which triggers the scale-out/scale-in action on Scaling service. The idea is quite simple.

Auto-scaling with the traditional approach with Heat & Telemetry

This approach consists of two main components which are packaged and distributed together with OpenStack infrastructure.

• Orchestration: The core component providing auto-scaling is Orchestration (heat). Use Orchestration to define rules using human-readable YAML templates. These rules are applied to evaluate system load based on Telemetry data to find out whether you need to add more instances into the stack. When the load drops, Orchestration can automatically remove the unused instances again.
• Telemetry: Telemetry does OpenStack environment performance monitoring, collecting data on CPU, storage, and memory utilization for instances and physical hosts. Telemetry also involves three smaller services:

• Ceilometer – Monitoring Service: a data collection service that provides the ability to normalize and transform data across all OpenStack core components.
• Gnocchi – Storage Service: provides a time-series resource indexing, metrics storage.
• Aodh – Alarming Service: triggers actions based on defined rules against sample or event data collected by Ceilometer and stored in Gnocchi.

The Telemetry issues

• Telemetry projects (in the Rocky release, at the time we took a survey) have both lack of contribution from the opensource community, which is one of the factors defining open-sourced project longevity.

• As of the Queens release, the Telemetry project had lost most developers. Development on Aodh and Panko stopped.
• Gnocchi was moved out of OpenStack in June 2017. Gnocchi is unmaintained.

• The three projects are dependent on each other, however, which are developed separately. Leading to a small change in one project can make the whole system collapses. Moreover, since Gnocchi is no longer a part of OpenStack, unsynchronized development issues between these are unavoidable.
• In our experiences, RabbitMQ was under heavy load due to Ceilometer workload. If Ceilometer gets stuck, its queues overflow.
• Telemetry only collects information about virtual machines operating systems and OpenStack physical hosts. It is impossible for Telemetry to monitor services and applications running on virtual machines. Additionally, changing and implementing more features for Telemetry are not simple tasks. That includes writing codes, compiling codes, packaging, testing, and deploying to real system.
• Telemetry doesn’t fit our ecosystem. We chose Prometheus to monitor the whole infrastructure system as well as applications. Switching/Setup other systems like Telemetry alongside Prometheus is not ideal.

Auto-scaling with Faythe and Prometheus

How it works

Actually, this is not a completely new approach, it leverages Orchestration to do scaling action with new Alarming Service – Faythe and Monitoring Service – Prometheus. As mentioned before, auto-scaling is the combination of three steps: Metering, Monitoring, and Scaling. OpenStack Orchestration is already providing superb scaling service, Heat. Instead of using OpenStack Telemetry for Metering and Alarming, the new approach introduces Faythe, an open-sourced software that we built ourselves, working as a bridge between Cloud platform (OpenStack) to the Monitoring system (Prometheus),  and Prometheus.

Why Faythe?

Faythe is one of our proudly created projects, which is designed, developed and implemented while we are working at Viettlet Network. Faythe emerges to fill in the gap between Prometheus and OpenStack.

• Faythe supports Prometheus to collect metrics and PromQL to evaluate and decide whether it is necessary for an alarm.
• PromQL is a very nice query language.
• A wide range of exporters: this is the most valuable point of Prometheus. The open-source community is creating more and more new exporters, which means there are fresh metrics that can be collected by Prometheus.
• Flexible: besides the system factor like CPU or Memory usage, users can evaluate any metrics that can be collected by any of the exporters, for example, JVM metrics, HTTP requests related metrics. For example, you can define a scaling rule depend on the number of Django HTTP requests.

• Faythe can be easily configured.
• Faythe has its own cluster mechanism.

Faythe provides not only a user-friendly UI but also complete APIs.

For more details, you can take a look at Faythe repository.

The post Auto-scaling in Viettel Networks Without OpenStack Telemetry appeared first on Superuser.

This is the beginning of a short series detailing how to deploy OpenStack on ARM64, using Docker containers with Kolla and Kolla-ansible.

The objective of this first post is to create the ARM64 container images and push them to a remote registry in order to be used later on, when deploying our OpenStack cloud. We are going to use Azure Container Registry to store the images, but any other OCI compliant registry will do.

Create a container registry

Let’s start by creating the container registry and related access credentials. This can be done anywhere, e.g. from a laptop, doesn’t need to be ARM. All we need is to have the Azure CLI installed.

az login
# If you have more than one Azure subscription, choose one:
az account list --output table
az account set --subscription "Your subscription"

Next, let’s create a resource group and a container registry with a unique name. Choose also an Azure region based on your location.

RG=kolla
ACR_NAME=your_registry_name_here
LOCATION=eastus

az group create --name $RG --location $LOCATION
az acr create --resource-group $RG --name $ACR_NAME --sku Basic

We’re now creating two sets of credentials, one with push and pull access to be used when creating the images and one with pull only access to be used later on during the OpenStack deployment.

ACR_REGISTRY_ID=$(az acr show --name $ACR_NAME --query id --output tsv)
SERVICE_PRINCIPAL_NAME=acr-kolla-sp-push
SP_PASSWD=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpush --query password --output tsv)
SP_APP_ID=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)
echo "SP_APP_ID=$SP_APP_ID"
echo "SP_PASSWD=$SP_PASSWD"

SERVICE_PRINCIPAL_NAME=acr-kolla-sp-pull
SP_PASSWD_PULL_ONLY=$(az ad sp create-for-rbac --name http://$SERVICE_PRINCIPAL_NAME --scopes $ACR_REGISTRY_ID --role acrpull --query password --output tsv)
SP_APP_ID_PULL_ONLY=$(az ad sp show --id http://$SERVICE_PRINCIPAL_NAME --query appId --output tsv)
echo "SP_APP_ID_PULL_ONLY=$SP_APP_ID_PULL_ONLY"
echo "SP_PASSWD_PULL_ONLY=$SP_PASSWD_PULL_ONLY"

Create and push the OpenStack Kolla container images

It’s now time to switch to an ARM server where the Kolla container images will be built. We are going to use a Lenovo server with an eMAG 32 cores Armv8 64-bit CPU provided by Ampere Computing. The host operating system is Ubuntu 20.04, but the following instructions can be easily adapted to other Linux distros.

Let’s start with installing the dependencies and add your current user to the docker group (or create a separate user).

sudo apt update
sudo apt install -y docker-ce python3-venv git
sudo usermod -aG docker $USER
newgrp docker

Let’s get Docker to login into the remote registry that we just created. Set ACR_NAME, SP_APP_ID and SP_PASSWD as obtained in the previous steps.

REGISTRY=$ACR_NAME.azurecr.io
docker login $REGISTRY --username $SP_APP_ID --password $SP_PASSWD

Now we can install Kolla in a Python virtual environment and get ready to start building our container images. The OpenStack version is the recently released Victoria but a previous version can be used if needed (e.g. Ussuri).

mkdir kolla-build
cd kolla-build
python3 -m venv venv
source venv/bin/activate
pip install wheel
# Install Kolla, Victoria version
pip install "kolla>=11,<12"

Edit: the following step can be skipped on Victoria since it defaults to Ubuntu 20.04 where pmdk-tools is available. Additionally, thanks to a recent patch, it can be skipped on Ussuri and Train.

The pmdk-tools Ubuntu package is not available on ARM, so we need to remove it from the nova-compute docker image build. This is done by creating a “template override” that we are going to pass to the build process.

tee template-overrides.j2 << EOT
{% extends parent_template %}

# nova-compute
{% set nova_compute_packages_remove = ['pmdk-tools'] %}
EOT

We can now build the container images and push them to the registry. This will take a while since it’s building and pushing container images for all OpenStack projects and services. Alternatively, it is possible to reduce the number of containers to a subset by creating a profile in kolla-build.conf as explained here.

kolla-build -b ubuntu --registry $REGISTRY --push
# If you created a template override run:
# kolla-build -b ubuntu --registry $REGISTRY --template-override template-overrides.j2 --push

We are finally ready for our OpenStack AMR64 deployment with Kolla-ansible in the next post!

The post OpenStack on ARM64 – Kolla container images appeared first on Cloudbase Solutions.

VEXXHOST is a Canadian cloud computing provider with an impressive global presence and record. As a former Superuser Awards winner and a major contributor to the open source community, VEXXHOST’s growth trajectory is of keen interest to any member of the community. 

From small businesses to governments, VEXXHOST’s operations are spread over 150 countries. The company started as a web hosting provider in 2006 and later transitioned into cloud solutions. They adopted OpenStack for their cloud operations in 2011, coinciding with the platform’s second release, Bexar. VEXXHOST now offers OpenStack-based public cloud, private cloud, consulting, and other enterprise-grade cloud solutions. 

VEXXHOST recently announced a massive revamp in its public cloud offerings during its keynote address at the Open Infrastructure Summit 2020. There are some new and exciting developments, including a brand new Data Center in Amsterdam. The company says that the revamp is part of their new growth strategy, powered by their association with various open source foundations and communities. They are a reputed member of the Open Infrastructure Foundation, Linux Foundation, CNCF, and the Ceph Foundation, to name a few. 

With this revamp, VEXXHOST joins the range of community-driven companies evolving and growing strong even during these testing times (i.e., pandemic time). Without further delay, here’s what they have included in the cloud revamp. 

Amsterdam – New Region

The Amsterdam data center is VEXXHOST’s third region to date, the other two being Montreal and Santa Clara. VEXXHOST also manages several private cloud operations for different enterprises and organizations in various other parts of the world, in association with local data centers. The launch of the new region not only puts them physically in a different continent but also cements their global footprint.

The state-of-the-art data center used in Amsterdam provides unparalleled service in terms of connectivity, efficiency, server protection mechanisms,  security, and reliability. The facility is built to Uptime Institute Tier 3 standards, and has  PCI/SOC, and AMS-IX, and multiple ISO certifications. The Amsterdam data center also boasts of a multi-tier security system, 24×7 on-site support, high energy efficiency with green standards, advanced global connectivity via being linked to AMS-IX, among other features. 

Modified Public Cloud Pricing 

Another change that is part of the revamp is VEXXHOST’s new pricing strategy. The modified pricing is available to all new users as well as existing customers who make the switch to their latest solutions. 

According to VEXXHOST’s new introductory and limited-time pricing model, the hourly rate for a standard 2 core, 8 GB offering is just $0.055, compared to the market rate, which stands at $0.086. The difference between the current market rate and the new VEXXHOST rate is 36%. Higher core/memory offerings such as 4 core – 16 GB, 8 core – 32 GB, and 16 core – GB, will also carry proportionate price reduction.

This new pricing strategy couldn’t have come at a better time. Many businesses are facing a lot of challenges due to the pandemic situation and there is also a need for companies to find remote and cloud solutions to adapt to the rapid changes around. Getting users enterprise-grade cloud solutions at great pricing is beneficial not only for the parties involved but also for the overall community.

Montreal Region – 2nd Gen AMD EPYC-powered Servers 

Next in VEXXHOST’s revamp is the addition of new servers equipped with 2nd Gen AMD EPYC processors for their Montreal region. Driven by AMD Infinity Architecture, these processors are the first x86 processor of their kind for servers based on 7nm process technology. Also boasting of a hybrid, multi-die architecture, they offer up to 64 high-performance cores per SOC and are equipped with AMD Infinity Guard and PCIe® Gen 4 I/O security features. 

The new AMD processors bring about significant change to VEXXHOST’s servers in better performance and secure computing, improving workload acceleration, data protection, and overall infrastructure to the users.

Montreal Region – Upgraded Storage from SSD to NVMe

VEXXHOST has upgraded storage from SSD to NVMe in their Montreal region, offering its public cloud users the fastest hard drive in the market. A major benefit available with this storage type is faster parallel read and write capabilities.  

NVMe drives can work at speeds of more than 2,000MB/s compared to the typical SATA III SSD running under 600MB/s. This is possible because of the interaction with flash memory, made through a PCIe interface. This interface is bi-directional and runs at a stellar speed. NVMe storage is also better in power-efficiency, reducing consumption by significant percentages in standby mode. NVMe is also a scalable alternative going beyond the four lanes found in most conventional PCIe SSDs. 

Moving forward, VEXXHOST has a lot planned in terms of cloud offerings to customers. If you would like to know more about the company and its extensive cloud solutions, check out VEXXHOST’s website.

The post New in OpenInfra: VEXXHOST’s Public Cloud Revamp Includes a Lot More Than the New Data Center in Amsterdam appeared first on Superuser.

What changed with CentOS?

What’s the current situation in RDO?

What’s our plan for RDO roadmap?

• RDO Wallaby (ETA is end of April 2021) will be built, tested and released only on CentOS 8 Stream.
• RDO CloudSIG repos for Victoria and Ussuri will be updated and tested for both CentOS Stream and CentOS Linux 8 until end of 2021 and then continue on CentOS Stream.
• We will create and test new RDO CloudSIG repos for Victoria and Ussuri on CentOS Stream 8.
• The RDO Trunk repositories (aka DLRN repos) will be built and tested using CentOS 8 Stream buildroot for all releases currently using CentOS Linux 8 (since Train on)

How do we plan to implement these changes?

• We will keep building packages just once. We will move buildroots for both DLRN and CloudSIG to use CentOS Stream 8 in the near future.
• For Ussuri and Victoria CloudSIG repos, while we are supporting both C8 and C8S, we will be utilizing separated CBS Tags. This will allow us to have separated repositories, promotion jobs and package versions for each OS.
• In order to reduce the impact of potential issues and discover issues related to C8S as soon as possible, we will put more focus on periodic jobs on C8S.
• At a later stage, we will move the CI jobs used to gate changes in distgits to use C8S instead of C8 for all RDO releases where we use CentOS 8.
• The CentOS/RHEL team has made public their interest in applying Continuous Delivery approach to CentOS Stream to provide a stable CentOS Stream using gating integration jobs. Our intent is to collaborate with the CentOS team on any initiatives that will help to validate RDO as early in the delivery pipeline as possible and reduce the impact on potential issues.

What’s next?

Codership is pleased to announce a new Generally Available (GA) release of the multi-master Galera Cluster for MySQL 5.6, 5.7 and 8.0, consisting of MySQL-wsrep 5.6.50 (release notes, download), 5.7.32 (release notes, download) and 8.0.22 (release notes, download) with Galera replication library 3.32 (release notes, download) implementing wsrep API version 25, and Galera replication library 4.7 (release notes, download) implementing wsrep API version 26 for 8.0. This release incorporates all changes to MySQL 5.6.50, 5.7.32 and 8.0.22 respectively, adding a synchronous option for your MySQL High Availability solutions.

This release is notable because there have been improvements with CRC32 hardware support compatibility on ARM based architectures for the Galera replication library 3.32 and 4.7. GCache locking has also been improved.

From a lower level, the build system has changed from using SCons to CMake, so while SCons scripts are still included they will not be actively maintained any longer.

In MySQL 5.6.50, we improved the XtraBackup v2 SST script for FreeBSD users.

In MySQL 5.7.32, we added a new variable wsrep_mode. The first application of the variable is to allow ignoring native replication filter rules if configured with replicate-do-db. We also improved LOAD DATA splitting to generate Xid events for binary log events with intermediate commits.

In MySQL 8.0.22, we now have a new SST method based on the CLONE plugin. Similar improvements around wsrep_mode exist too. Improvements around foreign key handling are also present in this release: load data with foreign keys had intermittent failures that have been fixed, as well as BF-BF conflicts between OPTIMIZE/REPAIR/ALTER TABLE and DML in presence of foreign key constraints were suppressed. We would also like to note that Percona XtraBackup version 8.0.22-15.0 and greater is required to perform XtraBackup based SSTs.

For now, MySQL 5.6.50 and 5.7.32 are the last official builds for CentOS 6 and Red Hat Enterprise Linux 6. It is also important to remember that MySQL 5.6 is nearing End Of Life (EOL) so we do recommend that you upgrade to MySQL-wsrep 5.7.

Cyborg is an accelerator resource (GPU, vGPU, FPGA, NVMe SSD, QAT, DPDK, SmartNIC etc.) management project, and it uses a micro-service architecture to support distributing deployment, which contains cyborg-api, cyborg-conductor and cyborg-agent services. Cyborg-agent collects accelerator resources information and reports it to the cyborg-conductor. Cyborg-conductor stores accelerator resources information to the database, and reports it to Placement resource management service. Placement stores the resources information and provides available resources for Nova scheduling when the server is creating. Cyborg-api service provides interfaces to request accelerator resources information. Diagram 0 is the architecture of Cyborg.

With Cyborg, we can boot server with accelerators by interacting with Nova and Placement, and support batch boot servers with scheduling accelerators (this is an enhancement in Inspur InCloud OpenStack Enterprise edition that we plan to contribute to the Nova and Cyborg community.). Then users can use these devices in server to program with FPGA, progress images via GPU and so on. We can also bind and unbind accelerators to exist server by hot-plug and non-hot plug devices, which guarantees the convenient usage of accelerators. Diagram 1 is the interaction flow between Nova and Cyborg when booting a server.

On the side of enhancing server operations with accelerators, we have supported most operations for servers, such as creation and deletion, reboots (soft and hard), pause and unpause, stop and start, take a snapshot, backup, rescue and unrescue, rebuild, evacuate. And there are other operations going on, such as shelve and unshelve, suspend and resume that are close to merge. We plan to support migration (live and cold) and resize soon. With these enhancements, operators can use accelerators more flexibly. Diagram 2 is the sequence flow about booting a server with vGPU devices.

And in Inspur InCloud OpenStack Enterprise edition we have made enhancements to some features, such as batch boot servers, bind and unbind accelerator devices based on hot-plug which are mentioned above. Cyborg was used to managing virtual GPU and the utilization rate was improved by 80%. Data synchronization strategy makes Cyborg and Placement data transportation increased by 30% on efficiency.

From diagram 3, the main components of N3000 include the Intel Arria® 10 FPGA, Dual Intel Ethernet Converged Network Adapter XL710, Intel MAX® 10 FPGA Baseboard Management Controller (BMC), 9 GB DDR4, 144 Mb QDR-IV. It can support High-speed network with 10Gbps/25Gbps interface and High-speed host interface with PCIe* Gen 3×16.

Intel® FPGA Programmable Acceleration Card N3000 (Intel FPGA PAC N3000) is a highly customized platform that enables high-throughput, lower latency, and high-bandwidth applications. It allows the optimization of data plane performance to achieve lower costs while maintaining a high degree of flexibility. End-to-end industry-standard and open-source tool support allow users to quickly adapt to evolving workloads and industry standards. Intel is accelerating 5G and network functions virtualization (NFV) adoption for ecosystem partners, such as telecommunications equipment manufacturers (TEMs), virtual network functions (VNF) vendors, system integrators, and telcos, to bring scalable and high-performance solutions to market. Diagram 4 is the sequence flow for program in Cyborg.

For SmartNIC, we can program it with OVS image as NFVI function in OpenStack, e.g. N3000, Mellanox CX5 and BF2. Diagram 5 is the sequence for enabling accelerators for SmartNIC.

In Cyborg, we support several new features:

• Asynchronous program API.
• Program process API.
• Resource report dynamically.

Users can start a new program process request for N3000. It is an async API, and it is safe to detect intelligently whether there is a conflict in resource usage and decide on accepting or rejecting the request. We also support a friendly program process query API. Users can use it to check the stage of the process and what percentage is completed at any time. When the program is finished, the resource typed and quantified, then Cyborg can discover and report the change dynamically.

In the whole OpenStack, we have also made some new improvements.

• Cyborg supports new drivers for new SRIOV NIC. 
• Neutron supports a new option to create a port with accelerator device profile.
• Improvements on Nova, to retrieve the new VF NIC and pass-through to server as an interface.

After these improvements, the OpenStack can support SmartNIC more flexibly and conveniently, and the improvements include SmartNIC cards (as mentioned above), even other SRIOV card. 

All the new features and improvements on OpenStack will be upstream.

On November 25, Inspur InCloud OpenStack (ICOS) completed the 1000-nodes “The world’s largest single-cluster practice of Cloud, Bigdata and AI”, a practice for convergence of Cloud, Bigdata and Artificial Intelligence. It is the largest SPEC Cloud test and the first large-scale multi-dimensional fusion test in the industry. It has achieved a comprehensive breakthrough in scale, scene and performance, and completed the upgrade from 500 nodes to 1000 nodes, and achieved the sublimation from quantitative change to qualitative change. Inspur is working on the white paper for this large-scale test, which will be released soon. This will certainly be a reference for products in large-scale environments.

Get Involved

This article is a summary of the Open Infrastructure Summit session, Enhancement of new heterogeneous accelerators based on Cyborg.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

Thanks to the 2020 Open Infrastructure Summit sponsors for making the event possible:

Headline: Canonical (ubuntu), Huawei, VEXXHOST

Premier: Cisco, Tencent Cloud

Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post Enhancement of New Heterogeneous Accelerators Based on Cyborg appeared first on Superuser.

The post An Overview of Cloud File Sharing with OpenStack Manila appeared first on VEXXHOST.

For people who have attended the Open Infrastructure Summits in recent years, you have probably heard of the first winner of the Superuser Awards, CERN. CERN is the European Organization for Nuclear Research. It has a laboratory that is located at the border between France and Switzerland, and the main mission of CERN is to uncover what the universe is made of and how it works. 

For a few years, all physical servers deployed in the CERN IT data centers have been provisioned as part of the OpenStack private cloud offering, leveraging Nova and Ironic. What started with a few test nodes has now grown to more than 5,000 physical machines, with the aim to move to around 10,000 nodes.

CERN’s Nova and Ironic Deployment

CERN and CERN IT rely heavily on OpenStack, and we have deployed OpenStack in production since 2013 with around 8,500 compute nodes with 300k cores and 35k instances in around 80 cells. In addition, we deploy three regions mainly for scalability and to ease the rollout of the new features.

Issues and Solutions when Scaling Bare Metal with Nova and Ironic

When we scaled our bare metal deployment with Nova and Ironic to several thousand nodes, we encountered three main issues so far:

• Controller Crashes
• API Responsiveness
• Resource Discovery

Scaling Issue 1: Controller Crashes

The CERN Cloud Infrastructure Team uses the iscsi deploy interface for Ironic. It means that a node upon deployment is exporting an iscsi device to the controller, and the controller is then dumping the image downloaded from Glance onto it. (Note that this deploy interface will be deprecated in the future, so direct deploy interface should be used instead.) 

For deployment, since the images are tunneled through the controller, many parallel deployments will drive the conductor into out-of-memory (OOM) situations. In consequence, the controller would crash and leave the nodes in an error state. 

Solutions

To address this issue we horizontally scaled the controllers and introduced the “wing” controllers to have more controllers to handle the requests. Another solution would be to use a scalable deploy interface, such as direct or ansible which makes the node download the image directly.

Scaling Issue 2: API Responsiveness

When scaling up the infrastructure, we noticed that all requests involving the database were slow. Looking at the request logs, we realized that the inspector as the other component running on the controller gets a list of all nodes to clean up its database every 60 seconds. In addition, we had disabled pagination when we reached 1,000 nodes, which means that every request that went to the API assembled all the nodes in one giant request and tried to give that back to the requester. 

Solutions

To solve this issue, we re-enabled pagination and changed the sync interval from 60 seconds to one hour. You can see how this solution affected the response time from the graph below. For a more scalable solution, we introduced “Inspector Leader Election”, which we developed together with upstream and deployed now in production. Inspector Leader Election will be available in the Victoria release, and you can see more details about how it works here.

Scaling Issue 3: Resource Discovery

Another issue that we faced when increasing the number of physical nodes managed by Ironic is the time that the Resource Tracker (RT), which runs in each nova-compute, takes to report all the available resources to Placement. From the graph below, you can see the original OpenStack Nova and Ironic setup for the dedicated bare metal cell at CERN. 

There is a dedicated Nova cell for Ironic which is the standard way that we partition and manage the infrastructure. In the dedicated Nova cell, we only have the cell control plane, nova-conductor, RabbitMQ, and nova-compute. This single nova-compute is responsible for all the communication between Nova and Ironic using the Ironic API. It is possible to run several nova-computes in parallel based on a hash ring to manage the nodes, but when we were testing this functionality, several issues were observed. 

Since there is only one nova-compute that interacts with Ironic and the RT needs to report all the Ironic resources, the RT cycle takes a long time when the number of resources is increased in Ironic. In fact, it took more than three hours to complete our deployment with about 5,000 nodes. During the RT cycle, all the users’ actions are queued until all the resources are updated. This created a bad user experience with the new resources creation taking a few hours. 

Conductor Groups

In order to have failure domains in Ironic and to allow to split the infrastructure, a new feature was introduced in the Stein release of Ironic and Nova. It’s called “conductor groups”. A conductor group is an association between a set of physical nodes and a set of (one or more) Ironic conductors which manage these physical nodes. This association reduces the number of nodes a conductor is looking after, which is key in scaling the deployment. 

Conductor Groups Configuration

The conductor group an Ironic conductor is taking care of is configured in Ironic’s configuration file “ironic.conf” on each Ironic conductor:

[conductor]
conductor_group = MY_CONDUCTOR_GROUP

Next, each Ironic resource needs to be mapped to the conductor group that is selected, and this can be done in the Ironic API. 

openstack baremetal node set --conductor-group "MY_CONDUCTOR_GROUP" 
<node_uuid>

Finally, each group of nova-compute nodes needs to be configured to manage only a conductor group. This is done in Nova’s configuration file “nova.conf” for each “nova-compute” node:

[ironic]
partition_key = MY_CONDUCTOR_GROUP
peer_list = LIST_OF_HOSTNAMES

Now, there is one nova-compute per conductor group, and you can see how the deployment looks like in the graph below.

The Transition Steps

How did we deploy the conductor groups? The transition steps below are summarized from a CERN Tech Blog, Scaling Ironic with Conductor Groups. 

• Nova-compute and Ironic services stopped since the databases will be manually updated
• Updated the Ironic and Nova configuration files with the conductor groups’ names 
• Updated Ironic “nodes” tables to map conductor groups to nodes
• Updated Nova “compute_nodes” and “instances” tables to set the new nova-compute dedicated to the conductor groups
• Finally, start again all the services 

Impact on the Resource Tacker (RT) Cycle Time

In the graph below, you can see the number of Placement requests per conductor group. As a result, the RT cycle time now only takes about 15 minutes to complete instead of three hours previously since it’s now divided through the number of conductor groups.

Number of Resources Per Conductor Group

How did we decide the number of resources per conductor groups? The RT cycle time increases linearly with the number of resources. The compromise between management and RT cycle time is around 500 nodes per conductor group in our deployment. The deployment scales horizontally as we add conductor groups

Conclusion and Outlook

Scaling infrastructure is a constant challenge! By introducing conductor groups, the CERN Cloud Infrastructure Team was able to split this locking and reduce the effective lock time from three hours to 15 minutes (and even shorter in the “leading” group)

Although we have addressed various issues, some issues are still open and new ones will arise. Good monitoring is key to see and understand issues.

Get Involved

This article is a summary of the Open Infrastructure Summit session, Scaling Bare Metal Provisioning with Nova and Ironic at CERN, and the CERN Tech Blog, Scaling Ironic with Conductor Groups. 

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

The post Scaling Bare Metal Provisioning with Nova and Ironic at CERN: Challenges & Solutions appeared first on Superuser.

The post Adding Linux Images to OpenStack Using IaC  appeared first on VEXXHOST.

The post How Resource Pooling Works in an OpenStack Private Cloud   appeared first on VEXXHOST.

I want to be able to see the level of change between OpenStack releases. However, there are a relatively small number of changes with simply huge amounts of delta in them — they’re generally large refactors or the delete which happens when part of a repository is spun out into its own project.

I therefore wanted to explore what was a reasonable size for a change in OpenStack so that I could decide what maximum size to filter away as likely to be a refactor. After playing with a couple of approaches, including just randomly picking a number, it seems the logical way to decide is to simply plot a histogram of the various sizes, and then pick a reasonable place on the curve as the cutoff. Due to the large range of values (from zero lines of change to over a million!), I ended up deciding a logarithmic axis was the way to go.

For the projects listed in the OpenStack compute starter kit reference set, that produces the following histogram:I feel that filtering out commits over 10,000 lines of delta feels justified based on that graph. For reference, the raw histogram buckets are:

I wanted a quick summary of OpenStack git release tags for a talk I am working on, and it turned out to be way more complicated than I expected. I ended up having to compile a table, and then turn that into a code snippet. In case its useful to anyone else, here it is:

Or in python form for those so inclined:

RELEASE_TAGS = {
    'austin': {'all': '2010.1'},
    'bexar': {'all': '2011.1'},
    'cactus': {'all': '2011.2'},
    'diablo': {'all': '2011.3'},
    'essex': {'all': '2012.1.3'},
    'folsom': {'all': '2012.2.4'},
    'grizzly': {'all': '2013.1.5'},
    'havana': {'all': '2013.2.4'},
    'icehouse': {'all': '2014.1.5'},
    'juno': {'all': '2014.2.4'},
    'kilo': {'all': '2015.1.4'},
    'liberty': {
        'glance': '11.0.2',
        'keystone': '8.1.2',
        'neutron': '7.2.0',
        'nova': '12.0.6'
    },
    'mitaka': {
        'glance': '12.0.0',
        'keystone': '9.3.0',
        'neutron': '8.4.0',
        'nova': '13.1.4'
    },
    'newton': {
        'glance': '13.0.0',
        'keystone': '10.0.3',
        'neutron': '9.4.1',
        'nova': '14.1.0'
    },
    'ocata': {
        'glance': '14.0.1',
        'keystone': '11.0.4',
        'neutron': '10.0.7',
        'nova': '15.1.5'
    },
    'pike': {
        'glance': '15.0.2',
        'keystone': '12.0.3',
        'neutron': '11.0.8',
        'nova': '16.1.8'
    },
    'queens': {
        'glance': '16.0.1',
        'keystone': '13.0.4',
        'neutron': '12.1.1',
        'nova': '17.0.13'
    },
    'rocky': {
        'glance': '17.0.1',
        'keystone': '14.2.0',
        'neutron': '13.0.7',
        'nova': '18.3.0'
    },
    'stein': {
        'glance': '18.0.1',
        'keystone': '15.0.1',
        'neutron': '14.4.2',
        'nova': '19.3.2'
    },
    'train': {
        'glance': '19.0.4',
        'keystone': '16.0.1',
        'neutron': '15.3.0',
        'nova': '20.4.1'
    },
    'ussuri': {
        'glance': '20.0.1',
        'keystone': '17.0.0',
        'neutron': '16.2.0',
        'nova': '21.1.1'
    },
    'victoria': {
        'glance': '21.0.0',
        'keystone': '18.0.0',
        'neutron': '17.0.0',
        'nova': '22.0.1'
    }
}

This proposal was submitted for FOSDEM 2021. Given that acceptances were meant to be sent out on 25 December and its basically a week later I think we can assume that its been rejected. I’ve recently been writing up my rejected proposals, partially because I’ve put in the effort to write them and they might be useful elsewhere, but also because I think its important to demonstrate that its not unusual for experienced speakers to be rejected from these events.

OpenStack today is a complicated beast — not only does it try to perform well for large clusters, but it also embraces a diverse set of possible implementations from hypervisors, storage, networking, and more. This was a deliberate tactical choice made by the OpenStack community years ago, forming a so called “Big Tent” for vendors to collaborate in to build Open Source cloud options. It made a lot of sense at the time to be honest. However, OpenStack today finds itself constrained by the large number of permutations it must support, ten years of software and backwards compatability legacy, and a decreasing investment from those same vendors that OpenStack courted so actively.

Shaken Fist makes a series of simplifying assumptions that allow it to achieve a surprisingly large amount in not a lot of code. For example, it supports only one hypervisor, one hypervisor OS, one networking implementation, and lacks an image service. It tries hard to be respectful of compute resources while idle, and as fast as possible to deploy resources when requested — its entirely possible to deploy a new VM and start it booting in less than a second for example (if the boot image is already held in cache). Shaken Fist is likely a good choice for small deployments such as home labs and telco edge applications. It is unlikely to be a good choice for large scale compute however.

The post An Overview of OpenStack Barbican for Secure Key Management appeared first on VEXXHOST.

The post Santa’s Scalable Sack Storage & Delivery Automation appeared first on Aptira.

The post Why a Managed OpenStack Cloud Is Better For You Than Self-Managed appeared first on VEXXHOST.

In 2019, the term “digital sovereignty” gained momentum in Europe. For the first time in history, several ministries such as the German and French ministries for Economic Affairs pushed forward initiatives to establish digital sovereignty.

At the Open Infrastructure Summit keynotes, Johan Christenson, CEO of City Network, discussed the importance of digital sovereignty in the infrastructure world. Christenson mentioned that on July 16th 2020, the Court of Justice of the European Union invalidated “Privacy Shield”, which means companies seeking to transfer personal data from the European Economic Area to the United States must now use other mechanisms recognized by General Data Protection Regulation (GDPR) to appropriately safeguard personal data.

“As Europe grapples with an ever more centralized cloud-world and lack of true choice in providers, new initiatives are surging like GAIA-X. Europe is looking to have vendors whom can support both its values and laws in the infrastructure layer.” Christenson said. Therefore, digital sovereignty is becoming critically important from both the aspects of achieving digital independence and the long-term innovation for Europe as a whole.

But, what exactly is digital sovereignty? Why is open infrastructure important to achieve digital sovereignty? What is the European initiative GAIA-X project and how is it leveraging open infrastructure for its use case? In this article, we will focus on what digital sovereignty is and why open infrastructure matters, from the talk that Marius Feldmann, Chief Operation Officer of Cloud&Heat Technologies, and Kurt Garloff, leading the Sovereign Cloud Stack (SCS) initiative, delivered—Digital Sovereignty Why Open Infrastructure Matters—at the Open Infrastructure Summit.

“Digital sovereignty is about sovereignty, not about autarky.”

It’s easy to confuse digital sovereignty with digital autarky, and they are completely different concepts. Digital sovereignty is about integrating the rising and existing technologies and innovative ideas into an overall system and adding its own solutions in order to have the freedom of choice among different options that exist. 

“Sovereignty matters – for sovereignty you need control and choice.”

Just like Feldmann discussed in his Summit session, “digital sovereignty is built on freedom of choice.” Consumers should have particularly effective freedom of choice when they become the “active managers” of their own data (Digital Sovereignty Report by the Advisory Council for Consumer Affairs, 2017).

But what are the essential preconditions to achieve having the freedom of choice in digital sovereignty? One of the most important preconditions is that alternatives should be available to achieve the freedom of choice. In addition, the alternatives must differ regarding their functional and non-function properties. It’s also fundamentally important that discovery based on these properties must be rendered possible, which means there should be possibilities and means to discover the digital services that are offered based on the properties.

European Initiative GAIA-X

With the aim to develop common requirements for a European data infrastructure, GAIA-X is a project that has been initiated by the German ministry for economy and energy. Currently, hundreds of companies worldwide work together on GAIA-X to push forward the overall vision of digital sovereignty. 

“A strong open infrastructure ecosystem is the foundation of digital sovereignty” (GAIA-X: Technical Architecture, June 2020).

GAIA-X aims to develop common requirements for a European data infrastructure. Therefore openness, transparency and the ability to connect to other European countries are central to GAIA-X. This project is the cradle of an open, transparent digital ecosystem, where data and services can be made available, collated and shared in an environment of trust. Read more about GAIA-X on GAIA-X: Technical Architecture document. 

Towards Decentralization

To further explain what it needs to achieve digital sovereignty, let’s assume a heterogeneous / distributed ecosystem as the foundation for a digital sovereign future. 

Without having heterogeneous and decentralized infrastructure, there is no freedom of choice. Therefore, Having various providers and decentralized infrastructures is key to digital sovereignty. Feldmann summarized it well that 

“Future digital infrastructures are not monolithic and centralized but highly distributed and heterogeneous regarding their properties.”

However, to avoid a potential drawback of this future digital infrastructure, which is having various APIs to achieve mainly the same things on the infrastructure side, there should be a standardized API for all the cloud providers. Therefore, on one hand, the future digital infrastructures should be heterogeneous in order to have alternatives available due to the different properties. On the other hand, there should be standards for the API and for the platform itself. 

To clarify, when talking about standardized APIs as well as standardized tools to make an infrastructure platform operational, Feldmann was only talking about the software layer for commodity (compute, storage, networking). There is no need for many alternatives on this layer. The commodity layer should leverage open infrastructure projects to render possible collaborative development and to avoid API fragmentation. 

“Open Infrastructure is bridging the chasm on the commodity layer.”

Reinventing the wheel on the commodity layer may be an interesting exercise; however, it slows down innovative and technical progress.

On the path towards digital sovereignty, on one hand, it is crucial to provide a modular open infrastructure solution that renders it possible to enable various actors to set up an operational platform for hosting services quickly. On the other hand, open infrastructure projects should reduce development overhead and should ensure API interoperability in order to avoid vendor-lock-in contradicting the freedom of choice.

Just like what Christenson said, “open source is the only realistic way to a solid choice in vendors. Open source is also the foundation for Europe’s long term innovation and sovereignty in anything data.” 

Get Involved

This article is a summary of the Open Infrastructure Summit session, Digital Sovereignty – Why Open Infrastructure Matters.

Watch more Summit session videos like this on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source stories using the hashtag, #WeAreOpenInfra, on Twitter and Facebook.

Thanks to the 2020 Open Infrastructure Summit sponsors for making the event possible:=

Headline: Canonical (ubuntu), Huawei, VEXXHOST

Premier: Cisco, Tencent Cloud

Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post Digital Sovereignty – Why Open Infrastructure Matters appeared first on Superuser.

While reviewing  the comments on the Ironic spec, for Secure RBAC. I had to ask myself if the “project” construct makes sense for Ironic.  I still think it does, but I’ll write this down to see if I can clarify it for me, and maybe for you, too.

Baremetal servers change.  The whole point of Ironic is to control the change of Baremetal servers from inanimate pieces of metal to “really useful engines.”  This needs to happen in a controlled and unsurprising way.

Ironic the server does what it is told. If a new piece of metal starts sending out DHCP requests, Ironic is going to PXE boot it.  This is the start of this new piece of metals journey of self discovery.  At least as far as Ironic is concerned.

But really, someone had to rack and wire said piece of metal.  Likely the person that did this is not the person that is going to run workloads on it in the end.  They might not even work for the same company;  they might be a delivery person from Dell or Supermicro.  So, once they are done with it, they don’t own it any more.

Who does?  Who owns a piece of metal before it is enrolled in the OpenStack baremetal service?

No one.  It does not exist.

Ok, so lets go back to someone pushing the button, booting our server for the first time, and it doing its PXE boot thing.

Or, we get the MAC address and enter that into the ironic database, so that when it does boot, we know about it.

Either way, Ironic is really the playground monitor, just making sure it plays nice.

What if Ironic is a multi-tenant system?  Someone needs to be able to transfer the baremetal server from where ever it lands up front to the people that need to use it.

I suspect that ransferring metal from project to project is going to be one of the main use cases after the sun has set on day one.

So, who should be allowed to say what project a piece of baremetal can go to?

Well, in Keystone, we have the idea of hierarchy.  A Project is owned by a domain, and a project can be nested inside another project.

But this information is not passed down to Ironic.  There is no way to get a token for a project that shows its parent information.  But a remote service could query the project hierarchy from Keystone. 

https://docs.openstack.org/api-ref/identity/v3/?expanded=show-project-details-detail#show-project-details

Say I want to transfer a piece of metal from one project to another.  Should I have a token for the source project or the remote project.  Ok, dump question, I should definitely have a token for the source project.  The smart question is whether I should also have a token for the destination project.

Sure, why not.  Two tokens. One has the “delete” role and one that has the “create” role.

The only problem is that nothing like this exists in Open Stack.  But it should.

We could fake it with hierarchy;  I can pass things up and down the project tree.  But that really does not one bit of good.  People don’t really use the tree like that.  They should.  We built a perfectly nice tree and they ignore it.  Poor, ignored, sad, lonely tree.

Actually, it has no feelings.  Please stop anthropomorphising the tree.

What you could do is create the destination object, kind of a potential piece-of-metal or metal-receiver.  This receiver object gets  a UUID.  You pass this UUID to the “move” API. But you call the MOVE api with a token for the source project.   The move is done atomically. Lets call this thing identified by a UUID a move-request. 

The order of operations could be done in reverse.  The operator could create the move request on the source, and then pass that to the receiver.  This might actually make mores sense, as you need to know about the object before you can even think to move it.

Both workflows seem to have merit.

And…this concept seems to be something that OpenStack needs in general. 

Infact, why should the API not be a generic API. I mean, it would have to be per service, but the same API could be used to transfer VMs between projects in Nova nad between Volumes in Cinder. The API would have two verbs one for creating a new move request, and one for accepting it.

POST /thingy/v3.14/resource?resource_id=abcd&destination=project_id

If this is called with a token, it needs to be scoped. If it is scoped to the project_id in the API, it creates a receiving type request. If it is scoped to the project_id that owns the resource, it is a sending type request. Either way, it returns an URL. Call GET on that URL and you get information about the transfer. Call PATCH on it with the appropriately scoped token, and the resource is transferred. And maybe enough information to prove that you know what you are doing: maybe you have to specify the source and target projects in that patch request.

A foolish consistency is the hobgoblin of little minds.

Edit: OK, this is not a new idea. Cinder went through the same thought process according to Duncan Thomas. The result is this API: https://docs.openstack.org/api-ref/block-storage/v3/index.html#volume-transfer

Which looks like it then morphed to this one:

https://docs.openstack.org/api-ref/block-storage/v3/index.html#volume-transfers-volume-transfers-3-55-or-later

The post How OpenStack Sahara Works As a High-Performance Big Data Platform appeared first on VEXXHOST.

I've had to re-teach myself how to do this so I'm writing my own notes.

Prerequisites:

1. Get a working undercloud (perhaps from tripleo-lab)
2. git clone https://git.openstack.org/openstack/tripleo-ansible.git ; cd tripleo-ansible
3. Determine the test name: ls roles

Once you have your environment ready run a test with the name from step 3.

./scripts/run-local-test tripleo_derived_parameters

 export TRIPLEO_JOB_ANSIBLE_ARGS="--skip-tags run_ceph_ansible,run_uuid_ansible,ceph_client_rsync,clean_fetch_dir"
 ./scripts/run-local-test tripleo_ceph_run_ansible

This last tip should get added to the docs.

Codership is pleased to announce our first ever online training for Galera Cluster, with the launch of the Database Administration for Galera Cluster course. There are two sets of times for our EMEA and American attendees, with the former happening on January 2021, 13th and 14th, starting 10 AM CET and the latter happening on January 2021, 20th and 21st, starting 9 AM PST.

This is a hands on course that spans two days, and you will have about six contact hours per day in this instructor-led course. If you are a DBA or been given a Galera Cluster to manage, this is the course for you, as you’ll learn how to run your Galera Cluster in an optimal fashion, from setup, performance tuning, monitoring, as well as backups and more. We will cover not just Codership’s Galera Cluster distribution but also MariaDB Galera Cluster and Percona XtraDB Cluster in this course. For more information, please read the full content breakdown.

As an added bonus, we will also be covering database administration with Galera Manager our new GUI tool.

What are you waiting for? Sign up now, as the early bird rate ends 31 December 2020. There is also a volume discount available.

The post Ceph Storage Basics and How It Benefits Your OpenStack Private Cloud appeared first on VEXXHOST.

European Centre for Medium range Weather Forecasts (ECMWF), an intergovernmental organisation, was established in 1975. Based in Reading, UK and the data center soon moving to Bologna Italy, ECMWF spans 34 States in Europe. It operates one of the largest supercomputer complexes in Europe and the world’s largest archive of numerical weather prediction data. In terms of its IT infrastructure, ECMWF’s HPC (High-performance computing) facility is one of the largest weather sites globally. With cloud infrastructure for Copernicus Climate Change Service (C3S), Copernicus Atmosphere Monitoring Service (CAMS) and WEkEO, which is a Data and Information Access Service (DIAS) platform, and the European Weather Cloud, teams at ECMWF maintain an archive of climatological data with a size of 250 PB with a daily growth of 250TB.

European Weather Cloud:

The European Weather Cloud started three years ago in a collaboration between ECMWF and The European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) aiming to make it easier to work on weather and climate big data in a cloud-based infrastructure. With the goal of bringing the computation resources (Cloud) closer to their Big data (meteorological archive and satellite data), ECMWF’s pilot infrastructure was with open source software – Ceph and OpenStack using TripleO.

The graph below shows the current state of the European Weather Cloud overall infrastructure comprising two OpenStack clusters: one built with OpenStack Rocky and another one with OpenStack Ussuri. The total hardware of the current configuration comprises around 3,000 vCPUs, 21 TB RAM for both clusters, 1PB of storage and 2×5 NVIDIA Tesla V100 GPUs.

Integration with Ceph:

The graph below shows the cloud infrastructure of the European Weather Cloud. As you can see, Ceph is built and maintained separately from OpenStack which gives the teams at the European Weather Cloud a lot of flexibility in building different clusters on the same Ceph storage. Both of its OpenStack clusters use the same Ceph infrastructure and the same rbd pools. Besides some usual HDD failures, Ceph performs very well, and the teams at the European Weather Cloud are planning to gradually move to CentOS8 (due to partial support of CentOS7) and upgrade to Octopus and cephadm on a live cluster after a lot of testing to their development environment.

OpenStack with Rocky version:

The first OpenStack cluster in the European Weather Cloud, which was built in September 2019, is based on Rocky with TripleO installer. In the meantime, engineers at the European Weather Cloud also created another development environment with OpenStack and Ceph clusters that are similarly configured for testing-experimentation.

Experience and Problems:

Their deployment has about 2,600 VCPUs with 11TB RAM which doesn’t have any significant problem. The external Ceph cluster integration worked with minimum effort by simply configuring the ceph-config.yaml with little modifications. The two external networks (one public facing and another for fast access to their 300PB data archive) were straightforward. 

Most of their VMs are attached to both external networks with no floating IPs, which was a challenging VM routing issue without dynamic routing on the switches. To solve this issue, they used dhcp hooks and configured VM routing before they made the images available to the users.

There were some problems that they encountered with the NIC bond interface configuration and in conjunction with the configuration of their switches at the beginning. Therefore, the engineers decided to not use a Link Aggregation Control Protocol (LACP) configuration, and now they have a single network interface card (NIC) deployment for OpenStack. They also encountered some problems with Load-Balancing-as-a-Service (LBaas) due to Octavia problems with certificates overridden on each deployment. The problem is presented here. 

As soon as they found the solutions to these challenges, the engineers updated their live system and moved the whole cluster from a Single NIC to a Multiple NIC deployment which is transparent to the users with zero downtime. The whole first cluster was redeployed, and the network was re-configured with Distributed Virtual Routing (DVR) configuration for better network performance. 

OpenStack update efforts from Stein-Train-Ussuri:

In March 2020, engineers at the European Weather Cloud added more hardware for their OpenStack and Ceph cluster, and they decided to investigate upgrading to the newest versions of OpenStack. 

Experience and Problems:

First, they converted their Rockey undercloud to a VM for better management and as a safety net for backups and recovery. From March to May 2020, they investigated and tested upgrading to Stein (first undercloud and then overcloud upgrade to a test environment). Updating was possible from Rocky to Stein to Train and finally to Ussuri, but due to the CentOS7 to CentOS8 transition since Ussuri was based on CentOS8, it was considered impractical. Therefore, they made a big jump from Rocky to Ussuri by skipping three updates and decided to directly deploy their new systems on OpenStack Ussuri.

OpenStack Ussuri Cluster:

The second OpenStack cluster, based on Ussuri, was first built in May 2020, 17 days after the release of Ussuri on May 13. This cluster was a plain vanilla configuration which means that although the network was properly configured with OVN and provider networks with 25 nodes they haven’t had any integration with Ceph storage. 

Experience and Problems:

The new building method that was based on Ansible instead of Mistral had some hiccups, such as the switch from stack to heat-admin which is not what the users are used to deploy. In addition, they were trying to quickly understand and master the CentOS8 base operating system for both the host systems and service containers. Engineers at the European Weather Cloud also continued with OVS instead of OVN because of the implications in assigning floating IP addresses. With the help from the OpenStack community, the problems were solved, and the cluster was built again in mid-June 2020.

Regarding GPUs, the configuration of Nvidia GPUs was straightforward. However, since they haven’t implemented IPv6 to their Ussuri cluster when they installed and configured the GPUs drivers to a node, OVS was trying to bind to IPv6 addresses during the booting time which results in a considerable increase in booting time. A workaround was to explicitly remove PIv6 configuration to their GPU nodes. All nodes with a GPU also resolved as normal compute nodes, and they have configured nova.conf with their Ansible playbooks.  

Future Next Steps:

In terms of the European Weather Cloud’s infrastructure, the engineers are planning to integrate the infrastructure with other internal systems for better monitoring and logging. They are also planning to phase out the Rocky cluster and move all the nodes to Ussuri. Trying to follow the latest versions of OpenStack and Ceph, they will continue to operate, maintain and upgrade the Cloud’s infrastructure.

For the federation, the goal is to federate their Cloud infrastructure with infrastructures of their Member states. They have identified and will continue to explore potential good use cases to federate. 

Regarding the integration with other projects, the European Weather Cloud will be interfacing with the Digital Twin Earth which is a part of the Destination Earth Program of the EU. 

Teams at the European Weather Cloud are also planning to contribute code and help other users that are facing the same problems while deploying clusters in the OpenStack community.

Get Involved:

Watch this session video and more on the Open Infrastructure Foundation YouTube channel. Don’t forget to join the global Open Infrastructure community, and share your own personal open source story using #WeAreOpenInfra on social media.

Special thanks to our 2020 Open Infrastructure Summit sponsors for making the event possible:

• Headline: Canonical (ubuntu), Huawei, VEXXHOST
• Premier: Cisco, Tencent Cloud
• Exhibitor: InMotion Hosting, Mirantis, Red Hat, Trilio, VanillaStack, ZTE

The post OpenStack in Production and Integration with Ceph: A European Weather Cloud User Story appeared first on Superuser.

I have two business ideas to explore, and I decided that now is a good time to take the plunge and create a prototype. My hesitation throughout the last year was due to the time and financial investment required. After some inspiration, detailed thought, and self-evaluation, I am ready to go for it. Worst case scenario, this is going to eat up a lot of my time. Even if I lose time, I will learn a lot about cloud infrastructure, cloud networking, and cloud instance provisioning. My first business idea is in the realm of home and small business network cyber security. The second utilizes a private cloud platform to provision labs for IT and cyber security training. A small virtual lab isn’t going to cut it for these ventures.

My Current Setup

Before I can pursue these builds, I need to upgrade my home network and lab and select a platform. I currently have 3 old used servers (2 Dell PowerEdge R510s and an HP Proliant DL360) for the cloud. For networking, I have an ancient Cisco switch. I think I can get by with the old switch for now, but my small private cloud requires more servers. I can use the private cloud to provision networks to test out capabilities, learn, and design. These can also hold prototypes and proof of concepts for demonstrations. For the private cloud, I selected OpenStack as my platform. This will allow me to provision instances using Terraform, and have more flexibility with networking configuration. I can also avoid a large AWS and Azure bill while I experiment with different configurations. The only thing that will suffer is my power bill.

Project Goals

Based on the OpenStack documentation I will need at least 4-5 servers to support my configuration which is a small compute cloud. To use Juju and Metal as a Service (MAAS) to deploy the cloud, I will need 2 more servers, but I could probably use one of my servers and host 2 VMs instead of purchasing another server. I haven’t yet decided whether I am going to use Juju and MAAS to deploy OpenStack, but I do know that I need at least 2 more servers for my project. I also want to separate my private cloud from the rest of my network and still maintain network performance with the added security, so I will need a firewall / IPS appliance. Once complete, my home network will look something like this:

My Private Cloud Budget

I am trying to stay under $2,000 total for this project (including what I already spent). Below is the price I paid for everything I already have.

Existing devices with costs at the time of purchase

So, based on that I have about $1100 to spend. Although I have plenty of room, I am sticking with used equipment. The only exception I am making is my firewall appliance.

Purchasing New Equipment

I was able to find 2 Dell PowerEdge R610s for $157 each, well within budget. My shipping costs to my location are really high, so I have to keep that in mind. Even with the shipping costs, I still consider these a bargain and they meet my needs. These servers also come from the same vendor as my previous purchases (PC Server and Parts), so I know they will arrive in good condition and operate well.

Next I need a firewall appliance, for this I am going straight to a vendor because their site is a lot cheaper than Amazon. This appliance from Protectli has 4 NICs, a quad core processor, and a small SSD. This is more than enough to run pfsense (and it was already tested for it), so it will easily meet my needs and be a step up from my current options for under $300.

Total Costs

With those 2 purchases, I have all the equipment I will need, and significantly under my max budget! The only other purchase I might make is a rack to store the equipment and a PDU. For now, I just have to wait for them to arrive. I plan to start sometime in December. While I wait, I am going to work on my remote access solutions, determine what IDS/IPS I am going to use (Suricata, Snort, or Bro), and finalize my design of how this will all fit together.

Existing devices with costs at time of purchase

This article was originally posted on mattglass-it.com. See the original article here.

The post Embarking on a New Venture: Creating a Private Cloud with OpenStack for Under $1700 appeared first on Superuser.

The post Why OpenStack Private Cloud for DevOps is the Way Forward appeared first on VEXXHOST.

The post A Brief History of the Open Infrastructure Foundation appeared first on VEXXHOST.

Kata containers are containers that use hardware virtualization technologies for workload isolation almost without performance penalties. Top use cases are untrusted workloads and tenant isolation (for example in a shared Kubernetes cluster). This blog post describes how to run Percona Kubernetes Operator for Percona XtraDB Cluster (PXC Operator) using Kata containers.

Prepare Your Kubernetes Cluster

Setting up Kata containers and Kubernetes is well documented in the official github repo (cri-o, containerd, Kubernetes DaemonSet). We will just cover the most important steps and pitfalls.

Virtualization Support

First of all, remember that Kata containers require hardware virtualization support from the CPU on the nodes. To check if your linux system supports it run on the node.

$ egrep ‘(vmx|svm)’ /proc/cpuinfo

VMX (Virtual Machine Extension) and SVM (Secure Virtual Machine) are Intel and AMD features that add various instructions to allow running a guest OS with full privileges, but still keeping host OS protected.

For example, on AWS only i3.metal and r5.metal instances provide VMX capability.

Containerd

Kata containers are OCI (Open Container Interface) compliant, which means that they work pretty well with CRI (Container Runtime Interface) and hence well supported by Kubernetes. To use Kata containers please make sure your Kubernetes nodes run using CRI-O or containerd runtimes.

The image below describes pretty well how Kubernetes works with Kata.

Hint: GKE or kops allows you to start your cluster with containerd out of the box and skip manual steps.

Setting Up Nodes

To run Kata containers, k8s nodes need to have kata-runtime installed and runtime configured properly. The easiest way is to use DaemonSet which installs required packages on every node and reconfigures containerd. As a first step apply the following yamls to create the DaemonSet:

$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-rbac/base/kata-rbac.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/kata-deploy/base/kata-deploy.yaml

DaemonSet reconfigures containerd to support multiple runtimes. It does that by changing /etc/containerd/config.toml. Please note that some tools (ex. kops) keep containerd in a separate configuration file config-kops.toml. You need to copy the configuration created by DaemonSet to the corresponding file and restart containerd.

Create runtimeClasses for Kata. RuntimeClass is a feature that allows you to pick runtime for the container during its creation. It has been available since Kubernetes 1.14 as Beta.

$ kubectl apply -f https://raw.githubusercontent.com/kata-containers/packaging/master/kata-deploy/k8s-1.14/kata-qemu-runtimeClass.yaml

Everything is set. Deploy test nginx pod and set the runtime:

$ cat nginx-kata.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-kata
spec:
  runtimeClassName: kata-qemu
  containers:
    - name: nginx
      image: nginx

$ kubectl apply -f nginx-kata.yaml
$ kubectl describe pod nginx-kata | grep “Container ID”
    Container ID:   containerd://3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111

On the node verify if Kata is used for this container through ctr tool:

# ctr --namespace k8s.io containers list | grep 3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111
3ba8d62be5ee8cd57a35081359a0c08059cf08d8a53bedef3384d18699d13111    sha256:f35646e83998b844c3f067e5a2cff84cdf0967627031aeda3042d78996b68d35 io.containerd.kata-qemu.v2cat 

Runtime is showing kata-qemu.v2 as requested.

The current latest stable PXC Operator version (1.6) does not support runtimeClassName. It is still possible to run Kata containers by specifying io.kubernetes.cri.untrusted-workload annotation. To ensure containerd supports this annotation add the following into the configuration toml file on the node:

# cat <> /etc/containerd/config.toml
[plugins.cri.containerd.untrusted_workload_runtime]
  runtime_type = "io.containerd.kata-qemu.v2"
EOF

# systemctl restart containerd

Install the Operator

We will install the operator with regular runtime but will put the PXC cluster into Kata containers.

Create the namespace and switch the context:

$ kubectl create namespace pxc-operator
$ kubectl config set-context $(kubectl config current-context) --namespace=pxc-operator

Get the operator from github:

$ git clone -b v1.6.0 https://github.com/percona/percona-xtradb-cluster-operator

Deploy the operator into your Kubernetes cluster:

$ cd percona-xtradb-cluster-operator
$ kubectl apply -f deploy/bundle.yaml

Now let’s deploy the cluster, but before that, we need to explicitly add an annotation to PXC pods and mark them untrusted to enforce Kubernetes to use Kata containers runtime. Edit deploy/cr.yaml:

pxc:
  size: 3
  image: percona/percona-xtradb-cluster:8.0.20-11.1
  …
  annotations:
 
      io.kubernetes.cri.untrusted-workload: "true"

Now, let’s deploy the PXC cluster:

$ kubectl apply -f deploy/cr.yaml

The cluster is up and running (using 1 node for the sake of experiment):

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
pxc-kata-haproxy-0                                 2/2     Running   0          5m32s
pxc-kata-pxc-0                                     1/1     Running   0          8m16s
percona-xtradb-cluster-operator-749b86b678-zcnsp   1/1     Running   0          44m

In crt output you should see percona-xtradb cluster running using Kata runtime:

# ctr --namespace k8s.io containers list | grep percona-xtradb-cluster | grep kata
448a985c82ae45effd678515f6cf8e11a6dfca159c9abf05a906c7090d297cba    docker.io/percona/percona-xtradb-cluster:8.0.20-11.2 io.containerd.kata-qemu.v2

We are working on adding the support for runtimeClassName option for our operators. The support of this feature enables users to freely choose any container runtime.

Conclusions

Running databases in containers is an ongoing trend and keeping data safe is always the top priority for a business. Kata containers provide security isolation through mature and extensively tested qemu virtualization with little-to-none changes to the existing environment.

Deploy Percona XtraDB Cluster with ease in your Kubernetes cluster with our Operator and Kata containers for better isolation without performance penalties.

This article was originally posted on percona.com/blog. See the original article here.

The post Running Percona Kubernetes Operator for Percona XtraDB Cluster with Kata Containers appeared first on Superuser.

OpenStack upstream CI/CD tests the things on defined  LTS or stable distribution versions. OpenStack Technical Committee defines each cycle testing runtime. As per OpenStack Victoria testing runtime, defined versions are:

• Ubuntu 20.04
• CentOS 8
• openSUSE Leap 15

Ubuntu Focal (Ubuntu LTS 20.04) was released on April 23, 2020, and in OpenStack Victoria (released on 14th Oct 2020), we migrated the upstream CI/CD testing on the above-defined testing runtime. This work is done as one of the community-wide goals “Migrate CI/CD jobs to new Ubuntu LTS Focal“.

    What is this migration:

OpenStack CI/CD is implemented with Zuul jobs prepare the node to deploy the OpenStack using Devstack and run tests (Tempest or its plugins, project in-tree tests, rally tests etc). Base OS installed on the node is where OpenStack will be deployed by DevStack.

Till OpenStack Ussuri release, the base OS on the majority of the job’s node was Ubuntu Bionic (18.04). So DevStack used to deploy OpenStack on Ubuntu Bionic and then run tests.

With the new version of Ubuntu Focal (20.04), the node’s base OS has been moved from Ubuntu Bionic -> Ubuntu Focal. On every code change, it will make sure OpenStack work properly on Ubuntu Focal.

NOTE: This migration target only zuulv3 native jobs. Legacy jobs are left to keep running on Bionic and plan to be migrated on Focal while they migrate to zuulv3 native jobs . We had another community-goal to migrate all the legacy jobs to zuulv3 native.

    Perform migration testing in advance:

We started the work in June and prepared the devstack, tempest, and tox-based base jobs on Focal so that all projects gate can be tested and fixed in advance before devstack and Tempest base jobs merge. The idea behind the advance testing is to avoid or minimize the gate failure in any of the repo under any projects. This advance testing includes integration as well as tox based unit, functional, doc, pep8, and lower-constraint testing.

    Bugs & fixes:

This migration had more things to fix compared to the previous migration from Ubuntu Xenial to Bionic. One reason for that was Ubuntu Focal and python dependencies dropping python2.7 support and MySQL 8.0 compatibility. OpenStack already dropped the Python2.7 in the Ussuri release but OpenStack dependencies lower constraints were not updated to their python-3 only version because many of them were not python3-only at that time.  So in Ubuntu Focal, those dependencies versions are python3-only which caused many failures in our lower constraints jobs.

A few of the key issues we had to fix for this migrations are:

1. Failing device detachments on Focal: Bug#1882521
2. Few of the lower constraints are not compatible with python3.8: Bug#1886298.
3. Migrations broken on MySQL 8.x: story/2007732
4. Duplicate check constraint name ‘ck_started_before_ended’: story/2008121
5. No implicit user creation with GRANT syntax in MySQL 8.0: Bug#1885825
6.  pyflakes till 2.1.0 not compatible with py3.8: Bug#1886296.

    Actual Migration:

Fixing these bugs took a lot of time for us and that is the reason this migration was late and missed the initial deadlines.

• Tox based job migration happened on Sept 9th.
• devstack and Tempest base jobs migration happened on Sept 24th.

All the work for this migration are tracked on: https://storyboard.openstack.org/#!/story/2007865

All changes are: https://review.opendev.org/q/topic:%2522migrate-to-focal%2522+(status:open+OR+status:merged)

    How to migrate the third-party CI:

If your 3rd party CI jobs are still not migrated to zuulv3 then you need to first migrate legacy jobs to zuulv3 native. Refer to this community-goal for details.

For zuulv3 native jobs, like upstream jobs you need to switch the jobs nodeset from ubuntu Bionic to ubuntu Focal.

Below Diagram gives a quick glance of changing the nodeset to Ubuntu Focal:

If you want to verify the nodeset used in your zuul jobs, you can see the hostname and label in job-output.txt

In the same way, you can migrate your third-party CI also to Focal. If third-party job is using the base job without overriding the ‘nodeset’ then the job is automatically switched to Focal. If the job overrides the ‘nodeset’ then, you need to switch to Focal nodeset like shown above. All the Ubuntu Focal nodeset for a single node to multinode jobs are defined in devstack.

We encourage all the 3rd party jobs to migrate to Focal asap as devstack will not support the Bionic related fixes from Victoria onwards.

    Dependencies compatibility versions for upgrades to OpenStack Victoria:

There are many dependencies constraints that need to be bumped to upgrade the OpenStack cloud to the Victoria release. To know all those compatible versions, check the project-specific patches merged from here or from this Bug#1886298. This can help to have prepared the smooth upgrades.

    Completion Summary Report:

• A lot of work was involved in this. There are many changes required for this migration as compare to previous migration from Xenial to Bionic.
  • Many dependencies with lower constraints compatible with py3.8 and Focal distro had to be updated. Almost in all the repos.
  • Mysql 8.0 caused many incompatible DB issues.
• I missed the original deadline of m-2 to complete this goal due to falling gates and to avoid any gate block for any projects.
• More than 300 repo had been fixed or tested in advance before migration happened. This helped a lot to keep the gate green during this migration.

I would like to convey special Thanks to everyone who helped in this goal and made it possible to complete it in the Victoria cycle itself.

Would you like to know about OpenStack Cloud? So download our white paper and get reading!

Conquer The Competition With OpenStack Cloud

Download White Paper

The post Why Cloud Networking with OpenStack Neutron Works Great for Enterprises appeared first on VEXXHOST.