August 06, 2020

OpenStack Superuser

Verizon’s Optimum Performance Relies on Owning the Testing Process

Verizon’s cloud platform, built on an OpenStack infrastructure distributed across 32 global nodes, is used by the networking team to provide network services to commercial customers. Because these applications are commercial products such as firewalls, SD WAN and routing functions provided by third party companies, not owned by Verizon, the applications sometimes had odd interactions with the underlying infrastructure and each other.

Issue: Uneven vendor application performance

The product engineering team realized that the applications owned by Verizon’s partners were each configured differently, and depending on how they were each configured had a significant effect on how they behaved in the environment.

SDN vendor performance variance became a pressing issue that significantly affected throughput in the field. For example, it was discovered that in many cases, when encryption was turned on, throughput was reduced by half. With traffic moving through multiple systems, it became difficult to determine the cause (or in some cases, causes) of problems and determine the fixes needed. Dramatic variation in vendor capabilities to fully take advantage of virtualized applications and infrastructures, to optimize those applications to OpenStack became a major challenge.

Solution: Create platform and processes to address issues

Verizon tackled this issue of inconsistency by building a production engineering lab with full testing capabilities. This lab environment, used for product development, production support, and troubleshooting customer configurations, gives a clear and efficient feedback loop that is useful for informing product managers, sales teams and customers with real world results. For instance, when a customer decides to run voice traffic through a firewall (not a common configuration), with the lab Verizon can access and analyze all the different nuances of that configuration. The lab is also used to work closely with vendors to optimize their virtualized applications. It supports the capacity to test both data center environments and edge devices.

As a consequence of developing the production engineering lab, Verizon now has the ability to insist on thorough and consistent testing of each vendor’s application. Verizon is able to take customer their production traffic and run it through the lab, making it possible to reproduce customers’ issues in the lab environment. Through verifying each application, testing them for performance-based on factors like encryption, and making full performance testing on all integrated service chains automated and mandatory, Verizon is able to provide a much higher level of value to their customers to prevent potentially unpleasant surprises.

User Story: Financial Services firm with high security and performance requirements

Customers look to Verizon with high expectations, and Verizon makes sure to work with each of their vendors to provide the testing and support that they need most to meet their SLAs.

One of Verizon’s customers, began to experience problems with low bandwidth and application microfreezing. This was a big problem for their security application. After some testing it was soon obvious that this behavior was common to all security applications running on the virtualized environment. Immediately, the team at Verizon began to make changes to how the VMs were stood up in the environment, without needing to change any aspects of the underlying hosted infrastructure itself.

Because the results of this case study affected nearly every single one of Verizon’s vendor applications, particularly where the customers had latency sensitive deployments and transports larger than 100 Mbps, the company has now developed new standards to support customer configurations.. All VMs for future applications are now automatically configured to be pinned to resources to avoid resource contention, vendors are mandated to support SRIOV networking deployment, and customers are cautioned about throughput behavior if they choose to turn on traffic encryption.

Customers want reliable performance, but they not uncommonly put unexpected demands on their services. By building a full lab and testing center, Verizon was able to test customer configurations and troubleshoot issues down to the individual feature level. As a result, Verizon as a large operator now has even more capacity to facilitate cooperation with all of its vendors—ensuring the integrated service chains perform as expected, solving issues uncovered during development or production, and quickly addressing any customer related issues.

The 2020 OpenDev event series was held virtually. The photo is from OpenDev 2017 that was focused on edge computing where Beth Cohen also presented a session. 

The post Verizon’s Optimum Performance Relies on Owning the Testing Process appeared first on Superuser.

by Ella Cathey at August 06, 2020 02:00 PM

August 05, 2020

OpenStack Superuser

Where are they now? Superuser Awards winner: NTT Group

We’re spotlighting previous Superuser winners who are on the front lines deploying OpenStack in their organizations to drive business success. These users are taking risks, contributing back to the community and working to secure the success of their organization in today’s software-defined economy.

Nippon Telegraph and Telephone (NTT Group)  has various OpenStack deployments in production including large scale public cloud, private cloud for internal services and NFV infrastructure. The following are its major OpenStack deployments.

Note: Abbreviations in brackets are used to describe each deployment throughout this document:

  • [Com] NTT Communications provide large scale public cloud service for enterprise, called ECL 2.0
  • [Data] NTT Data has two OpenStack deployments: one is for internal application developers and the other is to host its enterprise customers.
  • [Resonant] NTT Resonant provides various web based services, including web portal service called “goo,” and has a large internal cloud to host its services.
  • [TX] NTT Technocross provides OpenStack support service and has their internal cloud for developers.
  • [R&D] NTT Software Innovation Center operates internal R&D Cloud, “XFarm,” for researchers and developers.

What has changed in your OpenStack environment since you won the Superuser Awards?

[Com] We have scaled out to accommodate our customers. We also stabilized our system by fixing bugs to improve our service quality. We also overcame the challenge of OpenStack version up in production.

[Data] We have deployed two large scale OpenStack clusters: one is for internal app development and the other is for external enterprise customers. We have started these two services since 2017 and are still expanding the scale.

[Resonant] We have doubled our virtual machine (VM) numbers since 2015. We are now working on upgrading our environment due to hardware end-of-life since last year. We will be migrating to OpenStack Queens and introducing all-flash storages.

[TX] We have upgraded OpenStack version and the deployment was automated by using Ansible.

[R&D] In 2015 we had two clusters with 15 nodes mainly for evaluation purpose. Now we have three clusters in production with 22 nodes hosting R&D workloads.

What is the current size of your OpenStack environment?

[Com] More than 3,000 compute nodes, 30,000 VMs and 80,000 virtual cores in production.

[Data] Approx. 80 compute nodes / 7,000 VMs / 21,000 virtual cores for internal app development. Approximately 20 compute nodes / 600 VMs / 2,000 virtual cores for external enterprise customers.

[Resonant] Our new deployment has about 60 compute nodes hosting around 3,500 VMs.

[TX] About 20 compute nodes.

[R&D] 22 Compute nodes, 1550 VMs, 8700 vCPUs

What version of OpenStack are you running?

[Com] Mitaka and Queens.

[Data] Newton and Queens.

[Resonant] Our new deployment runs on Queens

[TX] Mitaka and Queens

[R&D] Mitaka and Queens

What open source technologies does your team integrate with OpenStack?

[Com] Ansible, Chef, Docker, Contrail (TungstenFabric), fluentd, Influxdb, Grafana, telegraph, kibana, elasticsearch, filebeat, metricbeat, haproxy, rabbitmq, percona

[Data] Ansible, Docker, Grafana, Prometheus, Kibana, Elasticsearch, Metricbeat, Hinemos, HAproxy , Pacemaker, Corosync, MariaDB and RabbitMQ.

[Resonant] Puppet,Ansible,Docker,Docker-compose, kibana, elasticsearch, fluentd, haproxy, keepalived, CloudFoundry.

[TX] Ansible, haproxy, rabbitmq, percona, kubernetes, docker, gitlab

[R&D] Sheepodg, Ceph

What workloads are you running on OpenStack?

[Com] More than 1,000 enterprise customers including web application, call center, video conference, data processing, NFVs and IoTs.

[Data] More than 1,000 application development projects are using our internal OpenStack environment for develop and testing their apps. Target industries of the projects cover almost all sectors like public, BFSI, manufacturing, healthcare, retail, etc.

For the external customers, OpenStack is mainly used as a platform for various applications in BFSI customers.

[Resonant] 80+ web services with more than 1 Billion page view/month

[TX] Development software and OSS (e.g. K8s, Docker)

[R&D] Our cluster is an IaaS service for R&D employee to conduct research and development and variety of workloads are running.

How big is your OpenStack team?

[Com] About 30 members for OpenStack related development and tier3 engineering.

[Data] We have 10-20 OpenStack engineers.

[Resonant] About 8 OpenStack operators and hundreds of engineers developing applications on OpenStack.

[TX] About 100 engineers including our business partners.

[R&D] About 10 engineers including our business partners are operating OpenStack cluster. We also have around 10 developers contributing to OpenStack including 3 PTLs.

How is your team currently contributing back to the OpenStack project?

[Com] Reporting bug tickets and requests for engineering (RFEs), mainly in Masakari project. Participating community events to present our experience.

[Data] We’re mainly using Red Hat OpenStack so reporting bugs/challenges to Red Hat, which directly contributes to the community. Also, we have about 10 engineers contributing to the OpenStack project (code review, commits, etc.).

[Resonant] Reporting bugs and sending patches through our business partners.

[TX] Reporting bug tickets.

[R&D] Contributing in various OpenStack projects including Tacker, Nova/Placement, Swift and Masakari. We also report bugs and upstream patches.

Also NTT Group is actively contributing to regional OpenStack community activities including organizing meetups, OpenStack Days event, and running slack groups for operators.

What kind of challenges has your team overcome using OpenStack?

[Com] We increased the capability and agility to integrate our infrastructure with our customer system through OpenStack APIs.

[Data] We have successfully reduced the cost of infrastructure for many application development projects by consolidating their platforms into one. Also, we have successfully provided agility and flexibility to BFSI customers for their digital transformation.

[Resonant] We have managed to reduce server procurement cost by having our infrastructure team conduct collective procurement and assign resources appropriately, instead of having each development team to purchase servers separately.

[TX] Compute resources for developers can be prepared on an on-demand basis.

[R&D] We managed to reduce server cost by having our internal IaaS and at the same time, we were able to advance our capabilities in operating open source cloud, which we share among other group companies.

 

The post Where are they now? Superuser Awards winner: NTT Group appeared first on Superuser.

by Superuser at August 05, 2020 01:00 PM

August 03, 2020

OpenStack Blog

10 Years of OpenStack – Gary Kevorkian at Cisco

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at August 03, 2020 03:00 PM

OpenStack Superuser

Contributing to Open Infrastructure: Everybody Wins, So Let’s Get Started

During the 10th Anniversary celebration of OpenStack last month (what a great huge milestone for the community!), the OpenStack Foundation celebrated some remarkable achievements made during the last decade, not the least of which was the number of code contributions made by the global OpenStack community:

  • 500,000+ changes merged
  • 8,000+ individual developers authoring those changes
  • Every day, 900 changes are proposed and 18,000 tests are run to evaluate them

In fact, based on code contributions, OpenStack ranks as the one of the three most active open source projects in the world, along with the Linux kernel and Chromium (the upstream code base for the Chrome browser).

Those metrics are pretty impressive, but they don’t tell the whole story. For starters, they don’t include the growing number of code contributions to other open source projects supporting by OSF, like Airship, Kata Containers, StarlingX, and Zuul. Nor do those metrics include all the other forms of contributions—beyond code—that are made by the community in support of open infrastructure.

It’s important to remember that ‘contribution’ entails more than technical contributions. There are many ways to make valuable contributions to the open infrastructure community: sponsoring and hosting events, serving on working groups and committees, leading meetups, presenting at conferences, contributing to mentoring and diversity efforts, donating to scholarships, voting in elections, serving on the board, just to name a few. Each of these contributions plays a critical role in helping our community thrive and make progress.

City Network comes to mind as a company that contributes extensively in non-technical ways. City Network is the organization behind OpenStack Days Nordic and leads the public cloud group within OpenStack. Johan Christenson, the founder and CEO of City Network, is a member of the OpenStack Foundation board of directors and has volunteered to spearhead an outreach effort to encourage more upstream contributions.

“City Network has been a non-technical contributor to the community for many years, but recently we have been making a concerted effort to become more involved as a technical contributor as well, making some modest code contributions to Stein, Train and Ussuri,” said Johan. “We have been inspired by those who have shown that if you contribute technically, you learn the code and become a better operator. Plus, we believe companies like ours who are making money by using OpenStack in a serious way should be contributing more ourselves.”

Johan also described a collaboration City Network has formed with VEXXHOST.
“One of the companies that we admire for its technical contributions to the community is VEXXHOST,” explained Johan. “Mohammed Naser and his team have been ‘teaching us the ropes,’ helping us to get started on the technical side.”

So, I reached out to Mohammed, and I asked him to let me share some best practices with the broader community as well. Here’s a bit of our conversation:

So, Mohammed, how long have VEXXHOST employees been contributing upstream?

Mohammed Naser: The first time some of our code merged was in 2011, so we’ve been contributing to OpenStack for a long time.

What is your organizational philosophy about contributing upstream to OpenStack?

MN: We don’t believe in keeping “fixes” in our internal team notes. We believe in fixing it upstream. I consider OpenStack to be a core part of our business, so contributing not only helps make OpenStack better for others but helps secure our future as well.

What impact has contributing upstream had on your team’s experience actually using and deploying the software?

MN: Our “upstream-only-first” approach pays dividends for us, because we’re essentially getting free code review from the people who are most knowledgeable about OpenStack. That’s true business value. The reviewer can give suggestions or give assurance that you’re doing the right thing. Or the reviewer might actually find a better solution and explain why and how we should be doing it differently. If you’re making changes on your own, there’s no way you’ll get a cleaner solution. You don’t have to hire someone to evaluate the code. That’s kind of how open source works.

Also, by being heavily involved in upstream, we have a bigger familiarity with the code. We can fix bugs and identify issues more easily, because we can reach out to the community and ask how to fix something. Conversely, by being in close relationship with upstream, we don’t waste time isolated on an island building features that aren’t really needed or won’t be accepted—features that we’ll then end up having to support and maintain by ourselves. Community means we can get valuable feedback before wasting time and resources on coding no one besides us finds valuable.

I can give you many examples of indirect value, too. For instance, there’s a kind of “Contributor’s Karma”—you develop a close relationship with upstream, and they keep you in their thoughts and reach out to you as an operator to get your input. A lot of developers on these projects reach out to us and ask for advice. “Will it make your life easier or harder? Are we doing the right thing here? Can we make it better?” You definitely have a voice that’s heard, even if you are not the one writing the code. It’s a win-win for everyone because we’re building better software by working with each other.

What are some of the biggest obstacles to contributing upstream?

MN: On a corporate level, I think the biggest obstacle is when the management of companies don’t prioritize upstream work. Developers tend to want to contribute, but they can find it difficult to participate when management doesn’t prioritize or encourage it.

On an individual level, it can be intimidating to push up code, especially when you are new to the community, because you’re worried about how people might perceive you or judge you on your code. I truly don’t see that happen much in OpenStack, but I’ve seen that kind of reticence among young developers in general.

What advice would you give to organizations who use OpenStack and who are not currently contributing?

MN: My advice is to have some internal policy that is strict on the fact that you won’t run any local patches of OpenStack. That’s Step 0. If you commit to that, you’ll become an upstream company. You have no choice but to always ship things upstream and get it merged in order to put it in production. If you’re running a fork with local patches, your code is not going to get any performance improvements. It’s going to sit stale and start “bit rotting” with time. By contributing everything upstream, you are no longer maintaining any technical debt, you’ve got a whole community evolving and improving your code, and you’re ready to run on release day.

What’s a good way to get started?

MN: Here’s a practical suggestion: Pick one day every month or two, and call it the day where we all go upstream. Just find a bug and fix it, almost like a hack-a-thon, then everyone shares what they did upstream at the end of the day. You dedicate a whole day to this task, incentivize it, and, this way, no one feels like they are wasting their employer’s time by going upstream.

You might also start by unrolling all of the different local patches you’re maintaining in OpenStack. Document the work-arounds and submit bug fix reports. Start getting rid of your technical debt by pushing that upstream. If you’re a few releases behind, you might discover that some of those bugs are already fixed.

And, finally, if you just use OpenStack as is and it does everything you need it to, you can contribute by sponsoring or funding another company who is doing upstream work. That’s also a fair way to do it.

My thanks to Mohammed for sharing his thoughts. I agree with Johan that VEXXHOST is a great role model for contributing upstream, and we need more companies to follow suit.

I’m particularly pleased that Johan is willing to be the standard bearer for a concerted effort to get more folks involved.

“We have a huge community that relies on our collective efforts,” Johan says. “So, here’s the message we need to get out: ‘If you are a developer, please contribute. If you’re a DevOps manager or CxO, instill upstream contributions in your corporate culture.

“And to everyone in the Open Infrastructure community, I’d appreciate your help in spreading the word about the value of contributing. Let me know if you have ideas, questions, or would like to share your story about how contributing upstream has benefitted you and/or your organization.”

You can reach Johan at johan.christenson@citynetwork.eu. I hope you’ll reach out to him today.

The post Contributing to Open Infrastructure: Everybody Wins, So Let’s Get Started appeared first on Superuser.

by Mark Collier at August 03, 2020 11:00 AM

July 28, 2020

OpenStack Superuser

OpenDev Hardware Automation Recap and Next Steps

Last week, the OpenStack Foundation (OSF) held its second virtual OpenDev event. We’ve been amazed by how many people have joined us online to collaborate across different time zones. This OpenDev event focused on Hardware Automation including topics around hardware provisioning lifecycle for bare metal, bare metal infrastructure, networking and network security. Attendees shared various perspectives on the challenges, asked questions for how to improve, and identified next steps that the community can collectively collaborate on to ease these operator challenges.

This virtual event recruited over 400 participants located in more than 60 countries representing 200+ companies, who spent three days sharing their knowledge and discussing their experience of building and operating software that automates their hardware in the cloud environments.

EdZDLDuWoAERNzq
OpenDev brought developers and operators together to collaborate across boundaries.

Thanks to the OSF platinum and gold members who are committed to the Open Infrastructure community’s success. Also, thank you to the Programming Committee members: Mark Collier, Keith Berger, James Penick, Julia Kreger, Mohammed Naser. You helped to make these discussions possible!

If you didn’t tune in—or if you did and want a replaybelow is a snapshot of the conversations that took place, but I want to encourage you to check out the event recordings as well as the discussion etherpads found in the OpenDev event’s schedule to join the discussion.

Day 1: 

Mark Collier, OpenStack Foundation COO, kicked off the first day by explaining why the OSF chose Hardware Automation as one of the OpenDev topics. According to Help Net Security, the global data center networking market will reach $40.9 billion USD by 2025. Among our users, we’ve been seeing more complex hardware coming into the data centers such as ARM, GPUs for AI/machine learning and FPGAs that people have to manage.

OpenDev: Hardware Automation created an online space for community members to share their best practices and collaborate without boundaries. As more and more open source communities, such as Ironic, MAAS, tinkerbell and metal3, start to grow and solve these challenges, there is a huge demand for hardware automation. Ironic now has more code merged per day than in the history of OpenStack – showing that people want to work together on these problems more than ever! If you are interested in knowing more about OpenStack bare metal and how Ironic allows users to manage bare metal infrastructure, check out  the latest white paper from the OpenStack Bare Metal SIG “Building the Future on Bare Metal, How Ironic Delivers Abstraction and Automation using Open Source Infrastructure“.

Screen Shot 2020-07-24 at 2.21.45 PM (1)

Part One: Hardware Provisioning Lifecycle for Bare Metal

It’s common for users, regardless of their scale, to have a system to manage IT Asset Management (ITAM), Datacenter Infrastructure Management (DCIM), IP Address Management (IPAM), Configuration management database (CMDB). Part One of OpenDev’s topic on hardware provisioning lifecycle for bare metal was moderated by Mohammed Naser, VEXXHOST CEO and OpenStack Technical Committee (TC) chair. Mohammed kicked off the discussion by asking everyone to share how they organize data infrastructure & IP addresses inside their organization, what they wish they can do better and why they have not switched to automation. Later, community members from Verizon Media, China Mobile, CERN, SchwarzIT, VEXXHOST, and Red Hat have shared their various experiences on vendor selection, intake and deployment to the facility floor.

At the end of this discussion, participants signed up to collaborate further after the event in areas, such as collaborating on a set of SOPs/Documents on managing keys in a Trusted Platform Module (TPM) and creating a matrix of firmware installation processes per vendor per platform and building a common database of how to upgrade firmware automatically. If you are interested in discussing these topics and collaborating with the fellow operators, please sign up here, line 205 & line 217.

Part Two: Hardware Provisioning Lifecycle for Bare Metal

Part Two was moderated by James Penick, Verizon Media Architect Director. James continued the discussion on hardware provisioning lifecycle for bare metal, BIOS/firmware automation, how to keep the hardware secure, and how to detect attacks. The topics of this discussion included day-to-day consumption of hardware and power & thermal optimization automation. If you are interested in continuing the discussion on power affinity/grouping/weighting after the event, make sure to sign up on the etherpad, line 245.

As can be expected, this session included a lot of discussion about end-to-end hardware provisioning lifecycle for bare metal / cradle to grave for hypervisors. Check out the full discussion notes on OpenDev: Hardware Automation Day 1 etherpad, and watch the day 1 discussion recording.

Day 2: 

Part One: Bare Metal Infrastructure

Mohammed Naser returned as moderator and opened up the discussion on bare metal infrastructure by asking attendees on their own definition of “hyperconverge” to make sure everyone was on the same page. Arne Wiebalck, CERN Computing Engineer, gave two use cases that are considered as “hyperconverged” which are around massive storage systems that are developed in-house across thousands of servers and combining Ceph with each cell to achieve low latency IO for the VMs. Community members from China Mobile shared a use case on how different types of services can be converged within one type of hardware in the edge scenario. 

Later, community members dived into the discussions about autoscaled bare metal for cloud infrastructure, servers for serverless workloads. If you are interested in forming up a working group to look at some of these use cases and models, sign up here, line 206.

Part Two: Bare Metal Infrastructure

After a short break, Julia Kreger from Red Hat moderated the second half of the discussion on the topic of consuming bare metal infrastructure to provision cloud based workloads. Attendees from various companies gave use cases for turning ‘unused’ bare metal into cloud infrastructure orchestration. If you are interested in continuing the discussion on requirements regarding preemptable/bare metal workloads, please sign up here, line 246.

After the discussions on managing hardware using open standards such as Redfish and IPMI, it was apparent that many people are using both and interacting with their hardware. Questions such as why users care which protocol to use and what sort of issues people are encountering prompted people to share their experiences on how to make the job easier. Check out the full discussion notes on OpenDev: Hardware Automation Day 2 etherpad, and watch the day 2 discussion recording.

Day 3: 

Networking and Network Security

Under the umbrella of Hardware Automation, there is a wide variety of technologies, approaches, and solutions to networking. Open Infrastructure allows us to embrace these differences, and leverage our common ground. This discussion about networking was moderated by Mark Goddard, StackHPC Cloud Engineer with active speakers from China Mobile, Ericsson Software Technology, Verizon and more. The first half of the discussion was around network architectures and network automation. After a short break, in continuing with the theme of hardware automation, the attendees dug deeper into network security and how it relates to hardware and automation. 

The discussion about network security, moderated by Ian Jolliffe, Wind River Vice President Research and Development, explored questions such as how we are operationalizing the developer workflow to ensure network security in a dev ops world. Attendees shared their processes around automated firewall management as well as the security change management and tooling to do batch configuration or continuous configuration management of firewalls. Check out the full discussion notes on OpenDev: Hardware Automation Day 3 etherpad, and watch the day 3 discussion recording.

Wrap Up:

To wrap up, check out the etherpad that includes the OpenDev event feedback and follow up activities from the OpenStack Bare Metal SIG. We encourage you to continue the discussion at the Bare Metal SIG or sign up on the discussion etherpads in the coming weeks after the event. 

Screen Shot 2020-07-24 at 6.34.10 PM

Next Steps:

The goal with the OpenDev events is to extend this week’s learnings into future work and collaboration, so Jonathan Bryce and the event moderators, wrapped up the event to discuss next steps. These include:

Upcoming OpenStack Foundation (OSF) Events:

OpenDev:

Open Infrastructure Summit:

  • The annual Open Infrastructure Summit is going to be virtual (and free!). Register for the virtual Summit and join the global community at the virtual Open Infrastructure Summit on October 19-23 directly from your browser! 
  • The Call For Presentations (CFP) is open now! The CFP deadline is August 4 at 11:59pm PT, so start drafting your presentations and panels around Open Infrastructure use cases like AI/Machine Learning, CI/CD, Container Infrastructure, Edge Computing and of course, Public, Private and Hybrid Clouds.

The post OpenDev Hardware Automation Recap and Next Steps appeared first on Superuser.

by Sunny Cai at July 28, 2020 07:00 PM

StackHPC Team Blog

Kayobe & Kolla - sane OpenStack deployment

Coming up at the London and Manchester (virtual) OpenInfra meetup on Thursday July 30th 6pm-9pm UK time (17:00-19:00 UTC): Mark Goddard, Kolla PTL and StackHPC team member, will be talking on "Kayobe & Kolla - sane OpenStack deployment".

Mark at RCUK Cloud Workshop 2019

In this talk Mark will introduce Kayobe, the latest addition to the OpenStack Kolla project. Learn how Kayobe uses Bifrost to support bare metal provisioning, and extends Kolla Ansible to offer an end-to-end cloud deployment tool.

Mark will be joined by:

  • Ildikó Vancsa and Gergely Csatari, who will present on Edge ecosystem, use cases and architectures.
  • Belmiro Moreira will present on 7 years of CERN Cloud - From 0 to 300k cores

Add to your calendar.

If you're interested in finding out more about OpenStack and Kayobe, check out our OpenStack HIIT training courses.

Get in touch

If you would like to get in touch we would love to hear from you. Reach out to us via Twitter or directly via our contact page.

by Stig Telfer at July 28, 2020 09:00 AM

July 27, 2020

OpenStack Superuser

Submission Tips for the Virtual Open Infrastructure Summit

The world of Open Infrastructure is broad. At the Open Infrastructure Summits, presentations cover 30+ open source projects, diverse use cases like 5G, container orchestration, private cloud architectures, and CI/CD across a large set of industries. This is my first Open Infrastructure Summit, so I know how daunting it can be to look at the set of topics and wonder where to begin.

If you haven’t heard, the upcoming Summit hosted by the OpenStack Foundation (OSF) will be virtual. The great news is that this now opens up the opportunity to speak (and attend!) for folks who have never been able to travel to an in person Summit—like me!

To help folks who are submitting sessions for the first time—or those returning who need a fresh idea—we have asked the Summit Programming Committee to share specific topics within their Tracks on what kind of presentations they want to be submitted.

The deadline to submit a session is Tuesday, August 4 at 11:59pm PT—see what this means in your time zone.

5G, NFV & Edge Computing

  • Bare metal provisioning and pre-deployment validation are a couple of edge topics that need attention.
  • Real-world use cases and stories about how 5G and Edge Computing are being deployed and changing how people are approaching communications, industrial or health care.
  • Where the edge can be taken, provide clarity on definitions, and where the future is.
  • 5G, NFV, Edge – Impact of 5G Rel16, ETSI NFV4, cloud-native micro data centers efforts on:
    • Managed Container Infrastructure Object (MCIO) Pods
    • Interfaces IFA040 and APIs Changes for MANO Orchestrator & VIM framework including OpenStack CNTT Rx1s and Kubernetes Rx2s.
    • Changes to VNF, VNFC, CP a Tosca/Helm templates/Charts for VNFD, VDU, CPD schemas, & CRDs. Updating Config, FCAPS on Deployments, Deamonsets, Clusters, Namespaces starting Bare Metal hosts, nodes, Devices like GPU, FPGA, NVME, and other heterogeneous components.
    • Open Infrastructure role in 5G core to Edge and Edge to Nomadic to Mobile Gateways and Consumer & Enterprise for 5G, IoT, and Edge use cases.
    • Performance optimization for latency-sensitive, bandwidth sensitive and compute-intensive network functions including service function chaining, slicing, mapping, placements.
    • Edge platform automation and scheduling for parallel life cycle management along with local loop monitoring and controls.
    • Data-driven architectures, design, and enabled through hybrid VNF and Container workloads.
    • Effect of merchant Silicon for NPU, GPU, DSPs in programmable solutions on in-line accelerations, and user/kernel offloads for latency optimizations.
    • Language (py3.8, golang 1.4, DPC++, Java18, SYCL) Compilers and Dynamic Builds and Multi-CI and Mid-Stream integrations+Operations for both Upstream codes and Downstream deployments for Edge, NFV and 5G stacks.
    • Impact of pandemic & glocalization of the supply chain on Open technology for different Edge, NFV & 5G, Markets.”
  • Edge computing Stories
  • Typical use cases for different industries
    • telco
    • manufacturing
    • financial banking
    • others
  • How users use software/solutions about edge computing?
    • StarlingX
    • KubeEdge
    • K3s
    • Arkraino
    • others
  • What are the biggest challenges for building an edge cloud?
    • security
    • networking
    • small footprint
  • The relationship between 5G and edge computing
  • How to build a mobile edge computing (MEC) platform for telco to support 5G
  • Edge application innovations
  • How to optimize the edge cloud to meet the requirement, such as low latency and high bandwidth?
  • AI in edge computing environments
  • Edge computing solution based on OpenStack, NFV issues when big scalable deployment, container-based VNF application running on OpenStack system.

Programming Committee: 

  • David Paterson, Sr. Principal Software Engineer, Dell Technologies
  • Ian Jolliffe, VP R&D, Wind River
  • Prakash Ramchandran, Technical Staff, Dell
  • Shuquan Huang, Technical Director, 99Cloud
    Xiaoguang Zhang, Project Manager, China Mobile

Artificial Intelligence (AI), Machine Learning & High Performance Computing (HPC)

  • Submissions that addresses (but not limited to the following) scenarios on an OpenStack cloud:
    • Implementing a test bench or frameworks for running HPC  workloads for scientific research.
    • Benchmark various machine learning algorithms, which show performance metrics on various use cases.
    • How to prevent/eliminate biases in AI/machine learning predictions.
    • Applying AI/machine learning algorithms in the healthcare domain such as predicting cancer for early treatment.
    • Using the machine learning technique to prevent financial frauds
    • Using machine learning algorithms such as NLP and supervised learning to improve the software development process and quality.
    • Implementing AI/machine learning technique to maintain social distances or related preventive mechanisms in fighting COVID-19.
  • The challenges of expanding and scaling a cloud while maintaining availability.
  • Maximizing the performance of workloads on the cloud
  • The use of GPUs and FPGAs in the cloud.
  • Innovative uses of machine learning both in and on the infrastructure, as well as basic topics that introduce users to AI and machine learning’s particular infrastructure quirks.

Programming Committee:

  • Armstrong Foundjem, Researcher, École Polytechnique montréal
  • Alexander Dibbo, Cloud Architect, Science, and Technology Facilities Council (UKRI)
  • Hector Augusto Garcia-Baleon
  • Nick Chase, Head of Technical Content, Mirantis

CI/CD

  • CI/CD innovation to support scenarios, including telco, edge, public and private cloud.
  • Major technical development to address challenges in modern CI/CD
  • CI/CD for cross-vendor/cross-community/cross-toolchain scenarios
  • How reducing test duration can help to improve developer velocity on OpenStack itself.
  • What consumers of OpenStack are doing with OpenStack for their CI.
  • Sharing the knowledge from their CI/CD downstream or adjacent upstream among different communities.
  • How we can speed up the execution time and more important things are stability so that we keep the development process more smooth, less-breaking, and quality verification.

Programming Committee: 

  • Chris MacNaughton, Software Engineer, Canonical
  • Ghanshyam Mann, Cloud Consultant, NEC
  • Qiao Fu, Technical Manager, Chine Mobile

Container Infrastructure

  • Next-generation container orchestration – a world post-Kubernetes…
  • Securing containers
  • Serverless
  • Innovation technologies and practice in a container and cloud-native area, including container runtime, image building and delivering improvement, container networking, container practice in large organizations, public cloud services, or in the edge computing environment, etc.
  • How the telecommunication industry is evolving into cloud-native.
  • Gap analysis between current implementation and telco requirements on container infrastructure/cloud-native platform.

Programming Committee:

  • Matt Jarvis, Director of Community, D2iQ
  • Qihui Zhao, Project Manager, China Mobile
  • Xu Wang, Senior Staff Engineer, Ant Financial Services Group

Getting Started

  • How to get started in contributing to whether code or docs. Getting involved from the operations side. Diversity and inclusion talks about getting started and being a part of the community
  • Talks that break down complex components of OpenStack
  • Talks that introduce to users/developers on how to adapt and contribute.
  • First-time speakers showing their own experiences adopting OpenStack, the challenges and their tips on addressing them
  • Developers from the community presenting on how new users can contribute upstream
  • Speakers from the OpenStack governance talking about how to involved and how to seek help (communication channels ….)
  • A talk showing a user’s use-case of migrating from commercial platforms generically to OpenStack
  • Help more people know OpenStack and 4 opens.

Programming Committee: 

  • Amy Marrich, Principle Technical Marketing Manager, Red Hat
  • Mohamed Elsakhawy, Operational Lead / System Administrator III, Compute Canada / SHARCNET
  • Zhiqiang Yu, Open Source Program Manager, China Mobile Research Institute

Hands-on Workshops

  • Material that is engaging and in an environment that promotes questions and discussions.
  • “Taste of tech” – hands-on sessions designed to allow the experience of new and emerging tech without the intricacies of installation and deployment.
  • Advanced sessions – troubleshooting, scaling, CI/CD demonstrations, maybe even sessions on application modernization and migration, ie, the monolithic to container model.

Programming Committee: 

  • Keith Berger, Senior Software Engineer, SUSE
  • Mark Korondi
  • Russell Tweed, Infrastructure Architect, Red Hat

Open Development

Here is what previous Planning Committees looked for:

  • Content showcasing the power of the “Four Opens
  • What is open development
  • Why do we need open development
  • How does open development work
  • What is the relationship between open development and Conway’s law
  • How community success relates to a successful development
  • How to participate in open development in modern open-source software projects

Programming Committee: 

  • Meghan Heisler, AT&T

Private & Hybrid Cloud

Here is what previous Planning Committees looked for:

  • Private cloud and hybrid cloud implementation success stories, from small to large-scale (20 to 1,000 nodes), especially for industries with stringent requirements such as financial services and government
  • Different methods to deploy OpenStack;
    • How to scale and what, if any, are the bottlenecks?
    • Large deployment references
    • Maintaining cloud deployments
    • User stories, devs updates
    • New approaches for services integration
  • Cross-community collaboration
  • Presentations from actual superusers (providing user experience, testing and upstream at the same time)
  • OpenStack operational challenges

Programming Committee: 

  • Belmiro Moreira, Cloud Architect, CERN
  • Narinder Gupta, Lead Partner Solutions, Canonical
  • Kenneth Tan, Executive, Sardina Systems
  • Danny Abukalam, Solutions Architect, SoftIron

Public Cloud

  • Content for operators of public clouds as well as end user-driven content.
  • For operators, it’s super interesting to hear how others have solved things like billing, image life cycle management, monitoring, and auto-healing, etc, operator stuff that comes super important to keep track of your cloud(s).
  • Talks on how to utilize OpenStack in a great way, what tools are out there, how can I use them, etc, could be tools like Terraform for example.
  • Use case driven talks on how other end users are designing their infrastructure, how they automate that, how they leverage Kubernetes in OpenStack etc.
  • Adaption of OpenStack clouds with other frameworks like Kubernetes, cloud resource selling to end customers, global cloud initiative.
  • Adoption of containers with ease of use
  • The future of OpenStack and public cloud co-existence
  • Multi-cloud and ease of application portability
  • Domain-specific computing
  • Case studies on adoption and knowledge sharing
  • Cloud in the emerging world and how others can help them catch up

Programming Committee: 

  • Frank Kloeker, Deutsche Telekom AG
  • Krishna Kumar, Cloud Architect Sr., Accenture
  • Tobias Rydberg, Senior Developer, City Network

Security

  • Security content providing practical strategies to mitigate cloud security vulnerabilities and strengthen security posture for cloud users.

Programming Committee: 

  • Ashish Kurmi, Senior Cloud Security Engineer II, Uber Technologies

The post Submission Tips for the Virtual Open Infrastructure Summit appeared first on Superuser.

by Helena Spease at July 27, 2020 11:09 PM

OpenStack Blog

10 Years of OpenStack – Alan Clark at SUSE

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at July 27, 2020 03:00 PM

July 23, 2020

OpenStack Superuser

Inside open infrastructure: The latest from the OpenStack Foundation

Spotlight on: 10 years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. OpenStack, one of the top three most active open source projects, marks its 10th anniversary this month. Thank you for the 10 incredible years of contribution and support.

9fbeee0d69c09abf9c0ff697_882x616

Vietnam Open Infrastructure User Group celebrating 10 years

Launched in 2010 as a joint project between Rackspace and NASA, OpenStack started as two open source projects to orchestrate virtual machines and object storage. It has evolved into one of the three most active open source projects in the world.

Supported by a global community of over 105,000 individuals, the OpenStack project has accomplished several big milestones:

  • 21 on-time releases, from “Austin” to “Ussuri”
  • 451 Research projects a $7.7 billion USD OpenStack market by 2023, citing the most growth in Asia (36%), Latin America (27%), Europe (22%), and North America (17%).
  • From two projects in 2010 to 42 projects in 2020
  • Over 10 million cores in production
  • 500,000+ changes merged
  • 8,000+ individual developers authoring those changes
  • Every day, 900 changes are proposed and 18,000 tests are run to evaluate them

The community has hosted a 10 years of OpenStack virtual birthday celebration today at 8 am PT (1500 UTC). Watch the birthday celebration recording here.

Check out the 10 years of OpenStack blog and celebrate the incredible 10 years with us!

OpenStack Foundation news

Airship: Elevate your infrastructure

  • Congratulations to the 2020 Airship committee members elected last month:
    • Alex Bailey
    • Alexander Hughes
    • Alexey Odinokov
    • Jan-Erik Mangs
    • John (JT) Williams
  • The Technical Committee is excited to announce the 2020-2021 Working Committee election! This election will follow the same two-week cycle that the Technical Committee election followed last month.
    • Nomination Period: 20-26 July
    • Voting Period: 27 July-02 August
  • As the community grows and new contributors want to get involved, we hope to have a consistent definition to help familiarize them with the project. Contribute to the Airship glossary by defining and adding terms that we can use to educate future contributors

Kata Containers: The speed of containers, the security of VMs

  • The community is planning to set up a meeting to review and clean the Kata open issues backlog. The idea is then to add additional labels to the cleaned backlog for identifying the high priority ones being actively worked on and track them closely. We can then help focus on completing those issues and unblocking them if needed. Here is a doodle poll with 5 proposed time slots for next week: 
  • We have just released Kata Containers 2.0.0-alpha21.12.0-alpha01.11.2, and 1.10.6 releases. The 1.10.6 and 1.11.2 stable releases include the latest stable fix backports and users are encouraged to upgrade.
  • Check out The Road to Kata Containers 2.0 on The New Stack!
  • As the community grows and new contributors want to get involved, we hope to have a consistent definition to help familiarize them with the project. Contribute to the Kata Container glossary by defining and adding terms that we can use to educate future contributors.

OpenStack: Open source software for creating private and public clouds

  • The Victoria development cycle is now well underway. With more than 10,000 changes already merged, we are past the victoria-1 milestone, headed toward the victoria-2 milestone on July 30, for a final release planned October 14, 2020.
  • Starting August 1st, the entire OpenStack community will be represented by a single elected body, including developers, operators, and end-users of the OpenStack software, and inclusive of all types of contributions to the project. The User Committee will no longer operate as a separate entity from the Technical Committee. Read the updated UC charter for more details.
  • As the community grows and new contributors want to get involved, we hope to have a consistent definition to help familiarize them with the project. Take a look at the current OpenStack glossary and add missing terms or definitions to the etherpad that we can use to educate future contributors.

StarlingX: A fully featured cloud for the distributed edge

  • The StarlingX community is now in the final testing and bug fixing phase of the 4.0 release cycle. The release is planned to come out in a few weeks.
  • As the community grows and new contributors want to get involved, we hope to have a consistent definition to help familiarize them with the project. Take a look at the current StarlingX glossary and add missing terms or definitions to the etherpad that we can use to educate future contributors.

Zuul: Stop merging broken code

  • We’re looking for folks to help draft a Wikipedia entry about Zuul, so if you’re interested please lend a hand.
  • As the community grows and new contributors want to get involved, we hope to have a consistent definition to help familiarize them with the project. Take a look at the current Zuul glossary and add missing terms or definitions to the etherpad that we can use to educate future contributors.

Check out these Open Infrastructure Community Events!

For more information about these events, please contact denise@openstack.org

Questions / feedback / contribute

This newsletter is written and edited by the OSF staff to highlight open infrastructure communities. We want to hear from you! If you have feedback, news or stories that you want to share, reach us through community@openstack.org . To receive the newsletter, sign up here.

The post Inside open infrastructure: The latest from the OpenStack Foundation appeared first on Superuser.

by Helena Spease at July 23, 2020 06:00 PM

July 21, 2020

OpenStack Superuser

Thank You to the Last Decade, Hello to the Next!

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support.

Wow, 10-years’ today was when OpenStack was first launched. Many amazing tech milestones happened in 2010. Steve Jobs launched the first iPad. Sprint announced its first 4G phone. Facebook reached 500 million users. For us perhaps, one of the most important milestones was the birth of the OpenStack project in 2010.

In real time, the pace of change in the tech industry often feels glacial, but looking at things over a 10-year span, a lot of stark differences have emerged since 2010. So before you plug back your AirPods, fire up Fornite and watch a new show on Disney+, let’s take a look at how OpenStack has transformed the open source industry in the past 10 years.

The Decade Challenge—OpenStack Edition

What began as an endeavor to bring greater choice in cloud solutions to users, combining Nova for compute from NASA with Swift for object storage from Rackspace, has since grown into a strong base for open infrastructure. In 2010, three years before the OpenStack Summit Hong Kong, “the cloud” was barely a thing. Having a standardized, open source platform for public and private clouds was a dream. Now OpenStack is not only the de facto standard, but also a massive market that critical industries rely on.

Looking back to OpenStack in 2010, we were ecstatic to celebrate our first year of growth from a couple dozen developers to nearly 250 unique contributors in the Austin release (the first OpenStack release). Fast Forward to 2020, OpenStack ranks among the top three most active open source projects in the world, and is the most widely deployed open source cloud infrastructure software. Developers from around the world work together on a six-month release cycle with developmental milestones. 10 years in, OpenStack has had

  • 21 on-time releases in 10 years, from “Austin” to “Ussuri”
  • 451 Research projects a $7.7 billion USD OpenStack market by 2023, citing the most growth in Asia (36%), Latin America (27%), Europe (22%), and North America (17%).
  • From two projects in 2010 to 42 projects in 2020
  • Over 10 million cores in production
  • 500,000+ changes merged
  • 8,000+ individual developers authoring those changes

Every day, 900 changes are proposed and 18,000 tests are run to evaluate them.

Last year was a busy year for the open infrastructure community. The community combined 58,000 lines of code changes to support the output of more open source infrastructure software. In the same year, with 100,000 members and millions of visitors to the OpenStack website (check out the new homepage that we just launched, yay!), the community has made significant progress in supporting the goal of OpenStack’s $7.7 billion commercial market size worldwide and the future of over $12 billion OpenStack and container market by 2023.

The achievements of the global community in the past decade have been phenomenal. We are honored to work with many enthusiastic and talented individuals from all around the world to achieve the same goal, and also work together to build and operate open infrastructure.

Top 10 Favorites of OpenStack

There are so many milestones to celebrate in the past 10 years of OpenStack with our community. Here we have gathered some of the top 10 favorites of OpenStack from the community members around the world:

#ForTheLoveOfOpen

Throughout OpenStack’s history, we have been committed to creating an open culture that invites diverse contributions. Looking back to 10 years ago, we didn’t have enough experience to define what the “openness” means to us. In 2010, we described open source with the freedom to innovate, consume and redefine. Even though we didn’t have our “perfect” definition on “open” and “open source”, we drafted three commitments to our community in 2010:

COMMITMENT #1: We are producing truly open source software.

COMMITMENT #2: We are committed to an open design process.

COMMITMENT #3: All development will be done in the open.

OpenStack was started with the belief that a community of equals, working together in an open collaboration, would produce better software, more aligned to the needs of its users and more largely adopted. It was therefore started from day 0 as an open collaboration model that includes as many individuals and organizations as possible, on a level playing field, with everyone invited to design open infrastructure software. It was from these conditions that “The Four Opens” were born:

  • Open Source
  • Open Design
  • Open Development
  • Open Community

After 10 years, the Four Opens have proved very resilient, consistently managing to capture the OpenStack way of doing upstream open source development. They are instrumental in the success, the quality and the visibility of the OpenStack software.

Collaboration without Boundaries

Stackers also collaborate without boundaries. Truly representative of cross-project collaboration, this Open Infrastructure umbrella now encompasses components that can be used to address existing and emerging use cases across data center and edge. Today’s applications span enterprise, artificial intelligence, machine learning, 5G networking and more. Adoption ranges from retail, financial services, academia and telecom to manufacturing, public cloud and transportation.

We have too many examples of community collaborations. Here is a list of the OpenStack cross community collaboration and integration highlights in the past year:

Where Are They Now 

Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make OpenStack and the global Open Infrastructure community successful. Check out the stories from Tim Bell, Yumeng Bao and Prakash Ramchandran on how they got started with OpenStack and their favorite memories from the last 10 years of OpenStack.

Stay tuned for more content from the community members on the OpenStack stories throughout the year!

Thank You

None of it would be possible without the consistent growth of the OpenStack community. The momentum of growth is not slowing down. Today, we are not only celebrating our community’s achievement for the past 10 years, but also looking forward to the continuous prosperity of the community in the next 10 years.

Thanks to the OpenStack Foundation platinum members, gold members and corporate sponsors who are committed to the Open Infrastructure community’s success. Check out the blogs from China Mobile, Ericsson and Red Hat on what 10 years of OpenStack meant to them and how they have contributed to OpenStack in the past decade.

Check out the video from Red Hat about 10 years of OpenStack:

Stay tuned for more content from the ecosystem companies on what 10 years of OpenStack means to them throughout this year!

Looking Forward

“When the OpenStack project launched in 2010, few people imagined use cases for cloud as diverse as edge, containers, AI and machine learning,” said Jonathan Bryce. It’s crazy to see how far we’ve come for the past 10 years. Let’s remember how we got here — and realize that many of our milestones today may well be a fuzzy memory come 2030. Let’s build on the experience of the last years to continue to grow our software communities. Here’s to another exciting 10 years together as a community.

Get Involved

Follow the #10YearsofOpenStack hashtag on Twitter and Facebook and share your favorite memories of OpenStack with us!

Follow us on

Twitter: twitter.com/OpenStack

Facebook: facebook.com/OpenStack

WeChat ID: OpenStack

The post Thank You to the Last Decade, Hello to the Next! appeared first on Superuser.

by Sunny Cai at July 21, 2020 03:00 PM

VEXXHOST Inc.

How To Choose An Open Source Project

In the world of software development, there is a progressive division, better known as the open source community. The open design and development concept helps everyone through a multitude of available open source projects. Consequently, users don’t have to create projects that have already been developed by another community member.

With a mix of developers contributing to open source projects, there is a variety of experience and expertise clubbed together. Therefore, tinkering projects multiple times makes them more secure and robust. However, not all open source projects are created equal.

These projects vary in terms of security and agility. Thus, choosing the correct one for your use case can get tricky. To help you work out this problem, you can focus on the following key areas:

Users

User statistics are a great indicator of how good an open source project is. If a large number of people are using the project, it is usually a good sign. Other indicators are the number of downloads, reviews, comments, contributors, forks etc.

The important thing is to deep dive into who the users are. Is the development team using their own project? That is a good sign for you to pick it up.

Builders

Are the contributors participating avidly? Pushing changes to open source projects regularly is essential. If the developer team or other community members are not actively involved in updating and improving the project, then it is not worth choosing.

The stakeholder activity associated with an open source project is an excellent indicator of its relevancy. So keep an eye out for that!

Documentation

The open source project you want to adopt or implement must come up with regular release updates. Changelogs show project upgrades and the improvements that came into play with every change. Project lifecycle activity and documentation help determine how secure and updated the code is.

Moreover, the documentation must be well written and easy to understand to deem it an adequate open source project. Readable code is simpler to build upon, secure and fix.

Compatibility

Amid your hunt, do not lose track of your goals. The project in question must be in line with the goals you are trying to achieve. You must be mindful of certain aspects to determine compatibility between the two. They are technologies in use, licensing and code languages.

Using Open Source Project

It is not necessary to find the exact match for your use case. But that’s where the flexibility of open source is of significance. You can still utilize the project with modifications of your own.

You must be responsible for your copy of the project. Moreover, if you are an upstream contributor, do not fret about improving on it. The value of the open source community is to give back. So maintaining the health of the project in use is crucial.

VEXXHOST and Open Source

VEXXHOST has long been involved with the open source community and is an active member of the OpenStack Foundation. Among the plethora of open source projects available out there, we provide Managed Zuul and Kubernetes Enablement as solution offerings.

Know more about our history with OpenStack and our involvement with the community here.

Would you like to know more about Zuul? So download our white paper and get reading!

How to up your DevOps game with Project Gating

How to Up Your DevOps Game with Project Gating:
Zuul – A CI/CD Gating Tool

The post How To Choose An Open Source Project appeared first on VEXXHOST.

by Samridhi Sharma at July 21, 2020 01:42 PM

July 20, 2020

VEXXHOST Inc.

Looking Back At OpenStack’s Ten Year Journey

As OpenStack turns 10, VEXXHOST is here to honor the presence of the community by revisiting some fun and impactful milestones over the years.

OpenStack’s journey began in 2010 – shoutout to RackSpace and Anso Labs for converging their efforts ten years ago which formed the base of OpenStack. If it weren’t for their grit and passion to open source the code, we wouldn’t be where we are today.

OpenStack year 0
openstack year 1

Well, VEXXHOST wasn’t too late in joining in on the fun. With the second release of OpenStack, Bexar, that came out nine years ago, VEXXHOST adopted OpenStack as a base for our cloud infrastructure services. It was definitely a special year for us as we have grown alongside OpenStack ever since!

Two years into OpenStack, the OpenStack Foundation was formed. As an attempt to empower, protect and add more shared resources to OpenStack, a vibrant open-source community came into place eight years ago. Being a member of the Foundation has been such an honour and pleasure for the VEXXHOST team.

OpenStack year 2
OpenStack year 3

Neutron, OpenStack’s networking as a service, took a front seat among OpenStack projects seven years ago. Feels like just yesterday, but with new releases and upgrades made available by OpenStack, Neutron is an essential part of OpenStack cloud infrastructure.

OpenStack’s path and presence were soon being felt worldwide. Six years ago the first OpenStack Design Summit took place in Paris. It is notable as the inaugural event for the SuperUser Award ceremony. It would be hard not to flex about VEXXHOST winning the same award in 2019 at the Denver Summit.

openstack year 4
OpenStack year 5

The global network of OpenStack gained traction, and the community, along with the Foundation staff, made great strides toward cloud interoperability. The mission began five years ago and our team is proud to have supported the OpenStack community in driving adoption by launching the first interoperability testing program for OpenStack Powered products, including public clouds and distributions.

Let’s not forget the first OpenStack Days, from four years ago, that took place in New York. The first of many OpenStack Days that brought the community closer all over Canada, Australia, and Tokyo, to name a few of the many countries where OpenStack is now deployed.

OpenStack year 6
openstack year 7

Three years ago, the OpenStack Foundation spread its wings and embraced some strategic focus areas like container infrastructure, CI/CD and Edge Computing. OpenStack Foundation kick-started the initiative with pilot projects in these areas that are now better known as Kata Containers, Zuul and the Edge Computing Group.

Soon after, OpenStack came out with its 18th release Rocky. This was a significant milestone for the VEXXHOST team as we were running the release on the day of the launch itself, two years ago. We have maintained our momentum by doing the same for OpenStack’s 19th and 20th releases, too!

openstack year 8

No one is a stranger to the role Ironic, the bare metal project, plays to OpenStack deployments. It was one year ago that the OpenStack Foundation launched the OpenStack Bare Metal Program to bring more attention to its usage in an Openstack cloud and encourage collaboration.

The past ten years have been instrumental in making the Openstack community what it is today. Every stakeholder of OpenStack has walked the path of collaborative development and we are all extremely excited to be here today to celebrate OpenStack’s big day. With immense gratitude, the VEXXHOST team would like to wish OpenStack a very HAPPY BIRTHDAY!

Like what you’re reading?
Deep dive into a hands-on ebook about how you can build a successful infrastructure from the ground up!

The post Looking Back At OpenStack’s Ten Year Journey appeared first on VEXXHOST.

by Samridhi Sharma at July 20, 2020 04:13 PM

July 17, 2020

VEXXHOST Inc.

10 Years Of OpenStack – With OpenStack!

OpenStack completed a decade around the sun, and we couldn’t be more thrilled to have walked alongside the vibrant community for nine long years. So, we are taking this trip down memory lane of how it all began for us and how far VEXXHOST has come with OpenStack!

The open source platform was created to provide public and private cloud services to enterprises of all sizes. The Four Opens: Open Source, Open Community, Open Development and Open Design, are the foundation stones for the OpenStack Foundation. It was these fundamentals that drove us to become a part of the community in 2011. Since then, VEXXHOST has been an avid contributor and user of OpenStack ever since. We went on to become both infrastructure donors and corporate members of the OpenStack Foundation.

OpenStack Services & Solutions

With our involvement in the community, we got started with Public and Private cloud services. We deliver consistent performance throughout our OpenStack public cloud. With no vendor lock-in hassles, one can make use of 13 Openstack projects ranging from networking to block storage, identity management to container orchestration and so much more! Through our Private Cloud service, we offer a customizable experience that can be tailored to fit any business needs. We provision the availability of bare metal, virtual machine and kubernetes all in one cloud environment.

As VEXXHOST captured interest among users, we expanded on our OpenStack related solutions through consulting and upgrades offering. Moreover, we have even had the chance to venture outside of the OpenStack software and explore other projects under the OpenStack Foundation, such as Zuul. The CI tool has been tested against our cloud and is a part of our solution offerings as well.

OpenStack has genuinely formed a well-knit community that is always working to improve the product and its ecosystem. That in itself is something worth celebrating.

Right from the second release, Bexar, our journey began as an Openstack based IaaS provider. We are now on Openstack’s twentieth release, Ussuri, and VEXXHOST has been among some of the first to offer it as part of OpenStack Upgrades solution and as part of our Private cloud service.

Events and Interactions

VEXXHOST at OpenStack events

OpenStack has helped us get closer to our users and the community through events, summits and meetups. We have received some astounding exposure as members of the community. You can say we have travelled the world together, from Boston to Sydney, Berlin to Shanghai! It was because of OpenStack that we had the chance to speak at CERN, during an OpenStack Days event. Moreover, by being awarded the Superuser award in 2019, we received the same recognition from the community.

OpenStack itself has evolved plenty since 2010. The vast innovations in OpenStack have enabled great flexibility in cloud computing. Openstack is the leading open source option for public cloud environments. We are also members of the OpenStack Passport Program, allowing us to provide resources at scale to those who wish to test open source projects actively.

Even for private cloud environments, OpenStack has proven to be highly customizable. Be it bare metal, virtual machines or containers, your OpenStack environment can harness the power of all three at once! Furthermore, if you are looking for more affordable alternatives, OpenStack also supports hyper-converged infrastructure.

With the power of OpenStack, VEXXHOST has gone from local to global. We have gained the trust of enterprise clients all over the world. It is our pleasure to be a part of OpenStack when they cross this milestone. The entire VEXXHOST team congratulates OpenStack and its community on their big day, and we express our gratitude for being a part of it!

Like what you’re reading?
Deep dive into a step-by-step guide about how you can bulletproof your cloud strategy with OpenStack!

The post 10 Years Of OpenStack – With OpenStack! appeared first on VEXXHOST.

by Samridhi Sharma at July 17, 2020 05:22 PM

July 16, 2020

Ed Leafe

Day 52: Happy 10th Birthday, OpenStack!

I just saw this announcement from the OpenStack Foundation about OpenStack’s 10th birthday! Yes, 10 years ago this week was the first OpenStack Summit, in Austin, TX, with the public announcement the following week at O’Rielly OSCON. Yet most people don’t know that I played a very critical role in the beginning! OpenStack began as … Continue reading "Day 52: Happy 10th Birthday, OpenStack!"

by ed at July 16, 2020 08:45 PM

OpenStack Blog

Thank You to the Last Decade, Hello to the Next!

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support.  Wow, 10 years. Many amazing tech milestones happened in 2010. Steve Jobs launched the first iPad. Sprint announced its first 4G phone. Facebook reached 500... Read more »

by Sunny at July 16, 2020 02:00 PM

Favorite OpenStack Swag—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered people’s favorite OpenStack swag... Read more »

by Sunny at July 16, 2020 11:00 AM

First OpenStack Event—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we gathered the first OpenStack events from... Read more »

by Sunny at July 16, 2020 11:00 AM

Most Unusual Place That an OpenStack Deployment Has Been Built—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered the most unusual place... Read more »

by Sunny at July 16, 2020 11:00 AM

Favorite Cities You’ve Visited for OpenStack—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered community members’ favorite cities... Read more »

by Sunny at July 16, 2020 11:00 AM

Predictions of OpenStack—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered community members’ predictions on... Read more »

by Sunny at July 16, 2020 11:00 AM

First Contribution—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we gathered the first contribution from the... Read more »

by Sunny at July 16, 2020 11:00 AM

Most Memorable OpenStack Moments—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered the most memorable OpenStack... Read more »

by Sunny at July 16, 2020 11:00 AM

Most Used Features In OpenStack—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we have gathered the most used features... Read more »

by Sunny at July 16, 2020 11:00 AM

Favorite OpenStack Events—10 Years of OpenStack

Millions of cores, 100,000 community members, 10 years of you. Powering the world’s infrastructure, now and in the future. Thank you for the 10 incredible years of contribution and support. There are so many milestones to celebrate in the past 10 years of OpenStack with the community. Here we gathered people’s favorite OpenStack events from... Read more »

by Sunny at July 16, 2020 11:00 AM

10 Years of OpenStack – Prakash Ramchandran at Dell Technologies

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at July 16, 2020 10:58 AM

10 Years of OpenStack – Yumeng Bao at ZTE Corporation

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at July 16, 2020 10:55 AM

10 Years of OpenStack – Tim Bell at CERN

Happy 10 years of OpenStack! Millions of cores, 100,000 community members, 10 years of you. Storytelling is one of the most powerful means to influence, teach, and inspire the people around us. To celebrate OpenStack’s 10th anniversary, we are spotlighting stories from the individuals in various roles from the community who have helped to make... Read more »

by Sunny at July 16, 2020 10:50 AM

July 12, 2020

Julio Villarreal Pelegrino

An introduction to Enhanced Platform Awareness (EPA) capabilities in OpenStack

An introduction to Enhanced Platform Awareness (EPA) capabilities in OpenStack

One of the main topics of discussion when architecting an OpenStack cloud to support Telco workloads is around Enhanced Platform Awareness (EPA) capabilities. After all, for an NFV platform to be useful, its virtualized functions must meet or exceed the performance of physical implementations. Some of the EPA features that contribute to the VNF performance are described briefly in this blog post and are fully supported by OpenStack.


Non-Uniform Memory Access (NUMA)

In previous hardware architectures (back in the early days of computing), CPUs ran slower than their memory. That changed considerably with the introduction of more modern architectures, where CPUs operate considerably faster than the memory they use. In this new paradigm, with faster CPUs, if all memory is equally accessible, the result will be that memory access times will most likely be the same regardless of which CPU in the system operates. This design is called Uniform Memory Access (UMA).  Then, the need for a new design: Non-Uniform Memory Access (NUMA).

In NUMA,  system memory is divided into zones, called nodes, allocated to particular CPUs or sockets. Access to memory local to a CPU is faster than memory connected to remote CPUs on that system. In a NUMA system, each CPU socket will have access to a local memory node faster than a memory in a node that is local to another CPU socket or memory on a bus shared by all CPUs.  According to multiple testing for high-performance applications, NUMA misses impacts performance significantly, generally starting at around 10% performance hit or higher.

To mitigate performance degradation, OpenStack compute service (nova) should be configured to make smart scheduling decisions when launching instances to take advantage of NUMA topologies. The main objective is simple, to avoid the penalties associated with CPUs accessing memory that are not local to them.

In the following image, we see a Virtual Machine running without NUMA. In this example, the virtual instance is getting with vCPU (virtual CPU) from CPU_0 in Socket_0 and accessing memory segments local to CPU_1 in Socket_1. The systems need to cross the UPI (Intel Ultra Path Interconnect) and get penalized performance-wise.

An introduction to Enhanced Platform Awareness (EPA) capabilities in OpenStack

TIP: Not all instances require NUMA to be enabled; these configurations should be used only when the application requires it. Most general type workloads will perform well without NUMA enabled in the system.

In this second image, we see a system that is consuming vCPU and Memory resources from the same CPU. In this topology with NUMA enabled, UPI is not used to access system memory, as the resources are local to the CPU.

An introduction to Enhanced Platform Awareness (EPA) capabilities in OpenStack

CPU Pinning

The main objective of CPU pinning (sometimes called Processor affinity or cache affinity) is to run a process (in our case, a Virtual Machine) on a specific physical CPU in a given host. Since virtual machines run as userspace tasks on the host operating system, pinning increases cache efficiency. Combining vCPU pinning with numatune can avoid NUMA misses.

By enabling CPU pinning, the operator implicitly is configuring a guest NUMA topology. Each NUMA node of this NUMA topology maps to a separate host NUMA node. The OpenStack operator or administrator must understand the NUMA topology of the host systems to configure CPU pinning and NUMA settings correctly.


Huge Pages

Before we go any further into discussing what Huge Pages are and what is the importance of enabling them, let us talk a bit about how Linux handles system memory. If memory management in Linux could be described in one word, that word should be complex.

In computing, as the processor executes a program, it reads an instruction from memory and decodes it. In this operation of decoding the instruction, it may need to get or store the contents of a location in memory. The processor then executes the instruction and moves onto the next instruction in the program. In this way, the processor is always accessing memory to fetch instructions or fetch and store data.

Linux uses a different approach to manage system memory, using Virtual Memory. In a virtual memory system, all of these addresses are virtual addresses and not physical addresses. These virtual addresses are converted into physical addresses by the processor based on information held in tables maintained by the operating system.

To make this translation, virtual and physical memory is divided into small chunks called pages. By default, Linux on Intel x86 systems it uses 4 Kbyte pages. Each of these pages is given a unique number; the page frame number (PFN).

As described before, the system fetches memory by accessing entire pages instead of individual bytes of memory. This translation happens when the system looks in the Translation Lookaside Buffers (TLB), which contain the physical to virtual address mappings for the most recently or frequently used pages. When a mapping being searched for is not in the TLB, the processor must iterate through all the page tables to determine the correct address mappings. This action causes a performance penalty. Therefore, it is preferable to optimize the TLB to ensure the target process can avoid the TLB misses.

The recommendation to optimize the TLB is to enable Huge Pages. Huge Pages changes from the typical page size of 4KB, to a larger size. Larger page sizes mean that there are fewer pages overall, increasing the amount of system memory that can have it is virtual to physical address translation stored in the TLB, and as a result, lowers the potential for TLB misses, which increases performance.

It is essential to notice that by enabling Huge Pages, there is a potential for memory to be wasted as processes must allocate in pages where not all memory is required.  For NFV deployments where page optimization is required, we usually see 1 GB size pages as VNF vendors' recommendation to optimize performance in the application stack.




by Julio Villarreal Pelegrino at July 12, 2020 07:13 PM

July 09, 2020

Fleio Blog

Fleio 2020.07.1: Automated invoices documentation, new information regarding the staff Angular pages and more

Fleio 2020.07.1 was released. The latest version was published on 2020-07-09 and it’s a stable release. In the beta release we have first released the automated invoicing feature. This was added in order to help end users with the adding credit process, thus automating this part. In the stable release we have added a detailed […]

by Marian Chelmus at July 09, 2020 10:46 AM

July 07, 2020

Doug Hellmann

beagle 0.3.0

beagle is a command line tool for querying a hound code search service such as http://codesearch.openstack.org What’s new in 0.3.0? Add repo-pattern usages examples in the doc (contributed by Hervé Beraud) Add an option to filter repositories in search results Refresh python’s versions and their usages (contributed by Hervé Beraud)

by doug at July 07, 2020 02:12 PM

July 06, 2020

OpenStack Superuser

Women of Open Infrastructure: Meet Yumeng Bao from the OpenStack Cyborg Project

This post is part of the Women of Open Infrastructure series to spotlighting the women in various roles in the community who have helped make the Open Infrastructure successful. With each post, we learn more about each woman’s involvement in the community and how they see the future of Open Infrastructure taking shape. If you’re interested in being featured or would like to nominate someone to tell their stories, please email editor@openstack.org.

This time we’re talking to Yumeng Bao from the OpenStack Cyborg Project. She tells Superuser about how the Open Infrastructure community has helped her to grow and excel at her role in the community.

What’s your role(s) in the Open Infrastructure community?

I currently serve as the OpenStack Cyborg Project Team Lead (PTL) for the Victoria Release Cycle, which is a new challenge for me. Before this role, all of my mind was on the code as a core reviewer, such as feature development, code enhancement, bug issue fix, patch review, and etc. While in the new role of being a PTL, I also need to think about the people, organizations, and cross-project collaboration.

What obstacles do you think women face when getting involved in the Open Infrastructure community?   

For women, I think that it’s not easy to be different in a male-dominated community, and, inevitably, people have many stereotypes, which are leftover from the development of the society. Limited by the stereotypes, women are more likely to take the so-called “safe” or “easy” tasks, with the fear of making mistakes. However, we all make mistakes. You might see it as a mistake, while others see it as your engagement.

Stereotypes may take generations to break. In this open source community, we, as contributors, can do our best to build a better community with fewer stereotypes, more open minds, and encouragement for the rise of women’s power. What I think is most important to women is that we shouldn’t allow the fear of failure to stop you from taking challenges.

Why do you think it’s important for women to get involved with open source?

Because that’s a win-win choice! Open source needs women’s potential to grow with the open source community. Some people might believe that men have an advantage of pioneering and innovating, and women have a natural advantage of handling details and aspiring perfection. The development of open source projects is always inseparable from these two stages: founding and further perfection, in which women can harness and mold the unique female strengths to their advantage.

For women ourselves, contributing in open source community is an opportunity to gain professional experiences in developing such a great project, to stay updated with leading technology and great framework design patterns, to learn from peers and experts from different countries, companies and with different genders, ages, and races.

Efforts have been made to get women involved in open source, what are some initiatives that have worked, and why?

Trust and support!

When facing challenges, people tend to strive to overcome difficulties when they believe they can do, but hesitate when they lack confidence. Same for women. Trust, either self-trust or trust from others, is a magic power to inspire someone to take difficult challenges. Whatsmore, providing appropriate support where necessary is also important to encourage new contributors, especially, to get involved. OpenStack Outreachy, Docs, IRC channels, and mailing lists (ML) are proved to be a great start for new contributors to get involved. However, I’ve noticed that not all upstream developers/projects attach the supportive documentation. We can do more to maintain the high priority of supportive materials.

Open source moves very quickly. How do you stay on top of things and what resources have been important for you during this process?

As you know, that’s impossible for sure to catch up with everything, but it is possible to stay on top of things in the scope of your priorities in a period of time. For me, the most important strategy is synchronization with community leaders and those within reach! Within the period of a release cycle, I usually have some goals or topics with high priorities. Once the goals are determined, I regularly follow the ML  threads with related tags, star and track correlative patches in Gerrit, and find clues in the meeting agenda of related projects. Once I’m done with all the above research, I can easily find who the people are on top. Then bring my questions and initiatives, directly, and talk to them! The only disadvantage is that we might be in different time zones, but hey, no worries! Our community has a lot of good remote collaboration tools, and I think you can finally find one that perfectly matches your needs.

What advice would you give to a woman considering a career in open source? What do you wish you had known?

For me, I feel that it’s well worth taking the open source upstream developer role as my career’s starting point, especially in the Open Infrastructure community. It is a good opportunity to gain career experiences in many areas such as technical experience in distributed cloud technology, cross-culture collaboration with excellent engineers from all over the world to solve common problems in real production, and witness the evolution of such an influential technology as one of the contributors. I joined the community after my college graduation in 2016. If I had known cloud computing and open source models earlier, I think I would give it a try sooner and probably catch its early evolution period. Therefore, if a woman is considering a career in the open source, I would recommend that just go ahead and try! Open source has the natural advantage that you can start as a volunteer without the need to assign any long-time contracts. I’m sure you won’t regret whatever you’ve tried in the Open Infrastructure community!

The post Women of Open Infrastructure: Meet Yumeng Bao from the OpenStack Cyborg Project appeared first on Superuser.

by Superuser at July 06, 2020 01:00 PM

July 02, 2020

OpenStack Superuser

Takeaways and next steps from OpenDev: Large Scale Usage of Open Infrastructure

Event organizers and participants experience a variety of emotions during the lifecycle of an event: excitement, anticipation, stress, and sadness when it’s over. OpenDev: Large Scale Usage of Open Infrastructure was no exception. To be honest, baby Yoda put it best.

This week, the OpenStack Foundation (OSF) held its third OpenDev conference. This iteration focused on scaling challenges for open infrastructure across a variety of topics including upgrades, high availability (HA) networking, bare metal, and more. Jonathan Bryce, OSF executive director, kicked off the event explaining how OpenDev events are an example of how the global Open Infrastructure community collaborates without boundaries. He encouraged participants to think broadly and remember that collaboration is built by people who are actively engaged and participating by sharing their knowledge and experiences so they can learn from each other.

This virtual event recruited participants from over 70 countries who spent three days asking questions, sharing scaling challenges, and explaining implementations that worked for their local environments. Each moderated session combined various perspectives on the challenges, questions for how to improve, and next steps that the community can collectively collaborate on to ease these operator challenges.

Thank you to the OpenDev: Large-scale Usage of Open Infrastructure Software programming committee members: Beth Cohen, Roman Gorshunov, Belmiro Moreira, Masahito Muroi, and Allison Price. You helped to make these discussions possible!

So, what happened? Below is a snapshot of the conversations that took place, but I want to encourage you to check out the event recordings as well as the discussion etherpads found in the OpenDev event’s schedule to join the discussion.

Jump to the User Stories recap
Jump to the Upgrades recap
Jump to the Tools recap
Jump to the HA Networking Solutions recap
Jump to the Software Stack recap
Jump to the Bare Metal recap
Jump to the Edge Computing recap
Jump to the OpenDev event’s next steps


User Stories

Ahead of the weeks’ discussions, speakers from Blizzard Entertainment. OpenInfra Labs, and Verizon shared their own scaling challenges via short user story presentations Monday morning followed by audience questions moderated by Mark Collier, OSF COO.

Colin Cashin, Blizzard Entertainment senior director of cloud engineering, and Erik Andersson, Blizzard Entertainment technical lead, senior cloud engineer, discussed four key scaling challenges that affect their 12,000 node OpenStack deployment that spans multiple regions. Their challenges were: Nova scheduling w/ NUMA pinning, scaling RabbitMQ (a frequent challenge repeated throughout the week), scaling Neutron, and compute fleet maintenance. The Blizzard team received a flurry of questions about their specific challenges and their multi-cloud approach (OpenStack, Amazon, Google, and Alibaba). They also used the opportunity to share that they’re hiring engineers to help manage their massive OpenStack footprint, so if you’re interested, check out their openings.

Michael Daitzman provided an overview of OpenInfra Labs, the newest OSF pilot project, that is a collaboration among several research institutions including Boston University, Harvard University, MIT, Northeastern University, and University of Massachusetts. He covered some challenges for integrating different open source technologies into a single environment including specific problem areas like monitoring and what happens when your Ceph clusters aren’t properly backed up (TL;DR: you lose all your data).

Beth Cohen, OpenDev veteran and moderator of the HA networking solutions discussion, presented an updated Verizon case study centered around some of their challenges. Her team is seeing that application configuration (written by partners) has a significant effect on how the applications behave in the environment (i.e. w/ encryption turned on, throughput halved). With traffic going through all of these systems, it can be hard to identify the source of the problem. In order to isolate and reproduce this, they have built a full production engineering lab with full testing capabilities that is also used by their development team, ultimately providing a really nice feedback loop.


Upgrades

When asking OpenStack operators their number one challenge with their deployments, upgrades are often at the top of the list. This discussion, moderated by Belmiro Moreira, cloud architect at CERN, explored the different setbacks around upgrades, including why operators find them so daunting. Top reasons shared included upgrading across versions, the amount of resources needed, the feeling of “once you’re behind, you’re behind,” and API downtime. Some suggestions included the recent pre-upgrade checks that the OpenStack TC set as a goal in the Stein cycle in 2019 as well as fast forward upgrades.

As can be expected, this session included a lot of discussion about specific upgrade scenarios. One of the next steps that Jonathan mentioned in the event wrap-up was the opportunity to use the OpenDev event as a launching off point for some upgrade documentation. If you’re an operator who has successfully upgraded your OpenStack environment and would like to collaborate on a set of tips, please sign up here.


Tools

Moderated by John Garbutt, principal engineer at StackHPC, the Tools discussion covered a few topics: Different ways to measure scale, the different considerations into what you’re trying to scale and how that will affect tools, and finding and removing bottlenecks, which is naturally where a lot of the conversation centered. Shared challenges included RabbitMQ flatlining, legacy Keystone UUID tokens, and losing access to Cinder volumes.

The next step for this topic was largely centered around sharing tooling, which included the idea of revisiting the OsOps initiative that was started a few years ago so that operators could share tooling with each other and collaborate more closely.


HA Networking Solutions

What does HA mean from a network perspective? Are you concerned with networking connectivity failures in your environment and if so what are you doing about them? Do you have a diverse network? Beth Cohen, cloud product technologist at Verizon, started the discussion with these questions to make sure that everyone knows what networking high availability means in this context and get to know how important HA is in the participants’ deployments.

Later, the participants dived into discussions around different factors that apply to scaling, where you put failure domains and what the most cost-effective way is to do that. Should we pay for a lot of redundant and expensive hardware or handle failure at the rack level, and trust the software to be able to work around that? The interactive discussion continues. If you are interested in adding your thoughts to the etherpad, please add it here.


Software Stack

The Software Stack discussion, moderated by Masahito Muroi, senior software engineer at LINE Corp., centered around what the open source mix will be over time and how to plan for deployment infrastructure software and the resources that the end users will want. A poll during this session helped show that networking and deployment were the two biggest challenges that participants faced.


Bare Metal

James Penick, architect director at Verizon Media, moderated an active discussion around bare metal, including multiple opportunities where he activated his own team to implement learnings back in their own environment. The conversation included use cases for bare metal and where participants see those use cases evolving to, including scenarios like 5G, the tooling required around bare metal management, and lifecycle management. With only an hour available for discussion, participants were only able to scratch the surface on the bare metal conversation, but luckily, the next OpenDev event (Hardware Automation, July 20-22) will be centered around this topic, including an entire day on lifecycle management.


Edge Computing

The last session of the event was around the edge computing use case and was moderated by Shuquan Huang, technical director at 99cloud. The discussion kicked off with use cases that included farming, high performance computing (HPC), 5G, facial recognition, and an impressive China Tower edge computing use case that was planning to leverage StarlingX across 2 million antenna towers in China. The different architectural requirements for the varying edge use cases was a popular question and unsurprising as participants had earlier identified networking as their number one pain point in the Software Stack discussion. This was a good opportunity to promote the OSF Edge Computing Group’s latest whitepaper, Edge Computing: Next Steps in Architecture, Design, and Testing.


Next Steps

The goal with the OpenDev events is to extend this week’s learnings into future work and collaboration, so Jonathan Bryce and Thierry Carrez, OSF VP of Engineering, wrapped up the event to discuss next steps. These include:

  • Join the OpenStack Large Scale SIG to continue sharing challenges and solutions around scaling
  • Take the OpenStack User Survey to share feedback with the upstream community
  • Collaborate on documentation around common pain points, like upgrades
  • Revive OsOps Tools
  • Join the OpenDev: Hardware Automation event to continue discussing bare metal use cases
  • Check out OpenInfra Labs and get involved in their mission to refine a model for fully integrated clouds supporting bare metal, VMs, and container orchestration.

Let’s keep talking about open infrastructure! Check out the next two OpenDev events:

The post Takeaways and next steps from OpenDev: Large Scale Usage of Open Infrastructure appeared first on Superuser.

by Allison Price and Sunny Cai at July 02, 2020 01:00 PM

July 01, 2020

OpenStack Superuser

Turning your challenges to opportunities in Nokia NESC OpenStack

Starting with just a handful of servers 10 year ago, Nokia Enterprise and Services Cloud (NESC) has grown into one of the world’s largest OpenStack private clouds providing 484,000 virtual computing cores, 40PB of storage and a sustained monthly usage in excess of 230 million active core hours in month. NESC is distributed globally across three continents in six Tier three data centers and is used by Nokia for hosting mission critical research and development workloads and customer interfacing business applications. NESC provides standard Infrastructure As A Service components strengthened with ISO27001 certification.

Key to the success of NESC has been constant evolution, empowering our Nokia R&D and applications teams to reliably and rapidly build the virtual environments via self-service and automation. Every day in NESC there are over 200,000 new virtual server instance starts and close to 100,000 API calls per minute during peak hours, a challenge for any cloud.

Fortunately NESC has a great developer team who focus not on telling users what is not possible, but instead on figuring out how. This has driven the team to develop and excel in some key areas like:

Monitoring, logging and metrics analytics: NESC Logstash solution collects around 2 million log lines per minute which is then indexed and enriched in real-time to maximize the value of the data. In addition, NESC continuously collects huge volumes of metrics that are used not only for looking at current status, but also by applying Machine learning, are able to find anomalies,  performance degradations and even to make future predictions of outcomes based on trends.

Operation Automation:  “The best sysadmin is a lazy one”. They don’t want to fix same things over and over again.  Anything that can be automated in NESC is, from upgrades to database clean-ups and other resource leaks. 

Master data:  The key to successful and reliable automation is the reliability of programmable inventory data. You need to know with certainty on which assets what services are running so you can automate them. NESC tried enterprise commercial auto-discovery tools and other methods for many years, but none of them were reliable enough or able to keep dependencies and context consistent. Today, we have developed and implemented our own auto-discovery solution for inventory and configuration management as single source of truth.  

Security: With NESC software development tools & processes and the overall service being ISO 27001 certified, the security journey has been filled with automated controls and dedicated toolsets that are mostly open source based.  

Today NESC is stronger and better as a result, a dependable service within Nokia. The ability to embrace challenges forced the team to learn faster and adopt new ideas and services like GeoDNS, tiered storage solutions or NESC’s latest fast growing Kubernetes as service offering.  NESC DNA of fast development cycles,  fail sooner than later and extensive automation has pushed us this far.

The next chapter for NESC is to offer it as a managed service to Nokia customers worldwide, either as an on-Premises or off-premises solution.

The post Turning your challenges to opportunities in Nokia NESC OpenStack appeared first on Superuser.

by Yilmaz Karayilan at July 01, 2020 01:00 PM

June 27, 2020

Lee Yarwood

Openstack Nova Victoria Cycle Gerrit Dashboards

As in previous cycles I’ve updated some of the Nova specific dashboards available within the excellent gerrit-dash-creator project and started using them prior to dropping offline on paternity. I’d really like to see more use of these dashboards within Nova to help focus our limited review bandwidth on active and mergeable changes so if you do have any ideas please fire off reviews and add me in!

June 27, 2020 11:51 PM

June 26, 2020

John Likes OpenStack

Running tripleo-ansible molecule locally for dummies

I've had to re-teach myself how to do this so I'm writing my own notes.

Prerequisites:

  1. Get a working undercloud (perhaps from tripleo-lab)
  2. git clone https://git.openstack.org/openstack/tripleo-ansible.git ; cd tripleo-ansible
  3. Determine the test name: ls roles

Once you have your environment ready run a test with the name from step 3.


./scripts/run-local-test tripleo_derived_parameters
Some tests in CI are configured to use `--skip-tags`. You can do this for your local tests too by setting the appropriate environment variables. For example:

export TRIPLEO_JOB_ANSIBLE_ARGS="--skip-tags run_ceph_ansible,run_uuid_ansible"
./scripts/run-local-test tripleo_ceph_run_ansible

This last tip should get added to the docs.

by Unknown (noreply@blogger.com) at June 26, 2020 06:39 PM

June 22, 2020

OpenStack Blog

Submit your first OpenStack patch in three steps

If you are new to the OpenStack Community and want to start the contribution, this document can help you in a quick way. OpenStack does not use github pull request instead it uses Gerrit for code collaboration tool. Also, there is some accounts setup required for using the Gerrit system. This guide will quickly help you to... Read more »

by Ghanshyam Mann at June 22, 2020 06:31 PM

OpenStack Superuser

OSF Edge Computing Group defines architectures, open source components, and testing activities for massively distributed systems

The OpenStack Foundation (OSF) Edge Computing Group is excited to announce we published our latest white paper, a result of collaboration among fellow open infrastructure operators and vendors! With deployments of IoT devices and the arrival of 5G networks, edge computing has rapidly gained popularity over the past few years. While the popularity has rapidly increased, there are still countless debates about the definition of related terms and the right business models, architectures and technologies required to satisfy the seemingly endless number of emerging use cases of this novel way of deploying applications over distributed networks.

In our previous white paper, the OSF Edge Computing Group defined cloud edge computing as resources and functionality delivered to the end users by extending the capabilities of traditional data centers out to the edge, either by connecting each individual edge node directly back to a central cloud or several regional data centers, or in some cases connected to each other in a mesh. From a bird’s eye view, most of those edge solutions look loosely like interconnected spider webs of varying sizes and complexity.

Our second white paper, Edge Computing: Next Steps in Architecture, Design and Testing, delivers the specific ways open source communities are shaping the future of edge computing by collecting use cases, identifying technology requirements and contributing architectures, open source components and considerations for testing activities.

This white paper also highlights the OSF Edge Computing Group’s work to more precisely define and test the validity of various edge reference architectures. To help with understanding the challenges, there are use cases from a variety of industry segments, demonstrating how the new paradigms for deploying and distributing cloud resources can use reference architecture models that satisfy these requirements. 

Check out the lastest edge computing white paper here!

The post OSF Edge Computing Group defines architectures, open source components, and testing activities for massively distributed systems appeared first on Superuser.

by Ildiko Vancsa at June 22, 2020 01:00 PM

Ghanshyam Mann

Submit your first OpenStack patch in 3 steps.

If you are new to OpenStack Community and want to start the contribution, this document can help you in a quick way. OpenStack does not use github pull request instead it uses Gerrit for code collaboration tool. Also, there is some accounts setup required for using the Gerrit system. This guide will quickly help you to set up those accounts and the minimal steps

  Step1: Set up accounts

To get started, first set up the required accounts.

  • Setup OpenStack Foundation Account.

    • Go to the OpenStack Foundation sign up page.
    • Under individual members, click the Foundation Member button.
    • Few tips for filling the form:
      • Use the same e-mail address at every step of the registration procedure.
      • Add your affiliation information if you want otherwise you are contributing on behalf of your  ‘Individual Contributors’.
  • After you submit the application, you will get the email once it is approved.
  • Setup Your Task Tracker Account

    • Go to the https://login.launchpad.net/.
    • If you don’t have a ubuntu One Account, click the I don’t have an Ubuntu One account.
      • Use the same email address that was used during the OpenStack Foundation account setup.
    • Fill all information and ‘Create Account’.

  • Install git:

    • Mac OS
      • Go to the Git download page and click Mac OS X.
      • The downloaded file should be a dmg in your downloads folder. Open that dmg file and follow the instructions on screen.
      • If you use the package manager Homebrew, open a terminal and type:
        brew install git
    • Linux
      • For distributions like Debian, Ubuntu, or Mint open a terminal and type:
            sudo apt install git
      • For distributions like RedHat, Fedora 21 or earlier, or CentOS open a terminal and type:
            sudo yum install git
      • For Fedora 22 or later open a terminal and type:
            sudo dnf install git
      • For SUSE distributions open a terminal and type:
            sudo zypper in git
    • Configure Git
            git config --global user.name "Firstname Lastname"
            git config --global user.email "your_email@youremail.com"
      • Use the same email address that was used during the OpenStack Foundation account setup.
  • Setup Your Gerrit Account

    • Visit OpenStack’s Gerrit page and click the sign in link.
    • You will be prompted to select a username. Choose and type your username carefully. Once it is set, you cannot change the username.
    • From here on out when you sign in to Gerrit, you’ll be prompted to enter your Launchpad login info. This is because Gerrit uses it as an OpenID single sign on.
    • Sign Individual Contributor License Agreement. If you want to contribute from company then ask your company to sign the Company CLA.
      • In Gerrit’s settings click the “New Contributor Agreement” link and sign the agreement.

  • Setup SSH Keys

    • In order to push patches to Gerrit we need to have a way to identify ourselves and ssh key is the one way to authenticate. You can submit patches from any machine but you need to update the ssh key of that machine.
    • Generate SSH Key Pairs
         ssh-keygen -t rsa
    • Copy Public Key
        cat ~/.ssh/id_rsa.pub
    • Add Public Key Into Gerrit

  • Install git-review tool

    • pip install gitreview
    • git config global gitreview.username <username>

  Step2: Push your change

  • Clone the repository.

    • Clone the repo which you want to push the changes:
        git clone https://opendev.org/openstack/<PROJECT_NAME>

                    Here you can find all the OpenStack projects and repo under those projects.

  • Create your local branch to do the changes

    git checkout -b <branch_name>
  • Do the changes in code and perform all required unit or functional tests.

    • To check the files that have been updated in your branch:
        git status
  • Add your changes to the branch

    git add -A

  • Write commit changes

    git commit
    or 
    git commit --amend (if you are ammending the previously written commit msg)
  • Submit your changes

    git review
  • Tracking your Changes

    • You can track the submitted changes at Code Review. After logging in, click on ‘My’ then ‘Changes’ and there you can see all your changes.

  Step3: Practice the above steps using Sandbox Project

To make sure everything is set up correctly or to understand the workflow without trying it on actual projects, you can practice in a Sandbox using the How to Use the Sandbox Projects Guide. Sandbox project is just for practice so do not hesitate or worry about anything.

For more details on OpenStack community contributor, please refer to The OpenStack Contributor Guide or ping me on IRC (#openstack-dev or openstack-upstream-institute channels), I am available with ‘gmann’ nickname.

by Ghanshyam Mann at June 22, 2020 12:53 AM

June 18, 2020

OpenStack Superuser

Inside open infrastructure: The latest from the OpenStack Foundation

Spotlight on: Project Teams Gathering (PTG) Recap

It has been a week since the first virtual Project Teams Gathering (PTG)! While it was not the same experience as our traditional in-person PTGs, the community made the best of the current situation. We’ve been amazed by how many people have joined us online this year to collaborate on OSF-supported projects. We not only had the highest attendance and gender diversity in this PTG, but also had 20 more countries represented than the Denver PTG last year. Thank you to all the community members who have worked together virtually in this unprecedented time to collaborate without boundaries.

PTG Participation by Day*

*Attendees’ attendance might be counted twice in the same day if they are participating in multiple sessions.

If you didn’t attend—or if you did and want a replay—check out the PTG spotlight on Superuser where we have collected the project announcements, community updates, and discussions you may have missed.

OpenStack Foundation news

  • The OSF Edge Computing Group published its second white paper, Edge Computing: Next Steps in Architecture, Design and Testing.
  • There will be two community meetings next week to cover OSF-supported project updates and event updates. One is on Thursday, June 25 at 1500 UTC and the other is Friday, June 26 at 0200 UTC. You can find the dial-in information here.
  • The OSF has entered into a partnership with ETSI. The partnership will help strengthen collaboration between standardization and open source activities, including the area of edge computing.
  • The OSF Board of Directors approved the 2020.06 RefStack guidelines.
  • OpenDev: Large-scale Usage of Open Infrastructure Software, the first of three virtual OpenDev events, kicks off June 29. Register here.

Airship: Elevate your infrastructure

  • Nominations for the Airship Technical Committee are now open, and will remain open until EOD June 21. More information here.
  • Airship 2.0’s alpha milestone was completed in May. Stay updated on progress toward the beta through the blog.
  • Running or evaluating Airship? The User Survey is available here. 

Kata Containers: The speed of containers, the security of VMs

  • We published two stable releases at the end of last week – 1.11.1 and 1.10.5.
    • Among other bug fixes, these releases include security fixes for CVE-2020-2023 and CVE-2020-2026.
    • Kata Security Advisory for the above CVEs describing impact and mitigation has been published here.
  • All Kata versions prior to 1.11.1 and 1.10.5 are impacted. It is recommended to upgrade to the latest stable releases. The security fixes have been pushed to master as well.
  • We tagged Kata Containers 2.0.0-alpha1 release last week. This release uses the rust agent as default and makes the switch to ttrpc from grpc as the communication protocol. We have also consolidated the agent and runtime repositories and moved them to kata-containers/kata-containers repository for better maintenance and release management.
  • Now you can find the release information for 2.0.0-alpha1 here, and you can also find the features that we are planning for 2.0 here.

OpenStack: Open source software for creating private and public clouds

  • The OpenStack community had a great virtual Project Teams Gathering. While we missed seeing each other in person, it was a productive event, putting the Victoria development cycle to a good start. You can still access all the etherpads for the event, and find summaries posted on the discuss mailing-list.
  • Speaking of the Victoria cycle, the Technical Committee finally selected two community-wide goals around CI/CD for this cycle: switch legacy Zuul jobs to native, and migrate jobs to new Ubuntu 20.04 LTS. For more details, check out Ghanshyam Mann’s post on the mailing-list.
  • Hot on the heels of the recent Ussuri release, Thomas Goirand announced the availability of packages for Debian sid, as well as the buster-ussuri.debian.net repositories. Peter Matulis announced the availability of the 20.05 OpenStack charms release, introducing support for OpenStack Ussuri on Ubuntu 18.04 LTS and 20.04 LTS. Amy Marrich announced the general availability of the RDO build for OpenStack Ussuri for RPM-based distributions, CentOS Linux and Red Hat Enterprise Linux.
  • Following discussions at the PTG, the Technical Committee and the User Committee have started to move forward with merging into a single governance body for the OpenStack open source project, representing all contributors (developers, operators, and end users of the software). As a first step, the separate user-committee mailing-list was discontinued and future discussions will be held on the openstack-discuss mailing-list.

StarlingX: A fully featured cloud for the distributed edge

  • StarlingX is now a confirmed top-level Open Infrastructure project supported by the OpenStack Foundation.
  • The community is currently working on the 4.0 version of the platform that they are planning to release in July.
  • You can check out this blog post to find out more about the community’s achievements since the project’s launch and their plans for the next release.

Zuul: Stop merging broken code

  • Zuul 3.19.0 and Nodepool 3.13.0 are the last planned series 3 releases, incorporating a new Ansible 2.9 default, branch guessing for tags, ability to pause mergers, TLS encryption for Zookeeper connections, a new “serial” pipeline manager, a timezone selector for the dashboard, and more; work is underway for version 4 which sets the stage for distributed schedulers, stateful restarts, and high availability across all services.
  • Zuul: A Wazo Platform Case Study
    • Learn why Wazo Platform, An open source software programmable telecommunication platform, leverages Zuul’s cross-repository dependencies for its repositories.
  • Zuul: A T-Systems Case Study
    • Learn why global IT company T-Systems leverages Zuul’s ability to easily test workflows.

Check out these Open Infrastructure Community Events!

For more information about these events, please contact denise@openstack.org

Questions / feedback / contribute

This newsletter is written and edited by the OSF staff to highlight open infrastructure communities. We want to hear from you! If you have feedback, news or stories that you want to share, reach us through community@openstack.org . To receive the newsletter, sign up here.

The post Inside open infrastructure: The latest from the OpenStack Foundation appeared first on Superuser.

by Helena Spease at June 18, 2020 04:00 PM

StackHPC Team Blog

Software RAID support in OpenStack Ironic

OpenStack Ironic operates in a curious world. Each release of Ironic introduces ever more inventive implementations of the abstractions of virtualisation. However, bare metal is wrapped up in hardware-defined concrete: devices and configurations that have no equivalent in software-defined cloud. To exist, Ironic must provide pure abstractions, but to succeed it must also offer real-world circumventions.

For decades the conventional role of an HPC system administrator has included deploying bare metal machines, sometimes at large scale. Automation becomes essential beyond trivial numbers of systems to ensure repeatability, scalability and efficiency. Thus far, that automation has evolved in domain-specific ways, loaded with simplifying assumptions that enable large-scale infrastructure to be provisioned and managed from a minimal service. Ironic is the first framework to define the provisioning of bare metal infrastructure in the paradigm of cloud.

So much for the theory: working with hardware has always been a little hairy, never as predictable or reliable as expected. Software-defined infrastructure, the method underpinning the modern mantra of agility, accelerates the interactions with hardware services by orders of magnitude. Ironic strives to deliver results in the face of unreliability (minimising the need to ask someone in the data centre to whack a machine with a large stick).

HPC Infrastructure for Seismic Analysis

As a leader in the seismic processing industry, ION Geophysical maintains a hyperscale production HPC infrastructure, and operates a phased procurement model that results in several generations of hardware being active within the production environment at any time. Field failures and replacements add further divergence. Providing a consistent software environment across multiple hardware configurations can be a challenge.

ION is migrating on-premise HPC infrastructure into an OpenStack private cloud. The OpenStack infrastructure is deployed and configured using Kayobe, a project that integrates Ironic (for hardware deployment) and Kolla-Ansible (for OpenStack deployment), all within an Ansible framework. Ansible provides a consistent interface to everything, from the physical layer to the application workloads themselves.

This journey began with some older-generation HPE SL230 compute nodes and a transfer of control to OpenStack management. Each node has two HDDs. To meet the workload requirements these are provisioned as two RAID volumes - one mirrored (for the OS) and one striped (for scratch space for the workloads).

Each node also has a hardware RAID controller, and standard practice in Ironic would be to make use of this. However, after analysing the hardware it was found that:

  • The hardware RAID controller needed to be enabled via the BIOS, but the BIOS administration tool failed on many nodes because the 'personality board' had failed, preventing the tool from retrieving the server model number.
  • The RAID controller required a proprietary kernel driver which was not available for recent CentOS releases. The driver was not just required for administering the controller, but for mounting the RAID volumes.

Taking these and other factors into account, it was decided that the hardware RAID controller was unusable. Thankfully, Ironic developed a software-based alternative.

Provisioning to Software RAID

Linux servers are often deployed with their root filesystem on a mirrored RAID-1 volume. This requirement exemplifies the inherent tensions within the Ironic project. The abstractions of virtualisation demand that the guest OS is treated like a black box, but the software RAID implementation is Linux-specific. However, not supporting Linux software RAID would be a limitation for the primary use case. Without losing Ironic's generalised capability, the guest OS “black box” becomes a white box in exceptional cases such as this. Recent work led by CERN has contributed software RAID support to the Ironic Train release.

The CERN team have documented the software RAID support on their tech blog.

In its initial implementation, the software RAID capability is constrained. A bare metal node is assigned a persistent software RAID configuration, applied whenever a node is cleaned and used for all instance deployments. Prior work involving the StackHPC team to develop instance-driven RAID configurations is not yet available for software RAID. However, the current driver implementation provides exactly the right amount of functionality for Kayobe's cloud infrastructure deployment.

The Method

RAID configuration in Ironic is described in greater detail in the Ironic Admin Guide. A higher-level overview is presented here.

Software RAID with UEFI boot is not supported until the Ussuri release, where it can be used in conjuction with a rootfs UUID hint stored as image meta data in a service such as Glance. For Bifrost users this means that legacy BIOS boot mode is the only choice, ruling out secure boot and NVMe devices for now.

In this case the task was to provision a large number of compute nodes with OpenStack Train, each with two physical spinning disks and configured for legacy BIOS boot mode. These were provisioned according to the OpenStack documentation with some background provided by the CERN blog article. Two RAID devices were specified in the RAID configuration set on each node; the first for the operating system, and the second for use by Nova as scratch space for VMs.

{
  "logical_disks": [
    {
      "raid_level": "1",
      "size_gb"   : 100,
      "controller": "software"
    },
    {
      "raid_level": "0",
      "size_gb"   : "800",
      "controller": "software"
    }
  ]
}

Note that although you can use all remaining space when creating a logical disk by setting size_gb to MAX, you may wish to leave a little spare to ensure that a failed disk can be rebuilt if it is replaced by a model with marginally different capacity.

The RAID configuration was then applied with the following cleaning steps as detailed in the OpenStack documentation:

[{
  "interface": "raid",
  "step": "delete_configuration"
 },
 {
  "interface": "deploy",
  "step": "erase_devices_metadata"
 },
 {
  "interface": "raid",
  "step": "create_configuration"
 }]

A RAID-1 device was selected for the OS so that the hypervisor would remain functional in the event of a single disk failure. RAID-0 was used for the scratch space to take advantage of the performance benefit and additional storage space offered by this configuration. It should be noted that this configuration is specific to the intended use case, and may not be optimal for all deployments.

As noted in the CERN blog article, the mdadm package was installed into the Ironic Python Agent (IPA) ramdisk for the purpose of configuring the RAID array during cleaning. mdadm was also installed into the deploy image to support the installation of the grub2 bootloader onto the physical disks for the purposes of loading the operating system from either disk should one fail. Finally, mdadm was added to the deploy image ramdisk, so that when the node booted from disk, it could pivot into the root filesystem. Although we would generally use Disk Image Builder, a simple trick for the last step is to use virt-customize:

virt-customize -a deployment_image.qcow2 --run-command 'dracut --regenerate-all -fv --mdadmconf --fstab --add=mdraid --add-driver="raid1 raid0"'

Open Source, Open Development

As an open source project, Ironic depends on a thriving user base contributing back to the project. Our experiences covered new ground: hardware not used before by the software RAID driver. Inevitably, new problems are found.

The first observation was that configuration of the RAID devices during cleaning would fail on about 25% of the nodes from a sample of 56. The nodes which failed logged the following message:

mdadm: super1.x cannot open /dev/sdXY: Device or resource busy

where X was either a or b and Y either 1 or 2, denoting the physical disk and partition number respectively. These nodes had previously been deployed with software RAID, either by Ironic or by other means.

Inspection of the kernel logs showed that in all cases, the device marked as busy had been ejected from the array by the kernel:

md: kicking non-fresh sdXY from array!

The device which had been ejected, which may or may not have been synchronised, appeared in /proc/mdstat as part of a RAID-1 array. The other drive, having been erased, was missing from the output. It was concluded that the ejected device had bypassed the cleaning steps designed to remove all previous configuration, and had later resurrected itself, thereby preventing the formation of the array during the create_configuration cleaning step.

For cleaning to succeed, a manual workaround of stopping this RAID-1 device and zeroing signatures in the superblocks was applied:

mdadm --zero-superblock /dev/sdXY

Removal of all pre-existing state greatly increased the reliability of software RAID device creation by Ironic. The remaining question was why some servers exhibited this issue and others did not. Further inspection showed that although many of the disks were old, there were no reported SMART failures, the disks passed self tests and although generally close, had not exceeded their mean time before failure (MTBF). No signs of failure were reported by the kernel in addition to the removal of a device from the array. Actively seeking errors, for example by running tools such as badblocks to exercise the entire disk media, showed that only a very small number of disks had issues. Benchmarking, burn-in and anomaly detection may have identified those devices sooner.

Further research may help us identify whether the disks that exhibit this behaviour are at fault in any other way. An additional line of investigation could be to increase thresholds such as retries and timeouts for the drives in the kernel. For now the details are noted in a bug report.

The second issue observed occurred when the nodes booted from the RAID-1 device. These nodes, running IPA and deploy images based on Centos 7.7.1908 with kernel version 3.10.0-1062, would show degraded RAID-1 arrays, with the same message seen during failed cleaning cycles:

md: kicking non-fresh sdXY from array!

A workaround for this issue was developed by running a Kayobe custom playbook against the nodes to add sdXY back into the array. In all cases the ejected device was observed to resync with the RAID device. The state of the RAID arrays is monitored using OpenStack Monasca, ingesting data from a recent release candidate of Prometheus Node Exporter containing some enhancements around MD/RAID monitoring. Software RAID status can be visualised using a simple dashboard:

Monasca dashboard

Monasca MD/RAID Grafana dashboard using data scraped from Prometheus node exporter.

The plot in the top left shows the percentage of blocks synchronised on each RAID device. A single RAID-1 array can be seen recovering after a device was forcibly failed and added back to simulate the failure and replacement of a disk. Unfortunately it is not yet possible to differentiate between the RAID-0 and RAID-1 devices on each node since Ironic does not support the name field for software RAID. The names for the RAID-0 and RAID-1 arrays therefore alternate randomly between md126 and md127. Top right: The simulated failed device is visible within seconds. This is a good metric to generate an alert from. Bottom left: The device is marked as recovering whilst the array rebuilds. Bottom right: No manual re-sync was initiated. The device is seen as recovering by MD/RAID and does not show up in this figure.

The root cause of these two issues is not yet identified, but they are likely to be connected, and relate to an interaction between these disks and the kernel MD/RAID code.

Open Source, Open Community

Software that interacts with hardware soon builds up an extensive "case law" of exceptions and workarounds. Open projects like Ironic survive and indeed thrive when users become contributors. Equivalent projects that do not draw on community contribution have ultimately fallen short.

The original contribution made by the team at CERN (and others in the OpenStack community) enabled StackHPC and ION Geophysical to deploy infrastructure for seismic processing in an optimal way. Whilst in this case we would have liked to have gone further with our own contributions, we hope that by sharing our experience we can inspire other users to get involved with the project.

Get in touch

If you would like to get in touch we would love to hear from you. Reach out to us via Twitter or directly via our contact page.

by Stig Telfer at June 18, 2020 12:00 PM

OpenStack Superuser

Where are they now? Superuser Awards: City Network

Local European financial services companies rely on infrastructure that adheres to local regulatory requirements, often partnering with OpenStack cloud provider, City Network. A Gold Member of OSF, City Network provides public and private cloud environments, receiving the Superuser Award specifically for their research and development, professional services, education and engineering teams.

Keep reading to see how they’ve evolved since winning the Superuser Award at the Berlin Summit in 2018.

What has changed in your OpenStack environment since you won the Superuser Awards?

A lot! First, we are adamant about upgrading, so currently most locations run Train, but a few are still on older versions. Ideally, before summer, all of our locations and clouds will run Train. That in itself of course allows for more functionality and yet again improved stability.

In addition, we have looked to start engaging other projects we have not run before. So before going to Train, we activated Magnum as well as Barbican. Now with Train, we are also activating Designate and Manila. Manila is something many customers have wanted for a long time.

We also had a soft launch of our new cloud management system, written in React. It adds a bit of functionality that our public cloud requires.

What is the current size of City Network’s OpenStack environment?

We currently run well over 10,000 VMs in eight different locations from New York to Tokyo. Our focus is the European enterprise where many times regulatory challenges put a little extra work around how each workload is handled.

What version of OpenStack is City Network running?

Most locations are running Train, but we do have a few installations on Rocky as well. Those on Rocky will go to Train before summer.

What open source technologies does your team integrate with OpenStack?

As a whole, there are about 20 open source projects to make it all come together. OpenStack is of course the foundation to create a stack that can be fully automated. No doubt Kubernetes is something we also make sure our customers can run to orchestrate their containers. Magnum is a part of that container strategy. As with most container work loads, VMs are at the heart of those containers and we run it similarly. We are also looking into Kata Containers and are curious about the other OSF projects.

What workloads are you running on OpenStack?

With 1,200 customers in our public cloud we would say just about any and all types of workloads– from more traditional LAMP installations to modern container workloads with the latest CI/CD implementations pushing code at a fierce rate. As a company we focus on enterprise customers where anything from standard applications to the customer meeting is built out in our public cloud. In our Compliant Cloud, we run workloads mostly related to higher level regulatory challenges from banks to security companies. In Europe many of the leading digital ID companies run their workloads with City Network. So overall there is a very broad set of workloads.

How big is your OpenStack team?

We are about 25 people working with and around OpenStack.

How is your team currently contributing back to the OpenStack project? Is your team contributing to any other projects supported by the OpenStack Foundation (Airship, Kata Containers, StarlingX, Zuul)?

Many years ago we started by contributing more by helping out in other ways than just technical– from hosting OpenStack Days to leading the public cloud group to being on the board, helping out in various committees. Another way was to become a gold member and thus support OSF financially as well. One part of this decision has been because we have felt we are not the programmers that can always jump in and add. However this is changing and we have for some time contributed smaller additions like reviews and some code when we find aspects that need corrections.

At this point we have a more aggressive vision of how we want to contribute technically and all around. We truly believe that we will become better operators if we know the code, as well as the people around the code. The more we engage the better we will become. So we have started to assign people in various parts of our engineering team that will start to contribute code. We just added 20% to Designate, and there are a few people that will work on OpenStack-Ansible for instance. From there we expect to get deeper into other projects as well.

What kind of challenges has your team overcome using OpenStack?

We simply would not be the company we are. By allowing our customers to fully automate the stack with the many projects of OpenStack we offer, we are a huge part of their digital transformation. We are on a journey made possible with OpenStack and of course a number of other open source projects. We do not think anybody can build what OpenStack offers unless you are a hyper scaler. For us, open source and the four opens of the OpenStack Foundation have formed us and how we like to see open infrastructure evolve.

That said, it has not always been easy. OpenStack was difficult to operate a few years back, and is still pretty complex. However, it has made huge strides towards being better in all ways including operation aspects. Today, a majority of our large-scale upgrades go by without incidents and no down time. Five years ago that was not always the case. It also allows for a ton more functionality, truly giving our customers all they need. Combining it with Kubernetes, there are next to no work loads we can not take on today in a dynamic way, allowing for that platform of innovation our customers require, whether a bank or a gaming company.

Another challenge is knowledge. We continue to educate internally but also love to see how the OpenStack Foundation is helping spread the word and trying to engage in educating more people. This will continue to be something we would love to see more of.

OpenStack continues to evolve at a rapid rate and just about all serious challenges are today dealt with and overcome. OpenStack has proven to work well with the most critical workloads and with significant volume. We are delighted to not only have OpenStack as our foundation but to also have joined one of the largest and fun open source communities in the world to work with.

 

Stay tuned for more updates from previous Superuser Award winners!

 

Cover Image: City Network

The post Where are they now? Superuser Awards: City Network appeared first on Superuser.

by Ashlee Ferguson at June 18, 2020 11:00 AM

Ghanshyam Mann

OpenStack Ussuri is Python3-Only: Upgrade Impact

    A brief history of Python2 -> Python3:

Python version 2.0 was officially released in 2000, OpenStack was founded in 2010 and has since used Python 2.0 as its base language. The Python Foundation realized that in order to prevent users from having to perform tasks in a backward or difficult way, big improvements needed to be made to the software.

We released Python 2.0 in 2000. We realized a few years later that we needed
to make big changes to improve Python. So in 2006, we started Python 3.0.
Many people did not upgrade, and we did not want to hurt them. So, for many
years, we have kept improving and publishing both Python 2 and Python 3.

In 2015, The Python Foundation made a very clear announcement on multiple platforms to migrate to Python 3 and discontinue Python 2. This initial plan was later extended to 2020.

We have decided that January 1, 2020, was the day that we sunset Python 2.
That means that we will not improve it anymore after that day, even if
someone finds a security problem in it. You should upgrade to Python 3
as soon as you can.

    OpenStack Starting Support of Python 3:

With the announcement of the sunset of Python 2, it became very clear that OpenStack also could not support Python 2 for much longer. Because it would have been impossible to fix any security bugs on Python 2, it was better for OpenStack to drop its support completely and instead concentrate on Python 3.

OpenStack’s support of  Python 3 started in 2013, and many developers contributed towards the enormous task of transitioning the software. After so much hard work from the community, the Stein cycle (September 2018) was the time when running OpenStack under Python3 as default work became a community goal. The community goal is a way to achieve common changes in OpenStack. “OpenStack runs under Python3 as default” was a great effort and includes a lot of hard work by many developers. Doug Hellmann was one of the key developers and showed coordination and leadership with other developers and projects to finish this goal.

    OpenStack Train (Oct 2019): Python3 by default:

In the OpenStack Train release (October 2019), OpenStack was tested on Python 3 by default. This meant that you could upgrade your cloud to Python 3 environment with full confidence. OpenStack Train was released with well tested Python 3 support, but still also supported Python 2.7. At the same time, we kept testing the latest Python 3 version, and the OpenStack Technical Committee (TC) started defining the testing runtime for each cycle. OpenStack will targeting Python 3.8 in the ‘‘Victoria’‘ development cycle.

    OpenStack Ussuri (May 2020): Python3-Only: Dropped the support of Python2:

With the Ussuri cycle, OpenStack dropped all support of Python 2. All the projects have completed updating their CI jobs to work under Python 3. This achievement allows the software to be able to remove all Python 2 testing as well as the configuration that goes along with it..

Very first thing in the Ussuri cycle, we started planning for the drop of Python 2.7 support. Dropping Python 2.7 was not an easy task when many projects depend on each other and also integrate CI/CD. For example, if Nova drops Python 2.7 support and becomes Python 3 only, it can break Cinder and many other projects’s CI/CD. We prepared a schedule and divided the work into three phases, dropping support from services first, then library or testing tools.

    Phase-1: Start of Ussuri -> Ussuri-1 milestone: OpenStack Services to start

             dropping the py2.7 support.

    Phase-2: milestone-1 -> milestone-2:  Common libraries and testing tooling

    Phase-3: at milestone-2: Final audit.

Even still, a few things got broken in initial work. So, we made DevStack as Python 3 by default which really helped move things forward. In phase 2, when I started making Tempest and other testing tools as python3-only, a lot of stable branch testing for python2.7 started breaking. That was obvious because Tempest and many other testing tools are branchless, meaning the master version is being used for testing both the current and older releases of OpenStack. So all Python 2.7 testing jobs were using the Tempest master version. Finally, capping Tempest version or fixing Tempest installed in py3 venv made all stable branches and master testing green.

Just a couple of weeks before the Ussuri release, we completed this work and made OpenStack as python3-only, with an updated wiki page. Two projects, Swift and Storlets, are going to keep supporting Python 2.7 for another one or two cycles.

    What “OpenStack is Python3-Only” means for Users/Upgrades:

If your existing cloud is on Python 3 env, then you do not need to worry at all. If it is on Python 2.7 and you are upgrading to Ussuri,then you need to check that your env has the Python 3.6 or higher version available. From the Ussuri release onwards, OpenStack will be working on Python 3.6 or higher only. For example, if you want to install Nova Ussuri version, then it will give an error if Python 3.6 or higher is not available. It is done via metadata (“python-requires = >=3.6”) in setup configuration file. Below is the screenshot of how the setup config file looks in the Ussuri release onwards:

python-requires = >=3.6

classifier =

      Environment :: OpenStack

      Intended Audience :: Information Technology

      Intended Audience :: System Administrators

      License :: OSI Approved :: Apache Software License

      Operating System :: POSIX :: Linux

      Programming Language :: Python

      Programming Language :: Python :: 3

      Programming Language :: Python :: 3.6

      Programming Language :: Python :: 3.7

      Programming Language :: Python :: 3 :: Only

      Programming Language :: Python :: Implementation :: CPython

If you are using a distribution that does not have Python 3.6 or higher available, then you need to upgrade your distro first. There is no workaround or any compatible way to keep running OpenStack on Python 2.7. We have sunset the Python 2.7 support from Ussuri onwards, and the only way is to also upgrade your python version. There are a few questions on the python upgrade which are covered in the FAQ section below.

    FAQ:

Q1: Is Python 2 to Python 3 upgrade being tested in Upstream CI/CD?

Answer: Not directly, but it is being tested indirectly. We did not set up the grenade testing (upstream upgrade testing) for py2 setup to py3 setup. However, previous OpenStack releases like Stein and Train were tested on both python versions. This means that the OpenStack code was working or well-tested on both python versions before it was python3 only. This makes sure that upgrading the py2->py3 for OpenStack has been tested indirectly. If you are upgrading OpenStack from Stein or Train to Ussuri, then there should not be any issues.

Q2: How are the backport changes from Ussuri onwards to old stable branches going to be python2.7 compatible?

Answer: We still run the Python 2.7 jobs until Stable Train testing so that any backport from Ussuri or higher (which are tested on Python 3 only) will be backported on Train or older stable branches with testing on Python 2.7 also. If anything breaks on Python 2.7, it will be fixed before backporting. That way we will keep Python 2.7 support for all stable branches before Ussuri.

Q3: Will testing frameworks like Tempest which are branchless (using the master version for older release testing) keep working for Python 2.7 as well?

Answer: No. We have released the last compatible version for Python 2.7 for Tempest and other branchless deliverables. Branchless means that the tools master version is being used to test the current or older OpenStack releases. For example, Tempest 23.0.0 can be used as a Python 2.7 supported version and Tempest 24.0.0 or master is Python 3 only. But there is a way to keep testing the older Python 2.7 release also (until you upgrade your cloud and want Tempest master to test your cloud). You can run Tempest on a Python 3 node or virtual env and keep using the master version for testing Python 2.7 cloud. Tempest does not need to be installed on the same system as other OpenStack services, as long as the APIs are accessible from the separate testing node, or the virtual env Tempest is functioning.

For any other questions, feel free to ping on the #openstack-dev IRC channel.

by Ghanshyam Mann at June 18, 2020 01:56 AM

June 17, 2020

OpenStack Superuser

Project Teams Gathering (PTG) Recap

It has been a week since the first virtual Project Teams Gathering (PTG)! While it was not the same experience as our traditional in-person PTGs, the community made the best of the current situation. We’ve been amazed by how many people have joined us online this year to collaborate across different time zones on the OSF projects. We not only had the highest attendance and gender diversity in this PTG, but also had 20 more countries represented than the Denver PTG last year. Thank you to all the community members who have worked together virtually in this unprecedented time to collaborate without boundaries. 

PTG Participation by Day *

If you didn’t attend—or if you did and want a replay—we have collected the project announcements, community updates, and discussions you may have missed.

Airship:

The Airship community has participated in the June Virtual PTG and made progress on secrets, deployment configurations, and AirshipUI. The community saw higher participation than usually seen in person, and good cross-team collaboration with 3 other groups: StarlingX, Ironic, and the Edge Working Group. You can see the Airship project PTG agenda and recordings here.

Kata Containers:

The Kata Containers community had its first PTG event on June 2-3, 2020, including contributors and users from what felt like all of the timezones. The sun did not set on the discussion! 

Peng Tao from Ant Financial and Eric Ernst from Ampere facilitated several sessions covering a range of topics. While it didn’t involve as much hands-on-hacking as in past meetups, the community was able to take the time for more in depth discussions, and demonstrations of ongoing work. Check out the Kata Containers PTG update here.

OpenStack:

The OpenStack community had a great virtual PTG, and several teams have posted summaries on the mailing-list. Kendall Nelson from the OpenStack Technical Committee has summarized and posted the Victoria vPTG summaries and PTG OpenStack TC update: Victoria vPTG Summary of Conversations and Action Items on the OpenStack blog. If there is a particular action item you are interested in taking, please reply to the mailing list thread.

StarlingX

The StarlingX community participated in the first virtual PTG that was held online. At the virtual PTG on June 1-5, the StarlingX community spent almost ten hours discussing current technical challenges and new proposals as well as community-related topics. Check out the StarlingX PTG recap and learn about the discussion that happened at the event. 

In addition, you might also be interested in checking out the summary of the OSF Edge Computing Group sessions at the event.

*Attendees’ attendance might be counted twice in the same day if they are participating in multiple sessions

The post Project Teams Gathering (PTG) Recap appeared first on Superuser.

by Sunny Cai at June 17, 2020 08:00 PM

OSF Edge Computing Group PTG Overview

The first virtual Project Teams Gathering (PTG) was held on June 1-5 in all time zones around the globe providing the opportunity to anyone to join including contributors of the OSF Edge Computing Group (ECG).

The group had three sessions during the first three days of the week where we spent a total of seven hours to discuss topics relevant to edge computing use cases, technologies, and architectures. To take advantage of the virtual format, we also invited adjacent communities to participate like CNTT, OPNFV and the Kubernetes edge groups.

We started the sessions with the introduction of the work that the Edge Computing Group has been doing to define reference models and architectures that satisfy the requirements of most edge computing use cases and prepare for some common error cases. The main discussion point is the level of autonomy that an edge site requires which, among other things, affects the available functionality in case of losing the network connectivity towards the central data center. The two identified models are the Centralized Control Plane and the Distributed Control Plane.

The ECG has defined reference architectures to realize the above models with OpenStack services and started testing activities as well to verify and validate functionality. The purpose of the sessions at the PTG was to gather feedback about the models and to improve the reference architectures with adding new components and discuss the options to run all types of workloads at the edge.

We touched on TripleO’s Distributed Compute Node (DCN) architecture which is an example of the Centralized Control Plane model. Our discussions were circulating around challenges of edge deployments, such as latency: “100ms is a killer for distributed systems over WAN”; nodes getting out of sync can be a big issue. We also talked about improvements like Ceph being available since the OpenStack Train release compared to only ephemeral storage prior to that, and increased number of edge nodes that are running compute services and workloads.

We spent a longer amount of time discussing the Distributed Control Plane model which was in interest for the CNTT community as well therefore we discussed details about ways to implement this option. Some of the meeting participants have already been deploying OpenStack on edge sites which requires shrinking the footprint to prepare for limited hardware resources which is one of the common constraints of edge use cases. In case of running all the controller services at the edge, the resource usage can be a challenging factor, but it’s not an unsolved problem. Another popular option to discuss is the federated approach that is supported by components such as the OpenStack Identity Service (Keystone).

As an example to the distributed model, we had a short discussion about StarlingX and some of the design decisions the project has made to shape the project’s architecture. StarlingX is integrating well-known open source components such as OpenStack, Kubernetes, Ceph, etc into one platform along with services for software and hardware management that are developed by the community. During the PTG session, we discussed the Distributed Cloud feature in more details to understand how StarlingX manages the edge sites which can have full autonomy in case of network failures while still managed centrally. Discussion topics included understanding what is synchronized and shared between the nodes to ensure smooth operation in different scenarios and essential functionality for edge, such as zero touch provisioning.

StarlingX is running the majority of the platform services in containers and also provides the possibility to have edge sites with only container workloads in the architecture. The mention of containers lead the discussion towards better understanding the requirements towards container orchestration tools such as Kubernetes in edge infrastructure. We talked a bit about concepts such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Container as a Service (CaaS) and how the lines between these have started to disappear recently. The focus was on requirements on Kubernetes from a CaaS viewpoint while we also took a look at how it impacts the reference architectures. We need to understand storage and networking configurations as well as handling data crucial to run containers like quotas and certificates.

During the PTG sessions the participants took the opportunity to talk about storage which is an area that the ECG hasn’t gotten the chance yet to look into. We concentrated on object storage this time as the block storage strategies are a bit more straight forward. We agreed that the primary usage of object storage is to provide service for the applications, but it is useful for the platform too, like sharing images as a backend to the OpenStack Image Management (Glance) service. We had participants on the meeting from the OpenStack Object Storage (Swift) team to identify use cases and requirements for this component to take into account during the design and development process. The main use case we discussed was Content Delivery Networks (CDN) to leverage this functionality while online backup and gaming can also be considered. For design decisions we started to discuss architectural considerations and the effects of circumstances such as latency to the components of Swift.

As the PTG is a great opportunity to sync with project teams, we had joint sessions with the OpenStack Network Connectivity as a Service (Neutron) team and the OpenStack Accelerator (Cyborg) team. To also cover cross-community discussions, we had a session with the Airship project as well as KubeEdge from the CNCF community.

One of the ongoing discussions with Neutron is the handling of network segment ranges and to make sure they are defined and handled properly in the distributed environment in both architectural models. There is a feature request for Neutron that is already approved and has the potential to get priority during the Victoria release cycle. The first step is to put together a proof of concept deployment to test the planned configurations and changes. The Neutron team and ECG contributors will work on testing as a joint effort. A further relevant project is Neutron Interconnection which is mainly API definitions at this point as an effort to provide interconnection between OpenStack deployments and regions over WAN. Further networking related topics included functionality such as Precision Time Protocol that the StarlingX community is already working on along with Time Sensitive Networking (TSN).

The next cross-project session was the sync with the Cyborg team. This session was in big interest as the ability to use hardware accelerators is crucial for many edge use cases. During the session we were focusing on learning about the current activities within the project such as the implementation and next steps for the new v2 API. We also touched on device driver integration. Cyborg is concentrating on the ability of programming the acceleration devices made available to the applications and will not include low level device driver integration in the upstream code. The Cyborg team is working with Neutron, OpenStack Placement and OpenStack Compute (Nova) teams to ensure smooth integration and full functionality in these areas.

During the sync sessions we were also focusing on relevant topics such as lifecycle management of the platform services. One of the main challenges is to handle upgrades throughout the whole edge infrastructure which can be a big headache in massively distributed systems. In some use cases downtime is almost unacceptable which means that we need to prepare the edge site to have enough resources to keep services running while you are performing an upgrade. When it is not doable, we need to identify processes to minimize the time when the services are unavailable.

In connection to this theme we were talking to the Airship community as the project’s main mission is to address deployment and lifecycle management of software such as OpenStack and Kubernetes and therefore can be an option to address the aforementioned challenges. Airship is utilizing more and more components from the CNCF landscape as their concepts are using containers heavily for flexibility. For in place upgrades, Airship will use Cluster API and the concept of dual box deployment at edge sites which would ensure that there is always a replica of each service that provides availability during an upgrade process.

Our last session was with the KubeEdge project which is focusing on the usage of Kubernetes for edge computing use cases. It is built on top of Kubernetes with extensions such as application management, resource management and device management for IoT devices. Their main focus areas are IoT, CDN and Multi-access Edge Computing (MEC) scenarios. The project is releasing every three months and has an increased focus on IoT protocols. Their architectural model follows the Centralized Control Plane and running only worker nodes at the edge. As we had limited time during the PTG, we agreed to follow up with the collaboration after the event as well to work together on providing better solutions for edge computing use cases.

After the PTG we have a lot of action items to follow up on to evolve the architecture models as well as to help projects and adjacent communities with our learnings that they can use when they are working on their architecture solutions. We will keep looking closer into the object storage use cases and requirements to add that to our reference models and we are also working on setting up discussions with further Kubernetes groups such as the Kubernetes Edge and IoT group. We will also follow up on relevant implementation work and start testing activities when the software components are ready.

As a project highly relevant for edge, you might also be interested in checking out the StarlingX virtual PTG summary to learn about the community’s current activities and plans for the upcoming releases.

If you are interested in participating in the OSF Edge Computing Group, sign up to our mailing list and check out our weekly calls. You can find more information about the group’s activities on our wiki page. For a more cohesive summary of our work read the new white paper the group just published!

The post OSF Edge Computing Group PTG Overview appeared first on Superuser.

by Ildiko Vancsa at June 17, 2020 01:00 PM

June 16, 2020

OpenStack Blog

Victoria vPTG Summaries

The OpenStack community had a great virtual Project Teams Gathering (PTG). The first virtual event hosted 44 projects and spanned all timezones. Since the event concluded, many of those teams have posted summaries of the discussions they have had and the decisions that were made during the PTG.   As vice chair of the Technical Committee,... Read more »

by Kendall Nelson at June 16, 2020 10:48 PM

OpenStack Superuser

Zuul: A Wazo Platform Case Study

Wazo Platform is an An open source software programmable telecommunication platform. Wazo Platform allows you can pick and choose the components you need to build your infrastructures. Most of Wazo Platform’s 229 git repositories on GitHub use Zuul. Zuul allows Wazo Platforms to have cross repository dependencies in pull requests at the source code level, the Debian packaging level and at the container level. Similarly the nodes support their private OpenStack containers hosted in AWS CM Instance.

We got talking with Frederic Lepied to get some answers to why Wazo Platforms chose Zuul, an open source CI tool, and how they use it with GitHub and OpenStack.

How did your organization get started with Zuul?

We had people coming into the organization with a lot of expertise in OpenStack so it was a natural move for us.

Describe how you’re using it:

Zuul is used with GitHub on most of our 229 git repositories at https://github.com/wazo-platform. The Zuul instance is hosted on AWS and is public at https://zuul.wazo.community/zuul/t/local/status. The nodes are VMs from our private OpenStack and containers hosted in AWS CM instance.

What is your current scale?

The scale in term of nodes is small (around 10) and big in term of repositories (229).

What benefits has your organization seen from using Zuul?

Zuul allows us to have cross repository dependencies in pull requests at the source code level, the Debian packaging level, and at the container level.

It also allows us to reuse job definitions among all the repository of the same type, which is very helpful.

What have the challenges been (and how have you solved them)?

Operations are difficult and we got a lot of help from the Software Factory team.

What are your future plans with Zuul?

We plan to continue to use Zuul for more tasks.

Are there specific features that drew you to Zuul?

Definitely the cross-repository dependencies.

 

The post Zuul: A Wazo Platform Case Study appeared first on Superuser.

by Helena Spease at June 16, 2020 01:00 PM

June 15, 2020

VEXXHOST Inc.

4 Benefits of Going Serverless

Serverless architecture is forecasted to gain massive traction in the years to come. According to researchers, by 2020, 20% of global enterprises will operate in serverless computing. So what is serverless that it garners such interest? Serverless is a cloud service utilized without the worry of an operating system and underlying infrastructure. The definition of serverless architecture itself is a benefit.

Serverless, known as Function-as-a-service does use servers, both physical and virtual. But since the developers do not interact with them, they have the illusion of operating without servers! Therefore, serverless does not mean the elimination of servers from distributed applications.

Why Serverless

In terms of cloud service adoption, serverless leads among all other cloud services. Serverless architecture is suitable for those building light-weight applications and are concerned about time to market. Let’s have a look at what are some known advantages of going serverless.

Operation Cost

The number one reported benefit of going serverless is the reduction in operation cost. Serverless is the alternative to buying racks and racks of servers to operate a full-fledged cloud. You tend to pay for more than the actual resources used when opting for a complete cloud environment. But with serverless, payment is for per unit of resources consumed.

Scalability

Your traditional cloud deployment is according to maximum usage. Also, the scalability of such a set up is not entirely customizable. Whereas, with serverless, resources added are according to current usage. Furthermore, scalability is for the exact amount as needed by your organization. Therefore, no idle cloud resources are running the background.

Business Value

Aren’t we always looking to increase productivity? Be it in terms of the tool you use or the people who run those tools for you; efficiency is vital. With serverless architecture, your developers can do what they do best: code! So, the infrastructure layer is taken care of by the provider while you focus on your business’ core operations. The technology is best utilized for its agility, allowing you to test and make changes in applications through a quicker turnaround.

Multi-Platform Integration

It is noteworthy that serverless technology is increasingly becoming a part of hybrid platforms. Traditional cloud deployments like a public or private cloud do provide a lot more features when building applications. But serverless is for the in-between. Decision-makers are transitioning towards a multi-platform world, making use of PaaS, containers, and serverless to maximize productivity.

OpenStack and Serverless Architecture

If you did not already know, OpenStack too supports serverless architecture. The OpenStack project Qinling provides a platform for serverless functionality. Additionally, Qinling supports container orchestration platforms and function package storage backends.

If serverless architecture is for you or not depends on your use case. There are multiple guides out there to help you get started on OpenStack serverless architecture. But with over nine years of OpenStack experience, VEXXHOST is here to walk you through everything you need. Our OpenStack Consulting service can help you determine if you should reap the benefits of serverless or go the traditional way.

Would you like to know about how you can get Virtual Machines, Bare Metal and Kubernetes in one environment? Download our white paper and get reading!  

Virtual Machines, Bare Metal and Kubernetes: All in One Environment!

The post 4 Benefits of Going Serverless appeared first on VEXXHOST.

by Samridhi Sharma at June 15, 2020 05:28 PM

Mirantis

The ultimate guide to Kubernetes

Here at Mirantis we're committed to making things easy for you to get your work done, so we've decided to put together this guide to Kubernetes.

by Nick Chase at June 15, 2020 04:47 PM

Galera Cluster by Codership

Improved security audit features in Galera Cluster for MySQL 5.7.30, and an updated 5.6.48

Codership is pleased to announce a new Generally Available (GA) release of the multi-master Galera Cluster for MySQL 5.6 and 5.7, consisting of MySQL-wsrep 5.6.48 (release notes, download) and MySQL-wsrep 5.7.30 (release notes, download) with a new Galera Replication library 3.30 (release notes, download), implementing wsrep API version 25. This release incorporates all changes to MySQL 5.6.48 and 5.7.30 respectively, making it a MySQL High Availability solution.

A highlight of this release is that with MySQL 5.7.30, you will now have access to using the Percona audit log plugin, which will help with monitoring and logging connection and query activity that has been performed on specific servers. This implementation is provided as an alternative to the MySQL Enterprise Audit Log Plugin.

The Galera Replication library 3.30 has an enhancement to ensure that upon GCache recovery, all available space will be reclaimed in the ring buffer. Frequent cluster configuration changes handling of errors are also improved.

MySQL-wsrep 5.6.48 is an updated rebase to the 5.6.48 release, but also includes improvements around crash recovery: when binary logs are enabled, there is a more consistent recovery. SSL initialization has seen improvements, and error handling of cluster wide conflicts have been improved when the cluster itself is acting as an asynchronous secondary to a MySQL primary.

MySQL-wsrep 5.7.30 is an updated rebase to the 5.7.30 release, and in addition to what is present in 5.6.48, there is also the audit log plugin as mentioned above. One important note is that for your Galera Cluster, ensure that InnoDB tablespaces are kept within the data directory (if kept outside, they may not be copied over during SST).

Across the board, there is now also support and packages for CentOS 8 and RHEL 8.

You can get the latest release of Galera Cluster from http://www.galeracluster.com. There are package repositories for Debian, Ubuntu, CentOS, RHEL, OpenSUSE and SLES. The latest versions are also available via the FreeBSD Ports Collection.

 

by Sakari Keskitalo at June 15, 2020 10:45 AM

June 12, 2020

OpenStack Blog

OpenStack TC: Victoria vPTG Summary of Conversations and Action Items

I hope you all had a productive and enjoyable PTG! While it’s still reasonably fresh, I wanted to take a moment to summarize discussions and actions that came out of TC discussions.  If there is a particular action item you are interested in taking, please reply on the mailing list thread. For the long version,... Read more »

by Kendall Nelson at June 12, 2020 07:39 PM

OpenStack @ NetApp

My Name is Ussuri!

The OpenStack Ussuri release is now available! NetApp is proud to have contributed in the development of Cinder and Manila. The Ussuri release includes the following new features and feature enhancements that are supported by NetApp’s ONTAP and SolidFire storage platforms for your OpenStack deployment:  Ability to create shares from snapshots in aggregates and controllers other than the source share using Manila  Manila now efficiently creates shares from snapshots […]

The post My Name is Ussuri! appeared first on thePub.

by Carlos Da Silva at June 12, 2020 07:03 PM

Fleio Blog

Fleio 2020.06: Beta release, more angular news, scheduled backups with celery beat and more

Fleio 2020.06.1 was released. The latest version was published on 2020-06-17 and it’s a stable release. Beta release Starting with 2020.06 release we are changing the way on how we release Fleio. The first version (2020.06.0) will be a beta one, and will not be pushed to the public repository. Packages will be available only […]

by Marian Chelmus at June 12, 2020 06:13 AM

June 11, 2020

OpenStack Superuser

Women of Open Infrastructure: Meet Victoria Martinez de la Cruz from the OpenStack Manila Project

This post is part of the Women of Open Infrastructure series to spotlighting the women in various roles in the community who have helped make the Open Infrastructure successful. With each post, we learn more about each woman’s involvement in the community and how they see the future of Open Infrastructure taking shape. If you’re interested in being featured or would like to nominate someone to tell their stories, please email editor@openstack.org.

This time we’re talking to Victoria Martinez de la Cruz from the OpenStack Manila project. She tells Superuser about how the Open Infrastructure community has helped her to grow and excel at her role in the community.

What’s your role (roles) in the Open Infrastructure community?

Currently, I’m mainly focused on contributing to feature development, code enhancements and bug fixing for the Shared Filesystems as a Service project for OpenStack (Manila). I also enjoy mentoring, so I’ve been helping with the mentoring efforts for the Outreachy program.

What obstacles do you think women face when getting involved in the Open Infrastructure community?   

There are two factors that I think lead us to this situation: on one hand, I believe that people in tech tend to have the idea that some tasks are harder than others. For example, some people think that web development is easier than drivers development, something I believe is a mistake. Historically everything related to infrastructure administration has been considered a hard branch of computing. On the other hand, we know that there is a tendency for women to aspire to perfection and try to know everything before applying for a position or for taking the lead on a task. And, the harder the task is considered to be, the less women there are. Both things lead to the fact that not many women feel they can contribute to the projects we have in the OpenInfra community. I do believe there are women with the expertise and potential that we need, it’s just a matter of reaching them, and giving them opportunities. Some effort I think is being done by many companies in the community and also by other inclusion efforts such as Outreachy. We could do more though, for sure.

Why do you think it’s important for women to get involved with open source?

It gives them exposure, something that I think is critical in this market: exposure to real-world use cases to learn from, exposure to other people from different backgrounds, cultures and levels of expertise to work with, and exposure for them to become well known in the field, to expand their networks, and find which is the next step in their careers that challenges them and takes them to the next level.

Efforts have been made to get women involved in open source, what are some initiatives that have worked and why?

Yes, there have been many efforts, that makes me very proud of my community. One of the initiatives that worked out very well has been Outreachy, and I think because it helped underrepresented individuals with no experience contributing to open source to their first steps. Guided by mentors, the interns learned how the processes in open source are and how they could contribute to. They also gained experience with the tools and technologies we use. Several good contributions have been made by our interns since they could contribute fresh perspectives and new ideas. Some of them continue to be involved with the community because they could connect with hiring companies and take a full-time role. This is something I’d love to see more. There are not so many remote entry level positions around for our interns to apply to, and I think that is a big loss, because we invest in people with high potential and then we let them go.

Open source moves very quickly. How do you stay on top of things and what resources have been important for you during this process?

It’s definitely one of the biggest challenges we face. I believe that having an open and sharing community is key. It’s easier to keep up with things by sharing curated information between the collaborators than going ahead and reading all the material that is around. Material is good, but we are constantly being overloaded with details and stuff, which I think is good when you have the time or you actually want to focus on that specific topic. Otherwise, a summary by your peers is all that you need. That has been my strategy this last couple of years: I get updates from the community (either over IRC, reading the mailing list or watching presentations at conferences) and if I need to get a better understanding of something, I just go ahead and dig into the sea of articles and blog posts.

What advice would you give to a woman considering a career in open source? What do you wish you had known?

DO IT! No hesitation. I’m very grateful to everything that open source, and especially the Open Infrastructure community, has given to me: my colleagues, the technical experience I gained, my personal and professional growth. All experiences are different, for sure, but you need to go ahead and try. And the OpenInfra community? 10/10 would recommend to a friend.

I wish I had known how to organize my time better. I struggled a bit with that in the first couple of years. The amount of things happening at the same time was too much for me, and the context change used to kill my productivity. Plus, working remotely is not so straightforward as some people might think. Nowadays, taking the advice from several people from the community, I learned how to organize myself better and end my days feeling good about what I have accomplished.

The post Women of Open Infrastructure: Meet Victoria Martinez de la Cruz from the OpenStack Manila Project appeared first on Superuser.

by Superuser at June 11, 2020 02:00 PM

June 10, 2020

Ben Nemec

Oslo Virtual PTG for Victoria

The Oslo team held its second virtual PTG this week. We had a number of good discussions and even ran slightly over the 2 hours we scheduled, so I think it was a successful event. The first hour was mostly topics relating to Oslo itself, while the second hour was set aside for some cross-project discussions with the Nova team. Read on for details of both hours.

oslo.metrics

Thierry gave us a quick update on the status of oslo.metrics. Currently we have all the necessary infrastructure in place to add oslo.metrics, so we are just waiting on a code drop. Once we have code, we'll get it imported and review the API with the Oslo team to ensure that it fits with our general design standards.

However, just creating the library is not the end of the story. Once the library is released, we'll need to add the integration code for services like oslo.messaging. Once that is done it should be possible for deployers to start benefiting from the functionality provided by oslo.metrics.

Update! The above represents what was discussed at the PTG, but since then it has come to our attention that the oslo.metrics code was proposed. So in fact we are ready to start making progress. If this project is of interest to you please review the changes and propose patches.

oslo.cache

There were a couple of things we discussed about oslo.cache because we've been doing a fair amount of work in that library to improve security and migrate to better supported client libraries. The first topic was functional testing for the drivers we have. Some functional testing has already been added, but there are a number of drivers still without functional coverage. It would be nice to add similar tests for those, so if anyone is interested in working on that please let us know.

We've also been looking to change memcached client libraries for a couple of cycles now. Unfortunately this is not trivial, so the current plan is to add the new library as a completely different driver and then deprecate the old driver. That way we just need a migration path, not complete compatibility between the old and new libraries. There was a question about what to do with the dead host behavior from the existing memcache_pool driver, but the outcome of the discussion for now was to continue with the non-pooled driver and leave the pool version for later.

Retrospective

The Good
First, we added a new core, Sean McGinnis. Big thanks to Sean for all his work on Oslo over the past couple of cycles!

The oslo-coresec team was updated. Prior to this cycle it had gotten quite out of date, to the point where I was the only active Oslo contributor still on it. That's not ideal since this is the group which handles private security bugs, so keeping the list current is important. We now have a solid group of Oslo cores to look at any such bugs that may come in now.

We had also held our first virtual PTG after the Shanghai event, and that was counted in the success column. With so many Oslo contributors being part-time on the project it's likely we'll want to continue these.

A couple of our cores were able to meet in person at FOSDEM. Remember meeting in person? Me neither. ;-)
The Bad
We missed completing the project contributor documentation community goal. This goal was more difficult for Oslo than for many other projects because we have so many repositories under our governance. By the end of the cycle we did come up with a plan to complete the goal and have made good progress on it.
Proposed Changes
We discussed assigning a driver for community goals in the future. One of the problems with the contributor docs goal was that everyone assumed someone else would take care of it. Having a person specifically assigned should help with that.

In addition, at least one of the community goals proposed for the Victoria cycle would not require explicit completion by Oslo. It involves users of oslo.rootwrap migrating to oslo.privsep, which would only require the Oslo team to assist other projects, and hopefully work to improve the oslo.privsep docs based on peoples' migration experiences. Otherwise, Oslo isn't a consumer of either library so there is no migration needed.

Another proposed community goal is to make ci jobs zuulv3 native, and I believe that Oslo is already done with that for the most part. I know we've migrated a few of our one-off jobs over the past couple of years since zuulv3 came out so we should be in good shape there too.

Policy Cross-Project with Nova

After the Nova Ussuri release, some deployers reported problems with the new policy rules, despite the use of the oslo.policy deprecation mechanism that is designed to prevent breakage on upgrade. It turned out that the problem was that they were using the sample policy generator tool to create JSON policy files. The problem with this is that JSON doesn't support comments, so when you create a sample file in that format it overrides all of the policy-in-code defaults. When that happens, the deprecation mechanism breaks because we don't mess with custom policies specified by the deployer. This is one of the reasons we don't recommend populating the policy file with default policies.

However, even though we've recommended YAML for policy files since policy-in-code happened, we never changed the default filename in oslo.policy from policy.json. This naturally can lead to deployers using JSON formatted files, even though all of the other oslo.policy tools now default to YAML. One of the main reasons we've never changed the default is that it is tricky to do without opening potential security holes. Policy has a huge impact on the security of a cloud and there isn't a great option for migrating the default filename.

The solution we came up with is documented in an Oslo spec. You can read up on the full details there, but the TLDR is that we are going to coordinate with all of the consumers of oslo.policy to add an upgrade check that warns deployers if a JSON-formatted policy file is found. In addition to release notes, this should give deployers ample warning about the coming change. oslo.policy itself will also log a warning if it detects that JSON is in use after JSON support has been deprecated. As part of this deprecation work, oslo.policy will need to provide a tool to migrate existing JSON policies to YAML, preferrably with the ability to detect default policy rules and comment them out in the YAML version.

Deprecating and eventually removing JSON policy file support should allow us to deprecate policies in the future without worrying about the situation we ran into this cycle. YAML sample files won't override any rules by default so we'll be able to sanely detect when default rules are in use. There was some talk of proposing this as a community goal given the broad cross-project nature of the work, but we'll probably wait and see how the initial effort goes.

Healthcheck Cross-Project with Nova

Another longstanding topic that has recently come up is a standard healthcheck endpoint for OpenStack services. In the process of enabling the existing healthcheck middleware there was some question of how the healthchecks should work. Currently it's a very simple check: if the api process is running it returns success. There is also an option to suppress the healthcheck based on the existence of a file. This allows a deployer to signal a loadbalancer that the api will be going down for maintenance.

However, there is obviously a lot more that goes into a given service's health. We've been discussing how to make the healthcheck more comprehensive since at least the Dublin PTG, but so far no one has been able to commit the time to make any of these plans happen. At the Denver PTG ~a year ago we agreed that the first step was to enable the healthcheck middleware by default in all services. Some progress has been made on that front, but when the change was proposed to Nova, they asked a number of the questions related to the future improvements.

We revisited some of those questions at this PTG and came up with a plan to move forward that everyone seemed happy with. One concern was that we don't want to trigger resource-intensive healthchecks on unauthenticated calls to an API. In the original discussions the plan was to have healthchecks running in the background, and then the API call would just return the latest results of the async checks. A small modification to that was made in this discussion. Instead of having explicit async processes to gather this data, it will be collected on regular authenticated API calls. In this way, regularly used functionality will be healthchecked more frequently, whereas less used areas of the service will not. In addition, only authenticated users will be able to trigger potentially resource intensive healthchecks.

Each project will be responsible for implementing these checks. Since each project has a different architecture only they can say what constitutes "healthy" for their service. It's possible we could provide some common code for things like messaging and database that are used in many services, but it's likely that many projects will also need some custom checks.

I think that covers the major outcomes of this discussion, but we have no notes from this session so if I forgot something let me know. ;-)

oslo.limit

There was quite a bit of discussion during the Keystone PTG sessions about oslo.limit and unified limits in general. There are a number of pieces of work underway for this already. Hierarchical quota support is proposed to oslo.limit and a POC for Nova to consume it is also available. The Glance team has expressed interest in using oslo.limit to add quotas to that service, and their team has already started to contribute patches to oslo.limit (such as supporting configuration by service name and region). This is terrific news! That work also prompted some discussion of how to handle the separate configuration needed for keystoneauth and oslo.limit itself.

There was quite a bit of other discussion, some of which doesn't involve oslo.limit, some of which does. We need to define a way to export limits from one project and import them into Keystone. This will probably be done in the [project]-manage commands and won't involve Oslo.

Some refinement of the usage callback may be in order too. I don't know that we came to any definite conclusions, but encouraging more projects to use Placement was discussed, although some projects are hesitant to do that due to the complexity of using Placement. In addition, there was discussion of passing a context object to the usage callback, but it wasn't entirely clear whether that would work for all projects or if it was necessary.

Finally, the topic of caching came up. Since there can be quite a few quota calls in a busy cloud, caching may be needed to avoid significant performance hits. It's something we've deferred making any decisions on in the past because it wasn't clear how badly it would be needed or exactly how caching should work for limits. We continued to push this decision off until we have unified limits implemented and can gather performance information.

That should cover my recollection of the limits discussion. For the raw notes from the PTG, see the Keystone PTG Etherpad, under Unified Limits.

That's All Folks!

The past few months have been quite...interesting. Everyone is doing the best they can with a tough situation, and this all-virtual PTG is yet another example of that. Huge thanks to all of the organizers and attendees for making it a productive event in spite of the challenges.

I hope this has been useful. If you have any comments or questions feel free to contact me in the usual ways.

by bnemec at June 10, 2020 05:09 PM

Slawek Kaplonski

My summary of the OpenStack Virtual_PTG July 2020

Retrospective From the good things team mentioned that migration of the networking-ovn driver to the core neutron went well. Also our CI stability improves in the last cycle. Another good thing was that we implemented all required in this cycle community goals and we even migrated almost all jobs to Zuul v3 syntax already. Not so good was progress on some important Blueprints, like adoptoion of the new engine facade. The other thing mentioned here was activity in the stadium projects and in the neutron-lib.

June 10, 2020 01:08 PM

OpenStack Superuser

Where are they now? Superuser Awards winner: China Mobile

We’re spotlighting previous Superuser winners who are on the front lines deploying OpenStack in their organizations to drive business success. These users are taking risks, contributing back to the community and working to secure the success of their organization in today’s software-defined economy.

China Mobile won the Superuser Award at the OpenStack Summit in Barcelona. As the world’s largest mobile phone operator with various OpenStack deployments, they are now not only building up the NFV network among world-leading operators based on OpenStack but also expanding the NFV platform to support 5G C in the near future. The following are their major OpenStack deployments.

What has changed in your OpenStack environment since you won the Superuser Awards?

We have carried out in-depth practice of OpenStack as the cloud infrastructure, especially in our Network Cloud, to build China Mobile’s NFV/SDN network. We believe it has directly promoted the maturity of the industry and the maturity of NFV/SDN solutions based on OpenStack, especially in a multi-vendor environment through our NFV/SDN pilot and NovoNet experimental network. China Mobile is now entered into the second year of NFV/SDN network construction since 2019. The network is about to commercialize in the second half of 2020.

What is the current size of your OpenStack environment?

The scale of China Mobiles NFV/SDN network cloud is huge, in eight regions across the whole country.

We believe China Mobile is now building up the biggest NFV network among worlds leading operators based on OpenStack, the total number of servers in the network is more than 50,000 now. For each OpenStack instance, it has to manage 500 to 1,500 servers. The virtual network functions (VNFs) running on top of the virtualization includes IP Multimedia Subsystem (IMS), Evolved Packet Core (EPC) and value added services etc. and We will also expand the NFV platform to support 5G C in the near future.

What version of OpenStack are you running?

Mitaka + (with some enhancements of Pike/Queens version are absorbed)

What open source technologies does your team integrate with OpenStack?

Our team mainly researches NFV system integration. Now we build the CI/CD process to carry out automatic software integration and testing. Different virtual infrastructure manager (VIM) (OpenStack based platform), NFV orchestrator (NFVO) and VNF are automatically deployed and tested through a unified platform. Currently, the CI/CD platform uses Docker technology

What workloads are you running on OpenStack?

  • 4G service, including 18 core network elements and service platforms, such as virtualized IMS, EPC, intelligent network and Short Message Service (SMS)/Multimedia Messaging Service (MMS) platform etc.
  • 5G service.

How big is your OpenStack team?

Its difficult to give the exact numbers, but we have different teams working on OpenStack, with one working on the additional telco requirements based on open source version, another team is to test the vendors product and make sure their commercial OpenStack software can support telcos communication services. After deployment, there would be another team continuously working on the operational aspects of the OpenStack.

How is your team currently contributing back to the OpenStack project?

At present, in terms of Network Cloud, China Mobile is mainly acted as an OpenStack user. With the emergence of Edge Computing, we are also closely following and contributing other OpenStack projects, such as StarlingX.

What kind of challenges has your team overcome using OpenStack?

The feasibility of OpenStack carrying telecommunication service has been fully verified. At present, in the aspect of network operation and management, OpenStack still needs to be enhanced and improved to achieve 5 Nines HA. In addition, more work needs to be done in the hyper-scale scenario to achieve high performance and high reliability of OpenStack.

 

The post Where are they now? Superuser Awards winner: China Mobile appeared first on Superuser.

by Superuser at June 10, 2020 07:00 AM

Stephen Finucane

What Is Nova?

This talk was delivered to a number of Red Hat interns at the start of their internship and served as a brief, high-level overview of the OpenStack Compute (nova) project.

June 10, 2020 12:00 AM

June 09, 2020

Galera Cluster by Codership

Galera Cluster 4 for MySQL 8 Release Webinar recording is now available

The much anticipated release of Galera Cluster 4 for MySQL 8 is now Generally Available. Please join Codership, the developers of Galera Cluster, and learn how we improve MySQL High Availability with the new features in Galera Cluster 4, and how you can benefit from using them. We will also give you an idea of the Galera 4 short term road map, as well as an overview of Galera 4 in MariaDB, MySQL and Percona.

Learn about how you can load data faster with streaming replication, how to use the new system tables in the mysql database, how your application can benefit from the new synchronization functions, and how Galera Cluster is now so much more robust in handling a bad network for Geo-distributed Multi-master MySQL.

WATCH THE RECORDING 

 

The slides for the webinar can found here

WEBINAR SLIDES

by Sakari Keskitalo at June 09, 2020 11:32 AM

June 08, 2020

OpenStack Superuser

Zuul: A T-Systems Case Study

Open Telekom Cloud is a major OpenStack-powered public cloud in Europe. It is operated for Deutsche Telekom Group by its subsidiary T-Systems International GmbH

Artem Goncharov, Open Telekom Cloud architect, shares why Open Telekom Cloud chose Zuul, the open source CI tool, and how they use it with GitHub and OpenStack.

How did your organization get started with Zuul

We started using Zuul for the development of OpenStack client components like SDKs, CLIs, and other internal operational software components. After we managed to get some changes merged into Zuul, we deployed it productively as our continuous integration system. Today it is our CI system for the development of all open source tooling we offer to our clients. Furthermore, Zuul is currently used for monitoring our platform services quality. For that, we periodically execute a set of tests. It also includes monitoring permanently our RefStack compliance. 

We prepare Zuul as an internal service for other departments inside Deutsche Telekom apart from our own projects in the future. We run Zuul on our own public cloud, the Open Telekom Cloud, and also spawn the VMs there. We are all-in OpenStack!

Describe how you’re using Zuul

Currently, we have Zuul working on a public domain interacting with GitHub. Although the CI workflow with Gerrit is very powerful, we observed that some users struggle with its complexity. We thus made a decision to stay with GitHub to allow more people in our community to participate in the development of our projects. Nodepool spins up the virtual machines for the jobs facilitating an OpenStack driver.

What is your current scale?

We have a five-node zookeeper cluster and each one scheduler, a nodepool-builder, and a nodepool-launcher. At present two Zuul executors satisfy our needs. We have about ten projects managed by Zuul but plan to increase this number up to 50 soon. On average we do 50 builds a day.

What benefits has your organization seen from using Zuul?

We are now prepared for growth. Today, our projects are clearly laid out in size and complexity, but we expect the complexity to grow. Therefore we are relieved to have gating in place ensuring all software is tested and consistent all the time. That allows us to scale the number of projects we cover.

Second, we have better control over where and how the build and test processes take place. Since we are testing real-life cloud scenarios, there are credentials for and access to actual cloud resources involved. With Zuul and Nodepool we have better control over these virtual machines and the stored data.

Last, but not least we have a rather complex integration and deployment workflow. It is not just software that we build and package, but we also create other artifacts like documentation, PyPI packages, and a lot more that requires extra steps. We like the flexibility of having Ansible playbooks defining those workflows.

What have the challenges been (and how have you solved them)?

It is important for us to test all aspects of our public cloud. This functional testing obviously includes logging into domains, creating resources, and dealing with all aspects of credentials. Since this setup is connected to GitHub and thus indirectly accessible for the public, we felt a bit uneasy to run the Zuul setup on the same platform where we conducted the actual tests and builds. Eventually, we segregated those scopes by means of several dedicated OpenStack domains, where only Zuul is having API access to. So in the worst case should credentials should ever leak, we just have to clean up and reset one of our test domains, but the Zuul infrastructure itself remains unaffected from that. We facilitate the “project cleanup” feature of the OpenStack SDK for that, to which we also contributed.

We also experienced functional tests or verification of refstack often leave a lot of debris behind, which was not cleaned up by the code, sometimes even because of failing API calls of OpenStack itself. We leverage “project cleanup” also to mitigate this behavior.

Zuul publishes also a lot of information in log files to public readable Swift containers. Our security teams complain about that, even if most of the information is harmless. In some cases, we patched Zuul or its jobs so this data does not accumulate in the first place.

Both for operational and security reasons, we’d like to containerize all workloads as much as possible. Zuul comes with a set of Docker containers. Unfortunately, especially the Nodepool-builder needs a lot of privileges, which is hard to implement with plain old Docker. Our approach is to leverage Podman as an alternative for that.

What are your future plans with Zuul?

The Gerrit code review system implements a sophisticated role model, which enables users to do code reviews, promote revisions, or to authorize the eventual merges. It is a real challenge to implement these access control features just with GitHub. As a workaround for the time being we use “/merge” comments on the pull requests.

Even though Zuul’s prime directive is to automate, sometimes it’s nice to be able to manually intervene. Unfortunately, there’s currently not really a UI for administrative tasks like re-building some artifacts. That would come in handy to migrate even more Jenkins jobs.

The operation of Zuul is complex and we currently don’t have a dedicated ops team. We decrease the effort of operations by implementing Ansible playbooks for that, but this an ongoing effort.

We work on transforming Zuul into an internal offering for other Deutsche Telekom subsidiaries and projects, so they also start using it. We’re also very interested in enabling Kubernetes and OpenShift to act as an operations platform for Zuul. Here the challenge is inherited from multi-cloud issues that are required by high availability.

Are there specific Zuul features that drew you to Zuul?

Zuul fuels the development of OpenStack, which is a remarkable job and a considerable responsibility. We are impressed by how scalable and flexible it is and have even adapted its architecture to internal projects. We’re confident that there is more to come.

The post Zuul: A T-Systems Case Study appeared first on Superuser.

by Helena Spease at June 08, 2020 02:00 PM

June 06, 2020

Ed Leafe

Day 12: Communities and Survivorship Bias

Communities, especially Open Source communities, tend to form some form of governance once they grow beyond a certain size. The actual size isn’t as important as the relationship among the members: when everyone knows everyone else, there’s really no need for governance. But when individuals come from different companies, or who otherwise may have different … Continue reading "Day 12: Communities and Survivorship Bias"

by ed at June 06, 2020 06:39 PM

About

Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.

Subscriptions

Last updated:
August 08, 2020 09:22 AM
All times are UTC.

Powered by:
Planet