April 20, 2018

RDO

Community Blogpost Round-up: April 20

The last month has been busy to say the least, which is why we haven’t gotten around to posting a recent Blogpost Roundup, but it looks like you all have been busy as well! Thanks as always for continuing to share your knowledge around RDO and OpenStack. Enjoy!

Lessons from OpenStack Telemetry: Deflation by Julien Danjou

This post is the second and final episode of Lessons from OpenStack Telemetry. If you have missed the first post, you can read it here.

Read more at https://julien.danjou.info/lessons-from-openstack-telemetry-deflation/

Unit tests on RDO package builds by jpena

Unit tests are used to verify that individual units of source code work according to a defined spec. While this may sound complicated to understand, in short it means that we try to verify that each part of our source code works as expected, without having to run the full program they belong to.

Read more at https://blogs.rdoproject.org/2018/04/unit-tests-on-rdo-package-builds/

Red Hatters To Present at More Than 50 OpenStack Summit Vancouver Sessions by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform

OpenStack Summit returns to Vancouver, Canada May 21-24, 2018, and Red Hat will be returning as well with as big of a presence as ever. Red Hat will be a headline sponsor of the event, and you’ll have plenty of ways to interact with us during the show.

Read more at https://redhatstackblog.redhat.com/2018/04/13/openstack-summit-vancouver-preview/

Lessons from OpenStack Telemetry: Incubation by Julien Danjou

It was mostly around that time in 2012 that I and a couple of fellow open-source enthusiasts started working on Ceilometer, the first piece of software from the OpenStack Telemetry project. Six years have passed since then. I’ve been thinking about this blog post for several months (even years, maybe), but lacked the time and the hindsight needed to lay out my thoughts properly. In a series of posts, I would like to share my observations about the Ceilometer development history.

Read more at https://julien.danjou.info/lessons-from-openstack-telemetry-incubation/

Comparing Keystone and Istio RBAC by Adam Young

To continue with my previous investigation to Istio, and to continue the comparison with the comparable parts of OpenStack, I want to dig deeper into how Istio performs RBAC. Specifically, I would love to answer the question: could Istio be used to perform the Role check?

Read more at https://adam.younglogic.com/2018/04/comparing-keystone-and-istio-rbac/

Scaling ARA to a million Ansible playbooks a month by David Moreau Simard

The OpenStack community runs over 300 000 CI jobs with Ansible every month with the help of the awesome Zuul.

Read more at https://dmsimard.com/2018/04/09/scaling-ara-to-a-million-ansible-playbooks-a-month/

Comparing Istio and Keystone Middleware by Adam Young

One way to learn a new technology is to compare it to what you already know. I’ve heard a lot about Istio, and I don’t really grok it yet, so this post is my attempt to get the ideas solid in my own head, and to spur conversations out there.

Read more at https://adam.younglogic.com/2018/04/comparing-istio-and-keystone-middleware/

Heading to Red Hat Summit? Here’s how you can learn more about OpenStack. by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform

Red Hat Summit is just around the corner, and we’re excited to share all the ways in which you can connect with OpenStack® and learn more about this powerful cloud infrastructure technology. If you’re lucky enough to be headed to the event in San Francisco, May 8-10, we’re looking forward to seeing you. If you can’t go, fear not, there will be ways to see some of what’s going on there remotely. And if you’re undecided, what are you waiting for? Register today. 

Read more at https://redhatstackblog.redhat.com/2018/03/29/red-hat-summit-2018-openstack-preview/

Multiple 1-Wire Buses on the Raspberry Pi by Lars Kellogg-Stedman

The DS18B20 is a popular temperature sensor that uses the 1-Wire protocol for communication. Recent versions of the Linux kernel include a kernel driver for this protocol, making it relatively convenient to connect one or more of these devices to a Raspberry Pi or similar device.

Read more at https://blog.oddbit.com/2018/03/27/multiple-1-wire-buses-on-the-/

An Introduction to Fast Forward Upgrades in Red Hat OpenStack Platform by Maria Bracho, Principal Product Manager OpenStack

OpenStack momentum continues to grow as an important component of hybrid cloud, particularly among enterprise and telco. At Red Hat, we continue to seek ways to make it easier to consume. We offer extensive, industry-leading training, an easy to use installation and lifecycle management tool, and the advantage of being able to support the deployment from the app layer to the OS layer.

Read more at https://redhatstackblog.redhat.com/2018/03/22/an-introduction-to-fast-forward-upgrades-in-red-hat-openstack-platform/

Ceph integration topics at OpenStack PTG by Giulio Fidente

I wanted to share a short summary of the discussions happened around the Ceph integration (in TripleO) at the OpenStack PTG.

Read more at http://giuliofidente.com/2018/03/ceph-integration-topics-at-openstack-ptg.html

Generating a list of URL patterns for OpenStack services. by Adam Young

Last year at the Boston OpenStack summit, I presented on an Idea of using URL patterns to enforce RBAC. While this idea is on hold for the time being, a related approach is moving forward building on top of application credentials. In this approach, the set of acceptable URLs is added to the role, so it is an additional check. This is a lower barrier to entry approach.

Read more at https://adam.younglogic.com/2018/03/generating-url-patterns/

by Mary Thengvall at April 20, 2018 02:57 PM

April 19, 2018

OpenStack Superuser

Vancouver Superuser Award Nominee: City Network

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

City Network is one of the seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Who are the team members?

City Network’s dev ops, professional services, education and engineering teams: Marcus Murwall, Tobias Rydberg, Magnus Bergman, Tobias Johansson, Alexander Roos, Johan Hedberg, Joakim Olsson, Emil Sundstedt, Mattias Nilsson, Lars Östergaard, Erik Johansson, Joel Svensson, Erik Arenhill, Ioannis Karamperis, Johan Hedberg, Namrata Sitlani, Christoffer Carlberg, Daniel Öhberg, Florian Haas, Adolfo Brandes, Syed Armani, Priscila Prado.

How has open infrastructure transformed your business? 

Shifting to 100 percent focus on OpenStack has been key to the global expansion of our organization in general and our cloud offerings in particular. City Network is now a global provider of public, compliant, private and hybrid-cloud solutions based on OpenStack.
With OpenStack and all its features, ease-of-use and popularity as the catalyst we have added value through multiple data centers in Europe, the U.S., Asia and the United Arab Emirates. With a clear strategy and implementation of data protection and regulatory aspects, City Network is leading the way for cloud and OpenStack adoption in the financial services industry.

How has the organization participated in or contributed to an open infrastructure community? 

  • Our CEO is a member of the OpenStack Foundation Board. His goal is to help the community and the ecosystem to leverage this open platform and drive the transformation for a more open, future-proof IT infrastructure.
  • City Network founded and is still organizing OpenStack Days Nordic, 2018 marks our third year. We are also involved in OpenStack Days Israel and India and attend multiple OpenStack Days events across the globe.
  • We have participated in every summit for the past six years and contribute to the user groups; public cloud and the security project.
  • City Network provides OpenStack training and is working to bridge the OpenStack and Open edX communities through mutual collaboration with the common ambition of providing quality education to anyone with access to a browser.

What open source technologies does the organization use in its IT environment?

We are very pro open source and use it in every case where open source is a viable option. A selection of the Open Source technologies we are currently using: CentOS, OpenBSD, Ubuntu, Nginx, Apache, PHP, Python, Ansible, MySQL, Mariadb, Mongodb and Ceph, Open edX.

What is the scale of the OpenStack deployment? 

We run our public OpenStack based cloud in eight regions across three continents. All our data centers are interconnected via private networks. In addition to our public cloud, we provide a pan-European cloud for verticals where regulatory compliance is paramount (e.g. banking and financial services, government, healthcare) addressing all regulatory challenges. Over 2,000 users of our infrastructure-as-a-service solutions run over 10,000 cores in production.

What operational challenges have you overcome during your experience with open infrastructure? 

Since we’re running OpenStack as public IaaS there have been a lot of hurdles to overcome as OpenStack is not yet fully adapted for public clouds. We had to build our own APIs in order to get network connectivity over several sites to work and also we had to add features such as volume copy and the ability to move volumes between sites. We have also had our fair share of issues with upgrading to new OpenStack versions, however we do feel as this process have been getting better with each upgrade.

How is this team innovating with open infrastructure? 

We innovate with OpenStack on two main focus areas. One is figuring out how we can interconnect all our OpenStack data centers over a global, private network and all the benefits that comes from doing so—including being able to provide customers with a direct, private access to our cloud services.

The other focus area has to do with our focus on helping regulated companies, mainly in the financial and healthcare industries with their digital transformation and cloud adoption. By building completely separated cloud services compliant with regulations such as ISO 9001, 27001, 27015, 27018, Basel, solvency and HIPAA, we allow for these industries to go cloud with a pay-as-you-go model and be truly agile.

How many Certified OpenStack Administrators (COAs) are on your team?

None.

Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: City Network appeared first on Superuser.

by Superuser at April 19, 2018 03:34 PM

Vancouver Superuser Award Nominee: Ontario Institute for Cancer Research (OICR)

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

OICR is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Who are the team members?
George Mihaiescu (cloud architect)
Jared baker (cloud engineer)
Francois Gerthoffert (project manager)
Rahul Verma (software developer)
Robert Tisma (software developer)
Dusan Andric (software architect)
Vincent Ferretti (principal investigator)
Lincoln Stein (principal investigator)
Junjun Zhang (Senior Bioinformatics Manager)

Site: www.cancercollaboratory.org

How has open infrastructure transformed your organization? 

OpenStack made it possible for OICR to build the Cancer Genome Collaboratory, a cloud that enables research on the world’s largest and most comprehensive cancer genome dataset.

Researchers can run complex analysis across a large repository of cancer genome sequences. Instead of spending weeks to months downloading hundreds of terabytes of data from a central repository before computations can begin, researchers can upload their analytic software into the Collaboratory cloud, run it and download the computed results in a secure fashion.

The Collaboratory is home to the data holdings of the International Cancer Genome Consortium (ICGC), a global collaboration involving more than 40 countries/jurisdictions to sequence the genomes across 50 major cancer types. Users of the Collaboratory have fast and easy access to this unique data set.

How has the organization participated in or contributed to an open infrastructure community? 

OICR is hosting the Toronto OpenStack meetups in 2018. George Mihaiescu and Jared Baker have presented at past OpenStack Summits (Barcelona, Boston and soon Vancouver) as well as reporting bugs and providing feedback on the mailing list and IRC channels.

Mihaiescu has been involved with OpenStack since the Cactus release, with the first conference attended being in Boston 2011 and active in the OpenStack scientific working group, as well as a member in the OpenStack Day Canada 2018 organizing committee.

The team also writes blog posts on http://softeng.oicr.on.ca/, sharing knowledge of open-source technologies.

What open source technologies does the organization use in its IT environment?

All the technologies used in Collaboratory are open source, including: Linux, OpenStack, Ceph, Ansible, Zabbix, Elasticsearch, Logstash, Kibana, Grafana, MaaS, ARA.

We are proud to always choose open source alternatives for our technology. We benefit from the flexibility and freedom provided and using the cost savings to invest in increased capacity for cancer research. Most importantly, we couldn’t offer cancer researchers a cloud environment at this scale and price point if we weren’t using open-source technologies like OpenStack and Ceph.

What is the scale of the OpenStack deployment? 

The Collaboratory has 2,600 cores and 18 terabytes of RAM, as well as 7.3 petabytes of storage managed by Ceph. More than 40 research labs and 95 researchers across four continents are using Collaboratory to access the 670 terabytes of protected cancer genome data stored.

The Collaboratory has contributed to the research underlying 43 peer-reviewed papers, with an additional 50 papers from the Pan-cancer project currently in preparation or review.

The genomic cancer research workloads are special in their needs, so the Collaboratory provides very large flavors to accommodate the storage, CPU and memory requirements. Because of its use of only open source and commodity hardware, the cost for using the Collaboratory is almost 40 percent less than that of the leading commercial cloud provider.

What operational challenges have you overcome during your experience with open infrastructure? 

We initially deployed OpenStack on the Juno release and upgraded live multiple times all the way to the Ocata release, with a planed upgrade to Pike in the weeks to come.

We also live upgraded the operating system from Ubuntu 14 to 16 and Ceph from Hammer to Jewel. From packaging issues to documentation mistakes, we gained a lot of operational experience over time managing OpenStack and Ceph.

With a infrastructure support team of just two people, we have to stay close to the latest OpenStack version and careful of the projects supported in our environment while keeping the environment stable and secure.

Both infrastructure support people are OpenStack certified (COA) and spend considerable time researching and testing new OpenStack releases and features.

How is this team innovating with open infrastructure? 

The Collaboratory team has developed two open-source applications for our users:

– A cost-recovery application providing the principal investigators with daily usage and cost metrics.

– An enrollment application, allowing researchers to request OpenStack tenants to be created, or users to be added to existing tenants.

Also, we developed and open sourced metadata and storage software used to provide granular and time-limited access to the protected data sets only to approved researchers.

Another open-source project developed at OICR is called Dockstore and its goal is to allow researchers to package and share bioinfomatics tools and workflows.

How many Certified OpenStack Administrators (COAs) are on your team?

Two.

Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: Ontario Institute for Cancer Research (OICR) appeared first on Superuser.

by Superuser at April 19, 2018 03:34 PM

Vancouver Superuser Award Nominee: SmartMe

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

SmartMe is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Who are the team members?

The core of SmartMe.io is a young, dynamic team of 15 at the University of Messina. Fujitsu also supports SmartMe with a team of 20. The teams work together on development, test and documentation; the joint team is based in Europe and Asia.

How has open infrastructure transformed the organization? 

Organizational culture has been forged by collaboration since the very beginning. In fact, the first step of SmartMe.io was a crowdfunding campaign that funded our initial development. After that, working with the OpenStack community enhanced the quality of the software. This philosophy helped SmartMe to be more agile, fast and gain a better understanding of the global market.

How has the organization participated in or contributed to an open infrastructure community? 

SmartMe works closely with the OpenStack Monasca team to develop Monasca and Stack4Things (IoTronic, under incubation) further. We’ve also participated in OpenStack Days and Summits to share our story and help the community with topics such as edge computing, artificial intelligence, anomaly detection and monitoring. SmartMe also collaborates with FIWARE to deploy our solution for a global audience. We have been working with the OpenStack community since our beginning in 2014.

The Italian government has recently launched a nation-wide contest (Open Community 2020) for the reuse of best practices and the transfer of results to other interested municipalities. The city of Turin, after being favourably impressed with Messina’s experience, gathered a consortium with the cities of Padua, Lecce, Syracuse, and together with the municipality of Messina and the University of Messina, submitted its proposal to reuse Messina’s practices. The consortium won E500K+ to spend on this project and the activities just started.

What open source technologies does the organization use in its IT environment?

The technology stack for edge computing at SmartMe.io includes OpenStack (especially IoTronic, Neutron, Monasca), Docker, LXD, InfluxDB, Node.js, Node-RED, WebSockets, WAMP.ws, FUSE, Darknet.

What’s the scale of the OpenStack deployment? 

The original test project in Messina was made of 5+ OpenStack instances, with 5+ tenants per instance, and 50+ VMs per tenant. Similar configurations were deployed for the initial production.

What operational challenges have you overcome during your experience with open infrastructure? 

Rolling upgrades are still a steep hurdle. Activities on internet of things infrastructure use IoTronic that supports remote software updates of the fleets of edge node.

How is this team innovating with open infrastructure? 

SmartMe uses new OpenStack subsystems for IoT/edge devices enrollment as infrastructure, their management and virtualization. Examples of used projects are: containerization, filesystem virtualization, SDN/NFV/SFC. We use these projects to develop smart cities, smart building, smart parking, facial recognition (particularly for security) and industry 4.0 etc.

How many Certified OpenStack Administrators (COAs) are on your team?

We believe in know-how, best practices and learning. Recently eight people attended COA training and one of them is already COA-certified. The others plan to get the certification too.

 Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: SmartMe appeared first on Superuser.

by Superuser at April 19, 2018 03:33 PM

Vancouver Superuser Award Nominee: Telia

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

Telia is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

 Who are the team members?

Telia – Aurimas Baubkus, IT products manager.

How has open infrastructure transformed the organization’s business? 

Telia became a primary advocate for OpenStack in the Baltic region after experiencing the transformative benefits of its public cloud offering. In their nine data centers, they use several integrations including VMware, Hyper-V and Citrix among others. Their extensive knowledge and first-hand experience with multiple distributors has competitively positioned the company and attracts customers who many not have considered building an OpenStack cloud. OpenStack has allowed Telia to provide infrastructure sets and new revenue streams to customers.

How has the organization participated in or contributed to an open infrastructure community? 

OpenStack is a fairly new product in the Baltic region and Telia has stepped up to be a huge advocate for building OpenStack awareness. The largest data center in Finland has over 5,000 racks, allowing perspective OpenStack users to see first-hand how powerful OpenStack is and why they should seriously consider starting their journey.

The team has extensive experience with other distributions and frequently uses first-hand examples to show tenants how OpenStack wins over other options, like VMware. Telia has taken a leadership role in the region, advising individual countries in the Baltics  how OpenStack can transform their cloud environment and reach this level of architecture. The Telia team attends the Summits and local meetups.

What open source technologies does the organization use in its IT environment?

Telia uses Open MNS, which is completely open source. Before adopting OpenStack it used the monitoring solution as well. The company is currently adopting the technology for their internal needs and has made it a main focus for 2018.

What is the scale of the OpenStack deployment? 

Telia is currently running 30 compute nodes in their environment and has plans to rapidly expand as they anticipate more customer needs.

What operational challenges have you overcome during your experience with open infrastructure? 

As any experienced OpenStack user knows, the process of adopting the platform varies for every organization. Telia was no different and ultimately had to switch distributors after a “failed” first attempt. The team started out with no experience building or managing OpenStack and heavily relied on the second distributor they hired, Red Hat, to produce a new platform that took into account the lessons learned from their first try. In the second initiative, Telia understood the importance of a strong partner, which allowed them to cater to more of their customer demands. Red Hat was able to proactively support their team and solve their human resources issue as it was a steep learning curve for everyone on their team as well as everyone else in the organization.

How is this team innovating with open infrastructure? 

At first, the Telia team was very ambitious, building a custom self-service platform platform using Swift. The second time around, it was imperative that they succeeded and the benefits were felt throughout the organization. They took the more traditional route to build a platform that was bullet-proof by avoiding any risky projects. Today, they take pride in their partnership with Red Hat and anything beyond minor bugs that their internal team can solve, they consult the Red Hat team or other distributors like Trilio who specialize in each component.

How many Certified OpenStack Administrators (COAs) are on your team?

Telia has four Certified OpenStack Administrators on their team.

Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: Telia appeared first on Superuser.

by Superuser at April 19, 2018 03:32 PM

Vancouver Superuser Award Nominee: T-Mobile

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

T-Mobile is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Please specify your team and organization for nomination. Who are the team members?

In the true spirit of supporting T-Mobile’s un-carrier efforts along with minimizing time to market and solving industry pain points around data growth, T-Mobile is deploying OpenStack-based virtual evolved packet core (EPC) so that we can continue to support large scale, consumer-facing data connectivity to devices and better prepare for upcoming 5G deployments. There are multiple OpenStack programs within T-Mobile with a mission to virtualize network functions. In this nomination, we’re presenting the largest use-case which is EPC VNF running on OpenStack. Team Leaders: Suliman Albasheir, Senthil Kaliappan, Rahul Pal, Viraj Silva, John Lazarte and Tom Browning.

How has open infrastructure transformed your business? 

T-Mobile is looking for exponential footprint growth in EPC network to serve the growing demands. OpenStack is a key component to address this demand and being used for other NFV programs. Working with OpenStack for NFV improved deployment cost and cross functional interaction among different teams. Having OpenStack as a baseline for telecom core systems helps in automation and shortens time-to-market for launching services. The T-Mobile NFV team has provided a tremendous number of improvements to the NFV implementation of EPC on OpenStack improving stability, increasing capacity and improving availability. These improvements were identified via NFV solution design, lab testing, live traffic and VNF rollouts in production network. Scale: approximately 6,000 bare-metal and VMs.

How has the organization participated in or contributed to an open infrastructure community? 

At T-Mobile, we did not contribute on code level in the past but this might change in future. We have identified numerous bugs with various vendors/partners during the solution design/validation phase and worked with partners to provide resolutions at the OpenStack level for the same. All these resolutions were identified, reported and resolved through collaborative NFV design with vendor, NFV lab testing, validation, live traffic stress testing and VNF rollouts in production network environment, thereby making deployment of telecom virtual applications more stable.

The T-Mobile NFV team has worked with many vendors and partners to excel the OpenStack integration for telco-grade NFV. Also, T-Mobile organizations have active/regular participation in OpenStack Summits.

What open source technologies does the organization use in its IT environment?

We use Ansible, Puppet, Red Hat Satellite, Zabbix, Kafka, etc. for managing life cycle but this is not an extensive list of tools that we use.

What is the scale of the OpenStack deployment? 

Our OpenStack footprint is supporting virtual EPC production traffic, with around 6,000 bare-metal servers and VMs.

What operational challenges have you overcome during your experience with open infrastructure? 

IP/NFV issues

  • Parity between EPC PNF and OpenStack based VNF.
  • Manageability of VNF from FCAPS standpoint, lengthy VNF validation process.
  • Layer2 convergence’s impact on RabbitMQ/OpenStack intra process calls.
  • Impacts on VNF state machine from: Transport link bundling design’s behavior during Spine/Leaf failover, lack of bi-directional flow detection, NIC bonding setup issues.
  • Uneven load balancing on Intel’s Chip’s hashing for SR-IOV stack.

OpenStack issues:

  • OpenStack upgrade is a key challenge.
  • Issues with OpenStack Fencing.
  • Several issues with RabbitMQ that impacted the VNF availability.
  • Issues with CEPH that caused VNF restart.
  • Ceilometer services causing the slowness in ceph cluster.
  • Issues with Galera DB availability.
  • Unstable IPMI/ STONITH network.
  • Cinder volume service failure.

How is this team innovating with open infrastructure? 

NFV use-case:
Currently T-Mobile U.S. is in the phase of deployment of mobility grade data call processing applications as a VNF within its OpenStack environment. EPC VNF is the largest use case. In this setup, IP infrastructure is still treated as PNF and the EPC application is running as a VNF within OpenStack Domain. IP infrastructure is a tag separated domain for various workstreams needed to bring up the VNF as a workload. VNF itself is a fixed workload and there is no elastic scaling in scope for this iteration. Elastic scaling is planned as future use-case.

Future workloads: 5G, IoT, Private APN, Edge computing, MVNO.

How many Certified OpenStack Administrators (COAs) are on your team?

Two people are certified as OpenStack administrators (Mirantis certification).

Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: T-Mobile appeared first on Superuser.

by Superuser at April 19, 2018 03:32 PM

Vancouver Superuser Awards Nominee: VEXXHOST

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

VEXXHOST is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Who is the nominee?

VEXXHOST is a leading Canadian public, private and hybrid cloud provider with an infrastructure powered by 100 percent vanilla OpenStack.

How has open infrastructure transformed your business? 

The open infrastructure of OpenStack allows VEXXHOST to speed up the delivery of new services to customers by accelerating the ability to innovate. The team can focus resources to deliver quality services without dealing with infrastructure issues which are solved by OpenStack. Every release of OpenStack has offered new features that were utilized to deliver higher performance.

How has the organization participated in or contributed to an open infrastructure community? 

VEXXHOST has been contributing to the OpenStack community since its second release in 2011. We have had a presence in the community by regularly attending OpenStack summits and being part of the interop challenge during the Boston summit. Our co-founder Mohammed Naser, who is the project team lead (PTL) for Puppet OpenStack, has given talks at Montreal, Ottawa and Toronto OpenStack meetups.

We also play a part in the community by actively contributing upstream code and sharing feedback with PTLs and developers. When we encounter bugs, we report them, diagnose them and work with the community to get a full fix in. We are active on the mailing list and provide feedback and fixes. We also contribute to the community by being a proud infrastructure donor for the OpenStack infrastructure team.

What open-source technologies does the organization use in its IT environment?

We run exclusively OpenStack services across our entire infrastructure. Our offering is fully open source without any proprietary licensed technology. Among many others, in our IT environment we use Nova with KVM with Libvirt, Ceph centralized storage, Pacemaker for high availability, MySQL Galera for database and Puppet for config management.

What’s the scale of the OpenStack deployment? 

Being a public cloud provider, we cannot disclose metrics regarding the scale of our users’ consumption. Our public cloud is able to handle several production-grade enterprise-scale workloads with private and hybrid cloud solutions delivering the same level of stability and robustness as our public cloud. Both our infrastructure and users production workloads are powered by OpenStack. OpenStack compute, network and storage are the backbones that are powering all our managed solutions.

What kind of operational challenges have you overcome during your experience with open infrastructure? 

Initially, we faced some challenges in terms of rolling upgrades as they were difficult though they have become much easier with new releases. After upgrading our infrastructure to Pike, we found a bug in the code which we reported. The developers at OpenStack were very responsive and happy to cooperate — as they always are — to help fix the bug. The bug was fixed in less than 24 hours in trunk and less than 48 hours in stable branches. This increases our trust the OpenStack CI and in turn grows our confidence in the software.

How is this team innovating with open infrastructure? 

As a public and private cloud provider, we’re heavily invested in improving and extending our list of managed services. Using OpenStack has helped us innovate in our managed services. In August 2017, we launched Kubernetes services using Magnum on the Pike release. We worked with the Magnum project team to ensure delivery of the best possible Kubernetes and OpenStack experience. VEXXHOST is currently one of the very few cloud providers to offer Magnum. OpenStack has also facilitated the delivery of big data solutions with the help of Sahara integration. We were also able to speed up the deployment of clusters with the help of transient clusters which provide huge cost savings.

How many Certified OpenStack Administrators (COAs) are on your team?

None.

 Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Awards Nominee: VEXXHOST appeared first on Superuser.

by Superuser at April 19, 2018 03:31 PM

Vancouver Superuser Award Nominee: Wingu

It’s time for the community to help determine the winner of the OpenStack Vancouver Summit Superuser Awards, sponsored by Zenko. Based on the community voting, the Superuser Editorial Advisory Board will review the nominees and determine the finalists and overall winner.

Now, it’s your turn.

Wingu is one of seven nominees for the Superuser Awards. Review the nomination criteria below, check out the other nominees and rate the nominees before the deadline Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

Cast your vote here!

Please specify the team and organization for nomination. Who are the team members?

Wingu – Thomas Lee, CEO.

How has open infrastructure transformed the organization’s business? 

Wingu’s entire business is built around OpenStack. Led by Thomas Lee, an OpenStack expert and self-described enthusiast, the company was an early adopter of OpenStack and has delivered innovative solutions to its enterprise customers throughout South Africa. In the past 18 months, Wingu has grown extensively, as an OpenStack public cloud provider, delivering full OpenStack technologies with APIs immensely improving customer agility and time-to-market.

How has the organization participated in or contributed to an open infrastructure community? 

Wingu has contributed extensively to the OpenStack community across South Africa, most notably by providing OpenStack infrastructure to the local Durban Infrastructure User Group. They also provide and track code improvements through their partner, Canonical. Lee’s over 20 years of experience make him a valuable expert in this region as Wingu supports OpenStack users starting their OpenStack journeys. Wingu proactively works with vendors and partners to build an enterprise-grade platform that will deliver not only a strong public computing platform, but also the skills and services to assist customers in successfully adopting cloud technologies. Wingu has attended several OpenStack Summit events since 2014.

What open-source technologies does the organization use in its IT environment?

Wingu is an early adopter and massive OpenStack user. It uses open source software on desktop machines and tools like Alfresco for their document management platforms, native monitoring and logging tools. There are no commercial tools in their environment except for Trilio. In addition, Wingu is the project sponsor for the Tachyonic open source web framework technology, powering NVF automation and a portal front-end for a OpenStack billing platform.

What is the scale of the OpenStack deployment? 

Wingu currently is running 10 compute nodes in their environment with the capability to host over 1,920 virtual machines. The platform runs Ubuntu OpenStack with more than 100 commercial customers on the platform.

What kind of operational challenges have you overcome during your experience with open infrastructure? 

A major operational challenge for Wingu is upgrades. They have overcome scaling network environments, so they are able to build OpenStack clouds in a custom way that is extremely scalable and very fast. Wingu is focusing on how to improve rolling upgrades and is currently utilizing tools like Livepatch to make the process much easier.

How is this team innovating with open infrastructure? 

Wingu’s customer base is extremely Microsoft-centric in South Africa. With over 40 percent of its enterprise customer base using Microsoft, Wingu has put extensive efforts into ensuring OpenStack is a strong partner of these workloads. By running features like VPN as-a-service, customers are able to extend their Microsoft directory into the cloud. Wingu is doing extensive work in bringing virtual network functions such as firewall-as-a Service (IP Tables, Juniper & FortiNet), load balancer-as-a-service (HA Proxy, F5, AVI Networks) and VPN-as-a-service to public cloud customers.

How many Certified OpenStack Administrators (COAs) are on your team?

None at this time, we are working on the certification testing.

Voting is limited to one ballot per person and closes Tuesday, April 24 at 11:59 p.m. Pacific Time Zone.

The post Vancouver Superuser Award Nominee: Wingu appeared first on Superuser.

by Superuser at April 19, 2018 03:30 PM

April 18, 2018

OpenStack Superuser

Return to the City of Glass: A guide to the Vancouver OpenStack Summit

 

The 2015 OpenStack Summit Vancouver was one of my favorites. Besides the obvious beauty of the water and mountains, there’s so much to see in this “Canadian Manhattan.” Much like the diversity of the OpenStack community, Vancouver reflects its rich past with influences from India, China, Japan as well as its sibling states in the U.S. Pacific Northwest. The result of this influence is a veritable cornucopia of food and lifestyle options in the city — I like to think of Vancouver as one of those cities that personify the OpenStack project and community.

Vancouver was also my very first North American OpenStack Summit. I remember being apprehensive about meeting the people that I’d only been rubbing virtual elbows with online. Even though I’d been working with OpenStack for over a year, I had never been involved with the larger OpenStack community. Suddenly, here I was in Vancouver, ready to take part in a summit with thousands of like-minded OpenStack aficionados.

Even with over a year of OpenStack under my belt, I still felt like an amateur before the summit. So I sought some guidance online to help me navigate the event. Some of the resources I found were published right here on Superuser (surprise!).  Articles like “Making your list and checking it twice for the OpenStack Vancouver Summit” that outlines planning, packing and arrival information for Vancouver were invaluable in preparing me for the logistics of travelling to Vancouver. Today, just like in 2015, there are already similar articles being published here on Superuser. Some are even about specific topics like,  “What’s new at the Vancouver Summit” and Must-see sessions at the Vancouver Summit that give readers some good suggestions about the general direction and even specific sessions to see.

“Wearing your 6-inch stilettos or funky pointed-toe Florsheim’s might leave you in flip flops by Tuesday”

 

However, even after reading everything I could get my hands on before the summit in 2015, I still remember feeling overwhelmed by all the sessions and the check-in process and deadlines. So let me give you some suggestions to help you relax and get the most out of your OpenStack Summit in Vancouver.

General tips

  • Register early. Typically there’s plenty of time to register prior to the Monday morning rush. If you’re like me, you don’t like standing in lines. Tear yourself away from snapping pics of the beautiful vistas, check-in, and get that OpenStack badge early!
  • Pick your sessions in advance and adjust at the conference. Go to the Summit Schedule and choose the sessions you really want to see first, then schedule the ones you are just curious about next. Get to sessions early, some fill up fast. Migrate to the center of the room if you don’t want standing room attendees breathing on you from the aisles.
  • Don’t schedule every single day with back-to-back sessions. Listen, I know you’re excited and don’t want to miss a single session, I get it. However, if you don’t schedule some downtime during the day (and no, lunch is “a” break, not “the” break), your brain will feel like butterscotch pudding by the last session and you’re not going to retain anything. Be nice to your brain.
  • Hydrate. Hydrate. Hydrate. Most conference centers do a great job of having water available in strategic areas, but a lot of times they aren’t exactly close-by. I recommend bringing a water bottle of your own and keeping it filled throughout the day. The brain needs water to function and I guarantee you’re going to be using the ol’ noodle a lot.
  • Wear comfy shoes. You’ll be walking a lot at the OpenStack Summit and afterwards. I routinely do 12-20,000 steps each day of the Summit. So if you look at your Fitbit right now and you’re only clocking in about 3,ooo steps a day, you’ll understand why wearing your 6-inch stilettos or funky pointed-toe Florsheim’s might leave you in flip flops by Tuesday.
  • Get out of your hotel room. There are plenty of parties and gatherings in the evening at every OpenStack Summit and Vancouver will be no different. Attending some of these events is a great way to see the city, plus it’s an excuse for a walk down to historic Gastown, an area chock full of eating and people-watching opportunities.

Tips for beginners

  • In the Summit Schedule, use the filter button that allows you to see only beginner sessions. I remember looking them when I was a rookie and wondering if everyone had a different definition of  “beginner.” Find some sessions that appeal to you and give them a try. If they’re too far over your head, use your mobile device to immediately find another session and head over there. No one expects you to be an expert, these Summits are for everyone. As a primer on OpenStack, check out my Sydney session on YouTube entitled “OpenStack for Absolute Amateur” it will get you well on your way to understanding the other sessions.
  • Talk to lots of people. I made the mistake my first Summits by not getting involved in the hallway conversations. They’re the best source of what’s really going on out there and what people are doing with OpenStack. OpenStack people are some of the friendliest I’ve ever met, don’t be afraid to be a joiner.
  • Ask questions. There are some really knowledgeable people at the Summit and it’s one of the few opportunities where you’ll get them all in one place. Plus, there are lots of operators, engineers and developers in attendance. So no matter what the question, someone will have the answer or will be able to point you in the right direction.
  • Get involved. There are so many ways to get involved with OpenStack, including contributing code. There will be a lot of OpenStack Ambassadors available as well folks from the User Group community who would love to talk to you about what they’re doing in your city or town. Check out the Summit schedule for community events where you’ll run into them.
  • Is AWS your thing? No problem. If you’re coming to OpenStack from AWS or from any other cloud provider, you might enjoy my session on OpenStack for AWS Architects. It will present a Rosetta Stone approach to translating AWS products to OpenStack projects and how the two stack up.

    A view of Vancouver. Photo: Ben Silverman.

Intermediate and advanced tips

  • Expand your mind. Now that OpenStack has matured, the OpenStack Foundation is moving into some other interrelated areas. Some of these areas like edge computing, Kata containers and other open-source container technologies. I would recommend branching out to some of the sessions in these new areas, they are very exciting talks and on the edge (pun intended) of cloud technology.
  • Bigger and better. OpenStack at scale has always been a favorite topic at the summit. But what happens when you start combining it with hybrid and multi-cloud architectures? What about cells now that they are a mandatory part of Nova? How do container technologies play a part in this? Good news for you, there are sessions for all of those questions. Get them on your schedule now!
  • Calling all telco/NFV fans. If you’re a service provider and telco/NFV OpenStack fanboy like me, you’re going to have a great week, there are over 40 telco/NFV sessions scheduled. You can hear sessions about everything from 5G, Edge, and autoscaling, VNF service chain orchestration to Ceph in Telco clouds.

Regardless of your experience level, you must do one thing: enjoy yourself in Vancouver and relax. Admittedly my last summit in Vancouver was anything but relaxing — and I’ve always regretted my high-strung activities in such a beautiful place.

Therefore, this year, I plan on relaxing before and after my session. My wish for all attendees is that they have fun, learn as much as they can (comfortably!) and get some time to unwind and process what they’ve learned throughout the week. I’ll be there all week, so if you see me staring out the enormous windows of the convention center overlooking the water, please come over say hello.  It might just remind us both to relax and take in the beauty of this City of Glass.

About the author

Ben Silverman is the Chief Cloud Officer for OnX. He considers himself an international cloud activist, and he’s also co-author of the book “OpenStack for Architects.”  Silverman started his OpenStack career in 2013 by designing and delivering American Express’ first OpenStack environment, worked for Mirantis as a senior architect and has been a contributing member of the OpenStack Documentation team since 2014. He’s also written for Superuser on how to get a job with OpenStack.

 

Superuser is always interested in community content, get in touch at editorATopenstack.org

Cover Photo // CC BY NC

The post Return to the City of Glass: A guide to the Vancouver OpenStack Summit appeared first on Superuser.

by Ben Silverman at April 18, 2018 04:19 PM

StackHPC Team Blog

The State of HPC Containers

HPCAC Lugano 2018

The recent HPC Advisory Council 2018 Lugano Conference was another great event, with a stimulating schedule of presentations that were once again themed on emerging trends in HPC. This year, the use of containers in HPC was at the forefront.

Swiss mountains

Motivations for Containers in Research Computing

The case for containerised compute is already well established, but here's a quick recap on our view of the most prominent advantages of supporting workload containerisation:

  • Flexibility: Conventional HPC infrastructure is built around a multi-user console system, a parallel filesystem and a batch-queued workload manager. This paradigm works for many applications, but there is an increasing number of workloads that do not fit this mould. The users who require applications classed as non-traditional HPC can often do so through the ability for containers to package a runtime that otherwise cannot be practically supported. Some software that wouldn’t be possible to run can do so by containerising it.
  • Convenience: For the scientist user, application development is a means to an end. The true metric in which they are interested is the "time to science". If a user can pull their application in as a container, requiring minimal consideration for adapting to the run-time environment of the HPC system, then they probably save themselves time and effort in their goal of conducting their research.
  • Consistency: Users of a research computing infrastructure have often arrived there after outgrowing the compute resources of their laptop or workstation. Their home workspace may be a very different runtime environment from the HPC system. The inconvenience and frustration that could be incurred by porting to a new environment (or maintaining portability between multiple environments) is an inconvenience that can be avoided if the user is able to containerise their workspace and carry over to other systems.
  • Reproducibility: The ability to repeat research performed in software environments has been a longstanding concern in science. A container, being a recipe for installing and running an application, can play a vital role in addressing this concern. If the recipe is sufficiently precise, it can describe the sources and versions of dependencies to enable others to regenerate the exact environment at a later time.

The Contenders for Containerised HPC

Representatives from all the prominent projects spoke, and thankfully Rich Brueckner from InsideHPC was there to capture each presentation. InsideHPC has posted all presentations from the conference online here.

This post attempts to capture the salient points of each technology and make a few comparisons that hopefully are not too odious.

Docker

Christian Kniep from Docker Inc spoke on the work he has been doing to enhance the integration of the Docker engine with HPC runtime environments. (Christian's slides uploaded here).

Christian's vision is of a set of runtime tools based on modular components standardised through the OCI, centred upon use of the Docker daemon. As it executes with root privileges, the Docker daemon has often been characterised as a security risk. Christian points out that this argument is unfairly singling out the Docker daemon, given the same could be said of slurmd (but rarely is).

Through use of a mainstream set of tools, the philosophy is that containerised scientific compute is not left behind when new capabilities are introduced. And with a common user experience for working with containerised applications, both 'scientific' and not, cross-fertilisation remains easy.

If the user requirements of scientific compute can be implemented through extension of the OCI's modular ecosystem, this could become a simple way of focussing on the differences, rather than creating and maintaining an entirely disjoint toolchain. Christian's work-in-progress doxy project aims to demonstrate this approach. Watch this space.

The Docker toolchain is the de facto standard implementation. The greatest technical challenges to this approach remain around scalability, process tree lineage and the integration of HPC network fabrics.

Kubernetes

Saverio Proto from SWITCH presented their new service for Kubernetes and how it integrates with their SWITCHEngines OpenStack infrastructure.

Kubernetes stands apart from the other projects covered here in that it is a Container Orchestration Engine rather than a runtime environment. The other projects described here use a conventional HPC workload manager to manage resources and application deployment.

Saverio picked out this OpenStack Summit talk that describes the many ways that Kubernetes can integrate with OpenStack infrastructure. At StackHPC we use the Magnum project (where available) to take advantage of the convenience of having Kubernetes provided - as a service.

Saverio and the SWITCH team have been blogging about how Kubernetes is used effectively in the SWITCHengines infrastructure here.

Singularity

Abhinav Thota from Indiana University presented on Singularity, and how it is used with success on IU infrastructure.

Singularity

A 2017 paper in PLOS ONE describes Singularity's motivation for mobility, reproducibility and user freedom, without sacrificing performance and access to HPC technologies. Singularity takes a iconoclastic position as the "anti-Docker" of containerisation. This contrarian stance also sees Singularity eschew common causes such as the Linux Foundation's Open Container Initiative, in favour of entirely home-grown implementations of the tool chain.

Singularity is also becoming widely used within research computing and is developing a self-sustaining community around it.

Singularity requires set-UID binaries for both building container images and for executing them. As an attack surface this may be an improvement over a daemon that continuously runs as root. However the unprivileged execution environment of Charliecloud goes further, and reduces its attack surface to the bare minimum - the kernel ABI and namespace implementations themselves.

The evident drawback of Singularity is that its policy of independence from the Docker ecosystem could lead to difficulties with portability and interoperability. Unlike the ubiquitous Docker image format, the Singularity Image Format depends on the ongoing existence, maintenance and development of the Singularity project. The sharing of scientific applications packaged in SIF is restricted to other users of Singularity, which must inevitably have an impact on the project's aims of mobility and reproducibility of science.

If these limitations were resolved then Singularity appears to be a good choice for it's rapidly-growing user base and evolving ecosystem. It also requires little administrative overhead to install, but may not be as secure as Charliecloud due to its requirement for set-UID.

Shifter

Alberto Madonna from CSCS gave an overview of Shifter, and an update on recent work at CSCS to improve it.

Shifter's genesis was a project between NERSC and Cray, to support the scalable deployment of HPC applications, packaged in Docker containers, in a batch-queued workload manager environment. Nowadays Shifter is generic and does not require a Cray to run it. However, if you do have a Cray system, Shifter is already part of the Cray software environment and supported by Cray.

Shifter's focus is on a user experience based around Docker's composition tools, using containers as a means of packaging complex application runtimes, and a bespoke runtime toolchain designed to be as similar as possible to the Docker tools. Shifter's implementation addresses the scalable deployment of the application into an HPC environment. Shifter also aims to restrict privilege escalation within the container and performs specific customisations to ensure containerisation incurs no performance overhead.

To ensure performance, the MPI libraries of the container environment are swapped for the native MPI support of the host environment (this requires use of the MPICH ABI).

To enable scalability, Shifter approaches docker container launches by creating a flattened image file on the shared parallel filesystem, and then mounting the image locally on each node using a loopback device. At NERSC, Shifter's scalability has been demonstrated to extend well to many thousands of processes on the Cori supercomputer.

CSCS work has removed several perceived issues with the Shifter architecture. CSCS have been developing Shifter to improve the pulling of images from Dockerhub (or local user-owned Docker image repositories), and have added the ability to import images from tar files.

Shifter appears to be a good choice for sites that have a conventional HPC batch-queued infrastructure and are seeking to provide a scalable and performant solution, but retaining as much compatibility as possible with the Docker work flow. Shifter requires more administrative setup than Singularity or Charliecloud.

Shifter is available on NERSC's github site.

Charliecloud

Michael Jennings from Los Alamos National Lab presented Charliecloud and the concepts upon which it is built.

Charliecloud's development at LANL resulted in a solution developed in a site with strict security requirements. Cluster systems in such an environment typically have no external internet connectivity. System applications are closely scrutinised, in particular those that involve privileged execution.

In these environments, Charliecloud's distinct advantage is the usage of the newly-introduced user namespace to support non-privileged launch of containerised applications. This technique was described in the 2017 Singularity paper as being "not deemed stable by multiple prominent distributions of Linux". It was actually introduced in 2013, but its use widened exposure to a new kernel attack surface. As a result its maturity has been complex and slow, but user namespaces are now a standard feature of the latest releases of all major Linux distributions. Configuration of Debian, RHEL and CentOS is described here. (For environments where unprivileged user namespaces cannot be supported, Charliecloud can fall back to using setuid binaries).

The user namespace is an unprivileged namespace. A user namespace can be created without requiring root privileges. Within a user namespace all other privileged namespaces can be created. In this way, a containerised application can be launched without requiring privileged access on the host.

Development for a Charliecloud environment involves using the Docker composition tools locally. Unlike Docker, a container is flattened to a single archive file in preparation for execution. Execution is scaled by the scalable distribution of this archive, which is unpacked into a tmpfs environment locally on each compute node. Charliecloud has been demonstrated scaling to 10,000 nodes on LANL's Trinity Cray system.

Charliecloud

Charliecloud appears to be a good choice for sites that are seeking scalability, but with strong requirements for runtime security. The Docker development environment and composition tools are also helpful for users on a learning curve for containerisation.

Further details on Charliecloud can be be found from the informative paper presented at Supercomputing 2017. Michael has provided a Charliecloud Vagrantfile to help people familiarise themselves with it. Charliecloud packages are expected to ship in the next major release of OpenHPC.

The Road Ahead

The ecosystem around container technology is rapidly evolving, and this is also true in the niche of HPC.

The Open Container Initiative

The tools for this niche are somewhat bespoke, but thanks to the efforts of the OCI to break down the established Docker tools into modular components, there is new scope to build a specialist solution upon a common foundation.

This initiative has brought about new innovation. Rootless RunC is an approach for using the runc tool for unprivileged container launch. This approach and its current limitations are well documented in the above link.

In a similar vein, the CRI-O project is working on a lightweight container runtime interface that displaces the Docker daemon from Kubernetes compute nodes, in favour of any OCI-compliant runtime.

Shifter, Charliecloud and Singularity are not OCI-compliant runtimes, as they predate OCI’s relativately recent existence. However, when the OCI's tools become suitably capable and mature they are likely be adopted in Charliecloud and Shifter.

Challenges for HPC

There are signs of activitiy around developing better support for RDMA in containerised environments. The RDMA Cgroup introduced in Linux 4.11 introduces support for controlling the consumption of RDMA communication resources. This is already being included in the spec for the OCI runtime.

RDMA isolation (for example, through a namespace) doesn’t seem to be currently possible. Current implementations can only pass-through the host’s RDMA context. This will work fine for HPC configurations with a scheduling policy not to share a compute node between workloads.

The greatest advantages of specialist solutions appear to address challenges that remain unique to scientific computing. For example:

  • Scalable launch of containerised workloads. The approach taken by Singularity, Shifter and Charliecloud involves using a parallel filesystem for the distribution of the application container image. This addresses one of the major differences in use case and design. Distributing the container as a single image file also greatly reduces filesystem metadata operations.
  • Launching multi-node MPI applications in research computing containers. The Docker runtime creates complications with interacting with MPI's Process Management Interface. Shifter's innovation around replacing container MPI libraries with host MPI libraries is an intriguing way of specialising a generalised environment. Given multi-node MPI applications are the standard environment of conventional HPC infrastructure, running containerised applications of this form is likely to be a tightly specialised niche use case.

(Most) Paths Converge

A future direction in which HPC runtime frameworks for containerised applications have greater commonality with the de facto standard ecosystem around OCI and/or Docker's tools has considerable appeal. The development environment for Docker containers is rich, and developing rapidly. The output is portable and interchangeable between different runtimes. As Michael Jennings of Charliecloud says, “If we can all play in the same playground, that saves everyone time, effort and money”.

The HPCAC 2018 Swiss Conference brought together experts from all the leading options for containerised scientific compute, and was a rare opportunity to make side-by-side comparisons. Given the rapid development of this technology I am sure things will have changed again in time for the 2019 conference.

by Stig Telfer at April 18, 2018 09:00 AM

April 17, 2018

StackHPC Team Blog

HPCAC Conference Keynote: Ceph on the Brain

The HPC Advisory Council held its 2018 workshop in Lugano in the beautiful region of Ticino, Switzerland.

InsideHPC has recorded footage and gathered presentation slides from much of the conference, available here. Thanks Rich!

I was able to return to the conference and deliver a keynote presentation, in partnership with Adrian Tate from Cray EMEA Research Lab. We spoke on the Cray-Intel pre-commercial procurement (PCP) project, codenamed JULIA, for the Human Brain Project at Jülich Supercomputer Centre. Adrian spoke on Cray's recent R&D work on data movement within application workflows. I spoke on work to optimise Ceph for NVMe and HPC network architectures.

InsideHPC covered it here. Their footage of the presentation is also available on their YouTube channel:

An HPC conference can bring together hot technologies such as software-defined storage and non-volatile memory, and appraise them in the context of HPC's long tradition of maximising the capability of computation at scale.

An exciting time for StackHPC to be standing at this crossroads!

by Stig Telfer at April 17, 2018 05:40 PM

RDO

Unit tests on RDO package builds

Unit tests are used to verify that individual units of source code work according to a defined spec. While this may sound complicated to understand, in short it means that we try to verify that each part of our source code works as expected, without having to run the full program they belong to.

All OpenStack projects come with their own set of unit tests, for example this is the unit test folder for the oslo.config project. Those tests are executed when a new patch is proposed for review, to ensure that existing (or new) functionality is not broken with the new code. For example, if you check this review, you can see that one of the CI jobs executed is “openstack-tox-py27”, which runs unit tests using Python 2.7.

Unit tests in action

How does this translate into the packaging world? As part of a spec file, we can define a %check section, where we add scripts to test the installed code. While this is not a mandatory section in the Fedora packaging guidelines, it is highly recommended, since it provides a good assurance that the code packaged is correct.

In many cases, RDO packages include this %check section in their specs, and the project’s unit tests are executed when the package is built. This is an example of the unit tests executed for the python-oslo-utils package.

“But why are these tests executed again when packaging?”, you may ask. After all, these same tests are executed by the Zuul gate before being merged. Well, there are quite a few reasons for this:

  • Those unit tests were run with a specific operating system version and a specific package set. Those are probably different from the ones used by RDO, so we need to ensure the project compatibility with those components.
  • The project dependencies are installed in the OpenStack gate using pip, and some versions may differ. This is because OpenStack projects support a range of versions for each dependency, but usually only test with one version. We have seen cases where a project stated support for version x.0 of a library, but then added code that required version x.1. This change would not be noticed by the OpenStack gate, but it would make unit tests fail while packaging.
  • They also allow us to detect issues before they happen in the upstream gate. OpenStack projects use the requirements project to decide which version of their own libraries should be used by other projects. This allows for some inter-dependency issues, where a change in an Oslo library may uncover a bug in another project, but it is not noticed until the requirements project is updated with a new version of the Oslo library. In the RDO case, we run an RDO Trunk builder using code from the master branch in all projects, which allows us to notify in advance, like in this example bug.
  • They give us an early warning when new dependencies have been added to a project, but they are not in the package spec yet. Since unit tests exercise most of the code, any missing dependency should make them fail.

Due to the way unit tests are executed during a package build, there are some details to keep in mind when defining them. If you as a developer follow them, you will make packagers’ life easier:

  • Do not create unit tests that depend on resources available from the Internet. Most packaging environments do not allow Internet access while the package is being built, so a unit test that depends on resolving an IP address via DNS will fail.

  • Try to keep unit test runtime within reasonable limits. If unit tests for a project take 1 hour to complete, it is likely they will not be executed during packaging, such as here.

  • Do not assume that unit tests will always be executed on a machine with 8 fast cores. We have seen cases of unit tests failing when run on a limited environment or when it takes them more than a certain time to finish.

Now that you know the importance of unit tests for RDO packaging, you can go ahead and make sure we use it on every package. Happy hacking!

by jpena at April 17, 2018 04:49 PM

OpenStack Superuser

What’s next in CI/CD

OpenDev  is an annual event focused at the intersection of composable open infrastructure and modern applications. The first edition centered  on edge computing while the 2018 collaborative two-day event focuses on continuous integration, deployment and delivery (CI/CD). This time it’s co-located with the OpenStack Summit, enabling Summit attendees to dive into the content, too.

Here we’re highlighting some of the sessions you’ll want to add to your schedule. Check out the whole program here and if you’re attending the OSF Summit you can add theses sessions here.

Deploy from CI/CD to production in one click

With complex deployment such as AT&T’s, it’s always a challenge to deploy artifacts frequently on production environments and to prevent human errors in case of manual deployments. In this intermediate session, Jerome Brette will talk about how his team has designed updates, upgrades and greenfields to be deployed with out any human intervention. The result bridges the gap between the development activities and deployment activities. Details here.

CI/CD build and pipelines for OpenStack at scale

Oath Inc. deploys a heavily customized version of OpenStack across many different regions and clusters. The company need a way to build each OpenStack component with its customizations into a self-contained package that’s deployable in data centers. They also required fine-grained control over deployments in order to swiftly deploy fixes/enhancements to single components in a single cluster with minimal downtime. Oath’s Ryan Bridges will talk about their journey and the solutions they’re using now in this intermediate session. Details here.

CI/CD in ETSI NFV environment

CI/CD and dev ops are common practices but require the automatic management of software components on live systems and the capability of sending feedback from live systems to developers. ETSI NFV defined a multi-layer MANO framework to manage the different layers of telecom services. OpenStack is a key enabler of dev ops in the NFV architecture, thanks to its capacity to dynamically manage workloads. There’s a push in the NFV community to introduce capabilities that allow  dev ops and CI/CD into this framework. In this beginner-level presentation, Nokia’s Gergely Csatari and Ixia Solutions Group Pierre Lynch will describe the challenges of introducing dev ops practices in the telecom environment and discuss planned solutions for these challenges. Details here.

We came together, now what?

As various initiatives across open source communities address the challenges to solve common problems, it’s clear that how well these components work together is increasingly critical. In this beginner-level session, Fatih Degirmenci of Ericsson, Melvin Hillsman Huawei and Robyn Bergeron of Red Hat will talk about how these efforts are shaping infrastructure, CI/CD, integration and testing as well as how communities can keep the conversation going. Details here.

Open CD for open infrastructure: Hybrid and multi-cloud deployments with Spinnaker

Spinnaker is an open source, multi-cloud continuous delivery platform built by Netflix, Google, Microsoft and others. At Netflix, Spinnaker powers over 4,000 deployments a day. In this talk, Spinnaker’s Andrew Phillips will introduce Spinnaker and how it enables continuous delivery automation from basic first steps through to advanced pipelines incorporating deployment safeguards, Canary analysis, out-of-the-box deployment strategies, sophisticated health checks, sharing Golden Path pipelines and more. The session will also cover muiti-cloud and Kubernetes support and talk about patterns for microservice delivery, before wrapping up with a quick tour of Spinnaker in action. Details here.

See you at OpenDev May 22-23, 2018!  Register here.

Cover Photo // CC BY NC

The post What’s next in CI/CD appeared first on Superuser.

by Superuser at April 17, 2018 04:17 PM

Chris Dent

TC Report 18-16

This the 16th week of the year, meaning I've been making these reports for a full year. The first one was in the 17th week of 2017. The reports have changed quite a bit since then: Back then there was still a once weekly TC meeting. That's since been replaced by less formal office hours, three times a week. That's had mixed results, reflected in the tone and content of these reports. To some extent the decompression of TC activity has meant less drama, intensity and apparent depth in the reports. But it may also be the case that the TC hasn't done as much as it could or should.

With elections in progress, we could take advantage of this time to reflect on the role of the TC and work to make sure the next term is more active in shaping and sustaining OpenStack. At the time of this writing there are ten hours left if you would like to nominate yourself. Info on the election page.

There are nine candidates (so far) for seven slots. When nominations close, there will be a week of "campaigning". This is an opportunity for community members to question the candidates about any concerns.

(I'm running for reelection, you can read my nomination if you like. Please ask me any questions you may have.)

Leadership Shadowing

Last Wednesday there was some comparison between the Kubernetes and OpenStack styles of "growing new leaders". In Kubernetes there is a shadowing system that has mixed success, depending on the group.

Forum and Summit

On Thursday there was final discussion on submitting sessions to the forum.

Prior to the summit proper there will be a joint leadership meeting. Thierry has started a thread seeking topics that the community would like to see raised at that meeting. These meetings are the most significant formal engagement the TC has with the board throughout the year.

More on Kolla

Also on Thursday there was more discussion on how kolla-k8s might evolve.

by Chris Dent at April 17, 2018 01:30 PM

Matthew Treinish

MQTT in the Cloud: Firehose

This is the first post in what’s going to be a multi post series exploring how MQTT can be leveraged in the cloud. The series will look at several different models for deploying applications in the cloud and how you can leverage MQTT as an event based protocol for those applications. This first post will be looking at using MQTT as unified event bus for heterogeneous service deployed using more traditional configuration managed services by looking at the OpenStack community infrastructure’s firehose effort.

MQTT

Before diving into firehose and why MQTT is used there it would to explain what MQTT is. MQTT is a machine to machine pub/sub protocol which is primarily targeted for IoT applications and sensor networks. It’s designed to be lightweight and work in environments where code size is a constraint or networking is unreliable or where bandwidth is at a premium. The protocol was originally written in 1999 and is now an ISO standard that is managed by the OASIS group.

MQTT is based on a centralized broker. Clients publish and/or subscribe to topics on the broker to send and receive messages to each other. There are a variety of different brokers both open and closed source available.  On the client side there are bindings available for a lot of different languages and environments.

The interesting pieces of the MQTT protocol when it comes to firehose and similar applications running in a cloud environment are topics, quality of servicem and client persistence.

Topics

The most obvious thing that makes MQTT different from a lot of other is how topics work. All message topics in MQTT are hierarchical and dynamic. This means that a topic is only created at message publish time and are dynamically matched with any clients subscribed. When coupled with wild carding is when this gets really useful. This enables you to build applications that listens only to the subset of messages you  are interested in.

For example, let’s say I was publishing messages from my laptop’s sensor I would use a topic schema like:

sensors/<hostname>/<sensor type>/<device id>

So if I wanted to publish a message with the SSD’s temperature, I would publish to a topic like:

sensors/sinanju/temperature/nvme0n1

Where sinanju is the laptop’s hostname. Now for a client subscribing you could subscribe to that exact topic and get message for just that one device. Or you could use wildcards to subscribe to multiple devices. For example, if you wanted all sensors from my laptop you would subscribe to:

sensors/sinanju/#

Which uses the multilevel wildcard ‘#’ to match any topics that start with “sensors/sinanjuin the hierarchy. Or if you wanted all temperature sensors on all devices you would subscribe to:

sensors/+/temperature/+

Which uses the single level wildcard ‘+’, which will match any field on that level of the hierarchy. You can see how powerful  using a combination of a detailed hierarchy and the wildcards let you dynamically subscribe to only messages your application is interested in.

For more examples and some suggestions on building topic hierarchies these 2 links have more details:
https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices
https://mosquitto.org/man/mqtt-7.html

QoS

MQTT supports 3 levels of quality of service 0, 1, and 2. QoS level 0 means there is no guarantee on delivery, QoS level 1 means the message is guaranteed to be delivered at least once, but may be recieved more than once, and QoS level 2 means the message will be delivered once and only once. The interesting piece on QoS in MQTT is that it’s per message publish or per subscription. Meaning that when a client publishes a message that   set it’s own QoS level for sending to the broker. Then when a client subscribes to a topic on a broker it sets QoS level for the subscription. These are independent from each other, meaning you can publish a message to a topic with QoS level 2 and subscribe to that topic with QoS level 0. This also means you can have an application hand pick the guarantees per message to optimize between the bandwidth and overhead for the individual messages if you need.

You can read more about QoS in MQTT here: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels

Persistent Sessions

The last aspect of the MQTT protocol that really makes it useful for an application like firehose running in a cloud environment is persistent sessions. Normally when a client connects to a broker it specifies it subscribes to the topics it’s interested in, but when the client disconnects those topics are lost. However, a client can specify a clientId and when that is combined with the higher QoS levels the broker will queue up messages to ensure that even if the subscribing client disconnects it will receive those messages on reconnect. This way you can ensure a client will never miss a message even if you lose connectivity.

The details on this are quite specific and instead of rehashing them all here the hivemq blog has documented the details quite well here: https://www.hivemq.com/blog/mqtt-essentials-part-7-persistent-session-queuing-messages

Firehose

So with some background on what MQTT is and some of the strengths it brings to the table it’s time to take a look at the firehose. The OpenStack community’s infrastructure runs completely in the open to support the OpenStack community. This includes running things like the gerrit review system, the upstream CI system, and other services. It ends up being a very large infrastructure running over 40 different services on 250 servers (not counting hosts used for running tests) in OpenStack clouds which are donated by various OpenStack service providers.  All of these services are managed using a combination of puppet to describe the configuration and packages and ansible to orchestrate running puppet on the different servers.

All of these services are generating events of some type, whether it’s the completion of a task, a user initiated action, a periodic status update, etc. Some of the services have native event streams but a lot of them don’t. The missing piece was a single place to handle the events from all of these different services. Right now if a service has an event stream at all it’s exposed as a separate thing implemented in it’s own way. For example, Gerrit‘s event stream is implemented as a command run via it’s ssh interface (which is inherently specific to gerrit). This is where firehose fits in, it provides a unified message bus for all the different services running in the community infrastructure. This gives both users and other applications a single place to go when they need to consume events from any infrastructure service.

We’ve documented how firehose is built and how to use it in the infra documentation: https://docs.openstack.org/infra/system-config/firehose.html. As that shows we’ve got a number of different services reporting events to the firehose and it’s pretty straightforward for anyone to subscribe to events. It’s even easy to do from JS natively in a browser like:


Which is subscribing to “#”, or all topics, and just printing the topic and payload colon separated. The code behind that is basically (just trimmed to work in my blog, the below is for a standalone page):

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

  </head>
  <body>
    <div id="title">
      <h1>MQTT Messages</h1>
    </div>
    <div class="content" style="height:240px;width:960px;border:1px solid #ccc;font:16px/26px Georgia, Garamond, Serif;overflow:auto;">
      <pre class="msglog" id="msglog"></pre>
    </div>
    <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"
            integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl"
            crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/paho-mqtt/1.0.1/mqttws31.js"
            type="text/javascript"></script>
    <script type="text/javascript">
      var mqttHost = "firehose.openstack.org";
      var client = new Paho.MQTT.Client(mqttHost, Number("80"), "client-" + Math.random());
  
      // set callback handlers
      client.onConnectionLost = onConnectionLost;
      client.onMessageArrived = onMessageArrived;
      
      // connect the client
      client.reconnect = true;
      client.connect({onSuccess: onConnect});
      
      // called when the client connects
      function onConnect() {
          // Once a connection has been made, make a subscription and send a message.
          console.log("onConnect");
          client.subscribe("#");
      }
      
      // called when the client loses its connection
      function onConnectionLost(responseObject) {
          if (responseObject.errorCode !== 0) {
              console.log("onConnectionLost:"+responseObject.errorMessage);
          }
      }
      
      // called when a message arrives
      function onMessageArrived(message) {
          console.log("onMessageArrived: "+message.destinationName + " " + message.payloadString);
          var content = message.destinationName + " " + message.payloadString + "\n";
          var pre = $("#msglog");
          pre.append(content);
          pre.scrollTop( pre.prop("scrollHeight") );
      }
    </script>
  </body>
</html>

The documentation linked above has a lot of code examples in a bunch of different languages. So you can use those examples to experiment with the firehose locally.

In addition to documentation linked above we also have extensive schema documentation for the firehose. Which documents the schema for both the topics and message payloads for all the services, which can be found at: https://docs.openstack.org/infra/system-config/firehose_schema.html. This documents how messages are constructed for all the services publishing messages to the firehose. This should enable anyone to build off the messages in the firehose for anything they need.

While this provides a description of what the firehose is and gives an idea on how to use it, it’ll be valuable to also talk about how it’s constructed and take a look how well it’s working.

firehose.openstack.org

For firehose we run the mosquitto MQTT broker on a single server. It’s actually a fairly small machine with only 2 vCPUs, 2GB of RAM, and a 40GB disk. Despite being a modest machine there is no issue with handling the load from all the services reporting. For example a typically message rate is:or you can see the current broker statistics at:
http://grafana.openstack.org/dashboard/db/mosquitto-status which was built using the mqtt_statsd project. Despite this message load it barely uses any resources Looking at the past month the CPU and RAM usage was minimal considering the amount of messages being published:

CPU UsageRam UsageYou can see the current numbers on cacti.openstack.org.

Load Testing

About a year ago when we started growing our usage of firehose we decided to do some manual load testing. We first tried to leverage the mqtt-bench project to do this, but unfortunately it’s not being actively maintained. So we ended up writing pymqttbench to do this task. When we did this load testing we were hindered by the bandwidth limitations of 200 Mbps for the server flavor we deployed. However despite hitting that hard wall limiting our testing we were able to see some interesting data.

So  the first thing we looked at the total number of message we were processing during the load testing:
We ran the testing over the period of a few hours while slowly ratcheting up the number of messages we were and were able to peak at about 2 million messages per minute going through the broker before we realized we were hitting a bandwidth limit and stopped the testing. The interesting thing with this test was looking at the system load at the time. First the CPU usage:
This graph shows a spike in the CPU utilization topping out at about 30-35% during the load test. The interesting thing with this though, is that spike occurred about an hour before the peak data usage of the test. Our expectation going into the test was that the load would be coupled directly to the number of messages and subscribers. But this graph clearly shows the spike was independent of either of those. Then looking at the RAM usage during the test:

We ended up using more RAM (percentage wise) during the testing peaking at about 1.25 GB being used by Mosquitto. But the big spike corresponds to the CPU usage spike which occurred before the max message throughput. When you look at the RAM usage when we were at the max messages it was at about it’s lowest value, at between 250MB and 350MB.

At some point we’ll have to revisit the load testing experiment. All this test showed was that the small server we have deployed is able to keep up with whatever load we throw at it, and have a large amount of headroom for more messages before we hit any limits of the small server we’re using, which admittedly was the goal of our testing. But, besides that it raised more questions than it answered about what those limits actually are, which is something we still need to investigate.

Conclusion

To tie everything together what has the firehose project shown us about using MQTT for applications in the cloud. The first thing is the lightweight nature of the protocol is a great fit for the cloud. In most cloud environments you pay for your resource utilization, so the less you use the less you have to pay. What firehose has demonstrated in this regard is that by using MQTT you can get away with minimal resource utilization and still have a solution that will work with a large load. Another key aspect is MQTT’s resiliency to unreliable networking. The old adage of treat your servers in the cloud like cattle not pets holds true. You can look at other aspects of  OpenStack’s community infrastructure and see all sorts of failures. For example, http://status.openstack.org/elastic-recheck/index.html#1298006 tracks a failure in the CI system where a test node can’t talk to the git servers. That graph shows how frequently we’re encountering random networking issues in a cloud. We’ve been running firehose since Autumn 2016 and I can’t recall of any instances when a service or client lost it’s connection and wasn’t able to recover seamlessly. This is including several broker restarts for service upgrades and configuration changes. The last aspect here is because MQTT has been around for 20 years there is a large ecosystem that already exists around the protocol. This means regardless of how you’re building your application, or what language it’s written in, there is likely already support for using MQTT there. This means you don’t have to spend time reinventing the wheel to add support for using MQTT or you can leverage existing projects to do common functions with MQTT.

What this whole project has demonstrated to me is that the application requirements for  IoT and remote sensor networks aren’t that dissimilar from writing applications in the cloud. The next post in this series will be looking at using MQTT in applications deployed on Kubernetes.

by Matthew Treinish at April 17, 2018 09:00 AM

OpenStack in Production

Maximizing resource utilization with Preemptible Instances

Motivation


The CERN cloud consists of around 8,500 hypervisors providing over 36,000
virtual machines. These provide the compute resources for both the laboratory's
physics program but also for the organisation's administrative operations such
as paying bills and reserving rooms at the hostel.

The resources themselves are generally ordered once to twice a year with servers being kept for around 5 years. Within the CERN budget, the resource planning teams looks at:
  • The resources required to run the computing services requirements for the CERN laboratory. These are projected using capacity planning trend data and upcoming projects such as video conferencing.
With the installation and commissioning of thousands of servers concurrently
(along with their associated decommissioning 5 years later), there are scenarios
to exploit underutilised servers. Programs such as LHC@Home are used but we have also been interested to expand the cloud to provide virtual machine instances which can be rapidly terminated in the event of
  • Resources being required for IT services as they scale out for events such as a large scale web cast on a popular topic or to provision instances for a new version of an application.
  • Partially full hypervisors where the last remaining cores are not being requested (the Tetris problem).
  • Compute servers at the end of their lifetime which are used to the full before being removed from the computer centre to make room for new deliveries which are more efficient and in warranty.
The characteristics of this workload is that it should be possible to stop an
instance within a short time (a few minutes) compared to a traditional physics job.

Resource Management In Openstack

Operators use project quotas for ensuring the fair sharing of their infrastructure. The problem with this, is that quotas pose as hard limits.This
leads to actually dedicating resources for workloads even if they are not used
all the time or to situations where resources are not available even though
there is quota still to use.

At the same time, the demand for cloud resources is increasing rapidly. Since
there is no cloud with infinite capabilities, operators need a way to optimize
the resource utilization before proceeding to the expansion of their infrastructure.

Resources in idle state can occur, showing lower cloud utilization than the full
potential of the acquired equipment while the users’ requirements are growing.

The concept of Preemptible Instances can be the solution to this problem. These
type of servers can be spawned on top of the project's quota, making use of the
underutilised  capabilities. When the resources are requested by tasks with
higher priority (such as approved quota), the preemptible instances are
terminated to make space for the new VM.

Preemptible Instances with Openstack

Supporting preemptible instances, would mirror the AWS Spot Market and the
Google Preemptible Instances. There are multiple things to be addressed here as
part of an implementation with OpenStack, but the most important can be reduced to these:
  1. Tagging Servers as Preemptible
In order to be able to distinguish between preemptible and non-preemptible
servers, there is the need to tag the instances at creation time. This property
should be immutable for the lifetime of the servers.
  1. Who gets to use preemptible instances
There is also the need to limit which user/project is allowed to use preemptible
instances. An operator should be able to choose which users are allowed to spawn this type of VMs.
  1. Selecting servers to be terminated
Considering that the preemptible instances can be scattered across the different cells/availability zones/aggregates, there has to be “someone” able to find the existing instances, decide the way to free up the requested resources according to the operator’s needs and, finally, terminate the appropriate VMs.
  1. Quota on top of project’s quota
In order to avoid possible misuse, there could to be a way to control the amount of preemptible resources that each user/project can use. This means that apart from the quota for the standard resource classes, there could be a way to enforce quotas on the preemptible resources too.

OPIE : IFCA and Indigo Dataclouds

In 2014, there were the first investigations into approaches by Alvaro Lopez
from IFCA (https://blueprints.launchpad.net/nova/+spec/preemptible-instances).
As part of the EU Indigo Datacloud project, this led to the development of the
OpenStack Pre-Emptible Instances package (https://github.com/indigo-dc/opie).
This was written up in a paper to Journal of Physics: Conference Series
(http://iopscience.iop.org/article/10.1088/1742-6596/898/9/092010/pdf) and
presented at the OpenStack summit (https://www.youtube.com/watch?v=eo5tQ1s9ZxM)

Prototype Reaper Service

At the OpenStack Forum during a recent OpenStack summit, a detailed discussion took place on how spot instances could be implemented without significant changes to Nova. The ideas were then followed up with the OpenStack Scientific Special Interest Group.

Trying to address the different aspects of the problem, we are currently
prototyping a “Reaper” service. This service acts as an orchestrator for
preemptible instances. It’s sole purpose is to decide the way to free up the
preemptible resources when they are requested for another task.

The reason for implementing this prototype, is mainly to help us identify
possible changes that are needed in Nova codebase to support Preemptible
Instances.

More on this WIP can be found here: 

Summary

The concept of Preemptible Instances gives operators the ability to provide a
more "elastic" capacity. At the same time, it enables the handling of increased
demand for resources, with the same infrastructure, by maximizing the cloud
utilization.

This type of servers is perfect for tasks/apps that can be terminated at any
time, enabling the users to take advantage of extra cpu power on demand without the fixed limits that quotas enforce.

Finally, here in CERN, there is an ongoing effort to provide a prototype
orchestrator for Preemptible Servers with Openstack, in order to pinpoint the
changes needed in Nova to support this feature optimally. This could also be
available in future for other OpenStack clouds in use by CERN such as the
T-Systems Open Telekom Cloud through the Helix Nebula Open Science Cloud
project.

Contributors

  • Theodoros Tsioutsias (CERN openlab fellow working on Huawei collaboration)
  • Spyridon Trigazis (CERN)
  • Belmiro Moreira (CERN)

References

by Theodoros Tsioutsias (noreply@blogger.com) at April 17, 2018 08:37 AM

April 16, 2018

OpenStack Superuser

Flipping the switch to 5G with open source

In a world where metrics are everything, some numbers are still eye-popping. Take AT&T, for example: One of the world’s largest telecom multinationals has been talking for years about “exploding” wireless traffic. How much are we talking about? On a typical business day, roughly 200 petabytes of data now flow across their wireless networks.

That number was center stage at the recent Open Networking Summit (ONS) organized by the Linux Foundation where Andre Fuetsch, AT&T’s CTO, keynoted. It represents nearly a 50 percent increase just in 12 months, he says, adding that “we kind of stopped doing the math on how many Libraries of Congress it equals, just because the numbers were just getting too ridiculous.”

Where are the new demands coming from? Streaming, video, augmented reality and gaming, he says. It all adds up to network demand that Fuetsch sees only accelerating in the future. To keep pace, AT&T plans to launch its mobile 5G network in a dozen U.S. cities before the end of 2018.

And what’s helping keep the lights on, Fuetsch says, is a strong open-source strategy.

The company plans to deploy over 60,000 white box routers in macro- and small-cell mobile infrastructures forming the core of AT&T’s 5g build. It’s an open-hardware design whose specs will be made available to the community later this year.

“Open source really helps us on two fronts: one is cost and the other speed,” Fuetsch says in an interview with TIA Now from ONS.  “If we were to approach software in a closed manner, we’d get saddled with all the costs — the life cycle costs, from birth to death… but if we’re able to put it into open source and show how others can take advantage and use it, we actually share the costs. Where the speed comes in is where we get others to help collaborate and advance that software not just for their needs but for our needs as well.”

To make 5G a reality, AT&T needed to devise a software stack at the edge, Fuetsch says. They decided to build it with open-source projects including OpenStack, Kubernetes and ONAP.

“That way we don’t start building all these one-off islands. We actually can work off a standard implementation that’s part of an open-source community that can evolve. We think it’s pretty powerful.”

Check out the entire AT&T keynote here or the TIA interview here.

What’s next

At the upcoming Vancouver Summit, you can hear more on 5G including a case study from China Unicom and sessions on network slicing, edge and autoscaling. AT&T, a platinum member of the OpenStack Foundation, will be out in force, too, heading up about 20 workshops, lightning talks and sessions covering everything from high-performance Ceph, OpenContrail-Helm and Kubernetes.

Cover Photo // CC BY NC

The post Flipping the switch to 5G with open source appeared first on Superuser.

by Superuser at April 16, 2018 04:11 PM

April 15, 2018

Michael Still

On Selecting a Well Engaged Open Source Vendor

Share

Aptira is in an interesting position in the Open Source market, because we don’t usually sell software. Instead, our customers come to us seeking assistance with deciding which OpenStack to use, or how to embed ONAP into their nationwide networks, or how to move their legacy networks to the software defined future. Therefore, our most common role is as a trusted advisor to help our customers decide which Open Source products to buy.

(My boss would insist that I point out here that we do customisation of Open Source for our customers, and have assisted many in the past with deploying pure upstream solutions. Basically, we do what is the right fit for the customer, and aren’t obsessed with fitting customers into pre-defined moulds that suit our partners.)

That makes it important that we recommend products from companies that are well engaged with their upstream Open Source communities. That might be OpenStack, or ONAP, or even something like Open Daylight. This raises the obvious question – what makes a company well engaged with an upstream project?

Read more over at my employer’s blog

Share

by mikal at April 15, 2018 11:35 AM

April 13, 2018

Red Hat Stack

Red Hatters To Present at More Than 50 OpenStack Summit Vancouver Sessions

OpenStack Summit returns to Vancouver, Canada May 21-24, 2018, and Red Hat will be returning as well with as big of a presence as ever. Red Hat will be a headline sponsor of the event, and you’ll have plenty of ways to interact with us during the show.

First, you can hear from our head of engineering and OpenStack Foundation board member, Mark McLoughlin, during the Monday morning Keynote sessions. Mark will be discussing OpenStack’s role in a hybrid cloud world, as well as the importance of OpenStack and Kubernetes integrations. After the keynotes, you’ll want to come by the Red Hat booth in the exhibit hall to score some cool SWAG (it goes quickly), talk with our experts, and check out our product demos. Finally, you’ll have the entire rest of the show to listen to Red Hatters present and co-present on a variety of topics, from specific OpenStack projects, to partner solutions, to OpenStack integrations with Kubernetes, Ansible, Ceph storage and more. These will be delivered via traditional sessions, labs, workshops, and lunch and learns. For a full list of general sessions featuring Red Hatters, see below.

Beyond meeting us at the Red Hat booth or listening to one of us speak in a session or during a keynote, here are the special events we’ll be sponsoring where you can also meet us. If you haven’t registered yet, use our sponsor code: REDHAT10 to get 10% off the list price. And check out the OpenStack Foundation’s great deals on hotels through early May.

Containers, Kubernetes and OpenShift on OpenStack Hands-on Training
Join the Red Hat’s OpenShift team for a full day of discussion and hands on lab to learn how OpenShift can help you deliver apps even faster on OpenStack.
Date: May 20th, 9:00 am-5:00 pm
Location: Vancouver Convention Centre West – Level Two – Room 218-219
RSVP required

Red Hat and Trilio Evening Social
All are invited to join Red Hat and Trilio for an evening of great food, drinks, and waterfront views of Vancouver Harbour.
When: Monday, May 21st, 7:30-10:30 pm
Location: TapShack Coal Harbour
RSVP required 

Women of OpenStack Networking Lunch sponsored by Red Hat
Meet with other women for lunch and discuss important topics affecting women in technology and business
Guest speaker: Margaret Dawson, Vice President of Product Marketing, Red Hat
Date: Wednesday, May 23 2018, 12:30-1:50 pm
Location: Vancouver Convention Centre West, Level 2, Room 215-216 
More information

 

Red Hat Training and Certification Lunch and Learn
Topic: Performance Optimization in Red Hat OpenStack Platform
Wednesday, May 23rd, 12:30-1:30 pm
Location: Vancouver Convention Centre West, Level 2, Room 213-214 

RSVP required 

Red Hat Jobs Social

Connect with Red Hatters and discover why working for the open source leader is a future worth exploring. We’ll have food, drinks, good vibes, and a chance to win some awesome swag.
Date: Wednesday, May 23, 6:00-8:00 pm
Location: Rogue Kitchen and Wetbar
RSVP required

General Sessions Featuring Red Hatters

Monday

Session Speaker Time
OpenStackSDKs – Project Update Monty Taylor 1:30 PM
Docs/i18n – Project Onboarding Stephen Finucane, Frank Kloeker (Deutsche Telekom), Ian Y. Choi (Fusetools Korea) 1:30 PM
Linux Containers Internal Lab Scott McCarty 1:30 PM
The Wonders of NUMA, or Why Your High Performance Application Doesn’t Perform Stephen Finucane 2:10 PM
Glance – Project Update Erno Kuvaja 3:10 PM
Call It Real: Virtual GPUs in Nova Silvain Bauza, Jianhua Wang (Citrix) 3:10 PM
A Unified Approach to Role-Based Access Control Adam Young 3:10 PM
Unlock Big Data Efficiency with CephData Lake Kyle Bader, Yong Fu (Intel), Jian Zhang (Intel), Yuan Zhuo (INTC) 4:20 PM
Storage for Data Platforms Kyle Bader, Uday Boppana 5:20 PM

 

Tuesday

Session Speaker Time
OpenStack with IPv6: Now You Can! Dustin Schoenbrun, Tiago Pasqualini (NetApp), Erlon Cruz (NetApp) 9:00 AM
Integrating Keystone with large-scale centralized authentication Ken Holden, Chris Janiszewski 9:50 AM
Sahara – Project Onboarding Telles Nobrega 11:00 AM
Lower the Barries: Or How To Make Hassle-Free Open Source Events Sven Michels 11:40 AM
Barbican – Project Update Ade Lee, Dave McCowan (Cisco) 11:50 AM
Glance – Project Onboarding Erno Kuvaja, Brian Rosmaita (Verizon) 11:50 AM
Kuryr – Project Update Daniel Mellado 12:15 PM
Sahara – Project Update Telles Nobrega 1:50 PM
Heat – Project Update Rabi Mishra, Thomas Herve, Rico Lin (EasyStack) 1:50 PM
Superfluidity: One Network To Rule Them All Daniel Mellado, Luis Tomas Bolivar, Irena Berezovsky (Huawei) 3:10 PM
Burnin’ Down the Cloud: Practical Private Cloud Management David Medberry, Steven Travis (Time Warner Cable) 3:30 PM
Infra – Project Onboarding David Moreau-Simard, Clark Boylon (OpenStack Foundation) 3:30 PM
Intro to Kata Containers Components: a Hands-on Lab Sachin Rathee, Sudhir Kethamakka 4:40 PM
Kubernetes Network-policies and Neutron Security Groups – Two Sides of the Same Coin? Daniel Mellado, Eyal Leshem (Huawei) 5:20 PM
How To Survice an OpenStack Cloud Meltdown with Ceph Federico Lucifredi, Sean Cohen, Sebatien Han 5:30 PM
OpenStack Internal Messaging at the Edge: In Depth Evaluation Kenneth Giusti, Matthieu Simonin, Javier Rojas Balderrama 5:30 PM


Wednesday

Session Speaker Time
Barbican – Project Onboarding Ade Lee, Dave McCowan (Cisco) 9:00 AM
Oslo – Project Update Ben Nemec 9:50 AM
Kuryr – Project Onboarding Daniel Mellado, Irena Berezovsky (Hauwei) 9:50 AM
How To Work with Adjacent Open Source Communities – User, Developer, Vendor, Board Perspective Mark McLoughlin, Anni Lai (Huawei), Davanum Srinivas (Mirantis), Christopher Price (Ericsson), Gnanavelkandan Kathirvel (AT&T) 11:50 AM
Nova – Project Update Melanie Witt 11:50 AM
TripleO – Project Onboarding Alex Schultz, Emilien Macchi, Dan Prince 11:50 AM
Distributed File Storage in Multi-Tenant Clouds using CephFS Tom Barron, Ramana Raja, Patrick Donnelly 12:20 PM
Lunch & Learn – Performance optimization in Red Hat OpenStack Platform Razique Mahroa 12:30 PM
Cinder Thin Provisioning: a Comprehensive Guide Gorka Eguileor, Tiago Pasqualini (NetApp), Erlon Cruz (NetApp) 1:50 PM
Nova – Project Onboarding Melanie Witt 1:50 PM
Glance’s Power of Image Import Plugins Erno Kuvaja 2:30 PM
Mistral – Project Update Dougal Matthews 3:55 PM
Mistral – Project Onboarding Dougal Matthews, Brad Crochet 4:40 PM
Friendly Coexistence of Virtual Machines and Containers on Kubernetes using KubeVirt Stu Gott, Stephen Gordon 5:30 PM
Intro to Container Security Thomas Cameron 11:50 AM


Thursday

Session Speaker Time
Manila – Project Update Tom Barron 9:00 AM
Oslo – Project Onboarding Ben Nemec, Kenneth Giusti, Jay Bryant (Lenovo) 9:00 AM
Walk Through of an Automated OpenStack Deployment Using Triple-O Coupled with OpenContrail – POC Kumythini Ratnasingham, Brent Roskos, Michael Henkel (Juniper Networks) 9:00 AM
Working Remotely in a Worldwide Community Doug Hellmann, Julia Kreger, Flavio Percoco, Kendall Nelson (OpenStack Foundation), Matthew Oliver (SUSE) 9:50 AM
Manila – Project Onboarding Tom Barron 9:50 AM
Centralized Policy Engine To Enable Multiple OpenStack Deployments for Telco/NFV Bertrand Rault, Marc Bailly (Orange), Ruan He (Orange) 11:00 AM
Multi Backend CNI for Building Hybrid Workload Clusters with Kuryr and Kubernetes Daniel Mellado, Irena Berezovsky (Huawei) 11:50 AM
Workshop/Lab: Containerize your Life! Joachim von Thadden 1:50 PM
Root Your OpenStack on a Solid Foundation of Leaf-Spine Architecture! Joe Antkowiak, Ken Holden 2:10 PM
Istio: How To Make Multicloud Applications Real Christian Posta, Chris Hoge (OpenStack Foundation), Steve Drake (Cisco), Lin Sun, Costin Monolanche (Google) 2:40 PM
Push Infrastructure to the Edge with Hyperconverged Cloudlets Kevin Jones 3:30 PM
A DevOps State of Mind: Continuous Security with Kubernetes Chris Van Tuin 3:30 PM
OpenStack Upgrades Strategy: the Fast Forward Upgrade Maria Angelica Bracho, Lee Yarwood 4:40 PM
Managing OpenStack with Ansible, a Hands-on Workshop Julio Villarreal Pelegrino, Roger Lopez 4:40 PM

 

We’re looking forward to seeing you there!

 

by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform at April 13, 2018 07:12 PM

OpenStack Superuser

A million playbooks a month: Scaling Ansible Run Analysis

The OpenStack community runs over 300,000 continuous integration jobs with Ansible every month with the help of the awesome Zuul.

Zuul Pipelines

It even provides ARA reports for ARA’s integration test jobs in a sort-of nested way. Zuul’s Ansible ends up installing Ansible and ARA. It makes my brain hurt sometimes… but in an awesome way.

Zuul ARA Report

As a core contributor of the infrastructure team there, I get to witness issues and get a lot of feedback directly from the users.

Static HTML report generation in ARA is simple but didn’t scale very well for us. One day, I was randomly chatting with Ian Wienand and he pointed out an attempt at a WSGI middleware that would serve extracted logs.

That inspired me to write something similar but for dynamically loading ARA sqlite databases instead… This resulted in an awesome feature that I had not yet taken the time to explain very well…until now.

An excerpt from the documentation:

To put this use case into perspective, it was “benchmarked” against a single job from the OpenStack-Ansible project:

  • 4 playbooks
  • 4,647 tasks
  • 4,760 results
  • 53 hosts, of which 39 had gathered host facts
  • 416 saved files

Generating a static report from that database takes ~1min30s on an average machine. The result contains 5321 files and 5243 directories for an aggregate size of 63MB (or 27MB recursively gzipped).
This middleware allows you to host the exact same report on your web server just by storing the sqlite database which is just one file and weighs 5.6MB.

This middleware can be useful if you’re not interested in aggregating data in a central database server like MySQL or PostgreSQL.

The OpenStack CI use case is decentralized: each of the >300,000 Zuul CI jobs have their own sqlite database uploaded as part of the log and artifact collection.

There’s a lot of benefits to doing things this way:

  • There’s no network latency to a remote database server: the first bottleneck is your local disk speed.
    • Even if it’s a 5ms road trip, this adds up over hundreds of hosts and thousands of tasks.
    • Oh, and contrary to popular belief, sqlite is pretty damn fast.
  • There’s no risk of a network interruption or central database server crash which would make ARA (and your sysadmins) panic.
  • Instead of one large database with lots of rows, you have more databases (“shards”) with fewer rows.
  • Instead of generating thousands of files and directories, you’re dealing with one small sqlite file.
  • There’s no database cluster to maintain, just standard file servers with a web server in front.

Another benefit is that you can easily have as many individual reports as you’d like, all you have to do is to configure ARA to use a custom database location.

When I announced that we’d be switching to the sqlite middleware on openstack-dev, I mentioned that projects could leverage this within their jobs and OpenStack-Ansible was the first to take a stab at it: https://review.openstack.org/#/c/557921/.

Their job’s logs now look like this:

ara-report/ansible.sqlite   # ARA report for this Zuul job
logs/                       # Job's logs
└── ara-report/             # ARA report for this OpenStack-Ansible deployment
    └── ansible.sqlite      # Database for this OpenStack-Ansible deployment

The performance improvements for the OpenStack community at large are significant.

Even if we’re spending one minute generating and transferring thousands of HTML files… That’s >300,000 minutes worth of compute that could be spent running other jobs.

How expensive are 300,000 minutes (or 208 days!) of compute? What about bandwidth and storage?

Unfreezing ARA’s stable release for development

The latest version of ARA is currently 0.14.6 and ARA was more or less in feature-freeze mode while all the work was focused on the next major release, “1.0”.

However, there is a growing amount of large scale users (me included!) who are really pushing the current limitations of ARA and 1.0 (or 2.0!) won’t be ready for a while still.

I couldn’t afford to leave performance issues and memory leaks ruin the experience of a tool that would otherwise be very useful to them.

These improvement opportunities have convinced me that there will be a 0.15.0 release for ARA.

Stay tuned for the 0.15.0 release notes and another update about 2.0 in the near future 🙂

Get involved

If you’d like to learn more, David Moreau Simard  will lead a session on Infra Project Onboarding along with the OSF’s Clark Boylan at the upcoming Vancouver Summit. Continuous integration is also one of the main themes of the Summit, check out all  of those sessions here.

 

This post first ran on David Moreau Simard‘s blog. Superuser is always interested in community content, get in touch at editorATopenstack.org

Cover Photo // CC BY NC

The post A million playbooks a month: Scaling Ansible Run Analysis appeared first on Superuser.

by David Moreau Simard at April 13, 2018 03:00 PM

Rackspace Developer Blog

Unlocking new capabilities with Syntribos 0.5.1

It has been a while since the last official release of Syntribos - a whole year, in fact! But we haven't been sitting idle. This article explores the new capabilities offered by this latest release.

What is Syntribos?

Syntribos is an open source automated API security testing tool that is maintained by members of the OpenStack Security Project.

Given a simple configuration file and a set of template HTTP requests, Syntribos generates and sends new requests that substitute parameters in the template with common attack payloads. The tool aims to automatically detect common security defects such as SQL injection, LDAP injection, buffer overflow, and so on. In addition, it can be used to help identify new security defects by automated fuzzing.

Although Syntribos ships with request templates and extensions that make it easy to test OpenStack applications, Syntribos is versatile enough to test any API.

At Rackspace, and previously under OSIC, our security teams have been using Syntribos to test a variety of APIs, both OpenStack and non-OpenStack. It has become an invaluable part of our toolset to keep the cloud secure. Already, Syntribos has been used to detect vulnerabilities before releases were launched.

New in Syntribos

Recently, we released Syntribos 0.5.1, which comes with significant usability and performance improvements. These enhancements were based, in large part, on feedback from Rackspace security engineers, who have been using Syntribos for their API security testing needs on a variety of projects. We hope that these changes make Syntribos better suited to your testing needs as well.

Multithreading

One of the biggest pain points that testers had when using Syntribos was its performance. Because Syntribos serialized its requests, a full run on a large project running on a remote host could sometimes take hours. In Syntribos 0.5.1, we have made wholesale changes to the runner, allowing a configurable number of workers to send requests in parallel. In practice, this offers a dramatic improvement in performance.

The following before and after results show the significant speed increase that the new multi-threaded runner in Syntribos 0.5.1 achieves when compared to the Syntribos 0.3.1 release. These tests were run against a small API with ~50ms network lag.

Syntribos 0.3.1 test

 ============================================
 Total: Ran 2655 tests in 349.914s
 Total: 6 unique failures and 0 unique errors
 ============================================

Syntribos 0.5.1 test

 ============================================
 Total: Ran 2655 tests in 78.639s
 Total: 6 unique failures and 0 unique errors
 ============================================

Of course, this behavior is fully configurable with command line options or configuration file parameters, in case you are testing against rate-limited endpoints or endpoints that do not handle concurrent connections well.

Template meta-variables

Request templates in Syntribos have always had the advantage of being very similar to raw HTTP requests, which makes it easy for testers to create them for a project. Syntribos 0.3.1 had the following shortcomings:

  • The template creation process usually involved a fair bit of tedious copy-and-pasting, even on attributes that are shared across templates. That means any changes to a group of templates required the user to open the templates individually and edit them one-by-one - a slow, manual process that was prone to mistakes.

  • To allow request templates to take full advantage of Syntribos' extensions or any other external libraries, they had to be cluttered with long, ugly CALL_EXTERNAL directives.

  • The request templates could only hold a limited amount of information. Syntribos treated nearly every part of the template identically, and there was no way for the user to specify, for example, that specific fields should be fuzzed only with URL-encoded characters, or with strings of a certain length.

To solve these problems, we introduced the concept of meta-variables. Meta-variables are JSON objects that are stored in separately from the request templates file and that define reusable variables, which can be referenced in request template parsing.

Any number of request templates can reference these meta-variables, and the meta-variables can even inherit values from other meta-variable files based on the directory where they are stored.

The following examples demonstrate the differences between the old and new Syntribos releases.

Syntribos 0.3.1 template example

The following example (post_image.template) demonstrates template meta-variable use:

POST /v2/images HTTP/1.1
X-Auth-Token: CALL_EXTERNAL|syntribos.extensions.identity.client:get_token_v2:["user"]|

{
    "name": "Test",
    "id": "CALL_EXTERNAL|syntribos.extensions.random_data.client:get_uuid:[]|"
    ...
}

Note Because Syntribos is fully backward-compatible, you can use this request template as-is with Syntribos 0.5.1.

With Syntribos 0.3.1, a change like adjusting the x-auth-token header for multiple templates requires manually updating each template individually.

Passing information to Syntribos 0.3.1 to assist API validation is not possible. For example, if the API accepts only ASCII characters for the name in the request body, Syntribos 0.3.1 has no way to pass that information.

Syntribos 0.3.1 code is not very pretty to look at, and you just have to live with it.

Syntribos 0.5.1 template example

With Syntribos 0.5.1, meta-variable files allow you to create a file, meta.json, in the same directory as your template, like the following example:

{
  "token": {
    "type": "function",
    "val": "syntribos.extensions.identity.client:get_scoped_token_v3",
    "args": ["user"]
  },
  "id": {
    "type": "function",
    "val": "syntribos.extensions.random_data.client:get_uuid"
  },
  "name": {
    "val": "Test"
    "fuzz_types": ["ascii"]
  }
}

Using the meta.json file, the template now looks like the following example:

POST /v2/images HTTP/1.1
X-Auth-Token: |token|

{
    "name": "|name|",
    "id": "|id|"
    ...
}

Now, your templates are simplified while containing more information. And of course, you can reference all those variables defined in meta.json in your other request templates as well.

For more detailed information on meta-variables, see https://docs.openstack.org/syntribos/latest/test-anatomy.html#meta-variable-file.

What's next?

We have big plans ahead for Syntribos, including the following:

  • Enable Syntribos to understand context beyond a single request, allowing more complex tests that create, modify, and clean up a resource.

  • Make Syntribos more CI/CD friendly.

  • Explore ways to integrate Syntribos into gate jobs and build pipelines.

  • Work on improvements to the test cases themselves, as well as to the documentation so that Syntribos is more accurate and user-friendly.

If you have any questions about Syntribos, we are available at #openstack-security on freenode. User feedback is very important to us as we continue development, and we'd love to hear from you!

April 13, 2018 12:00 AM

April 12, 2018

OpenStack Superuser

Kubernetes and Keystone: An integration test passed with flying colors

At Catalyst Cloud, we’re planning to deploy Magnum in our OpenStack based public cloud in New Zealand.

Magnum is an OpenStack service offering container clusters as-a-service, with support for Docker Swarm, Kubernetes, Mesos or DC/OS (Catalyst Cloud will support Kubernetes at the first step). Users of the service can deploy clusters of thousands of nodes in minutes and access them securely using their native APIs.

One of the feature requests coming from our existing customers is integration with OpenStack Keystone for both authentication and authorization, so that existing users within a tenant can access a Kubernetes cluster created by the tenant administrator in Magnum without too much extra user-management configuration inside the Kubernetes cluster.

The development team at Catalyst Cloud tested the integration between Keystone and Kubernetes, after some bug fixes and improvements, it works perfectly fine!

Before I walk you through it, you might want to look at a blog post about Keystone authentication in Kubernetes clusters by Saverio Proto from SWITCH. Here in this tutorial, we’ll talk about authentication and authorization, making the authorization configuration is very easy and flexible, especially after this pull request merged.  One last note before starting: we’d also like to give huge thanks to OpenLab — all of our tests are performed on infrastructure that they provided.

Prerequisites

  • An OpenStack deployment. I’m using Devstack to set up the OpenStack cloud of Queens release for the test.
  • We’re assuming that users for this tutorial already use OpenStack.
    In the real world, there are several types of users:

    • Cloud operators responsible for cloud operation and maintenance
    • Kubernetes cluster admin users who can create/delete Kubernetes cluster in OpenStack cloud and who are also administrators of the Kubernetes cluster
    • Kubernetes cluster normal users who can use the cluster resources, including, maybe different users with different authorities
  • Kubernetes version >= v1.9.3. For testing purposes, I won’t use Magnum in this post, the Kubernetes cluster admin can create the cluster in VMs by using kubeadm.
  • kubectl version >=v1.8.0. As Saverio Proto said in his blog post, kubectl has been capable of reading the openstack env variables since v1.8.0

Basic workflow

From the perspective of OpenStack cloud operator

As a cloud operator, there are some tasks to perform before exposing the Keystone authentication and authorization functionality to Kubernetes cluster admin:

  • Necessary Keystone roles for Kubernetes cluster operations need to be created for different users, e.g. kube-adm, kube-editor, kube-viewer
    • kube-adm role can create/update/delete Kubernetes cluster, can also associate roles to other normal users within the tenant
    • kube-editor can create/update/delete/watch Kubernetes cluster resources
    • kube-viewer can only have read access to Kubernetes cluster resources
source ~/openstack_admin_credentials
for role in "k8s-admin" "k8s-viewer" "k8s-editor"; do openstack role create $role; done
openstack role add --user demo --project demo k8s-viewer
openstack user create demo_editor --project demo --password password
openstack role add --user demo_editor --project demo k8s-editor
openstack user create demo_admin --project demo --password password
openstack role add --user demo_admin --project demo k8s-admin

From the perspective of a Kubernetes cluster admin

In this example, demo_admin user is a Kubernetes cluster admin in the demo tenant. demo_admin user is responsible for creating Kubernetes cluster inside the VMs, it could be done easily with Saverio Proto’s repo, I also have an ugly Ansible script repo here.

After the Kubernetes cluster is up and running, the cluster admin should make some configuration for Keystone authentication and authorization to work.

  1. Define Keystone authorization policy file.
    cat << EOF > /etc/kubernetes/pki/webhookpolicy.json
    [
      {
        "resource": {
          "verbs": ["get", "list", "watch"],
          "resources": ["pods"],
          "version": "*",
          "namespace": "default"
        },
        "match": [
          {
            "type": "role",
            "values": ["k8s-admin", "k8s-viewer", "k8s-editor"]
          },
          {
            "type": "project",
            "values": ["demo"]
          }
        ]
      },
      {
        "resource": {
          "verbs": ["create", "update", "delete"],
          "resources": ["pods"],
          "version": "*",
          "namespace": "default"
        },
        "match": [
          {
            "type": "role",
            "values": ["k8s-admin", "k8s-editor"]
          },
          {
            "type": "project",
            "values": ["demo"]
          }
        ]
      }
    ]
    EOF
    

    As an example, the above policy file definition is pretty straightforward. The Kubernetes cluster can only be accessed by the users in demo tenant, users with k8s-admin or k8s-editor role have both write and read permission to the pod resource, but users with k8s-viewer role can only get/list/watch pods.

  2. Deploy k8s-keystone-auth service. The implementation of k8s-keystone-auth service locates in the OpenStack cloud provider repo for Kubernetes. It’s running as a static pod (managed by kubelet) in the Kubernetes cluster.
    cat << EOF > /etc/kubernetes/manifests/k8s-keystone-auth.yaml
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        component: k8s-keystone-auth
        tier: control-plane
      name: k8s-keystone-auth
      namespace: kube-system
    spec:
      containers:
        - name: k8s-keystone-auth
          image: lingxiankong/k8s-keystone-auth:authorization-improved
          imagePullPolicy: Always
          args:
            - ./bin/k8s-keystone-auth
            - --tls-cert-file
            - /etc/kubernetes/pki/apiserver.crt
            - --tls-private-key-file
            - /etc/kubernetes/pki/apiserver.key
            - --keystone-policy-file
            - /etc/kubernetes/pki/webhookpolicy.json
            - --keystone-url
            - http://10.0.19.138/identity/v3
          volumeMounts:
            - mountPath: /etc/kubernetes/pki
              name: k8s-certs
              readOnly: true
            - mountPath: /etc/ssl/certs
              name: ca-certs
              readOnly: true
          resources:
            requests:
              cpu: 200m
          ports:
            - containerPort: 8443
              hostPort: 8443
              name: https
              protocol: TCP
      hostNetwork: true
      volumes:
      - hostPath:
          path: /etc/kubernetes/pki
          type: DirectoryOrCreate
        name: k8s-certs
      - hostPath:
          path: /etc/ssl/certs
          type: DirectoryOrCreate
        name: ca-certs
    status: {}
    EOF
    

    The image is built using the Dockerfile in the cloud-provider-openstack repo, you can build your own image if needed. Please replace the keystone-url param pointing to your own OpenStack cloud.

  3. Configure authentication and authorization webhook for Kubernetes API server, then wait for the API server to run.
    cat << EOF > /etc/kubernetes/pki/webhookconfig.yaml
    ---
    apiVersion: v1
    kind: Config
    preferences: {}
    clusters:
      - cluster:
          insecure-skip-tls-verify: true
          server: https://localhost:8443/webhook
        name: webhook
    users:
      - name: webhook
    contexts:
      - context:
          cluster: webhook
          user: webhook
        name: webhook
    current-context: webhook
    EOF
    sed -i "/authorization-mode/c \ \ \ \ - --authorization-mode=Node,Webhook,RBAC" /etc/kubernetes/manifests/kube-apiserver.yaml
    sed -i '/image:/ i \ \ \ \ - --authentication-token-webhook-config-file=/etc/kubernetes/pki/webhookconfig.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
    sed -i '/image:/ i \ \ \ \ - --authorization-webhook-config-file=/etc/kubernetes/pki/webhookconfig.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml
    

Now the Kubernetes cluster is ready for use by users within the demo tenant in OpenStack.

From the perspective of a Kubernetes cluster user

Cluster users only need to config the kubectl to work with OpenStack environment variables.

kubectl config set-credentials openstackuser --auth-provider=openstack
kubectl config set-context --cluster=kubernetes --user=openstackuser openstackuser@kubernetes --namespace=default
kubectl config use-context openstackuser@kubernetes
  1. demo user can list pods but can not create pods.
    $ source ~/openrc_demo
    $ kubectl get pods
    No resources found.
    $ cat << EOF > ~/test_pod.yaml
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: busybox-test
      namespace: default
    spec:
      containers:
      - name: busybox
        image: busybox
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
      restartPolicy: Never
    EOF
    $ kubectl create -f ~/test_pod.yaml
    Error from server (Forbidden): error when creating "test_pod.yaml": pods is forbidden: User "demo" cannot create pods in the namespace "default"
    
  2. demo_editor user can create and list pods.
    $ source ~/openrc_demoeditor
    $ kubectl create -f ~/test_pod.yaml
    pod "busybox-test" created
    $ kubectl get pods
    NAME           READY     STATUS    RESTARTS   AGE
    busybox-test   1/1       Running   0          3s
    
  3. users from other tenants don’t have access to the Kubernetes cluster
    $ source ~/openrc_alt_demo
    $ kubectl get pods
    Error from server (Forbidden): pods is forbidden: User "alt_demo" cannot list pods in the namespace "default"
    

All in one – live demo

asciicast

Future work

At Catalyst Cloud, we’re working on automating all the manual steps in Magnum, so in the near future, the Kubernetes cluster admin will only need to run a single command to create the cluster and Magnum will perform the rest automatically. We’ll also continue to work with sig-openstack group to keep improving stability and performance of the integration.

Author Lingxian Kong will be heading up four sessions at the Vancouver Summit, check them out here.

The post Kubernetes and Keystone: An integration test passed with flying colors appeared first on Superuser.

by Lingxian Kong at April 12, 2018 04:14 PM

Matthew Treinish

Python Test Runners

For the past year and half I’ve been working on a parallel test runner for python tests, stestr. I started this project to fit a need in the OpenStack community and it since has been adopted as the test runner used in the OpenStack project. But, it is generally a useful tool and something I think would provide general value for people writing python tests. I thought it would be valuable to explain two things in this post, first the testing stack underlying stestr and the history behind the project. The second aspect is how it compares to other popular python test runners are out there, and the purpose it fills.

Python unittest

Included in the python standard library is the unittest library. This provides the basic framework for writing, discovering, and running tests in python. It uses an object oriented model where tests are organized in classes (called test cases) with individual methods that represent a single test. Each test class has some standard fixtures like setUp and tearDown which define functions to return at certain phases of the text execution. These classes and their modules are combined to build test suites. These suites can either be manually constructed or use test discovery to build the suite automatically by scanning a directory.

The thing which is often misunderstood, especially given it’s name,  is that Python unittest is not strictly limited to unit testing or testing python code. The framework provides a mechanism for running structured python code and treat the execution of this code as test results. This enables you to write tests that do anything in python. I’ve personally seen examples that test a wide range of things outside of python code. Including hardware testing, CPU firmware testing, and Rest API service testing.

unittest2

As unittest is a python library included in the cpython standard library it gets improvements and new features with each release of cPython. This makes writing tests that support multiple versions of python a bit more difficult, especially if you want to leverage features in newer version of python. This is where the unittest2 library comes in, it provides backports of features from newer versions of python. This enables older versions of python to leverage features from newer unittest. It was originally written to leverage features included in python 2.7 unittest with python 2.6 and older. But, it also backports features from newer versions of python 3 to older versions of python.

Testtools

Building on the unittest framework is the testtools library. Testtools provides an extension on top of unittest to provide additional features like additional assert methods and a generic matcher framework to do more involved object comparisons.  While it is an extension on top of unittest testtools maintains compatibility with the upstream python standard lib unittest. So you can write tests that leverage this extra functionality and use them with any other unittest suite or runner.

One of the key things that testtools provides for the layers above it in this stack is it’s improved results stream. This includes the concept of details, which are like attachments that enable storing things like logging or stdout with test result.

Subunit

Subunit is streaming protocol for test results. It provides a mechanism for sharing test results from multiple sources in real time. It’s a language agnostic protocol with bindings for multiple languages including: python, perl, c, c++, go, js and others. I also recently learned that someone created a wikipedia page for the protocol: https://en.wikipedia.org/wiki/Subunit_(format)

The python implementation of the subunit library (the subunit library repository  is multilanguage) is built by extending testtools. It builds off of testtools’s test runner and the result stream additions that testtools adds on to base unittest. This means that any unittest suite (or testtools) can simply replace their test runner with subunit’s and get a real time subunit output stream. It’s this library that enables parallel execution and strict unittest compatibility in the tools above it on the stack.

The other thing which is worth pointing out is that because of OpenStack’s usage of testr and stestr we have developed a lot of tooling in the community around consuming subunit. Including things like stackviz, subunit2sql, and subunit2html. All of which can be reused easily by anything that uses subunit. There are also tools to convert between different test result formats, like junitxml,  and subunit.

Testrepository

Also known as testr, which is the command name, is a bit of a misunderstood project depending on who you talk too. Testrepository is technically a repository for storing test results, just as it’s name implies. As part of that it includes a test running mechanism to run any command which will generate a subunit results stream. It supports running those commands in parallel both locally and on remote machines. Having a repository results then enables using that data for future runs. For example, testr lets you rerun a test suite only with tests that failed in the previous run, or  use the previous run to bisect failures to try and find issues with test isolation. This has proven to be a very useful feature in practice, especially when working with large and/or slow test suites.

But for several years it was the default test runner used by the OpenStack project, and a lot of people just see it as the parallel test runner used by OpenStack. Even though the scope of the project is much larger than just python testing.

ostestr

Since the OpenStack project started using testr in late 2012/early 2013 there were a lot of UX issues and complaints people had with it.  People started working around these in a number of ways, there was the introduction of multiple setuptools entrypoints to expose commands off of setup.py to inovke testr. There were also multiple bash scripts floating around to run testr with an alternative UI called pretty_tox.sh. pretty_tox.sh started in the tempest project and was quickly copied into most projects using testr. However each copy tended to diverge and embed their own logic or preferences. ostestr was developed to try and unify those bash scripts, and it shows. It was essentially a bash script written in python that would literally subprocess out and call testr.

stestr

This is where stestr entered the field. After having maintained ostestr for a while I was getting frustrated with a number of bugs and quirks in testrepository itself. Instead of trying to work around them it would just be better to fix things at the source. However, given the lack of activity in the testrepository project this would have been difficult. I personally had pull requests sitting idle for years on the project. So I decided after a lot of personal deliberation to fork it.

I took the test runner UX lessons I learned from maintaining ostestr and started rewriting large chunks of testr to make stestr. I also tried to restructure the code to be easier to maintain and also leverage newer python features. testrepository was started 8 years ago and it supported python versions < 2.7 (having been started before 2.7’s release) this included a lot of code to implement things that were included standard in newer, more modern versions of the language.

The other aspect to stestr is that it’s scoped to just being a parallel python test runner. While testrepository is designed to be a generic test runner runner that will work with any test runner that emits a subunit result stream, stestr will only deal with python tests. Personally I always felt there was a tension in the project when using it strictly as a python test runner, some of the abstractions testr had to make caused a lot of extra work for people using it only for python testing. Which is why I rescoped the project to only be concerned with python testing.

Other Popular Test Runners

While I am partial to the tools described above and the way the stack is constructed (for the most part). These tools are far from the only way to run python tests. In fact outside of OpenStack this stack isn’t that popular. I haven’t seen it used in too many other places. So it’s worth looking at other popular test runners out there, and how they compare to stestr.

nosetests

nosetests at one time was the test runner used by the majority of python projects. (including OpenStack) It provided a lot  of missing functionality from python unittest; especially before python 2.7 which is when python unittest really started getting more mature. However it did this by basically writing it’s own library for testing and coupling that with the runner. While you can use nosetests for running unitttest suites in most cases, the real power with nose comes from using it’s library in conjunction with the runner. This made using other runners or test tooling with a nose test suite very difficult. Having personally worked with a large test suite that was written using nose migrating that to work with any unittest runner is not a small task. (it took several months to get it so tests could run with unittest)

Currently nosetests is a mostly inactive project in maintenance mode. There is a successor project, nose2, which was trying to fix some of the issues with nose and make it up to date. But it too is currently in maintenance mode, and not really super active anymore. (but it’s more active then nose proper) Instead the docs refer people to use pytest.

pytest

pytest is in my experience by far the most popular python test runner out there. It provides a great user experience (arguably the most pleasant), is pluggable, and seems to have the most momentum as a python test runner. This is with good reason there are a lot of nice features with pytest, including very impressive failure introspection which basically will just tell you why a test failed, making debugging failures much simpler.

But there are a few things to consider when using pytest, the biggest of which is it’s not strictly unittest compatible. While pytest is capable of running tests written using the unittest library it’s not actually a unittest based runner. So things like test discovery differ in how they work on pytest. (it’s worth noting pytest supports nose test suites too)

The other things that’s missing from pytest by default is parallel test execution. There is a plugin pytest-xdist which enables this, and it has come a long way in the last several years. It doesn’t provide all of the same features for parallel execution as stestr, especially around isolation and debugging failures. But for most people it’s probably enough.

Quite frankly, if I weren’t the maintainer of stestr and if I didn’t need or want things that stestr provides like first class parallel execution support, a local results repository, strict unittest compatibility, or subunit support I’d  probably just use pytest for my projects.

by Matthew Treinish at April 12, 2018 09:00 AM

April 11, 2018

OpenStack Superuser

What makes a Superuser

The Superuser Awards are open — you’ve got until April 15 to nominate your team for the Vancouver Summit.

Over the years, we’ve had a truly stellar group of winners (AT&T, CERN, China Mobile, Comcast, NTT Group) and finalists (GoDaddy, FICO, Walmart, Workday among many others). While only one wins the title, all of the finalists get a shoutout with an overview of what makes them super from the keynote stage at the Vancouver Summit.

When evaluating winners for the Superuser Award, nominees are judged on the unique nature of use case(s), as well as integrations and applications of OpenStack performed by a particular team.

Here are a few things that will make your application stand out:

  • Take a look at what works.  You can browse the applications of previous finalists here and the winners here.
  • Take your time. The application seems short — eight questions — but the majority of those questions cover a lot of ground.
  • Boil it down. It’s a balancing act: you’ll want to include significant milestones and contributions but stay within character count. (The space allotted for each answer is 800 characters, roughly 130 words.) Offers of libations in Vancouver will not persuade the judging committee or Foundation staff to accept supplemental materials!
  • Remember that you’re talking to your peers. Your nomination is first taken into consideration by the larger OpenStack community and then by the Superuser Editorial Advisory Board which makes the final call. Most of them are technically-minded folks who tend to be more impressed with metrics than marketing speak.
  • Lead with your most impressive accomplishments. All of the applications from nominees are published on Superuser for community voting. Most likely they’ll see a mention on social media, scan through your post, then click to vote. Make sure they see the best bits first.
  • Proofread and fact check with your team before submitting. The Superuser editorial staff goes through every finalist application and edits them for grammar and house style, but do keep in mind that the information you submit in the application will be published.

We’re looking forward to hearing more about your accomplishments with open infrastructure – deadline for Superuser Awards for the Vancouver Summit is April 15.  Superuser Awards application.

The post What makes a Superuser appeared first on Superuser.

by Superuser at April 11, 2018 04:09 PM

April 10, 2018

OpenStack Superuser

Inside container infrastructure: Must-see sessions at the Vancouver Summit

Join the people building and operating open infrastructure at the OpenStack Summit Vancouver in May.  The Summit schedule features over 300 sessions organized by use cases: including AI and machine learning, HPC, edge computing, NFV, container infrastructure and public, private and multi-cloud strategies. For the container curious, note that 60 of those sessions feature Kubernetes and another 25 feature Docker.

Here we’re highlighting some of the sessions you’ll want to add to your schedule about container infrastructure. Check out all the sessions, workshops and lightning talks focusing on container infrastructure here.

CERN experience with multi-cloud, federated Kubernetes

Using public cloud resources to cover for peak workloads is a practical and economical alternative to over provisioning on-premise resources. This is the case in environments like CERN, where its large internal computing infrastructure usually suffices but where periods prior to big international conferences or large event reconstruction campaigns see a significant spike in the amount of workloads submitted. CERN’s Ricardo Rocha and Clenimar Filemon Universidade Federal de Campina Grande will share their experiences in this intermediate-level talk. Details here.

OpenStack-Helm hands-on workshop: Deploy and upgrade OpenStack on Kubernetes

OpenStack-Helm is a Helm chart library that allows simple customization and deployment of OpenStack containers across Kubernetes clusters, from laptop-scale to data center-scale. Bring your own laptop for this beginner workshop led by AT&T’s Pete Birley, SK Telecom’s Jawon Choo and Siri Kim. Participants will deploy OpenStack on top of Kubernetes, deploy logging, monitoring, and alerting tools and perform OpenStack version upgrade on the running OpenStack cluster.
Details
here.

Kata Containers: The way to run virtualized containers

Kata Containers is a new open source project merging two hypervisor-based container runtime efforts: Hyper’s runV and Intel’s Clear Containers. Providing an OCI and CRI compatible runtime, it seamlessly integrates with OpenStack containers services. Each container, or each sandbox as defined by Zun, is hypervisor-isolated and runs inside a dedicated Linux VM. Intel’s Sebastien Boeuf will demo how Kata Containers can be as fast as a namespace-based container runtime while being run in a VM in this intermediate-level session. Details here.

Bonus session: If you’d like to go hands-on, check out the Kata workshop from Red Hat’s Sachin Rathee and Sudhir Kethamakka.

On the way to cloud native: Working with containers in a hybrid environment

Nokia’s Amy Fredj, Liat Pele and Gideon Agmon will showcase an example of a VNF implementation based on VMs and containers. Their option allows VNFs to be developed for hybrid environments, where installation is based on OS and networking uses Calico BGP to distribute Neutron networks to container domain. In this beginner-level talk, they’ll address lifecycle management, including installation, management, networking and root cause analysis in the hybrid world.
Details here.

Intro to container security

Application containerization solves numerous problems, allows for incredible application density, and can really increase flexibility and responsiveness. But container security is a lot more than what application is in the container. In this session, Red Hat’s Thomas Cameron will talk about the basic components of container security including kernel namespaces, Security Enhanced Linux, Linux control groups, the Docker daemon, etc. He’ll also talk about tips and tricks for planning a secure container environment, describe some “gotchas” about containers and debunk some of the security myths about containers. Details here.

See you at the OSF Summit in Vancouver, May 21-24, 2018! Register here.

Cover Photo // CC BY NC

The post Inside container infrastructure: Must-see sessions at the Vancouver Summit appeared first on Superuser.

by Superuser at April 10, 2018 02:51 PM

Chris Dent

TC Report 18-15

It feels like people are busy. Traffic in the #openstack-tc channel has been light for the past week.

Forum Topics

By this coming Thursday we hope to have determined which of several topics should be proposed for the forum.

Kolla Situation

In email and in IRC discussion Wednesday and Thursday the many faces of Kolla have been debated. This started as a discussion of whether kolla-kubernetes should be retired but has branched widely since then, including plenty of discussion on whether Kolla is doing containers "the right way" (whatever that might be), and whether the start scripts in Kolla images should be moved.

One of the side topics within this discussion is the nature of the multiple hats worn by members of the community when engaging in discussion. Does membership on the TC mean that everything you say is with the voice of the TC? I certainly hope not, however there is probably more that can be done to be clear which hat is being worn in any given situation.

(So it's clear, these reports are written wearing my Chris hat and are not the voice of the TC.)

Consumption Models

There was some discussion on Thursday about more effectively tracking the consumption (of OpenStack) models that are present in the community. ttx suggested some additional survey questions. Better data ought to help us decide if energy is being spent in the right ways.

Elections

The nomination period for TC elections starts tonight at 1 minute to midnight, UTC. There are seven positions open for this election. The potential forum topics give a pretty good overview of some of the things that are on the TC's radar for the coming months.

by Chris Dent at April 10, 2018 11:15 AM

Opensource.com

Install an OpenStack cloud with Jenkins

Learn to install and configure the Jenkins automation server and create a job to install an OpenStack cloud.

by architmodi at April 10, 2018 07:02 AM

April 09, 2018

Adam Young

Comparing Keystone and Istio RBAC

To continue with my previous investigation to Istio, and to continue the comparison with the comparable parts of OpenStack, I want to dig deeper into how Istio performs
RBAC. Specifically, I would love to answer the question: could Istio be used to perform the Role check?

Scoping

Let me reiterate what I’ve said in the past about scope checking. Oslo-policy performs the scope check deep in the code base, long after Middleware, once the resource has been fetched from the Database. Since we can’t do this in Middleware, I think it is safe to say that we can’t do this in Istio either. SO that part of the check is outside the scope of this discussion.

Istio RBAC Introduction

Lets look at how Istio performs RBAC.

The first thing to compare is the data that is used to represent the requester. In Istio, this is the requestcontext. This is comparable to the Auth-Data that Keystone Middleware populates as a result of a successful token validation. How does Istio populate the the requestcontext? My current assumption is that it makes an Remote call to Mixer with the authenticated REMOTE_USER name.

What is telling is that, in Istio, you have

      user: source.user | ""
      groups: ""
      properties:
         service: source.service | ""
         namespace: source.namespace | ""

Groups no roles. Kubernetes has RBAC, and Roles, but it is a late addition to the model. However…

Istio RBAC introduces ServiceRole and ServiceRoleBinding, both of which are defined as Kubernetes CustomResourceDefinition (CRD) objects.

ServiceRole defines a role for access to services in the mesh.
ServiceRoleBinding grants a role to subjects (e.g., a user, a group, a service)

This is interesting. Where-as Keystone requires a user to go to Keystone to get a token that is then associated with a a set of role assignments, Istio expands this assignment inside the service.

Keystone Aside: Query Auth Data without Tokens

This is actually not surprising. When looking into Keystone Middleware years ago, in the context of PKI tokens, I realized that we could do exactly the same thing; make a call to Keystone based on the identity, and look up all of the data associated with the token. This means that a user can go from a SAML provider right to the service without first getting a Keystone token.

What this means is that the Mixer can respond return the Roles assigned by Kubernetes as additional parameters in the “Properties” collection. However, with the ServiceRole, you would instead get the Service Role Binding list from Mixer and apply it in process.

We discussed Service Roles on multiple occasions in Keystone. I liked the idea, but wanted to make sure that we didn’t limit the assignments, or even the definitions, to just a service. I could see specific Endpoints varying in their roles even within the same service, and certainly have different Service Role Assignments. I’m not certain if Istio distinguishes between “services” and “different endpoints of the same service” yet…something I need to delve in to. However, assuming that it does distinguish, what Istio needs to be able to get request is “Give me the set of Role bindings for this specific endpoint.”

A history lesson in Endpoint Ids.

It was this last step that was a problem in Keystonemiddleware. An endpoint did not know its own ID, and the provisioning tools really did not like the workflow of

  1. create an endpoint for a service
  2. register endpoint with Keystone
  3. get back the endpoint ID
  4. add endpoint  ID to the config file
  5. restart the service

Even if we went with an URL based scheme, we would have had this problem.  An obvious (in hindsight) solution would be to pre-generate the Ids as a unique hash, and to pre-populate the configuration files as well as to post the IDs to Keystone.  These IDs could easily be tagged as a nickname, not even the canonical name of the service.

Istio Initialization

Istio does no have this problem, directly, as it knows the name of the service that it is protecting, and can use that to fetch the correct rules.  However, it does point to a chicken-egg problem that Istio has to solve; which is created first, the service itself, or the abstraction in Istio to cover it?  Since Kubernetes is going to orchestrate the Service deployment, it can make the sensible call;  Istio can cover the service and just reject calls until it is properly configured.

URL Matching Rules

If we look at the Policy enforcement in Nova, we can use the latest “Policy in Code” mechanisms to link from the URL pattern to the Policy rule key, and the key to the actual enforced policy.  For example, to delete a server we can look up the API

And see that it is

/servers/{server_id}

And from the Nova source code:

  policy.DocumentedRuleDefault(
        SERVERS % 'delete',
        RULE_AOO,
        "Delete a server",
        [
            {
                'method': 'DELETE',
                'path': '/servers/{server_id}'
            }
]),

With SERVERS %  expanding via :  SERVERS = 'os_compute_api:servers:%s'  to  os_compute_api:servers:delete.

Digging into Openstack Policy

Then, assuming you can get you hand on the policy file specific to that Nova server you could look at the policy for that rule. Nova no longer includes that generated file in the etc directory. But in my local repo I have:
"os_compute_api:servers:delete": "rule:admin_or_owner"

And the rule:admin_or_owner expanding to "admin_or_owner": "is_admin:True or project_id:%(project_id)s" which does not do a role check at all. The policy.yaml or policy.json file is not guaranteed to exist, in which case you can either use the tool to generate it, or read the source code. From the above link we see the Rule is:

RULE_AOO = base.RULE_ADMIN_OR_OWNER

and then we need to look where that is defined.

Lets assume, for the moment, that a Nova deployment has overridden the main rule to implement a custom role called Custodian which has the ability to execute this API. Could Istio match that? It really depends on whether it can match the URL-Pattern: '/servers/{server_id}'.

In ServiceRole, the combination of “namespace”+”services”+”paths”+”methods” defines “how a service (services) is allowed to be accessed”.

So we can match down to the Path level. However, there seems to be no way to tokenize a Path. Thus, while you could set a rule that says a client can call DELETE on a specific instance, or DELETE on /services, or even DELETE on all URLS in the catalog (whether they support that API or not) you could not say that it could call delete on all services within a specific Namespace. If the URL were defined like this:

DELETE /services?service_id={someuuid}

Istio would be able to match the service ID in the set of keys.

In order for Istio to be able to effectively match, all it really would need would be to identify that an URL that ends /services/feed1234 Matches the pattern /services/{service_id} which is all that the URL pattern matching inside the Web servers do.

Istio matching

It looks like paths can have wildcards. Scroll down a bit to the quote:

In addition, we support prefix match and suffix match for all the fields in a rule. For example, you can define a “tester” role that has the following permissions in “default” namespace:

which has the further example:

    - services: ["bookstore.default.svc.cluster.local"]
       paths: ["*/reviews"]
       methods: ["GET"]

Deep URL matching

So, while this is a good start, there are many more complicated URLs in the OpenStack world which are tokenized in the middle: for example, the new API for System role assignments has both the Role ID and the User ID embedded. The Istio match would be limited to matching: PUT /v3/system/users/* which might be OK in this case. But there are cases where a PUT at one level means one role, much more powerful than a PUT deeper in the URL chain.

For example: The base role assignments API itself is much more complex. To assign a role on a domain uses an URL fragment comparable to that to edit the domain specific configuration file. Both would have to be matched with

       paths: ["/v3/domains/*"]
       methods: ["PUT"]

But assigning a role is a far safer operation than setting a domain specific config, which is really an administrative only operation.

However, I had to dig deeply to find this conflict. I suspect that there are ways around it, and comparable conflicts in the catalog.

Conclusion

So, the tentative answer to my question is:

Yes, Istio could perform the Role check part of RBAC for OpenStack.

But it would take some work. Of Course. An early step would be to write a Mixer plugin to fetch the auth-data from Keystone based on a user. This would require knowing about Federated mappings and how to expand them, plus query the Role assignments. Of, and get the list of Groups for a user. And the project ID needs to be communicated, somehow.

by Adam Young at April 09, 2018 05:55 PM

OpenStack Superuser

The last mile: Where edge meets containers

Verizon’s Beth Cohen is on the front lines of edge. In her role as cloud networking product manager, she focuses on developing new cloud-based networking products and services.

She sat down, along with Ericsson’s Chris Price and the OpenStack Foundation’s Ildiko Vancsa, to talk all things edge with TIA now recently.  It’s a lively conversation about that the trio will be expanding on at the upcoming Vancouver Summit in a panel session along with Ericsson’s Alison Randal.

“It’s really important to understand that edge is not just one thing, it’s a lot of different things — it’s containers but not just containers…” Cohen says. “It’s a new way of looking at networks. Verizon has been deploying hundreds of thousands of devices for 50 years, but it’s the challenge of understanding the disaggregation between the hardware and the software and the VNFs and the underlying management.”

The tool sets that developed over long periods of time aren’t really designed to think about that disaggregation, she adds. An example? “The alarming systems have to take into consideration that there’s this underlying hardware… If a VNF or a network service gets alarmed, the underlying hardware might be just perfectly fine.”

Price agrees. “Containers are the key technology to allow you to get the applications deployed — get simple lifecycle management, get scale at the edge — but there’s a lot more that goes into actually enabling that.”

Culture plays a big part in the transition. “Just having the conversations with the operations people has been a challenge,” Cohen says. “I’ve spent hours and hours literally training them about how to work with virtual systems and virtual services and virtual applications because they don’t really think that way, that’s not in their DNA.”

While Price maintains that edge use cases are still being inched towards, Cohen sees a fairly straightforward path.
“Tech refresh is a huge use case for us and I think that’s across the board,” she says but adds there some other really more “far-out use cases” that will come about once companies have invested in the technology. She cites internet of things and augmented reality  as the “kind of sexy things” that will  come out of it but maintains not they’re not going to be the “driving force.”

Speaking of driving, “self-driving cars is actually the worst” use case for edge she says. “If you’re sending all the stuff back to the data center and it’s coming back, you really need very low latency and guaranteed connections. You don’t want to send off a request and not get a response if you’re trying to turn the corner.”

Vancsa underlines that there are commonalities, no matter the use case.  “It’s also important to mention that even if we are looking into telco or all the whole global industry, basically the requirements that it all boils down to are really similar in all these cases.”

They also talk about open source versus commercial edge, what’s next in tooling and current challenges in 16-minute interview.

You can check out all the Vancouver sessions, workshops and lightning talks focusing on edge here. And, if you’re interested in more about edge computing, read the new whitepaper created by the Edge Computing Group available at openstack.org/edge

Cover Photo // CC BY NC

The post The last mile: Where edge meets containers appeared first on Superuser.

by Nicole Martinelli at April 09, 2018 04:00 PM

Mirantis

How to create a Spinnaker pipeline

Ultimately we'll use Spinnaker for our whole CI/CD lifecycle management, but let's just start by creating an application that lets us resize a cluster based on feedback from an external system.

by Nick Chase at April 09, 2018 05:07 AM

David Moreau Simard

Scaling ARA to a million Ansible playbooks a month

The OpenStack community runs over 300 000 CI jobs with Ansible every month with the help of the awesome Zuul. It even provides ARA reports for ARA’s integration test jobs in a sort-of nested way. Zuul’s Ansible ends up installing Ansible and ARA. It makes my brain hurt sometimes… but in an awesome way. As a core contributor of the infrastructure team there, I get to witness issues and get a lot of feedback directly from the users.

April 09, 2018 12:00 AM

April 07, 2018

Adam Young

Comparing Istio and Keystone Middleware

One way to learn a new technology is to compare it to what you already know. I’ve heard a lot about Istio, and I don’t really grok it yet, so this post is my attempt to get the ideas solid in my own head, and to spur conversations out there.

I asked the great Google “Why is Istio important” and this was the most interesting response it gave me: “What is Istio and Its Importance to Container Security.” So I am going to start there. There are obviously many articles about Istio, and this might not even be the best starting point, but this is the internet: I’m sure Ill be told why something else is better!

Lets start with the definition:

Istio is an intelligent and robust web proxy for traffic within your Kubernetes cluster as well as incoming traffic to your cluster

At first blush, these seems to be nothing like Keystone. However, Lets take a look at the software definition of Proxy:

A proxy, in its most general form, is a class functioning as an interface to something else.

In the OpenStack code base, the package python-keystonemiddleware provides a Python class that complies with the WSGI contract that serves as a Proxy to the we application underneath. Keystone Middleware, then is an analogue to the Istio Proxy in that it performs some of the same functions.

Istio enables you to specify access control rules for web traffic between Kubernetes services

So…Keystone + Oslo-policy serves this role in OpenStack. The Kubernetes central control is a single web server, and thus it can implement Access control for all subcomponenets in a single process space. OpenStack is distributed, and thus the access control is also distributed. However, due to the way that OpenStack objects are stored, we cannot do the full RBAC enforcement in middleware (much as I would like to). IN order to check access to an existing resource object in OpenStack, you have to perform the policy enforcement check after the object has been fetched from the Database. That check needs to ensure that the project of the token matches the project of the resource. Since we don’t know this information based solely on the URL, we cannot perform it in Middleware.

What we can perform in Middleware, and what I presented on last year at the OpenStack Summit, is the ability to perform the Role check portion of RBAC in middleware, but defer the project check until later. While we are not going to be doing exactly that, we are pursuing a related effort for application credentials. However, that requires a remote call to a database to create those rules. Istio is not going to have that leeway. I think? Please correct me if I am wrong.

I don’t think Istio could perform this level of deep check, either. It requires parsing the URL and knowing the semantics of the segments, and having the ability to correlate them. That is a lot to ask.

Isito enables you to seamlessly enforce encryption and authentication between node

Keystone certainly does not do this. Nothing enforced TLS between services in OpenStack. Getting TLS everywhere in Tripleo was a huge effort, and it still needs to be explicitly enabled. OpenStack does not provide a CA. Tripleo, when deployed, depends on the Dogtag instance from the FreeIPA server to manage certificates.

By the time Keystone Middleware is executed, the TLS layer would be a distant memory.

Keystoneauth1 is the client piece from Keystone, and it could be responsible for making sure that only HTTPS is supported, but it does not do that today.

Istio collects traffic logs, and then parses and presents them for you:

Keystone does not do this, although it does produce some essential log entries about access.

At this point, I am wondering if Istio would be a viable complement to the security story in OpenStack. My understand thus far is that it would. It might conflict a minor bit with the RBAC enforcement, but I suspect that is no the key piece of what it is doing, and conflict there could be avoided.

Please post your comments, as I would really like to get to know this better, and we can share the discussion with the larger community.

by Adam Young at April 07, 2018 10:00 PM

April 06, 2018

OpenStack Superuser

How to make your OpenStack Summit talk a big success

You prepared, you submitted, you were accepted; congratulations! The OpenStack community is intelligent and engaged, so expectations are always high. Whether this is your 50th or first talk at an OpenStack Summit, here’s five little ways to make sure your talk is a success.

Focus on the non-obvious

Assume your audience is smart and that they’ve heard a talk about your subject before. Even if it’s a 101 talk where your goal is educating about the basics, what can you say that will be unique to your presentation? What could they not find out by Googling your topic? Make sure to present something new and unexpected.

A good presentation sells better than a sales pitch

Unfortunately, the quickest way to empty a room—particularly in the OpenStack community—is to use talk time to push a service or product. This might conflict with company expectations––someone probably wants to see an ROI on your talk and maybe even sent over talking points. Instead, create interest in your company or product by being an outstanding representative and demonstrating smarts, innovation and the ability to overcome the inevitable challenges. The “sales pitch” is not what you say about a product, but it is you and how you present.

Shorten your career path story

It’s very common for talks to begin with “first, a little about me,” which often sounds like reading a resume. While this can create an audience connection, it eats up valuable presentation time and takes the focus off the topic. Instead, share only the relevant pieces of your career to set up your expertise and the audience’s expectations.

Take a look at the difference between these examples:

Frequently done: “My name is Anne and I’m currently a marketing coordinator at the OpenStack Foundation. I started off in renewable energy, focusing on national energy policy and community engagement; then I became a content writer for a major footwear brand; then worked at an international e-commerce startup; and now I’m here! In my free time I race bicycles and like riding motorcycles.”

The audience has learned a lot about me (probably too much!), but it doesn’t give them a single area of expertise to focus on. It distracts the audience from the topic of my talk.

Alternative: “My name is Anne and as the marketing coordinator at the OpenStack Foundation, I work on our social media team.”

I’ve established my professional connection to the topic, explained why they should listen and foreshadowed that we’ll be talking about social media marketing.

Conversation, not recitation

Memorizing a script and having the script in front of you (like on a phone) is a common device to try to soothe presentation nerves. Ironically this makes your presentation more difficult and less enjoyable for the audience. When you trip up on a word (and we all do!), it can cause you to lose the paragraph that precedes it. Reading off a device will make your presentation sound artificial.

Instead, rehearse your presentation but use slide graphics or brief bullets to keep you on message. Pretend you’re having a conversation with the audience; just a cup of coffee over a very large table.

P.S. Make sure you budget time for conversation with your audience, and bring a few thought-provoking questions of your own to get the discussion started.

Humor doesn’t always work in international audiences

OpenStack has a wonderfully international community, which means that many people in your audience may not be native or fluent in the language you are presenting in. Idioms, turns of phrase or plays on words can be particularly difficult to understand. Instead of leaning on humor, tell a story about how something came to be, or a critical error that we can all see the humor in.

Looking forward to the incredible talks slated for the upcoming Summit; good luck, presenters!

Cover Photo // CC BY NC

The post How to make your OpenStack Summit talk a big success appeared first on Superuser.

by Anne Bertucio at April 06, 2018 11:02 AM

April 05, 2018

Chris Dent

Placement Container Playground 6

Updated 2018-04-10: The container developed from this work is now automatically built on docker hub.

This is the sixth and probably final post in a series of posts about experimenting with the OpenStack Placement service in containers.

The first got the basics working but was not satisfactory because it only used an already established database and created a rather large container with too many dependencies.

The second mostly discusses issues with placement accidentally importing more code from nova than it would like to, simply because of the structure of the nova code. This has implications for extraction. In fact, all of the container work has done an excellent job of finding speedbumps on that path.

The third continued the work to shrink things and tried it out with some earlier scale testing to confirm all was well.

The fourth gets things working with a standalone database, for experimentation without the rest of OpenStack, and also gets rudimentary Kubernetes support established.

The fifth enables an arbitrary remote database, which can be synchronized to generate tables and perform migrations, and extends the Kubernetes support to use a horizontal pod autoscaler.

Finally, today I merged all those changes (which were happening on a branch) back to the master of the placedock repo and added an extensive README that tries to explain it all in a useful fashion, including how to use the same container in multiple scenarios, controlled by the judicious use of environment variables.

  • Running the container with docker to sit beside a running OpenStack installation, such as devstack, talking to an established remote database.
  • Using a single sqlite3 database within the container to do placement-only testing (to do things like model nested resource providers and complex queries thereof).
  • Running in whatever fashion, but doing the work at startup to make sure that the target database has the proper tables.
  • Running in kubernetes with a horizontal pod autoscaler.

Doing this work revealed a great deal about how to make placement scale well, and also informed plenty of future work on extracting placement to an independent project. I think, however, what I would like to close on is how the container evolved from something that had a fair amount of state and dependency on the outside world to something that though flexible in what it can be used for is a static thing that just serves web requests, has no state, starts very quickly, and cannot be changed. If you want to change things you have to kill it, give it new variables, and run again.

This is incredibly good and powerful. Let's have more of that.

by Chris Dent at April 05, 2018 06:15 PM

OpenStack Superuser

Why open is the future of public and private cloud

Open like the sky. Open like possibility. Open to a future where anything might happen. Data centers are increasingly opting for systems built on open infrastructure for public and private clouds.

“Open allows for collaboration between public and private cloud, creating needed flexibility,” Johan Christenson CEO of City Network Hosting said in a recent interview at Markets Media. “It allows for the selection of several vendors working well together, not only positioning for your business at the right price. Open vs. closed should be a simple choice for any enterprise today, as open enables the solution of tomorrow and accelerates innovation.”

The Sweden-based company should know. A two-time Superuser award nominee, City Network operates multiple data centers in Europe, U.S., Asia and United Arab Emirates as well as providing a clear strategy and implementation for data protection and regulatory aspects across regions.

Their public OpenStack-based cloud operates eight regions across three continents; all of their data centers are interconnected through private networks. In addition to the public cloud, they are also responsible for a Pan-European cloud for a finance vertical that’s tasked with solving a vast number of regulatory challenges. Over 2,000 users of City’s infrastructure-as-a-service (IaaS) solutions run over 10,000 cores in production.

You can hear more about what’s next in public cloud at the upcoming Vancouver Summit, where City Network’s Tobias Rydberg, along with Huawei’s Zhipeng Huang, will appear in a session titled “The largest global public cloud footprint – Passport Program Phase II.”
The Passport Program is an initiative from the Public Cloud Working Group to provide a unified way for users to access free trial accounts from OpenStack public cloud providers around the world, which allows them to experience the freedom, performance and interoperability of open source infrastructure. 

Full story on how data centers are increasingly open to innovation over on Markets Media.

The post Why open is the future of public and private cloud appeared first on Superuser.

by Superuser at April 05, 2018 03:44 PM

April 04, 2018

OpenStack Superuser

Inside edge computing: Must-see sessions at the Vancouver Summit

Join the people building and operating open infrastructure at the OpenStack Summit Vancouver in May.  The Summit schedule features over 300 sessions organized by use cases including: artificial intelligence and machine learning, high performance computing, edge computing, network functions virtualization, container infrastructure and public, private and multi-cloud strategies.

Here we’re highlighting some of the sessions you’ll want to add to your schedule about edge computing.

Check out all the sessions, workshops and lightning talks focusing on edge here. And, if you’re interested in more about edge computing, read the new whitepaper created by the Edge Computing Group available at openstack.org/edge

Containers, bare-metal, VMs: Edge computing choices for workload flexibility

This panel features an all-star roster of experts including Verizon’s Beth Cohen, Ericsson’s Alison Randal, OPNFV’s Chris Price and the OSF’s Ildiko Vancsa. They’ll explore explore and discuss the strengths and weaknesses of each of these technologies, coupled with a discussion of technology maturity and market forces involved in delivering the right platform for creating practical solutions to fit the application. Details here.

China Unicom case study in edge/cloud architecture

Jianfeng (JF) Ding, Intel, and China Unicom’s Dan Chen will present OpenStack based edge-cloud architecture, its hardware configuration, applications and the MEC design that they deployed for one of China’s leading telcos. The pair will also discuss the gaps of OpenStack for edge use cases and the needs for hardware accelerator devices (e.g. FPGA, GPU) as well as NFV requirements. Details here.

Security Considerations for cloud edge computing

Beth Cohen moderates a panel that will discuss how security must be incorporated into cloud edge architectures and some of the current best practices for doing it. She’ll be joined by Rob Hirschfeld CEO of RackN, Glen McGowan of Dell EMC and Shuquan Huang from 99cloud. Details here.

Future edge cloud for China Mobile

In this beginner-level talk, you’ll hear from China Mobile project manager Qiao Fu about the future nationwide network for China Mobile as well as the architecture, design and requirements for this edge cloud.  She’ll also discuss issues that the telco is waiting for the open source communities to solve, including acceleration and container orchestration.
Details here.

Microclouds for fragmented markets: Getting OpenStack everywhere!

Benjamin Diaz and José Miguel Guzmán, both from Whitestack, offer this advanced-level talk about OpenStack Microclouds. The project aims to reduce entry costs and simplify the process of transitioning to cloud technologies for new clients by sharing geo-distributed controller nodes among multiple deployments. Using technologies like Ansible and Docker, they can present each client with an isolated and independent OpenStack deployment, making management of multiple deployments almost a trivial job. Details here.

OpenStack and Linux Foundation joint blueprint on edge computing

In this intermediate-level session, Gnanavelkandan Kathirvel, AT&T, and Melissa Evers-Hood, Intel, will share open-source based edge reference architecture that converges OpenStack and Linux Foundation communities to achieve the goals together without duplication of effort as the time, resources and funding are limited in the industry. Details here.

See you at the OSF Summit in Vancouver, May 21-24, 2018!
Register here.

The post Inside edge computing: Must-see sessions at the Vancouver Summit appeared first on Superuser.

by Superuser at April 04, 2018 04:01 PM

April 03, 2018

Chris Dent

TC Report 18-14

If the logs of #openstack-tc are any indicator of reality (they are not), then the only things that happened in the past week are that the next OpenStack release got a name, and the TC talked about how to evaluate projects applying to be official.

Stein

Yes, the people have spoken and their voices were almost heard. The first choice for the name of the "S" release of OpenStack, "Solar", foundered at the desk of legal and "Stein" won the day and there was much emojifying: 🍺.

From "Rocky" comes...another rock. Not ein Maß. Presumably such details will not limit the rejoicing.

Associated chatter.

Official Projects

The application of Adjutant continues to drive some discussion, both on the review and in IRC. On Wednesday I dropped a wall of text on the review, expressing my doubt and confusion over what rules we are supposed to be using when evaluating applicants.

Then at Thursday's office hour the discussion picked up with a larger group. There were at least three different threads of conversation happening at once:

  • comments related to the general topics I raised
  • evaluating Adjutant itself in terms of its impact on OpenStack
  • trying to get (and encourage the getting of) input from real operators about their thoughts on the usefulness of Adjutant (or something like it)

The last was an effort to stop speculating, which is something we do too much.

The second was an effort to not be moving the goalposts in the middle of an application, despite the confusion.

The first had a lot of ideas, but none were resolved (and there's a pattern there) so there's a plan to have a session about it at the Forum. If you look at the planning etherpad you'll see that there are two different topics related to project applications: one is for Adjutant specifically, in case things aren't resolved by then (we hope they will be); the other is a general session on really trying to dig deep into the questions and figure out what we're trying to do and be when we say "official". These are separate sessions very much on purpose.

The questions reach into the core of what OpenStack is, so it ought to be an interesting session.

by Chris Dent at April 03, 2018 04:30 PM

OpenStack Superuser

Kickstart your knowledge with the OpenStack Upstream Institute

With over 2,000 developers from over 300 different organizations worldwide, OpenStack is one of the largest collaborative software-development projects.

With that in mind, the OpenStack Foundation offers a training program to share knowledge about the different ways of contributing to OpenStack like providing new features, writing documentation, participating in working groups. Aimed at beginners, trainers are all-star volunteers from the community. It’s broken down into modules– so if you’re a developer, project manager or interested in Working Groups, you can follow what most interests you.

The next one-and-a-half day training of the Upstream Institute will take place Saturday, May 19 and Sunday, May 20 2018 at the Vancouver Summit. The class is offered gratis, but you must RSVP here.   Daylong versions will also be offered at upcoming OpenStack Days in June and July, more on those here.

The educational program is built on the principle of open collaboration and will teach the students how to find information and navigate the intricacies of the project’s technical tools and social interactions in order to get the contributions accepted. The live one and a half day class is focusing on hands-on practice like the students can use a development environment to work on real-life bug fixes or new features and learn how to test, prepare and upload them for review.

The Upstream Institute attendees are also given the opportunity to join a mentoring program to get further help and guidance on their journey to become an active and successful member of the OpenStack community. The program is run by Foundation staffers Kendall Nelson, upstream developer advocate and Ildiko Vancsa, ecosystem technical lead.

Want to know more about the course? The complete content of Upstream Institute Training can be found here:

Get involved!

If you’re interested in getting involved with Upstream Institute outside of this training, you can also participate in weekly meetings.
They happen in #openstack-meeting-3 (IRC webclient)

  •   Every two weeks (on odd weeks) on Monday at 2000 UTC
  •   Every two weeks (on even weeks) on Tuesday at 0900 UTC

You’re also welcome to hang out in the #openstack-upstream-institute channel in between meetings.

 

The post Kickstart your knowledge with the OpenStack Upstream Institute appeared first on Superuser.

by Superuser at April 03, 2018 11:30 AM

April 02, 2018

OpenStack Superuser

Get started with big data analytics on OpenStack by designing your ETL workflow process

OpenStack controls large pools of compute, storage and networking resources; This article primarily focuses on how OpenStack plays a key role in addressing a big data use case.

Big data on OpenStack!

These days, data is generated everywhere and is growing exponentially. Data from web servers, application servers, database servers in the form of user information, log files and system state information. A huge volume of data is also generated from internet of things devices like sensors, vehicles, industrial devices. Data generated from the scientific simulation models is also an example of a source of big data. Storing and performing analytics with data like this with traditional software tools can be difficult; Hadoop can address the issue.

Let me share my use case with you. I have a bulk amount of data stored in a relational database management system environment. The RDBMS doesn’t perform well when the data set grows larger — a problem that will only continue to grow along with it. At this stage, I’d like to avoid adopting NoSQL culture. I need to store and process a bulk amount of data in a cost-effective way. Should I rely on high-end server in a non virtualized environment? My requirement is to scale the cluster at any time and I also need a better dashboard to manage all of its components.

I planned to set up a Hadoop cluster on top of OpenStack and create my ETL job environment. Hadoop is an industry standard framework for storing and analyzing a large data sets with its fault tolerant Hadoop distributed file system and MapReduce implematation. Scalability, however, is a very common problem in a typical Hadoop cluster. Openstack has introduced a project called Sahara – data processing as a Service. Openstack Sahara aims to provision and manage data processing frameworks such as hadoop mapreduce, spark and storm in a cluster topology. This project is similar to the data analytics platform provided by Amazon Elastic MapReduce (EMR) service. Sahara deploys the cluster in a few minutes. Besides, Openstack Sahara can scale the cluster by adding or removing worker nodes based on demand.

Benefits of managing Hadoop cluster with Openstack Sahara

  • The cluster can be provisioned faster and easy to configure.
  • Like other OpenStack services, Sahara service can be managed through a powerful REST API, CLI and Horizon dashboard.
  • Plugins available to support mutliple Hadoop vendor such as Vannila (Apache Hadoop), HDP(ambari), CDH(Cloudera), MapR, Spark, Storm.
  • Cluster size can be scaled up and down based on demand.
  • It can be integrated with OpenStack Swift to store the data processed by Hadoop and Spark.
  • Cluster monitoring can be made simple.
  • Apart from cluster provisioning, Sahara can be used as analytics as-a-service for ad-hoc or bursty analytic workloads.

Architecture

Openstack Sahara is designed to leverage the core services and other fully managed services of OpenStack. It makes Sahara more reliable and ability to manage the Hadoop cluster efficiently and you have the option to use services including Trove and Swift. Let’s take a look at Sahara architecture.

  • Sahara service has an API server which responds to HTTP request from the end user and interacts with other OpenStack services to perform its function.
  • Keystone (identity as-a-service) – authenticates users and provides security tokens that are used to work with OpenStack, limiting a user’s abilities in Sahara to their OpenStack privileges.
  • Heat (orchestration as-a-service) – used to provision and orchestrate the deployment of data processing clusters.
  • Glance (virtual machine image as-a-service) – stores VM images with operating system and pre-installed Hadoop/Spark software packages to create a data processing cluster.
  • Nova (compute) – provisions a virual machine for data processing clusters.
  • Ironic (bare metal as-a-service) provisions a bare metal node for data processing clusters.
  • Neutron (networking) – facilitates networking services from basic to advanced topology to access the data processing clusters.
  • Cinder (block storage) – provides a persistent storage media for cluster nodes.
  • Swift (object storage) – provides a reliable storage to keep job binaries and the data processed by hadoop/spark.
  • Designate (DNS as-a-service) – provides a hosted zone to keep DNS records of the cluster instances. Hadoop services communicates with the cluster instances by their hostnames.
  • Ceilometer (telemetry) – collects and stores the metrics about the cluster for metering and monitoring purposes.
  • Manila (file share) – can be used to store job binaries and data created by the job.
  • Barbican (key management service) – stores sensitive data such as password and private keys securely.
  • Trove (database as-a-service) – provides data base instance for hive metastore and to store the states of the Hadoop services and other management services.

How to set up a Sahara cluster

Please follow the steps the installation guide for deploying Sahara in your environment. There are several ways where to deploy  it,  Kolla is also a good choice if you want to experiment with it.

You can also manage Sahara projects through the Horizon dashboard.

ETL (extract, transform and load) or ELT (extract, load and transform) with a Sahara cluster

There are numerous ETL tools available in the market;  traditional data warehouse have their own benefits and limitations, for example it might be in some other location other than your data source. I’m targeting Hadoop as an ideal platform to run ETL jobs. Data in your datastore has a variety of data including structured, semi-structured and unstructured. The Hadoop ecosystem has tools to ingest data from different data sources including databases, files and other data streams and store it in a centralized Hadoop Distributed File System (HDFS). As your data grows rapidly, the Hadoop cluster can be scaled and leverage OpenStack Sahara.

Apache Hive is the data warehouse project built on top of the Hadoop ecosystem and a proven tool for doing ETL analysis. Once the data is extracted from the data sources with tools (such as Sqoop, Flume, Kafka, etc.), it should be cleansed and transformed by Hive or pig scripts with the MapReduce technique.

Another advantage of Hive is that it is an interactive query engine and can be accessed via Hive Query Language. It resembles SQL. So, your database person can execute a job in the Hadoop ecosystem without prior knowledge of Java and MapReduce concepts. The Hive query execution engine parses the hive query and converts it in to a sequence of MapReduce/Spark job to a cluster. Hive can be accessed by JDBC/ODBC driver and thrift clients.

Oozie is a workflow engine available in Hadoop ecosystem, A workflow is simply a set of tasks that must to be executed as a sequence in a distributed environment. Oozie helps create a simple workflow to cascading multiple workflows and create coordinated jobs. It’s also ideal for creating workflows for complex ETL jobs, although it does not have modules to support all actions related to Hadoop. (A large organization might want their own workflow engine to execute their tasks, for example.) We can use any workflow engine to carryout our ETL job, for example Openstack Mistral – workflow as-a-service. Apache oozie resembles Openstack Mistral in some aspects, acting as a job scheduler that can be triggered at regular intervals.

Let’s look at a typical ETL job process with Hadoop where an application stores its data in a MySQL server. The data stored here needs to be analyzed at a minimum of cost and time.

Extract

The very first step is to extract data from MySQL and store it in HDFS. Apache Sqoop can be used to export/import from a structured data source such as RDBMS data store. If the data to be extracted is semi-structured or unstructured, you can use Apache Flume to ingest the data from data soruce such as web server log, twitter data stream or sensor data.

sqoop import \
--connect jdbc:mysql://<database host>/<database name> \
--driver com.mysql.jdbc.Driver–table <table name> \
--username <database user> \
--password <database password> \
--num-mappers 3 \
--target-dir hdfs://hadoop_sahara_local/user/rkkrishnaa/database

Transform

The data extracted from the above phase is not in a proper format, it’s just raw data. It should be cleansed by applying a proper filter and data aggregation from multiple tables. This is our staging and intermediate area for storing data in HDFS.

At this point we need to design hive schema for each table and create a database to transform the data stored in staging area. Typically, your data is in .csv format and each record is delimited by comma.

We don’t need to check HDFS data to know how it’s stored; with a few data type exceptions, it should be compatible with Hive.

Once the database is modeled, we can load the extracted data for cleaning. The data in the table is still de-normalized. Aggregate the required columns from the different table.

Open hive or beeline shell or if you have HUE (Hadoop User Experience) installed, please open the Hive editor and run the commands given below.

CREATE DATABASE <database name>;
USE <database name>;
CREATE TABLE <table name>
(variable1 DATATYPE ,
variable2 DATATYPE ,
variable3 DATATYPE ,
. ,
. ,
. ,
variablen DATATYPE
)
PARTITIONED BY (Year INT, Month INT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE;
LOAD DATA INPATH 'hdfs://hadoop_sahara_local/user/rkkrishnaa/database/files.csv'

Similarly, we can aggregate the data from the multiple tables with ‘OVERWRITE INTO TABLE’ statement.

Hive supports partitioning table to improve the query performance by distributing the execution load horizontally. We prefer to partition the column which stores year and month. Sometimes, partitioned table creates more tasks in a MapReduce job.

Load

We checked about the transform phase, now it’s time to load the transformed data in to a data warehouse dirctory in HDFS, which is the final state of the data. Here we can apply our SQL queries to get an appropriate results.

All DML commands can be used to analyse the warehouse data based on the use case.

Results can be download as a .csv, table or chart for analysis. It can be integrated with other popular BI tools such as Talend OpenStudio, Tabelau, etc.

Automate

Now we’re going to automate the ETL job using the Oozie workflow engine. If you have experience with Mistral, you can use that. Most Hadoop users are acquainted with Apache Oozie, so we’re going to use that as an example here.

Step 1: Create a job.properties file

nameNode=hdfs://hadoop_sahara_local:8020
jobTracker=<rm-host>:8050
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}
oozie.libpath=/user/root

Step 2: Create a workflow.xml file to define tasks for extracting data from MySQL data store to HDFS

<workflow-app xmlns=”uri:oozie:workflow:0.2″ name=”my-etl-job”>
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
</global>
 
<start to=”extract”/>
<action name=”extract”>
<sqoop xmlns=”uri:oozie:sqoop-action:0.2″>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<command>import –connect jdbc:mysql://<database host>:3306/<database name> –username <database user> –password <database password> –table <table name> –driver com.mysql.jdbc.Driver –target-dir hdfs://hadoop_sahara_local/user/rkkrishnaa/database${date} –m 3 </command>
</sqoop>
 
<ok to=”transform”/>
<error to=”fail”/>
 
<action name=”transform”>
<hive xmlns=”uri:oozie:hive-action:0.2″>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.hive.defaults</name>
<value>/user/rkkrishnaa/oozie/hive-site.xml</value>
</property>
</configuration>
<script>transform.q</script>
</hive>
 
<ok to=”load”/>
<error to=”fail”/>
 
<action name=”load”>
<hive xmlns=”uri:oozie:hive-action:0.2″>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>oozie.hive.defaults</name>
<value>/user/rkkrishnaa/oozie/hive-site.xml</value>
</property>
</configuration>
<script>load.q</script>
</hive>
 
<ok to=”end”/>
<error to=”fail”/>
</action>
<kill name=”fail”>
<message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
 
<end name=”end”/>
</workflow-app>

Step 3: Create a file transform.q

LOAD DATA INPATH 'hdfs://hadoop_sahara_local/user/rkkrishnaa/database${date}/files.csv'

Step 4: Create a file load.q

SELECT name, age, department from sales_department where experience > 5;

The above workflow job will do ETL operation. We can customize the workflow job based on our use case, we have lot of mechanisms to handle task failure in our workflow job. Email action helps to track the status of the workflow task. This ETL job can be executed on daily basis or weekly basis at any time. Usually organizations put these kind of long running tasks during weekends or at night time.

For more about the workflow: https://oozie.apache.org/docs/3.3.1/index.html

Conclusion

OpenStack has integrated a very large Hadoop ecosystem and many cloud providers offer Hadoop service in just a few clicks on their cloud management portals. Sahara supports most of the Hadoop vendor plugins Execute your ETL workflow with Openstack Sahara.

Enjoy learning!

References:
[1] https://www.openstack.org/software/sample-configs#big-data
[2] https://docs.openstack.org/sahara/latest/install/index.html
[3] https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML
[4] https://oozie.apache.org/docs/3.3.1/index.html

 

This article first appeared on the Hortonworks Community Connection. Superuser is always interested in community content, get in touch at editorATopenstack.org

Cover Photo // CC BY NC

The post Get started with big data analytics on OpenStack by designing your ETL workflow process appeared first on Superuser.

by Radhakrishnan Ramakrishnan at April 02, 2018 04:10 PM

April 01, 2018

NFVPE @ Red Hat

Install Cumulus Linux via ONIE

This quickstart tutorial will guide you through the steps to install Cumulus Linux via ONIE. ONIE (Open Network Install Environment) is a small operating system, pre-installed...

by Ricky Noriega at April 01, 2018 10:00 PM

March 30, 2018

OpenStack Superuser

An introduction to OpenStack fast forward upgrades

OpenStack momentum continues to grow as an important component of hybrid cloud, particularly among enterprise and telecomunications.

One area that Red Hat customers ask about is the rapid release cycle of OpenStack. And while this speed can be advantageous in getting key features to market faster, it can also be quite challenging to follow for customers looking for stability.

With the release of Red Hat OpenStack Platform 10 in December 2016, we introduced a solution to this challenge – we call it the Long Life release. This type of release includes support for a single OpenStack release for a minimum of three years plus an option to extend another two full years. We offer this via an ELS (Extended Life Support) allowing our customers to remain on a supported, production-grade OpenStack code base for far longer than the usual six-month upstream release cycle. Then, when it’s time to upgrade, they can upgrade in-place and without additional hardware to the next Long Life release. We aim to designate a Long Life release every third release, starting with Red Hat OpenStack Platform 10 (Newton).

Now, with the upcoming release of Red Hat OpenStack Platform 13 (Queens), we’re introducing our second Long Life release. This means we can, finally and with great excitement, introduce the world to our latest new feature: the fast forward upgrade.

Fast forward upgrades take customers easily between Long Life releases. It’s a first for our OpenStack distribution, using Red Hat OpenStack Platform director (known upstream as TripleO) and aims to change the game for OpenStack customers. With this feature, you can choose to stay on the “fast train” and get all the features from upstream every six months, or remain on a designated release for longer, greatly easing the upgrade treadmill some customers are feeling.

It’s a pretty clever procedure since it factorizes the process of three consecutive upgrades and therefore greatly reduces the number of steps needed to perform it. In particular, it reduces the number of reboots needed, which in the case of very large deployments makes a huge difference. This capability, combined with the extended support for security vulnerabilities and key backports from future releases, has made Red Hat OpenSack Platform 10 very popular with our customers.

openstack_graphic_large

Red Hat OpenStack Life Cycle dates

Under the covers of a fast forward upgrade

The fast forward upgrade starts like any other upgrade, with a Red Hat OpenStack Platform 10 minor update. A minor update may contain everything from security patches to functional enhancements to even backports from newer releases. There is nothing new about the update procedure for Red Hat OpenStack Platform 10, what changes is the packages that will be included such as kernel changes and other OpenStack specific changes. We’re placing all changes requiring a reboot in this minor update. The update procedure allows for a sequential update of the undercloud, control plane nodes and the compute nodes. It may also include an instance evacuation procedure so that there is no impact to running workloads even if they reside on a node scheduled for reboot after the update. The resulting Red Hat OpenStack Platform 10 cloud will have the necessary operating system components to operate in Red Hat OpenStack Platform 13 without further node reboots.

The next step is the sequential upgrade of the undercloud from Red Hat OpenStack Platform 10, to 11, to 12, to 13. There is no stopping during these steps and in case of a needed rollback of this portion you must return to Red Hat OpenStack Platform 10.

During the fast forward upgrade there are opportunities to perform backups. These should be performed since there’s no automated rewind included but rather a restore from these backups.

A lot of things have changed between Red Hat OpenStack Platform 10 and 13. The most notable is the introduction of OpenStack services in containers. But don’t worry! The fast forward upgrade procedure takes the cloud from a non-containerized deployment to a resulting cloud with OpenStack services running in containers, while abstracting and reducing the complexity of upgrading through the middle releases of Red Hat OpenStack Platform 11 and 12.

In the final steps of the fast forward upgrade procedure, the overcloud will move from Red Hat OpenStack Platform 10 to 13 using a procedure that syncs databases, generates templates for 13, and installs 13’s services in containers. While some of the content for these steps may be part of a release which is no longer supported, Red Hat will provide full support for the required code to perform the upgrade.

What’s next …

In order for this procedure to be supported, it needs to be validated with the released code, and carefully tested in many situations. For this reason, it is scheduled to be ready for testing from Red Hat OpenStack Platform 13 general availability (GA); however, we will warn users launching the procedure that it should not be used in production environments. We encourage you to try the procedure on test environments during this period, and report any issues you find via the normal support procedure.  This will greatly help us ensure that we are covering all cases. During this time support cases relating to fast forward upgrades will not be eligible for high priority response times.

Once we have thoroughly field tested the procedure, fixed bugs, and are confident that it is ready, we will remove this warning and make an announcement on this same blog. After this happens, it will be OK to proceed with fast forward upgrades in production environments. You can follow this progress of validation and testing by following this blog and staying in touch with your local Red Hat account and support teams.

Stay tuned for future fast forward upgrade blog posts where we dig deeper into the details of this procedure and share experiences and use cases that we’ve tested and validated.

What’s next

To hear more, check out the sessions for OpenStack and Ceph at the upcoming Red Hat Summit in San Francisco, May 8-10.

Red Hat will also be out in force with a host of workshops, lightning talks and sessions at the OpenStack Summit Vancouver, May 21-24.

Maria Bracho, along with colleague Lee Yarwood, will also be talking about OpenStack Upgrades Strategy: The Fast Forward Upgrade.

See you there!

This post first appeared on the Red Hat Stack blog. Superuser is always interested in community content, get in touch at editorATopenstack.org

 

The post An introduction to OpenStack fast forward upgrades appeared first on Superuser.

by Maria Bracho and August Simonelli at March 30, 2018 03:09 PM

March 29, 2018

OpenStack Superuser

Introducing OpenStack Event Listener

I wanted to write a little about a project that I enjoyed working on called the OpenStack Event Listener, or OSEL for short. This project bridges the OpenStack control plane and an external scanning facility provided by Qualys to initiate batched on-demand scanning of OpenStack instances when security group changes happen.

There were a number of interesting challenges. I was never able to really concentrate on it – this project took about 20 percent of my time for a period of three months. I started based on an initial proof-of-concept written by Charles Bitter, and also solicited contributions from Olivier Gagnon and Joseph Sleiman. I offer this partially as catharsis, to allow my brain to mark this part of my mental inventory as ripe for reclamation. I’m also writing on the off chance that someone might find this useful.

The setting

Let me paint a picture of the environment in which this development occurred.

The Comcast OpenStack environment was transitioning from the OpenStack Icehouse release (very old) to the Newton release (much more current). This development occurred within the context of the Icehouse environment.

Comcast’s security team uses S3 RiskFabric to manage auditing and tracking security vulnerabilities across the board. They also engage the services of Qualys to perform network scanning (in a manner very similar to Nessus) once a day against all the CIDR blocks that comprise Comcast’s Internet-routable IP addresses. Qualys scanning could also be triggered on-demand.

Technical requirements

First, let me describe the technical requirements for OSEL:

  • OSEL would connect to the OpenStack RabbitMQ message bus and register as a listener for “notification” events. This would allow OSEL to inspect all events, including security group changes.
  • When a security group change occurred, OSEL would ensure that it had the details of the change (ports permitted or blocked) as well as a list of all affected IP addresses.
  • OSEL would initiate a Qualys scan using the Qualys API. This would return a scan ID.
  • OSEL would log the change as well as the Qualys scan ID to the Security instance of Splunk to create an audit trail.
  • Qualys scan results would be imported into S3 RiskFabric for security audit management.

Implementation approach

My group does most of its development in Go and it was a good fit for this project by virtue of it’s ability to handle the stream of messages from RabbitMQ.

This is what the data I was getting back from the AMQP message looked like. All identifiers have been scrambled.


{
    "_context_roles":[
        "Member"
    ],
    "_context_request_id":"req-f96ea9a5-435e-4177-8e51-bfe60d0fae2a",
    "event_type":"security_group_rule.create.end",
    "timestamp":"2016-10-03 18:10:59.112712",
    "_context_tenant_id":"ada3b9b06482909f9361e803b54f5f32",
    "_unique_id":"eafc9362327442b49d8c03b0e88d0216",
    "_context_tenant_name":"EXAMPLEPROJECT",
    "_context_user":"bca89c1b248e4a78282899ece9e744cc54",
    "_context_user_id":"bca89c1b248e4a78282899ece9e744cc54",
    "payload":{
        "security_group_rule_id":"bf8318fc-f9cb-446b-ffae-a8de016c562"
    },
    "_context_project_name":"EXAMPLEPROJECT",
    "_context_read_deleted":"no",
    "_context_tenant":"ada3b9b06482909f9361e803b54f5f32",
    "priority":"INFO",
    "_context_is_admin":false,
    "_context_project_id":"ada3b9b06482909f9361e803b54f5f32",
    "_context_timestamp":"2016-10-03 18:10:59.079179",
    "_context_user_name":"admin",
    "publisher_id":"network.osctrl1",
    "message_id":"e75fb2ee-85bf-44ba-a083-2445eca2ae10"
}

You can see that this is a security group creation (“event_type”:”security_group_rule.create.end”), creating a security group rule “bf8318fc-f9cb-446b-ffae-a8de016c562” in project “EXAMPLEPROJECT”. That does not tell us much, sadly. In order to resolve what IP addresses were affected when this security group rule was created, OSEL queries neutron for all ports in that tenant, determines what the IP address and associated security groups are for each, and returns the list of IP addresses associated with the security group for which the rule was created.

Qualys is a service where you pay a certain amount of money and get a given number of API requests per time period. I did not find a maximum size for a single API request. OSEL implements a batching system: all requests that come in during a given time period get queued until a configurable interval is reached, then they are discharged in a single API request. Batching ensures that you can set a pace that does not exceed the number of API requests for which you have paid. This negates somewhat the real-time-scanning aspect of OSEL but it is necessitated by fiscal responsibility.

Testing pattern

I leaned heavily on dependency injection to make this code as testable as possible. For example, I needed an object that would contain the persistent `syslog.Writer`. I created a `SyslogActioner` interface to represent all interactions with syslog. When the code is operating normally, interactions with syslog occur through methods of the `SyslogActions` struct, but in unit testing mode the `SyslogTestActions` struct is used instead. The `SyslogTestActions` is limited to saving copies of all messages that would have been sent so they can be compared against the intended messages. This facilitates good testing.

Fate of the project

The OSEL project was implemented and installed into production. There were two problems with it.

The first problem to become visible was the lack of an exponential backoff for the AMQP connection to the OpenStack control plane’s RabbitMQ. When RabbitMQ had issues – which was surprisingly often – OSEL would hammer away, trying to reconnect. This would not be too much of an issue; despite what was effectively an infinite loop, CPU usage was not extreme. The real problem was that connection failures were logged – and logs could become several gigabytes in a matter of hours. This was mitigated by the OpenStack operations team rotating the logs hourly, and alerting if an hour’s worth of logs exceeded a set size.

The second – and fatal – issue is that S3 RiskFabric was not configured to ingest from Qualys scans more than once a day. Since Qualys was already scanning the CIDR block that corresponded to our OpenStack instances once a day, we were essentially just adding noise to the system. The frequency of the S3-Qualys imports could not be easily altered, and as a result the project was shelved.

Remaining work

If OSEL were ever to be un-shelved, here are a few things that I wish I had time to implement:

  • Neutron Port Events: The initial release of OSEL processed only security group rule additions, modifications, or deletions. That covered the base case for when a security group was already associated with a set of OpenStack Networking (neutron) ports. A scan should be similarly launched when a new port is created and associated to a security group. This is what happens when a new host is created.
  • Integrate with the really awesome Firewall as a Service project.
  • Modern OpenStack: In order to make this work with a more modern OpenStack, it would be best to integrate with events generated through Aodh. Aodh is built for this kind of reporting.
  • Implement exponential backoff for AMQP connections as mentioned earlier.

Get involved

If you are interested in contributing to the project, clone the project source from the OpenStack git repository and submit changes using the standard Gerrit-based OpenStack review process!

 

The post Introducing OpenStack Event Listener appeared first on Superuser.

by Nate Johnston, Charles Bitter, Olivier Gagnon and Joseph Sleiman at March 29, 2018 04:39 PM

Red Hat Stack

Heading to Red Hat Summit? Here’s how you can learn more about OpenStack.

Red Hat Summit is just around the corner, and we’re excited to share all the ways in which you can connect with OpenStack® and learn more about this powerful cloud infrastructure technology. If you’re lucky enough to be headed to the event in San Francisco, May 8-10, we’re looking forward to seeing you. If you can’t go, fear not, there will be ways to see some of what’s going on there remotely. And if you’re undecided, what are you waiting for? Register today

From the time Red Hat Summit begins you can find hands-on labs, general sessions, panel discussions, demos in our partner pavillion (Hybrid Cloud section), and more throughout the week. You’ll also hear from Red Hat OpenStack Platform customers on their successes during some of the keynote presentations. Need an open, massively scalable storage solution for your cloud infrastructure? We’ll also have sessions dedicated to our Red Hat Ceph Storage product.

Red Hat Summit has grown significantly over the years, and this year we’ll be holding activities in both the Moscone South and Moscone West. And with all of the OpenStack sessions and labs happening, it may seem daunting to make it to everything, especially if you need to transition from one building to the next. But worry not. Our good friends from Red Hat Virtualization will be sponsoring pedicabs to help transport you between the buildings.   

Here’s our list of sessions for OpenStack and Ceph at Red Hat Summit:

Tuesday

Session Speaker Time / Location
Lab – Deploy a containerized HCI IaaS with OpenStack and Ceph Rhys Oxenham, Greg Charot, Sebastien Han, John Fulton 10:00 am / Moscone South, room 156
Ironic, VM operability combined with bare-metal performances Cedric Morandin (Amadeus) 10:30 am / Moscone West, room 2006
Lab – Hands-on with OpenStack and OpenDaylight SDN Rhys Oxenham, Nir Yechiel, Andre Fredette, Tim Rozat 1:00 pm / Moscone South, room 158
Panel – OpenStack use cases: how business succeeds with OpenStack August Simonelli, Pete Pawelski 3:30 pm / Moscone West, room 2007
Lab – Understanding containerized Red Hat OpenStack Platform Ian Pilcher, Greg Charot 4:00 pm / Moscone South, room 153
Red Hat OpenStack Platform: the road ahead Nick Barcet, Mark McLoughlin 4:30 pm / Moscone West, room 2007


Wednesday

Session Speaker Time / Location
Lab – First time hands-on with Red Hat OpenStack Platform Rhys Oxenham, Jacob Liberman 10:00 am / Moscone South, room 158
Red Hat Ceph Storage roadmap: past, present, and future Neil Levine 10:30 am / Moscone West, room 2024
Optimize Ceph object storage for production in multisite clouds Michael Hackett, John Wilkins 11:45 am / Moscone South, room 208
Production-ready NFV at Telecom Italia (TIM) Fabrizio Pezzella, Matteo Bernacchi, Antonio Gianfreda (Telecom Italia) 11:45 am / Moscone West, room 2002
Workload portability using Red Hat CloudForms and Ansible Bill Helgeson, Jason Woods, Marco Berube 11:45 am / Moscone West, room 2009
Delivering Red Hat OpenShift at ease on Red Hat OpenStack Platform and Red Hat Virtualization Francesco Vollero, Natale Vinto 3:30 pm / Moscone South, room 206
The future of storage and how it is shaping our roadmap Sage Weil 3:30 pm / Moscone West, room 2020
Lab – Hands on with Red Hat OpenStack Platform Rhys Oxenham, Jacob Liberman 4:00 pm / Moscone South, room 153
OpenStack and OpenShift networking integration Russell Bryant, Antoni Segura Puimedon, and Jose Maria Ruesta (BBVA) 4:30 pm / Moscone West, room 2011

 

Thursday

Session Speaker Time / Location
Workshop – OpenStack roadmap in action Rhys Oxenham 10:45 am / Moscone South, room 214
Medical image processing with OpenShift and OpenStack Daniel McPherson, Ata Turk (Boston University), Rudolph Pienaar (Boston Children’s Hospital) 11:15 am / Moscone West, room 2006
Scalable application platform on Ceph, OpenStack, and Ansible Keith Hagberg (Fidelity), Senthivelrajan Lakshmanan (Fidelity), Michael Pagan, Sacha Dubois, Alexander Brovman (Solera Holdings) 1:00 pm / Moscone West, room 2007
Red Hat CloudForms: turbocharge your OpenStack Kevin Jones, Jason Ritenour 2:00 pm / Moscone West, room 201
What’s new in security for Red Hat OpenStack Platform? Nathan Kinder, Keith Basil 2:00 pm / Moscone West, room 2003
Ceph Object Storage for Apache Spark data platforms Kyle Bader, Mengmeng Liu 2:00 pm / Moscone South, room 207
OpenStack on FlexPod – like peanut butter and jelly Guil Barros, Amit Borulkar NetApp 3:00 pm / Moscone West, room 2009

 

Hope to see you there!

 

by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform at March 29, 2018 02:00 PM

March 28, 2018

OpenStack Superuser

How to set up notifications for OpenStack Monasca

Monasca is the monitoring and logging as-a-service solution in OpenStack. Monasca combines metric information, e.g. CPU, Disk space, with log messages. Monasca is a cloud native application based on microservices and it uses Kafka to stream real time data.

Monasca uses a push approach to prevent that data can be lost in case of network failure. Another key feature of Monasca is to store large quantities of data to allow deep analysis. Monasca is also multi-tenant, highly available, vertically and horizontally scalable.

Architecture

Monasca architecture is extensible in order to be easily integrated with other OpenStack services and legacy systems. This characteristic helps Monasca to notify other systems or to record issues in several tools.

Monasca already offers integration with email, PagerDuty, WebHook, Jira, Slack and HipChat. The notification engine is the Monasca component that consumes alarm state transition messages from the message queue and sends notifications for alarms, if required. As the architecture diagram shows, it is possible to extend the notification engine to send information to additional external systems.

The latest notification methods are Jira, Slack and HipChat. Here’s how to configure Monasca and these three systems in order to exchange information.

Notification plugins

In order to work with the notification engine, you must update monasca-notification and monasca-ui.

source /opt/monasca-notification/bin/activate
pip install --upgrade monasca-notification

systemctl restart monasca-notification
pip install --upgrade monasca-ui
systemctl restart httpd

After upgrading or updating the notification engine and the UI,  it’s possible to display the “Create Notification Method” UI that shows the default notification methods that can be used.

Jira

Some configuration updates are required to send notifications from Monasca to Jira.

Install Jira:

source /opt/monasca-notification/bin/activate

pip install jira

Update the /etc/monasca/notification.yaml config file:

notification_types:

plugins:

- monasca_notification.plugins.jira_notifier:JiraNotifier

 

jira:

user: user

password: password

# proxy: http://yourid:password@proxyserver:8080

Restart the notification engine:

systemctl restart monasca-notification

Now you can create a notification to be sent when an alarm is triggered, via either CLI or UI.

monasca notification-create jira-notification JIRA https://xxx.xxx.xxx.xxx/?project=MON

When an alarm is triggered (e.g. CPU higher than 10 percent) then Monasca will communicate with Jira to file a new issue. The issue is associated with an ID, so that if the alarm is updated (for example the alarm status changes in Monasca), that will be reflected in the Jira ticket.

Slack

A similar configuration is required for Slack.
Update the notification config file /etc/monasca/notification.yaml:

notification_types:
    plugins:
        - monasca_notification.plugins.slack_notifier:SlackNotifier

    slack:
        timeout: 20
        ca_certs: "/etc/ssl/certs/ca-bundle.crt"
        insecure: False
        # proxy: http://yourid:password@proxyserver:8080

Restart the notification engine:

systemctl restart monasca-notification

Create Monasca notification via either CLI or UI.

monasca notification-create Slack SLACK https://slack.com/api/chat.postMessage?token=&channel=#

Then trigger an alarm that invokes Slack.

 

HipChat

HipChat is similar to Jira and Slack.

Update the notification config file /etc/monasca/notification.yaml:

notification_types:

plugins:

- monasca_notification.plugins.hipchat_notifier:HipChatNotifier

 

hipchat:

timeout: 20

ca_certs: "/etc/ssl/certs/ca-bundle.crt"

insecure: False

proxy: http://yourid:password@proxyserver:8080

Create a notification via either CLI or UI.

monasca notification-create HipChatTest HIPCHAT https://api.hipchat.com/v2/room/<Room ID>/notification?auth_token=<access token>

Then let Monasca communicate with HipChat.

It’s possible to configure Monasca to notify external systems such as email, WebHook, Jira and Slack. Monasca architecture can be easily extended to communicate with other systems. In particular, the notification engine is the component responsible to communicate with external systems. Common use cases involve the communication between Monasca and team messaging tools that help engineers to understand cloud issues deeper and to react faster.

Get involved

If you want to know more about and contribute to Monasca check out the main page or get connected to the IRC channel or the meetings.

Author Cristiano Bellucci will be co-presenting a session at the upcoming OpenStack Summit Vancouver titled “An edge computing case study for Monasca: Smart City (AI-powered) Surveillance and Monitoring. ” Along with Giovanni Merlino and Antonio Puliafito, he’ll take you to the Italian city of Messina for a case study.

The post How to set up notifications for OpenStack Monasca appeared first on Superuser.

by Cristiano Bellucci at March 28, 2018 03:43 PM

Mirantis

Hope is not a strategy: Continuous delivery and real world business

Ensuring that development and deployment are handled properly can be a challenge at best, and a nightmare at worst. You can hope things will work out, but hope is not a strategy.

by Nick Chase at March 28, 2018 01:59 PM

March 27, 2018

OpenStack Superuser

Recommendations for high-performance computing on OpenStack

OpenStack is, without doubt, an exciting project and the leading open source infrastructure as-a-service platform. In the last couple of years, I’ve had the privilege to architect and deploy dozens of OpenStack clouds for multiple customers and use cases. Over the last year, I’ve been working on use cases with high-performance computing (HPC) on OpenStack.

In this post, I’ll offer some considerations about hosting high performance and high-throughput workloads.

First, let’s start with the three types of architectures that can be used when hosting HPC workloads on OpenStack:

  1. Virtualized HPC on OpenStack
    In this architecture, all components of the HPC cluster are virtualized in OpenStack
  2. Bare-metal HPC on OpenStack
    All components of the HPC cluster are deployed in bare metal servers using OpenStack Ironic
  3. Virtualized head node and bare-metal compute nodes
    The head node (scheduler, master and login node) are virtualized in OpenStack and the compute nodes are deployed in bare metal servers using OpenStack Ironic

Now that you have an overview of the three types of architecture that can deploy HPC software in OpenStack, I’m going to discuss a few OpenStack best practices when hosting these types of workloads.

Networking

For the networking aspect of OpenStack, there are two recommended configuration options:

  • Provider networks: The OpenStack administrator creates these networks and maps them directly to existing physical networks in the data center (L2). Because of the direct attachment to the L2 switching infrastructure, provider networks don’t need to route L3 traffic using the OpenStack control plane, since they should have an L3 gateway in the DC network topology.
  • SRIOV: SRIOV/SR-IOV (single root input/output virtualization) is recommended for HPC workloads based on performance requirements. SR-IOV enables OpenStack to extend the physical NIC’s capabilities directly through to the instance by using the available SRIOV NIC Virtual Functions (VF).  In addition, support for IEEE 802.1br allows virtual NICs to integrate with, and be managed by, the physical switch.
    • It’s important to mention that in tests conducted by various vendors, results show that SR-IOV can achieve near line-rate performance at a low CPU overhead cost per virtual machine/instance.
    • When implementing SRIOV, you need to take into consideration two essential limitations: not been able to use live migrations for instances using VF devices and bypassing OpenStack’s security groups.

Storage

For an HPC architecture, there are two major storage categories to consider:

  • OpenStack storage: image (Glance), ephemeral (Nova), and volume (Cinder)
  • HPC cluster file-based data storage: Used by the HPC cluster to store data

Based on both categories, here are a couple of recommendations to consider while designing your cluster:

OpenStack Storage:

  • Glance and Nova: For the Glance and Nova (ephemeral) storage, I recommend Ceph. One of the significant advantages of Ceph (besides the tight integration with OpenStack) are the performances benefits that you may obtain at instance creation time that image copy-on-write offers with this back end. Another advantage for the ephemeral workloads (not using SRIOV in this case) is the ability to live migrate between the members of the compute cluster.
  • Cinder: For the Cinder backend in this HPC use case, I recommend Ceph (same benefits apply from the previous point) and NFS/iSCSI backends like NetApp, EMC VNX and similar systems with supported cinder drivers.

HPC cluster file-based data storage:

Common used parallel file systems in HPC, like Lustre, GPFS, OrangeFS should be used by accessing them from dedicated SRIOV/Provider networks. Another recommended back end will be Ceph, also providing the access directly from the SRIOV/Provider networks for better performance.

Important information:
In general, Ceph offers a very flexible backend. A well-architected Ceph cluster can benefit multiple types of workloads in different configurations/architectures, for example:

  • Ethernet-based connectivity could benefit performance by higher throughput NIC interfaces for front end and back end storage traffic (10/25/40/50/100 Gbps), plus LACP configurations that could double the amount of bandwidth available
  • Storage servers components could be a combination of NVMe, SSD, SAS and SATA drives. Tailored to provide the required performance IO-wise
  • The distributed nature of the technology provides a flexible and resilient platform

The next thing to consider will be to automate the deployment of your HPC application on OpenStack. For that, multiple tools can be used: Heat, Ansible, or API calls from an orchestrator system.

Happy HPC on OpenStack hacking!

About the author

Julio Villarreal Pelegrino is an enterprise architect and systems engineer with over 15 years of experience in the software and IT industry. He’s currently a chief architect in the emerging technology practice at Red Hat. This post first appeared on his blog.

You can catch him at an Ansible hands-on workshop at the upcoming Vancouver Summit, where there’s also an entire track devoted to HPC.

Cover Photo // CC BY NC

The post Recommendations for high-performance computing on OpenStack appeared first on Superuser.

by Superuser at March 27, 2018 04:21 PM

Chris Dent

TC Report 18-13

It's a theme of no surprise, but this report, dedicated to ensuring people are just a bit more informed, has a fair bit to report on how staying informed can be difficult.

Global Communication

Last Wednesday, there was a wide ranging conversation about some of the difficulties that people living and working in the APAC area, especially China, experience when trying to interact with all of the OpenStack community. It's not solely a problem of time zone or language, there are also institutional limits on access to tools and networks, and different preferences.

One important question that was raised is that if an organization that is a member of the OpenStack Foundation (and thus obliged, at least in spirit, to contribute upstream) is knowingly limiting their employees access to IRC, email, and other tools of the OpenStack trade, are they violating the spirit of their agreement to be open?

Notably absent from this discussion were representatives of the impacted individuals or companies. So there was a lot of speculation going on.

This topic was picked back up later in the same day, eventually leading to the idea of extending the contributors guide, which is mostly for humans, to include a Contributing Organization Guide. Good idea!

Stackalytics Tired

Also on Wednesday there was discussion of how Stackalytics may be having some accuracy problems and whatever shall we do about that? Options include: kill it, fix it, replace it with something else, or switch over to a narrative report per cycle.

I'd be inclined to kill it first and see what organically emerges from the void. However we can't really do that because it's not run by OpenStack.

Adjutant

Last Thursday the application of Adjutant to be an official OpenStack project was discussed. There's still a bit of confusion about what it actually does, but to at least some people it sounds pretty cool.

Debate will continue on the review and if necessary, time will be set aside at the Forum to discuss the project in more detail.

Elections

This morning's topic was the forthcoming election for half the TC. There are at least two incumbents who will not be running for re-election. The office hours discussion had some interesting things to say about what governance is and to whom it is relevant.

by Chris Dent at March 27, 2018 03:30 PM

March 26, 2018

Cisco Cloud Blog

Cloud Unfiltered Podcast, Episode 39: OpenStack Update from VMware’s Mark Voelker

OK, I’ll cop to this right up front: I called my most recent guest “Vekler.” On tape. The problem with that is that his name is not Vekler. It’s Voelker....

by Ali Amagasu at March 26, 2018 07:17 PM

OpenStack Superuser

How you can influence the next OpenStack Forum

At the Forum, the entire OpenStack community gathers to brainstorm the requirements for the next release, gather feedback on the past version and engage in strategic discussions that go beyond goals for the next release cycle.

Now’s the time for you to weigh in for the upcoming Vancouver Summit. You can add ideas for sessions at the Forum in a couple of ways – there are catch-all Etherpads for the Technical Committee and the User Committee as well as those for specific projects listed on this Wiki page.

“We really want to get as many session ideas as possible so that the committee has too many to choose from,” says Melvin Hillsman, member of the User Committee.

Not sure what might work? Great forum sessions typically:

  • Involve both developers and users
  • Multiple projects/teams/working groups collaborating
  • Have a concrete outcome/a conclusion to work toward

Here’s more detail on the types of sessions that work for this event:

Project-specific sessions

Where developers can ask users specific questions about their experience, users can provide feedback from the last release and cross-community collaboration on the priorities and ‘blue sky’ ideas for the next release can occur.

Strategic, community-wide discussions

This is the time and place to think about the big picture, including beyond just one release cycle and new technologies.

Cross-project sessions

In a similar vein to what took place at past Design Summits, but with increased emphasis on issues that are of relevant to all areas of the community.

One more idea: if you’ve organized or attended any events in the past year, you ‘ve probably heard talks or been in discussions that are perfect for the Forum, Hillsman says. Pitch them!

A committee of Technical and User Committee representatives and Foundation Staff will then schedule the sessions into available times.

You have until April 2 15 – deadline extended – to propose your session.

The post How you can influence the next OpenStack Forum appeared first on Superuser.

by Superuser at March 26, 2018 01:34 PM

Opensource.com

How to create an open source stack using EFK

This tutorial shows you how to use Elasticsearch, Fluentd, and Kibana to build an open source stack that helps you manage complex data systems.

by mzamot at March 26, 2018 07:00 AM

Mirantis

How to deploy Spinnaker on Kubernetes: a quick and dirty guide

A reliable guide to deploying Spinnaker, including the magic steps it seems nobody ever talks about.

by Nick Chase at March 26, 2018 12:30 AM

March 24, 2018

Jian Li

openstack底层技术-netfilter框架概述

netfilter框架

netfilter是linux内核中的一个数据包处理框架,用于替代原有的ipfwadm和ipchains等数据包处理程序。netfilter的功能包括数据包过滤,修改,SNAT/DNAT等。netfilter在内核协议栈的不同位置实现了5个hook点,其它内核模块(比如ip_tables)可以向这些hook点注册处理函数,这样当数据包经过这些hook点时,其上注册的处理函数就被依次调用,用户层工具像iptables一般都需要相应内核模块ip_tables配合以完成与netfilter的交互。netfilter hooks、ip{6}_tables、connection tracking、和NAT子系统一起构成了netfilter框架的主要部分。可以参考Netfilter Architecture了解更多内容

下面这张图来自维基百科netfilter,它展示了netfilter框架在协议栈的位置,你们可能发现这张图在我的博客中出现多次了,那是因为这张图太经典了。它可以清楚地看到netfilter框架是如何处理通过不同协议栈路径上的数据包

netfilter

我们知道iptables的定位是IPv4 packet filter,它只处理IP数据包,而ebtables只工作在链路层Link Layer处理的是以太网帧(比如修改源目mac地址)。图中用有颜色的长方形方框表示iptables或ebtables的表和链,绿色小方框表示network level,即iptables的表和链。蓝色小方框表示bridge level,即ebtables的表和链,由于处理以太网帧相对简单,因此链路层的蓝色小方框相对较少。

我们还注意到一些代表iptables表和链的绿色小方框位于链路层,这是因为bridge_nf代码的作用(从2.6 kernel开始),bridge_nf的引入是为了解决在链路层Bridge中处理IP数据包的问题(需要通过内核参数开启),那为什么要在链路层Bridge中处理IP数据包,而不等数据包通过网络层时候再处理呢,这是因为不是所有的数据包都一定会通过网络层,比如外部机器与主机上虚拟机的通信流量,bridge_nf也是openstack中实现安全组功能的基础。

bridge_nf代码有时候会引起困惑,就像我们在图中看到的那样,代表iptables表和链的绿色小方框跑到了链路层,netfilter文档对此也有说明ebtables/iptables interaction on a Linux-based bridge

It should be noted that the br-nf code sometimes violates the TCP/IP Network Model. As will be seen later, it is possible, f.e., to do IP DNAT inside the Link Layer

ok,图中长方形小方框已经解释清楚了,还有一种椭圆形的方框conntrack,即connection tracking,这是netfilter提供的连接跟踪机制,此机制允许内核”审查”通过此处的所有网络数据包,并能识别出此数据包属于哪个网络连接(比如数据包a属于IP1:8888->IP2:80这个tcp连接,数据包b属于ip3:9999->IP4:53这个udp连接)。因此,连接跟踪机制使内核能够跟踪并记录通过此处的所有网络连接及其状态。图中可以清楚看到连接跟踪代码所处的网络栈位置,如果不想让某些数据包被跟踪(NOTRACK),那就要找位于椭圆形方框conntrack之前的表和链来设置规则。conntrack机制是iptables实现状态匹配(-m state)以及NAT的基础,它由单独的内核模块nf_conntrack实现。下面还会详细介绍

接着看图中左下方bridge check方框,数据包从主机上的某个网络接口进入(ingress), 在bridge check处会检查此网络接口是否属于某个Bridge的port,如果是就会进入Bridge代码处理逻辑(下方蓝色区域bridge level), 否则就会送入网络层Network Layer处理

图中下方中间位置的bridging decision类似普通二层交换机的查表转发功能,根据数据包目的MAC地址判断此数据包是转发还是交给上层处理,具体可以参考我另一篇文章linux-bridge

图中中心位置的routing decision就是路由选择,根据系统路由表(ip route查看), 决定数据包是forward,还是交给本地处理

总的来看,不同packet有不同的packet flow,packet总是从主机的某个接口进入(左下方ingress), 然后经过check/decision/一系列表和链处理,最后,目的地或是主机上某应用进程(上中位置local process),或是需要从主机另一个接口发出(右下方egress)。这里的接口即可以是物理网卡em1,也可以是虚拟网卡tun0/vnetx,还可以是Bridge上的一个port。

上面就是关于这张图的一些解释,如果还有其它疑问,欢迎留言讨论,下面说说netfilter框架的各个部分

connection tracking

当加载内核模块nf_conntrack后,conntrack机制就开始工作,如上图,椭圆形方框conntrack在内核中有两处位置(PREROUTING和OUTPUT之前)能够跟踪数据包。对于每个通过conntrack的数据包,内核都为其生成一个conntrack条目用以跟踪此连接,对于后续通过的数据包,内核会判断若此数据包属于一个已有的连接,则更新所对应的conntrack条目的状态(比如更新为ESTABLISHED状态),否则内核会为它新建一个conntrack条目。所有的conntrack条目都存放在一张表里,称为连接跟踪表

那么内核如何判断一个数据包是否属于一个已有的连接呢,我们先来了解下连接跟踪表

连接跟踪表

连接跟踪表存放于系统内存中,可以用cat /proc/net/nf_conntrack查看当前跟踪的所有conntrack条目。如下是代表一个tcp连接的conntrack条目,根据连接协议不同,下面显示的字段信息也不一样,比如icmp协议

ipv4 2 tcp 6 431955 ESTABLISHED src=172.16.207.231 dst=172.16.207.232 sport=51071 dport=5672 src=172.16.207.232 dst=172.16.207.231 sport=5672 dport=51071 [ASSURED] mark=0 zone=0 use=2

每个conntrack条目表示一个连接,连接协议可以是tcp,udp,icmp等,它包含了数据包的原始方向信息(蓝色)和期望的响应包信息(红色),这样内核能够在后续到来的数据包中识别出属于此连接的双向数据包,并更新此连接的状态,各字段意思的具体分析后面会说。连接跟踪表中能够存放的conntrack条目的最大值,即系统允许的最大连接跟踪数记作CONNTRACK_MAX

netfilter_hash_table

在内核中,连接跟踪表是一个二维数组结构的哈希表(hash table),哈希表的大小记作HASHSIZE,哈希表的每一项(hash table entry)称作bucket,因此哈希表中有HASHSIZE个bucket存在,每个bucket包含一个链表(linked list),每个链表能够存放若干个conntrack条目(bucket size)。对于一个新收到的数据包,内核使用如下步骤判断其是否属于一个已有连接:

  • 内核提取此数据包信息(源目IP,port,协议号)进行hash计算得到一个hash值,在哈希表中以此hash值做索引,索引结果为数据包所属的bucket(链表)。这一步hash计算时间固定并且很短
  • 遍历hash得到的bucket,查找是否有匹配的conntrack条目。这一步是比较耗时的操作,bucket size越大,遍历时间越长

如何设置最大连接跟踪数

根据上面对哈希表的解释,系统最大允许连接跟踪数CONNTRACK_MAX = 连接跟踪表大小(HASHSIZE) * Bucket大小(bucket size)。从连接跟踪表获取bucket是hash操作时间很短,而遍历bucket相对费时,因此为了conntrack性能考虑,bucket size越小越好,默认为8

#查看系统当前最大连接跟踪数CONNTRACK_MAX
sysctl -a | grep net.netfilter.nf_conntrack_max
#net.netfilter.nf_conntrack_max = 3203072

#查看当前连接跟踪表大小HASHSIZE
sysctl -a | grep net.netfilter.nf_conntrack_buckets
#400384
#或者这样
cat /sys/module/nf_conntrack/parameters/hashsize
#400384    

这两个的比值即为bucket size = 3203072 / 400384

如下,现在需求是设置系统最大连接跟踪数为320w,由于bucket size不能直接设置,为了使bucket size值为8,我们需要同时设置CONNTRACK_MAXHASHSIZE,因为他们的比值就是bucket size

#HASHSIZE (内核会自动格式化为最接近允许值)       
echo 400000 > /sys/module/nf_conntrack/parameters/hashsize

#系统最大连接跟踪数
sysctl -w net.netfilter.nf_conntrack_max=3200000      

#注意nf_conntrack内核模块需要加载     

为了使nf_conntrack模块重新加载或系统重启后生效

#nf_conntrack模块提供了设置HASHSIZE的参数          
echo "options nf_conntrack hashsize=400000" > /etc/modprobe.d/nf_conntrack.conf                  

只需要固化HASHSIZE值,nf_conntrack模块在重新加载时会自动设置CONNTRACK_MAX = hashsize * 8,当然前提是你bucket size使用系统默认值8。如果自定义bucket size值,就需要同时固化CONNTRACK_MAX,以保持其比值为你想要的bucket size

上面我们没有改变bucket size的默认值8,但是若内存足够并且性能很重要,你可以考虑每个bucket一个conntrack条目(bucket size = 1),最大可能降低遍历耗时,即HASHSIZE = CONNTRACK_MAX

#HASHSIZE
echo 3200000 > /sys/module/nf_conntrack/parameters/hashsize
#CONNTRACK_MAX
sysctl -w net.netfilter.nf_conntrack_max=3200000

如何计算连接跟踪所占内存

连接跟踪表存储在系统内存中,因此需要考虑内存占用问题,可以用下面公式计算设置不同的最大连接跟踪数所占最大系统内存

total_mem_used(bytes) = CONNTRACK_MAX * sizeof(struct ip_conntrack) + HASHSIZE * sizeof(struct list_head)

例如我们需要设置最大连接跟踪数为320w,在centos6/7系统上,sizeof(struct ip_conntrack) = 376,sizeof(struct list_head) = 16,并且bucket size使用默认值8,并且HASHSIZE = CONNTRACK_MAX / 8,因此

total_mem_used(bytes) = 3200000 * 376 + (3200000 / 8) * 16
# = 1153MB ~= 1GB

因此可以得到,在centos6/7系统上,设置320w的最大连接跟踪数,所消耗的内存大约为1GB,对现代服务器来说,占用内存并不多,但conntrack实在让人又爱又恨

关于上面两个sizeof(struct *)值在你系统上的大小,如果会C就好说,如果不会,可以使用如下python代码计算

import ctypes

#不同系统可能此库名不一样,需要修改             
LIBNETFILTER_CONNTRACK = 'libnetfilter_conntrack.so.3.6.0'

nfct = ctypes.CDLL(LIBNETFILTER_CONNTRACK)
print 'sizeof(struct nf_conntrack):', nfct.nfct_maxsize()
print 'sizeof(struct list_head):', ctypes.sizeof(ctypes.c_void_p) * 2

conntrack条目

conntrack从经过它的数据包中提取详细的,唯一的信息,因此能保持对每一个连接的跟踪。关于conntrack如何确定一个连接,对于tcp/udp,连接由他们的源目地址,源目端口唯一确定。对于icmp,由type,code和id字段确定。

ipv4     2 tcp      6 33 SYN_SENT src=172.16.200.119 dst=172.16.202.12 sport=54786 dport=10051 [UNREPLIED] src=172.16.202.12 dst=172.16.200.119 sport=10051 dport=54786 mark=0 zone=0 use=2

如上是一条conntrack条目,它代表当前已跟踪到的某个连接,conntrack维护的所有信息都包含在这个条目中,通过它就可以知道某个连接处于什么状态

  • 此连接使用ipv4协议,是一条tcp连接(tcp的协议类型代码是6)
  • 33是这条conntrack条目在当前时间点的生存时间(每个conntrack条目都会有生存时间,从设置值开始倒计时,倒计时完后此条目将被清除),可以使用sysctl -a |grep conntrack | grep timeout查看不同协议不同状态下生存时间设置值,当然这些设置值都可以调整,注意若后续有收到属于此连接的数据包,则此生存时间将被重置(重新从设置值开始倒计时),并且状态改变,生存时间设置值也会响应改为新状态的值
  • SYN_SENT是到此刻为止conntrack跟踪到的这个连接的状态(内核角度),SYN_SENT表示这个连接只在一个方向发送了一初始TCP SYN包,还未看到响应的SYN+ACK包(只有tcp才会有这个字段)。
  • src=172.16.200.119 dst=172.16.202.12 sport=54786 dport=10051是从数据包中提取的此连接的源目地址、源目端口,是conntrack首次看到此数据包时候的信息。
  • [UNREPLIED]说明此刻为止这个连接还没有收到任何响应,当一个连接已收到响应时,[UNREPLIED]标志就会被移除
  • 接下来的src=172.16.202.12 dst=172.16.200.119 sport=10051 dport=54786地址和端口和前面是相反的,这部分不是数据包中带有的信息,是conntrack填充的信息,代表conntrack希望收到的响应包信息。意思是若后续conntrack跟踪到某个数据包信息与此部分匹配,则此数据包就是此连接的响应数据包。注意这部分确定了conntrack如何判断响应包(tcp/udp),icmp是依据另外几个字段

上面是tcp连接的条目,而udp和icmp没有连接建立和关闭过程,因此条目字段会有所不同,后面iptables状态匹配部分我们会看到处于各个状态的conntrack条目

注意conntrack机制并不能够修改或过滤数据包,它只是跟踪网络连接并维护连接跟踪表,以提供给iptables做状态匹配使用,也就是说,如果你iptables中用不到状态匹配,那就没必要启用conntrack

iptables状态匹配

本部分参考自iptables文档iptables-tutorial#The state machine,我找不到比它更全面的文章了

先明确下conntrack在内核协议栈所处位置,上面也提过conntrack跟踪数据包的位置在PREROUTING和OUTPUT这两个hook点,主机自身进程产生的数据包会通过OUTPUT处的conntrack,从主机任意接口进入(包括Bridge的port)的数据包会通过PREROUTING处的conntrack,从netfilter框架图上可以看到conntrack位置很靠前,仅在iptables的raw表之后,raw表主要作用就是允许我们对某些特定的数据包打上NOTRACK标记,这样后面的conntrack就不会记录此类带有NOTRACK标签的数据包。conntrack位置很靠前一方面是保证其后面的iptables表和链都能使用状态匹配,另一方面使得conntrack能够跟踪到任何进出主机的原始数据包(比如数据包还未NAT/FORWARD)。

iptables状态匹配模块

iptables-status

iptables是带有状态匹配的防火墙,它使用-m state模块从连接跟踪表查找数据包状态。上面我们分析的那条conntrack条目处于SYN_SENT状态,这是内核记录的状态,数据包在内核中可能会有几种不同的状态,但是映射到用户空间iptables,只有5种状态可用:NEW,ESTABLISHED,RELATED,INVALID和UNTRACKED。注意这里说的状态不是tcp/ip协议中tcp连接的各种状态。下面表格说明了这5种状态分别能够匹配什么样的数据包,注意下面两点

  • 用户空间这5种状态是iptables用于完成状态匹配而定义的,不关联与特定连接协议
  • conntrack记录在前,iptables匹配在后(见netfilter框架图)
状态 解释
NEW NEW匹配连接的第一个包。意思就是,iptables从连接跟踪表中查到此包是某连接的第一个包。判断此包是某连接的第一个包是依据conntrack当前”只看到一个方向数据包”([UNREPLIED]),不关联特定协议,因此NEW并不单指tcp连接的SYN包
ESTABLISHED ESTABLISHED匹配连接的响应包及后续的包。意思是,iptables从连接跟踪表中查到此包是属于一个已经收到响应的连接(即没有[UNREPLIED]字段)。因此在iptables状态中,只要发送并接到响应,连接就认为是ESTABLISHED的了。这个特点使iptables可以控制由谁发起的连接才可以通过,比如A与B通信,A发给B数据包属于NEW状态,B回复给A的数据包就变为ESTABLISHED状态。ICMP的错误和重定向等信息包也被看作是ESTABLISHED,只要它们是我们所发出的信息的应答。
RELATED RELATED匹配那些属于RELATED连接的包,这句话说了跟没说一样。RELATED状态有点复杂,当一个连接与另一个已经是ESTABLISHED的连接有关时,这个连接就被认为是RELATED。这意味着,一个连接要想成为RELATED,必须首先有一个已经是ESTABLISHED的连接存在。这个ESTABLISHED连接再产生一个主连接之外的新连接,这个新连接就是RELATED状态了,当然首先conntrack模块要能”读懂”它是RELATED。拿ftp来说,FTP数据传输连接就是RELATED与先前已建立的FTP控制连接,还有通过IRC的DCC连接。有了RELATED这个状态,ICMP错误消息、FTP传输、DCC等才能穿过防火墙正常工作。有些依赖此机制的TCP协议和UDP协议非常复杂,他们的连接被封装在其它的TCP或UDP包的数据部分(可以了解下overlay/vxlan/gre),这使得conntrack需要借助其它辅助模块才能正确”读懂”这些复杂数据包,比如nf_conntrack_ftp这个辅助模块
INVALID INVALID匹配那些无法识别或没有任何状态的数据包。这可能是由于系统内存不足或收到不属于任何已知连接的ICMP错误消息。一般情况下我们应该DROP此类状态的包
UNTRACKED UNTRACKED状态比较简单,它匹配那些带有NOTRACK标签的数据包。需要注意的一点是,如果你在raw表中对某些数据包设置有NOTRACK标签,那上面的4种状态将无法匹配这样的数据包,因此你需要单独考虑NOTRACK包的放行规则

状态的使用使防火墙可以非常强大和有效,来看下面这个常见的防火墙规则,它允许本机主动访问外网,以及放开icmp协议

#iptables-save  -t filter
*filter
:INPUT DROP [1453341:537074675]
:FORWARD DROP [10976649:6291806497]
:OUTPUT ACCEPT [1221855153:247285484556]
-A INPUT -p icmp -j ACCEPT 
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 

主机进程与外网机器通信经历如下步骤,因为除了filter表,其它表没设置任何规则,所以下面步骤就省略其它表的匹配过程

  1. 进程产生要发送的数据包,数据包通过rawOUTPUT链(可以决定是否要NOTRACK)
  2. 数据包通过conntrack,conntrack记录此连接到连接跟踪表([UNREPLIED]) – NEW
  3. 通过OUTPUT链,然后从主机网卡发出 – NEW
  4. 外网目标主机收到请求,发出响应包
  5. 响应包从主机某个接口进入,到达rawPREROUTING链(可以决定是否要NOTRACK) – NEW
  6. 响应包通过conntrack,conntrack发现此数据包为一个连接的响应包,更新对应conntrack条目状态(去掉[UNREPLIED],至此两个方向都看到包了) – ESTABLISHED
  7. 响应包到达filterINPUT链,在这里匹配到--state RELATED,ESTABLISHED,因此放行 – ESTABLISHED

像上面这种允许本机主动出流量的需求,如果不用conntrack会很难实现,那你可能会说我也可以使用iptables

数据包在内核中的状态

从内核角度,不同协议有不同状态,这里我们来具体看下三种协议tcp/udp/icmp在连接跟踪表中的不同状态

tcp连接

下面是172.16.1.100向172.16.1.200建立tcp通信过程中,/proc/net/nf_conntrack中此连接的状态变化过程

ipv4     2 tcp      6 118 SYN_SENT src=172.16.1.100 dst=172.16.1.200 sport=36884 dport=8220 [UNREPLIED] src=172.16.1.200 dst=172.16.1.100 sport=8220 dport=36884 mark=0 zone=0 use=2

如上,首先172.16.1.100向172.16.1.200发送SYN包,172.16.1.200收到SYN包但尚未回复,由于是新连接,conntrack将此连接添加到连接跟踪表,并标记为SYN_SENT状态,[UNREPLIED]表示conntrack尚未跟踪到172.16.1.200的响应包。注意上面这条conntrack条目存在于两台主机的连接跟踪表中(当然,首先要两台主机都启用conntrack),对于172.16.1.100,数据包在经过OUTPUT这个hook点时触发conntrack,而对于172.16.1.200,数据包在PREROUTING这个hook点时触发conntrack

随后,172.16.1.200回复SYN/ACK包给172.16.1.100,通过conntrack更新连接状态为SYN_RECV,表示收到SYN/ACk包,去掉[UNREPLIED]字段

ipv4     2 tcp      6 59 SYN_RECV src=172.16.1.100 dst=172.16.1.200 sport=36884 dport=8220 src=172.16.1.200 dst=172.16.1.100 sport=8220 dport=36884 mark=0 zone=0 use=2

接着,172.16.1.100回复ACK给172.16.1.200,至此,三次握手完成,tcp连接已建立,conntrack更新连接状态为ESTABLISHED

ipv4     2 tcp      6 10799 ESTABLISHED src=172.16.1.100 dst=172.16.1.200 sport=36884 dport=8220 src=172.16.1.200 dst=172.16.1.100 sport=8220 dport=36884 [ASSURED] mark=0 zone=0 use=2

连接跟踪表中的conntrack条目不可能是永久存在,每个conntrack条目都有超时时间,可以如下方式查看tcp连接各个状态当前设置的超时时间

# sysctl -a |grep 'net.netfilter.nf_conntrack_tcp_timeout_'
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
...

正常的tcp连接是很短暂的,不太可能查看到一个tcp连接所有状态变化的,那如何构造处于特定状态的tcp连接呢,一个方法是利用iptables的--tcp-flags参数,其可以匹配tcp数据包的标志位,比如下面这两条

#对于本机发往172.16.1.200的tcp数据包,丢弃带有SYN/ACK flag的包       
iptables -A OUTPUT -o eth0 -p tcp --tcp-flags SYN,ACK SYN,ACK -d 172.16.1.200 -j DROP

#同样,这个是丢弃带有ACK flag的包 
iptables -A OUTPUT -o eth0 -p tcp --tcp-flags ACK ACK -d 172.16.1.100 -j DROP

同时利用tcp超时重传机制,待cat /proc/net/nf_conntrack获取到conntrack条目后,使用iptables -D OUTPUT X删除之前设置的DROP规则,这样tcp连接就会正常走下去,这个很容易测试出来

udp连接

UDP连接是无状态的,它没有连接的建立和关闭过程,连接跟踪表中的udp连接也没有像tcp那样的状态字段,但这不妨碍用户空间iptables对udp包的状态匹配,上面也说过,iptables中使用的各个状态与协议无关

#只收到udp连接第一个包
ipv4     2 udp      17 28 src=172.16.1.100 dst=172.16.1.200 sport=26741 dport=8991 [UNREPLIED] src=172.16.1.200 dst=172.16.1.100 sport=8991 dport=26741 mark=0 zone=0 use=2

#收到此连接的响应包    
ipv4     2 udp      17 29 src=172.16.1.100 dst=172.16.1.200 sport=26741 dport=8991 src=172.16.1.200 dst=172.16.1.100 sport=8991 dport=26741 mark=0 zone=0 use=2

同样可以查看udp超时时间

# sysctl -a | grep 'net.netfilter.nf_conntrack_udp_'
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180

icmp

icmp请求,在用户空间iptables看来,跟踪到echo request时连接处于NEW状态,当有echo reply时就是ESTABLISHED状态。

#icmp请求
ipv4     2 icmp     1 28 src=103.229.215.2 dst=113.31.136.7 type=8 code=0 id=35102 [UNREPLIED] src=113.31.136.7 dst=103.229.215.2 type=0 code=0 id=35102 mark=0 zone=0 use=2

#reply
ipv4     2 icmp     1 29 src=103.229.215.2 dst=113.31.136.7 type=8 code=0 id=35102 src=113.31.136.7 dst=103.229.215.2 type=0 code=0 id=35102 mark=0 zone=0 use=2

如何管理连接跟踪表

有一个用户空间工具conntrack,其提供了对连接跟踪表的增删改查功能,可以用yum install conntrack-tools来安装,下面是几个例子

#查看连接跟踪表所有条目  
conntrack -L
#清除连接跟踪表
conntrack -F
#删除连接跟踪表中所有源地址是1.2.3.4的条目
conntrack -D -s 1.2.3.4

如果openstack使用安全组IptablesFirewallDriver就会用到这个工具,但默认不会安装,未安装时,计算节点neutron-linuxbridge-agent日志会出现类似下面错误

2018-02-14 16:59:42.755 993 ERROR neutron.agent.linux.utils [req-4f84df17-534b-4032-b907-8d3463824726 - - - - -] Rootwrap error running command: ['conntrack', '-D', '-f', 'ipv4', '-d', '172.19.3.99', '-w', '1']
2018-02-14 16:59:42.872 993 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._remove_conntrack_entries_from_port_deleted(port)

需要这个工具是因为在对instance进行安全组/port/ip方面的变更后,需要同时删除连接跟踪表中旧的相关条目,比如某台instance之前放通本机80端口,某外部主机与此instance 80端口有连接并且有流量,那此时iptables从连接跟踪表中识别到此连接状态就是ESTABLISHED,然后此时突然有个需求需要禁止本机的80端口,如果不删除与本机80端口有关的处于ESTABLISHED状态的conntrack条目,而iptables又会放行处于RELATED,ESTABLISHED状态的连接,这样就会造成80端口仍能够连接

Bridge与netfilter

从netfilter框架图中可以看到,最下层蓝色区域为bridge level。Bridge的存在,使得主机可以充当一台虚拟的普通二层交换机来运作,这个虚拟交换机可以建多个port,连接到多个虚拟机。由此带来的问题是,外部机器与其上虚拟机通信流量只会经过主机二层(靠Bridge转发,此时不经过主机IP层),主机上的网卡类型变得复杂(物理网卡em1,网桥br0,虚拟网卡vnetX),进入主机的数据包可选路径变多(bridge转发/交给主机本地进程)。幸好,netfilter框架都可以解决,这部分内容在我的另一篇文章Bridge与netfilter,相信已经说得够清楚了。

conntrack与LVS

LVS的修改数据包功能也是依赖netfilter框架,在LVS机器上应用iptables时需要注意一个问题就是,LVS-DR(或LVS-Tun)模式下,不能在director上使用iptables的状态匹配(NEW,ESTABLISHED,INVALID,…)

LVS-DR模式下,client访问director,director转发流量到realserver,realserver直接回复client,不经过director,这种情况下,client与direcotr处于tcp半连接状态

因此如果在director机器上启用conntrack,此时conntrack只能看到client–>director的数据包,因为响应包不经过direcotr,conntrack无法看到反方向上的数据包,就表示iptables中的ESTABLISHED状态永远无法匹配,从而可能发生DROP

以上就是netfilter框架的内容,理论居多,后续会有一篇专门分析openstack安全组实现的文章,会包括本文大部分知识点,算是作为本文的一个实例应用
(本文完)

参考文章

Conntrack_tuning
Tuning The Linux Connection Tracking System
iptables-tutorial.html#STATEMACHINE
LVS-HOWTO:stateful filtering: LVS-DR
LVS/IPVS on Openstack - Bad Idea
conntrack_table_memory

March 24, 2018 12:00 AM

openstack底层技术-netfilter框架

netfilter框架

netfilter是linux内核中的一个数据包处理框架,用于替代原有的ipfwadm和ipchains等数据包处理程序。netfilter的功能包括数据包过滤,修改,NAT等。netfilter在内核协议栈的不同位置实现了5个钩子函数(hooks),不同类型的网络数据包会通过不同的hook点(比如数据包是要forward,或是交给local process处理),用户层工具iptables/ebtables通过操作不同的hook点以实现对通过此hook点的数据包的处理,关于netfilter框架,可以参考Netfilter Architecture

下面这张图来自维基百科netfilter, 它展示了netfilter注册的hook点在内核协议栈的分布,以及packet在通过内核协议栈时,所经过的hook点,你们可能发现这张图在我的博客中出现多次了,那是因为这张图太经典了。图中有清晰的网络模型划分,因此很容易看到数据包在经过某条链时所处的层次

netfilter

我们知道iptables只处理IP数据包(IP/PORT/SNAT/DNAT/…),而ebtables只工作在链路层Link Layer处理以太网帧(比如修改源/目mac地址)。图中用有颜色的长方形方框表示iptables或ebtables的表和链,绿色小方框表示network level,即iptables的表和链。蓝色小方框表示bridge level,即ebtables的表和链,由于处理以太网帧相对简单,因此链路层的蓝色小方框相对较少。

我们还注意到一些代表iptables表和链的绿色小方框位于链路层,这是因为bridge_nf代码的作用(从2.6 kernel开始),bridge_nf的引入是为了解决在链路层Bridge中处理IP数据包的问题(需要通过内核参数开启),那为什么要在链路层Bridge中处理IP数据包,而不等数据包通过网络层时候再处理呢,这是因为不是所有的数据包都一定会通过网络层,有可能数据包从Bridge的一个port进入,经过bridging decision后从另一个port发出,bridge_nf也是openstack中实现安全组功能的基础,关于Bridge下的数据包过滤,我的另一篇文章有总结Bridge与netfilter

bridge_nf代码有时候会引起困惑,就像我们在图中看到的那样,iptables的表和链(绿色小方框)跑到了链路层,netfilter文档对此也有说明ebtables/iptables interaction on a Linux-based bridge

It should be noted that the br-nf code sometimes violates the TCP/IP Network Model. As will be seen later, it is possible, f.e., to do IP DNAT inside the Link Layer

ok,图中长方形小方框已经解释清楚了,还有一种椭圆形的方框conntrack,即connection tracking,这是netfilter提供的连接跟踪机制,此机制允许内核”审查”通过此处的所有网络数据包,并能识别出此数据包属于哪个网络连接(比如数据包a属于IP1:8888->IP2:80这个tcp连接,数据包b属于ip3:9999->IP4:53这个udp连接)。因此,连接跟踪机制使内核能够跟踪并记录通过此处的所有网络连接及其状态。图中可以清楚看到连接跟踪代码所处的网络栈位置,如果不想让某些数据包被跟踪(NOTRACK),那就要找位于椭圆形方框conntrack之前的表和链来设置规则。conntrack机制是iptables实现状态匹配(-m state)以及NAT的基础,它由单独的内核模块nf_conntrack实现。下面还会详细介绍

接着看图中左下方bridge check方框,数据包从主机上的某个网络接口进入(ingress), 在bridge check处会检查此网络接口是否属于某个Bridge的port,如果是就会进入Bridge代码处理逻辑(broute->…), 否则就会送入网络层处理(raw–>…)

图中下方中间位置的bridging decision环节就是根据数据包目的MAC地址判断此数据包是转发还是交给上层处理,这个另一篇文章也有解释linux-bridge

图中中心位置的routing decision就是路由选择,根据系统路由表(ip route查看), 决定数据包是forward,还是交给本地处理

总的来看,图中整个packet flow表示的是数据包总是从主机的某个接口进入(左下方ingress), 然后经过一系列表和链处理,最后,或是交由主机上某应用进程处理(上中位置local process),或是从主机另一个接口发出(右下方egress),这里的接口即可以是物理网卡em1,也可以是虚拟网卡tun0/vnetx,还可以是Bridge上的一个port。

上面就是关于这张图的一些解释,如果还有其它疑问,欢迎留言讨论,下面说说netfilter的应用方面

connection tracking

当加载内核模块nf_conntrack后,conntrack机制就开始工作,如上图,椭圆形方框conntrack在内核中有两处位置(PREROUTING和OUTPUT之前)能够跟踪数据包。对于每个通过conntrack的数据包,内核都为其生成一个conntrack条目用以跟踪此连接,对于后续通过的数据包,内核会判断若此数据包属于一个已有的连接,则更新所对应的conntrack条目的状态(比如更新为ESTABLISHED状态),否则内核会为它新建一个conntrack条目。所有的conntrack条目都存放在一张表里,称为连接跟踪表

那么内核如何判断一个数据包是否属于一个已有的连接呢,我们先来了解下连接跟踪表

连接跟踪表

连接跟踪表存放于系统内存中,可以用cat /proc/net/nf_conntrack查看当前跟踪的所有conntrack条目。如下是代表一个tcp连接的conntrack条目,根据连接协议不同,下面显示的字段信息也不一样,比如icmp协议

ipv4 2 tcp 6 431955 ESTABLISHED src=172.16.207.231 dst=172.16.207.232 sport=51071 dport=5672 src=172.16.207.232 dst=172.16.207.231 sport=5672 dport=51071 [ASSURED] mark=0 zone=0 use=2

每个conntrack条目表示一个连接,连接协议可以是tcp,udp,icmp等,它包含了数据包的原始方向信息(蓝色)和期望的响应包信息(红色),这样内核能够在后续到来的数据包中识别出属于此连接的双向数据包,并更新此连接的状态。连接跟踪表中能够存放的conntrack条目的最大值,即系统允许的最大连接跟踪数记作CONNTRACK_MAX

netfilter_hash_table

在内核中,连接跟踪表是一个二维数组结构的哈希表(hash table),哈希表的大小称作HASHSIZE,哈希表的每一项(hash table entry)称作bucket,因此哈希表中有HASHSIZE个bucket存在,每个bucket包含一个链表(linked list),每个链表能够存放若干个conntrack条目(bucket size)。对于一个新收到的数据包,内核使用如下步骤判断其是否属于一个已有连接:

  • 内核将此数据包信息(源目IP,port,协议号)进行hash计算得到一个hash值,在哈希表中以此hash值做索引,得到数据包所属的bucket(链表)。这一步hash计算时间是固定的并且很短
  • 遍历bucket,查找是否有匹配的conntrack条目。这一步是比较耗时的操作,bucket size越大,遍历时间越长

如何设置最大连接跟踪数

根据上面对哈希表的解释,系统最大允许连接跟踪数CONNTRACK_MAX = 连接跟踪表大小(HASHSIZE) * Bucket大小(bucket size)。从连接跟踪表获取bucket是hash操作时间很短,而遍历bucket相对费时,因此为了conntrack性能考虑,bucket size越小越好,默认为8

#查看系统当前最大连接跟踪数CONNTRACK_MAX
sysctl -a | grep net.netfilter.nf_conntrack_max
#net.netfilter.nf_conntrack_max = 3200000

#查看当前连接跟踪表大小HASHSIZE
sysctl -a | grep net.netfilter.nf_conntrack_buckets
#400384
#或者
cat /sys/module/nf_conntrack/parameters/hashsize
#400384

#可以得出此时bucket size = 3200000 / 400384     

如下,现在需求是设置系统最大连接跟踪数为320w,由于bucket size值不能直接设置,为了使bucket size值为8,我们需要同时设置CONNTRACK_MAXHASHSIZE,因为他们的比值就是bucket size

#HASHSIZE = 3200000 / 8
echo 400000 > /sys/module/nf_conntrack/parameters/hashsize

#系统最大连接跟踪数
sysctl -w net.netfilter.nf_conntrack_max=3200000     

#注意nf_conntrack内核模块需要加载     

为了使重启后生效(nf_conntrack模块重新加载)

#nf_conntrack模块提供了设置HASHSIZE的参数          
echo "options nf_conntrack hashsize=400000" > /etc/modprobe.d/nf_conntrack.conf                  

只需要固化HASHSIZE值,nf_conntrack模块在重新加载时会自动设置CONNTRACK_MAX = hashsize * 8,但如果你bucket size不使用默认值8,则CONNTRACK_MAX也需要固化,以保持其比值为你想要的bucket size

上面我们没有改变bucket size的默认值8,但是若内存足够并且性能很重要,你可以考虑每个bucket一个conntrack条目(bucket size = 1),即HASHSIZE = CONNTRACK_MAX

#HASHSIZE
echo 3200000 > /sys/module/nf_conntrack/parameters/hashsize
#CONNTRACK_MAX
sysctl -w net.netfilter.nf_conntrack_max=3200000

如何计算连接跟踪所占内存

连接跟踪表存储在系统内存中,设置的最大连接跟踪数越多,消耗的最大系统内存就越多,可以用下面公式计算设置不同的最大连接跟踪数所占最大系统内存

size_of_mem_used_by_conntrack (in bytes) = CONNTRACK_MAX * sizeof(struct ip_conntrack) + HASHSIZE * sizeof(struct list_head)

假如我们需要设置最大连接跟踪数为320w,在centos7系统上,sizeof(struct ip_conntrack) = 376,sizeof(struct list_head) = 16,并且bucket size默认为8,因此HASHSIZE = CONNTRACK_MAX / 8,因此

size_of_mem_used_by_conntrack (in bytes) = 3200000 * 376 + (3200000 / 8) * 16
# = 1153MB ~= 1GB

因此可以得到,在centos7系统上,设置320w的最大连接跟踪数,所消耗的内存大约为1GB

关于上面两个sizeof(struct *)值在你系统上的大小,可以使用如下python代码计算

import ctypes

#不同系统可能此库名不一样  
LIBNETFILTER_CONNTRACK = 'libnetfilter_conntrack.so.3.6.0'

nfct = ctypes.CDLL(LIBNETFILTER_CONNTRACK)
print 'sizeof(struct nf_conntrack):', nfct.nfct_maxsize()
print 'sizeof(struct list_head):', ctypes.sizeof(ctypes.c_void_p) * 2

conntrack条目

conntrack从数据包中提取详细的,唯一的信息,因此能保持对每一个连接的跟踪。关于conntrack如何确定一个连接,对于tcp/udp,连接由他们的源目地址,源目端口唯一确定。对于icmp,由type,code和id字段确定。

ipv4     2 tcp      6 33 SYN_SENT src=172.16.200.119 dst=172.16.202.12 sport=54786 dport=10051 [UNREPLIED] src=172.16.202.12 dst=172.16.200.119 sport=10051 dport=54786 mark=0 zone=0 use=2

如上是一条conntrack条目,它代表当前某个被跟踪的连接,conntrack维护的所有信息都包含在这个条目中,通过它就可以知道某个连接处于什么状态:

  • 此连接使用ipv4协议,是一条tcp连接(tcp的协议类型代码是6)
  • 33是这条conntrack条目在当前时间点的生存时间(从设置值开始倒计时,倒计时完后此条目将被删除),可以使用sysctl -a |grep conntrack | grep timeout查看不同协议不同状态下生存时间设置值,当然这些设置值都可以调整,注意若后续有收到属于此连接的数据包,则此生存时间将被重置(重新从设置值开始倒计时),并且状态改变,生存时间设置值也会改变
  • SYN_SENT是这个连接在当前时间点的状态(内核角度),SYN_SENT表示这个连接只在一个方向发送了一TCP SYN包,还未看到响应的SYN+ACK包(只有tcp连接有这个字段)。
  • src=172.16.200.119 dst=172.16.202.12 sport=54786 dport=10051是此连接的源目地址、源目端口,代表此连接的初始数据包。即172.16.200.119:54786 -SYN-> 172.16.202.12:10051
  • [UNREPLIED]说明此时这个连接还没有收到任何响应,当一个连接已收到响应时,[UNREPLIED]标志就会被移除
  • 接下来的src=172.16.202.12 dst=172.16.200.119 sport=10051 dport=54786地址和端口和前面是相反的,这部分不是数据包中带有的信息,是conntrack填充的信息,代表conntrack希望收到的响应包的信息。意思是若后续conntrack跟踪到某个数据包信息与此匹配,则此数据包就是此连接的响应数据包,对于无状态的udp也同样合适,只要这里的ip端口匹配。

conntrack条目信息依据其连接类型的不同而不同,相比tcp连接,udp和icmp没有连接建立和关闭过程,因此其conntrack条目也有不同

#udp
ipv4     2 udp      17 22 src=172.16.200.119 dst=172.16.200.1 sport=33636 dport=53 [UNREPLIED] src=172.16.200.1 dst=172.16.200.119 sport=53 dport=33636 mark=0 zone=0 use=2    
#icmp
ipv4     2 icmp     1 29 src=114.14.240.77 dst=111.31.136.9 type=8 code=0 id=57960 src=111.31.136.9 dst=114.14.240.77 type=0 code=0 id=57960 mark=0 zone=0 use=2
#对于icmp,匹配“src=111.31.136.9 dst=114.14.240.77 type=0 code=0 id=57960”的包就是icmp的响应包         

这里要注意若conntrack条目中只看到一个方向的数据包,则会有[UNREPLIED]字段,后续收到属于此连接的响应包,[UNREPLIED]字段会被移除

还有要注意的是,conntrack机制作用只是跟踪并记录通过它的网络连接及其状态,并把信息更新在连接跟踪表,以提供给iptables做状态匹配使用

iptables状态匹配

先明确下conntrack跟踪的位置,上面也提过conntrack跟踪数据包的位置在PREROUTING和OUTPUT这两个hook点,本地产生的数据包会在OUTPUT处被跟踪,从外部进入(主机的任意接口,包括Bridge的port)的数据包会在PREROUTING处被跟踪,从netfilter框架图上可以看到conntrack跟踪位置很靠前,仅在iptablesraw表之后,raw表主要作用就是允许我们对不想被跟踪的数据包打NOTRACK标签,这样后面的conntrack看到此数据包带有NOTRACK标签就不会做任何处理。conntrack跟踪位置很靠前也就意味着其后面的iptables表和链都能使用状态匹配。并且conntrack能够跟踪到进出主机的任何原始数据包(比如还未NAT)。

iptables是带有状态匹配的防火墙,它使用-m state模块从连接跟踪表查找数据包状态。上面我们分析的那条conntrack条目处于SYN_SENT状态,这是内核记录的状态,数据包在内核中可能会有几种不同的状态,但是映射到用户空间iptables,只有4种状态可用:NEW,ESTABLISHED,RELATED 和INVALID。注意我们这里说的状态不是tcp/ip协议中tcp连接的各种状态,这里的4种状态只是iptables用于完成状态匹配而定义的,适用于各种连接协议。下面说明了这4种状态分别能够匹配什么样的数据包(注意conntrack记录在前,iptables匹配在后)

State 解释
NEW NEW匹配连接的第一个包。意思就是,iptables从连接跟踪表中查到此包是某连接的第一个包。这里的NEW状态是依据conntrack条目中只看到一个方向的数据包([UNREPLIED]),而不管conntrack条目的连接类型(tcp/udp/icmp),因此NEW并不单指tcp连接的第一个SYN包
ESTABLISHED ESTABLISHED匹配连接的响应包及后续的包。意思是,iptables从连接跟踪表中查到此包是属于一个已经收到响应的连接(即没有[UNREPLIED]字段)。因此在iptables状态中,只要发送并接到响应,连接就认为是ESTABLISHED的了。这个特点使iptables可以控制由谁发起的连接才可以通过,比如A与B通信,A发给B数据包属于NEW状态,B回复给A的数据包就变为ESTABLISHED状态。ICMP的错误和重定向等信息包也被看作是ESTABLISHED,只要它们是我们所发出的信息的应答。
RELATED RELATED匹配那些属于RELATED连接的包,这句话说了跟没说一样。RELATED状态有点复杂,当一个连接与另一个已经是ESTABLISHED的连接有关时,这个连接就被认为是RELATED。这意味着,一个连接要想成为RELATED,必须首先有一个已经是ESTABLISHED的连接存在。这个ESTABLISHED连接再产生一个主连接之外的新连接,这个新连接就是RELATED状态了,当然首先conntrack模块要能”读懂”它是RELATED。拿ftp来说,FTP数据传输连接就是RELATED与先前已建立的FTP控制连接,还有通过IRC的DCC连接。有了RELATED这个状态,ICMP错误消息、FTP传输、DCC等才能穿过防火墙正常工作。有些依赖此机制的TCP协议和UDP协议非常复杂,他们的连接被封装在其它的TCP或UDP包的数据部分(可以了解下overlay/vxlan/gre),这使得conntrack需要借助其它辅助模块才能正确”读懂”这些复杂数据包
INVALID INVALID说明数据包不能被识别属于哪个连接或没有任何状态。有几个原因可以产生这种情况,比如,内存溢出,收到不知属于哪个连接的ICMP 错误信息。一般地,我们DROP这个状态的任何东西。
UNTRACKED This is the UNTRACKED state. In brief, if a packet is marked within the raw table with the NOTRACK target, then that packet will show up as UNTRACKED in the state machine. This also means that all RELATED connections will not be seen, so some caution must be taken when dealing with the UNTRACKED connections since the state machine will not be able to see related ICMP messages et cetera.

本表格参考自

iptables会在PREROUTING链里从新计算所有的状态。如果我们发送一个流的初始化包,状态就会在OUTPUT链 里被设置为NEW,当我们收到回应的包时,状态就会在PREROUTING链里被设置为ESTABLISHED。如果第一个包不是本地产生的,那就会在PREROUTING链里被设置为NEW状 态。综上,所有状态的改变和计算都是在nat表中的PREROUTING链和OUTPUT链里完成的。

Bridge与netfilter

pass

netfilter与LVS

pass

openstack安全组实现

pass

参考文章

https://wiki.khnet.info/index.php/Conntrack_tuning
https://voipmagazine.wordpress.com/2015/02/27/tuning-the-linux-connection-tracking-system/
https://www.frozentux.net/iptables-tutorial/iptables-tutorial.html#STATEMACHINE

March 24, 2018 12:00 AM

March 23, 2018

OpenStack Superuser

Welcoming what’s new in Kubernetes 1.10

A big congratulations to the Kubernetes team on the 1.10 release slated to land on Monday, March 26.

Here’s a preview of some changes for how Kubernetes interfaces with OpenStack. One of the Kubernetes Enhancement Proposals introduced last year was the creation of a Cloud Controller Manager to allow cloud provider SIGs to directly manage their provider code that interfaces with Kubernetes.

During the 1.10 development cycle, Davanum Srinivas (a.k.a. Dims) maintained a fork of the upstream OpenStack provider code that uses the Cloud Controller Manager interface as a feature-for-feature replacement of the upstream provider. His repository holds more than just the provider code, it also has several other interface drivers developed by the Kubernetes and OpenStack community, including:

  • Two storage drivers that allow you to take advantage of the over 70 drivers available to Cinder over a single interface.
    • A Container Storage Interface (CSI) driver for Cinder
    • A Flex Interface driver for Cinder
  • A webhook-based Keystone authentication and authorization driver for Kubernetes

Coincident with the 1.10 release, the Cloud Provider Working Group is beginning the migration of cloud providers to their own Kubernetes-hosted repositories. Earlier this week, we moved Dims’s external repository into its permanent new Cloud Provider OpenStack home. With that milestone passed, we’re full steam ahead on continuing to support features like block storage and load balancers for Kubernetes running on OpenStack clouds.

In the 1.11 development cycle, the K8s-SIG-OpenStack plans to:

  • Collaborate with WG-Cloud-Provider and SIG-Testing to enable upstream e2e testing of the cloud provider
  • Deprecate the upstream provider in favor of the external provider
  • Continue to support the Flex and CSI drivers
  • Expand Keystone integration with Kubernetes role-based access control
  • Collaborate with the other cloud provider SIGs participating in WG-Cloud-Provider to give users a consistent, tested and well-documented way to interact with any cloud of their choosing

I’m thrilled about the strong support we have for running Kubernetes on OpenStack and even more excited about the developments coming in the weeks and months ahead.

We’re actively looking for people to help out with these efforts and welcome contributions to the provider code, documentation, bugs, feature requests, and installation experiences. SIG-OpenStack has a Slack channel, which is an excellent starting point to get involved.

At the upcoming OpenStack Summit in Vancouver, we’ll have over 60 sessions dedicated to Kubernetes integrations. In addition to the talks, it’s also a great place for some in-person time with contributors and community leaders. I’ll be attending and am looking forward to catching up with old and new friends.

A huge thanks to everyone who has contributed to these efforts, but especially:

  • Dims for his heroic efforts in both communities
  • David Lyle and Robert Morse for helping lead K8s-SIG-OpenStack
  • Fengyun Pan and Yuquan Ren for their numerous feature additions, bug fixes, and provider maintenance
  • Angus Lees for the creation and maintenance of the OpenStack provider
  • Saverio Proto for his numerous blog posts highlighting Kubernetes and OpenStack Integrations
  • Joe Topjian for maintaining the critical Gopher Cloud SDK that underlies the OpenStack provider
  • All of the users and contributors who are running a fully integrated open-source application platform with Kubernetes on OpenStack

Also, thanks goes out to the WG-Cloud-Provider team for helping to build out a common framework for all of the providers, making it possible to ensure a consistent and positive experience no matter what cloud host you use.

Finally, we’re excited to announce the release of the latest Cloud Native Computing Foundation dashboard, now featuring OpenStack as one of the target public clouds. This CI system runs nightly test jobs against CNCF projects. It uses a cross-cloud deployment tool to build a multi-node, highly available Kubernetes cluster to run end-to-end tests against, as well as install and test other projects on like Helm and Prometheus. You can even try the installer out for yourself! A special thanks goes out to Mohammed Naser and Vexxhost for their generous infrastructure contribution for this testing effort.

Congratulations to all on the forthcoming 1.10 release and now on to 1.11!

Chris Hoge is the OpenStack Foundation’s senior strategic program manager.

 

The post Welcoming what’s new in Kubernetes 1.10 appeared first on Superuser.

by Chris Hoge at March 23, 2018 02:13 PM

March 22, 2018

Julio Villarreal Pelegrino

High Performance Computing (HPC) on OpenStack: a few recommendations.

High Performance Computing (HPC) on OpenStack: a few recommendations.

OpenStack is, without doubt, an exciting project and the lead open source Infrastructure-as-a-Service platform. In the last couple years, I had the privilege to architect and deployed dozens of OpenStack clouds for multiple customers and use cases. One of the use cases that I worked on in the last year was High-Performance Computing (HPC) on OpenStack. In this blog, I am going to cover some of the considerations to host high performance and high throughput workloads.

First, let's’ start with three types of architectures that could be used when hosting HPC workloads on OpenStack:

  1. Virtualized HPC on OpenStack
    • In this architecture, all component of the HPC cluster is virtualized in OpenStack.
  2. Bare-metal HPC on OpenStack
    • In this architecture, all components of the HPC cluster are deployed in bare metal servers using OpenStack Ironic.
  3. Virtualized Head Node and Bare-metal compute nodes
    • In this architecture, the head node (scheduler, master and login node) are virtualized in OpenStack, and the compute nodes are deployed in bare metal servers using OpenStack Ironic.

Now that we discussed the 3 types of architecture that we could deploy HPC software in OpenStack, I am going to discuss a few OpenStack best practices when hosting this type of workloads.

Networking

For the networking aspect of OpenStack, there are two recommended configuration options:

  • Provider networks: The OpenStack administrator creates these networks and maps them directly to existing physical networks in the datacenter (L2). Because of the direct attachment to the L2 switching infrastructure, provider networks don’t need to route L3 traffic using the OpenStack control plane, as they should have an L3 gateway in the DC network topology.
  • SRIOV: SRIOV/SR-IOV (single root input/output virtualization) is recommended for HPC workloads based on performance requirements. SR-IOV enables OpenStack to extend the physical NIC’s capabilities directly through to the instance by using the available SRIOV NIC Virtual Functions (VF). Also, support for IEEE 802.1br allows virtual NICs to integrate with, and be managed by, the physical switch.
    • It’s important to mention that in tests conducted by various vendors, results show that SR-IOV can achieve near line rate performance at a low CPU overhead cost per Virtual Machine/Instance.
    • When implementing SRIOV, you need to take in consideration two essential limitations: not been able to use live migrations for instances using VF devices and bypassing OpenStack’s security groups.

Storage

For an HPC architecture, there are two major storage categories to consider:

  • OpenStack storage: image (glance), ephemeral (nova), and volume (cinder).
  • HPC cluster file-based data storage: Used by the HPC cluster to store data.

Based in both categories here are couple recommendations to consider while architecting your cluster:

OpenStack Storage:

  • Glance and Nova: For the Glance and Nova (ephemeral) storage, I like to recommend Ceph. One of the significant advantages of ceph (besides the tight integration with OpenStack) is the performances benefits that you could obtain at instance creation time that image copy-on-write offers with this backend. Another advantage for the ephemeral workloads (not using SRIOV in this case) is the ability to live migrate between the members of the compute cluster.
  • Cinder: For the cinder backend in this HPC use case, I like to recommend Ceph (same benefits apply from the previous point) and NFS/iSCSI backends like NetApp, EMC VNX, and similar systems with supported cinder drivers.

HPC Cluster file-based data storage:

Common used parallel file systems in HPC, like Lustre, GPFS, OrangeFS should be used by accessing them from dedicated SRIOV/Provider networks. Another recommended backend will be Ceph, also providing the access directly from the SRIOV/Provider networks for better performance.

Important Information:
Ceph as a backend, in general, is very flexible. A well-architected Ceph cluster could benefit multiple types of workloads in different configurations/architectures, e.g.:

  • Ethernet-based connectivity could benefit performance by higher throughput NIC interfaces for frontend and backend storage traffic (10/25/40/50/100 Gbps), plus LACP configurations that could double the amount of bandwidth available.
  • Storage servers components could be a combination of NVMe, SSD, SAS and SATA drives. Tailored to provide the required performance IO wise.
  • The distributed nature of the technology provides a flexible and resilient platform.

The next thing to consider after this will be to automate the deployment of your HPC application on OpenStack. For that multiple tools could be used: heat, Ansible, or API calls from an orchestrator system.

Happy HPC on OpenStack hacking!

by Julio Villarreal Pelegrino at March 22, 2018 07:05 PM

OpenStack Superuser

Got two minutes? Get up to speed on bare metal as-a-service

Faster than you can take a navy shower, Cody Hill can tell you what you need to know about bare metal.

“The reason you should care about bare metal as-a-service is that it allows you to provision your Hadoop workloads or SQL workloads or even a hypervisor onto bare metal infrastructure and treat them like cloud instances,” says Hill, who works at Platform9, in a two-minute video.

While Hill notes that bare metal provisioning has been around for a few years, “what really makes OpenStack Ironic different is that it allows you to use a unified way of managing virtual machines as well as physical assets in the exact same workflow.”

It works by booting your systems remotely and then being able to provision a image to it —just like the public cloud boots a virtual machine and provisions an image to it. That way you’re able to utilize the same tooling that you would use to provision virtual machine instances — such as orchestration (Heat) or application catalog (Murano) or even provision block storage volumes (in Cinder) to them — enabling you to treat physical assets just like you would virtual assets.

What are the requirements to get started? OpenStack Ironic requires two things of your servers: the capability to be booted from the intelligent platform management interface (or IPMI access to them) and they need to be able to network or PXE boot, requiring capable network interface cards (NICs.)

If that makes it sound simple, Hill concedes that it’s not. Any bare metal-as-a-service system can be difficult to get up and running, so Platform9 developed an integration for it, which you can check out on their sandbox.

And because there’s always something new-as-a-service, Hill will be speaking at the upcoming OpenStack Summit Vancouver on “Leveraging Serverless Functions in Heat to deliver Anything as as Service (XaaS).”  The talk will cover how Platform9 built a Heat module for Fission, an open source serverless framework built on Kubernetes.

The post Got two minutes? Get up to speed on bare metal as-a-service appeared first on Superuser.

by Superuser at March 22, 2018 04:08 PM

Red Hat Stack

An Introduction to Fast Forward Upgrades in Red Hat OpenStack Platform

OpenStack momentum continues to grow as an important component of hybrid cloud, particularly among enterprise and telco. At Red Hat, we continue to seek ways to make it easier to consume. We offer extensive, industry-leading training, an easy to use installation and lifecycle management tool, and the advantage of being able to support the deployment from the app layer to the OS layer.

One area that some of our customers ask about is the rapid release cycle of OpenStack. And while this speed can be advantageous in getting key features to market faster, it can also be quite challenging to follow for customers looking for stability.

With the release of Red Hat OpenStack Platform 10 in December 2016, we introduced a solution to this challenge – we call it the Long Life release. This type of release includes support for a single OpenStack release for a minimum of three years plus an option to extend another two full years. We offer this via an ELS (Extended Life Support) allowing our customers to remain on a supported, production-grade OpenStack code base for far longer than the usual 6 month upstream release cycle. Then, when it’s time to upgrade, they can upgrade in-place and without additional hardware to the next Long Life release. We aim to designate a Long Life release every third release, starting with Red Hat OpenStack Platform 10 (Newton).

Now, with the upcoming release of Red Hat OpenStack Platform 13 (Queens), we are introducing our second Long Life release. This means we can, finally and with great excitement, introduce the world to our latest new feature: the fast forward upgrade.

Screen Shot 2018-03-22 at 9.33.50 am

Fast forward upgrades take customers easily between Long Life releases. It is a first for our OpenStack distribution, using Red Hat OpenStack Platform director (known upstream as TripleO) and aims to change the game for OpenStack customers. With this feature, you can choose to stay on the “fast train” and get all the features from upstream every six months, or remain on a designated release for longer, greatly easing the upgrade treadmill some customers are feeling.

It’s a pretty clever procedure as it factorizes the process of three consecutive upgrades and therefore greatly reduces the number of steps needed to perform it. In particular, it reduces the number of reboots needed, which in the case of very large deployments makes a huge difference. This capability, combined with the extended support for security vulnerabilities and key backports from future releases, has made Red Hat OpenSack Platform 10 very popular with our customers.

openstack_graphic_large

Red Hat OpenStack Life Cycle dates

Under the covers of a Fast Forward Upgrade

The fast forward upgrade starts like any other upgrade, with a Red Hat OpenStack Platform 10 minor update. A minor update may contain everything from security patches to functional enhancements to even backports from newer releases. There is nothing new about the update procedure for Red Hat OpenStack Platform 10, what changes is the packages that will be included such as kernel changes and other OpenStack specific changes. We’re placing all changes requiring a reboot in this minor update. The update procedure allows for a sequential update of the undercloud, control plane nodes and the compute nodes. It may also include an instance evacuation procedure so that there is no impact to running workloads even if they reside on a node scheduled for reboot after the update. The resulting Red Hat OpenStack Platform 10 cloud will have the necessary operating system components to operate in Red Hat OpenStack Platform 13 without further node reboots.

The next step is the sequential upgrade of the undercloud from Red Hat OpenStack Platform 10, to 11, to 12, to 13. There is no stopping during these steps and in case of a needed rollback of this portion you must return to Red Hat OpenStack Platform 10.

During the fast forward upgrade there are opportunities to perform backups. These should be performed since there’s no automated rewind included but rather a restore from these backups.

A lot of things have changed between Red Hat OpenStack Platform 10 and 13. The most notable is the introduction of OpenStack services in containers. But don’t worry! The fast forward upgrade procedure takes the cloud from a non-containerized deployment to a resulting cloud with OpenStack services running in containers, while abstracting and reducing the complexity of upgrading through the middle releases of Red Hat OpenStack Platform 11 and 12.

In the final steps of the fast forward upgrade procedure, the overcloud will move from Red Hat OpenStack Platform 10 to 13 using a procedure that syncs databases, generates templates for 13, and installs 13’s services in containers. While some of the content for these steps may be part of a release which is no longer supported, Red Hat will provide full support for the required code to perform the upgrade.

What’s next …

In order for this procedure to be supported, it needs to be validated with the released code, and carefully tested in many situations. For this reason, it is scheduled to be ready for testing from Red Hat OpenStack Platform 13 general availability (GA); however, we will warn users launching the procedure that it should not be used in production environments. We encourage you to try the procedure on test environments during this period, and report any issues you find via the normal support procedure.  This will greatly help us ensure that we are covering all cases. During this time support cases relating to fast forward upgrades will not be eligible for high priority response times. 

Once we have thoroughly field tested the procedure, fixed bugs, and are confident that it is ready, we will remove this warning and make an announcement on this same blog. After this happens, it will be OK to proceed with fast forward upgrades in production environments. You can follow this progress of validation and testing by following this blog and staying in touch with your local Red Hat account and support teams.

Stay tuned for future fast forward upgrade blogs where we will dig deeper into the details of this procedure and share experiences and use cases that we’ve tested and validated.

Logotype_Event_RHSummit_RGB_Red

Additionally, we will give an in depth presentation on the fast forward upgrade process at this year’s Red Hat Summit May 8-10 in San Francisco and  OpenStack Summit in Vancouver May 21-24. Please come and visit us in San Francisco and Vancouver for exciting Red Hat news, demos, and direct access to Red Hatters from all over the world. See you there!

by Maria Bracho, Principal Product Manager OpenStack at March 22, 2018 09:15 AM

Opensource.com

Where does OpenStack fit in a public cloud world?

Public clouds are taking over the world. Every day, more and more companies are moving their infrastructure to services like AWS or Microsoft Azure to save capital and operational costs. This begs the question: Where does this leave OpenStack?

In this post, we'll explore how OpenStack is competing in a market dominated by public cloud providers, and how it is positioned to grow in the future, especially in the hybrid cloud business.

by mzamot at March 22, 2018 07:01 AM

OpenStack @ NetApp

Reducing Risk in OpenStack with NetApp Part 1: Encryption

Welcome to part one of a series on reducing risk in OpenStack with NetApp.  Whether you are already running OpenStack, or are considering running it, the security of your data should always be at the forefront of your thoughts.  NetApp has many features that work in concert with our OpenStack drivers to give you piece ... Read more

The post Reducing Risk in OpenStack with NetApp Part 1: Encryption  appeared first on thePub.

by David Blackwell at March 22, 2018 12:40 AM

March 21, 2018

NFVPE @ Red Hat

Spin up a Kubernetes cluster on CentOS, a choose-your-own-adventure

So you want to install Kubernetes on CentOS? Awesome, I’ve got a little choose-your-own-adventure here for you. If you choose to continue installing Kubernetes, keep reading. If you choose to not install Kubernetes, skip to the very bottom of the article. I’ve got just the recipe for you to brew it up. It’s been a year since my last article on installing Kubernetes on CentOS, and while it’s still probably useful – some of the Ansible playbooks we were using have changed significantly. Today we’ll use kube-ansible which is a playbook developed by my team and I to spin up Kubernetes clusters for development purposes. Our goal will be to get Kubernetes up (and we’ll use Flannel as the CNI plugin), and then spin up a test pod to make sure everything’s working swimmingly.

by Doug Smith at March 21, 2018 07:20 PM

About

Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.

Subscriptions

Last updated:
April 20, 2018 09:51 PM
All times are UTC.

Powered by:
Planet