July 20, 2018

OpenStack Superuser

How to set up container-based OpenStack with Open Virtual Network

Open Virtual Network (OVN) is a relatively new networking technology that provides a powerful and flexible software implementation of standard networking functionalities such as switches, routers, firewalls, etc.

Importantly, OVN is distributed in the sense that the aforementioned network entities can be realized over a distributed set of compute/networking resources. OVN is tightly coupled with OVS, essentially being a layer of abstraction which sits above a set of OVS switches and realizes the above networking components across these switches in a distributed manner.

A number of cloud computing platforms and more general compute resource management frameworks are working on OVN support, including oVirt, OpenStack, Kubernetes and Openshift – progress on this front is quite advanced. Interestingly and importantly, one dimension of the OVN vision is that it can act as a common networking substrate which could facilitate integration of more than one of the above systems, although the realization of that vision remains future work.

In the context of our work on developing an edge computing testbed, we set up a modest OpenStack cluster to emulate functionality deployed within an enterprise data center with OVN providing network capabilities to the cluster. This blog post provides a brief overview of the system architecture and notes some issues we had getting it up and running.

As our system is not a production system, providing high availability (HA) support was not one of the requirements; consequently, it was not necessary to consider HA OVN mode. As such, it was natural to host the OVN control services, including the Northbound and Southbound DBs and the Northbound daemon (ovn-northd) on the OpenStack controller node. Because is the node through which external traffic goes, we also needed to run an external facing OVS on this node which required its own OVN controller and local OVS database. Further, as this OVS chassis is intended for external traffic, it needed to be configured with ‘enable-chassis-as-gw‘.

We configured our system to use DHCP provided by OVN; consequently the Neutron DHCP agent was no longer necessary, we removed this process from our controller node. Similarly, L3 routing was done within OVN meaning that the neutron L3 agent was no longer necessary. OpenStack metadata support is implemented differently when OVN is used: instead of having a single metadata process running on a controller serving all metadata requests, the metadata service is deployed on each node and the OVS switch on each node routes requests to 169.254.169.254 to the local metadata agent; this then queries the nova metadata service to obtain the metadata for the specific VM.

The services deployed on the controller and compute nodes are shown in Figure 1 below.


Figure 1: Neutron containers with and without OVN

We used Kolla to deploy the system. Kolla does not currently have full support for OVN; however, specific Kolla containers for OVN have been created (e.g. kolla/ubuntu-binary-ovn-controller:queens, kolla/ubuntu-binary-neutron-server-ovn:queens). Hence, we used an approach which augments the standard Kolla-ansible deployment with manual configuration of the extra containers necessary to get the system running on OVN.

As always, many smaller issues were encountered while getting the system working – we won’t detail all these issues here, but rather focus on the more substantive issues. We divide these into three specific categories: OVN parameters which need to be configured, configuration specifics for the Kolla OVN containers and finally a point which arose due to assumptions made within Kolla that do not necessarily hold for OVN.

To enable OVN, it was necessary to modify the configuration of the OVS switches operating on all the nodes; the existing OVS containers and OVSDB could be used for this – the OVS version shipped with Kolla/Queens is v2.9.0 – but it was necessary to modify some settings. First, it was necessary to configure system-ids for all of the OVS chassis’ – we chose to select fixed UUIDs a priori and use these for each deployment such that we had a more systematic process for setting up the system but it’s possible to use a randomly generated UUID.

docker exec -ti openvswitch_vswitchd ovs-vsctl set open_vswitch . external-ids:system-id="$SYSTEM_ID"

On the controller node, it was also necessary to set the following parameters:

docker exec -ti openvswitch_vswitchd ovs-vsctl set Open_vSwitch . \
external_ids:ovn-remote="tcp:$HOST_IP:6642" \
external_ids:ovn-nb="tcp:$HOST_IP:6641" \
external_ids:ovn-encap-ip=$HOST_IP external_ids:ovn-encap type="geneve" \
external-ids:ovn-cms-options="enable-chassis-as-gw"

docker exec openvswitch_vswitchd ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-ex

On the compute nodes this was necessary:

docker exec -ti openvswitch_vswitchd ovs-vsctl set Open_vSwitch . \
external_ids:ovn-remote="tcp:$OVN_SB_HOST_IP:6642" \
external_ids:ovn-nb="tcp:$OVN_NB_HOST_IP:6641" \
external_ids:ovn-encap-ip=$HOST_IP \
external_ids:ovn-encap-type="geneve"

Having changed the OVS configuration on all the nodes, it was then necessary to get the services operational on the nodes. There are two specific aspects to this: modifying the service configuration files as necessary and starting the new services in the correct way.

Not many changes to the service configurations were required. The primary changes related to ensuring the the OVN mechanism driver was used and letting neutron know how to communicate with OVN. We also used the geneve tunnelling protocol in our deployment and this required the following configuration settings:For the neutron server OVN container

  • ml2_conf.ini
            mechanism_drivers = ovn
     	type_drivers = local,flat,vlan,geneve
     	tenant_network_types = geneve
    
     	[ml2_type_geneve]
     	vni_ranges = 1:65536
     	max_header_size = 38
    
     	[ovn]
     	ovn_nb_connection = tcp:172.30.0.101:6641
     	ovn_sb_connection = tcp:172.30.0.101:6642
     	ovn_l3_scheduler = leastloaded
     	ovn_metadata_enabled = true
    
  • neutron.conf
            core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
     	service_plugins = networking_ovn.l3.l3_ovn.OVNL3RouterPlugin
    

    For the metadata agent container (running on the compute nodes) it was necessary to configure it to point at the nova metadata service with the appropriate shared key as well as how to communicate with OVS running on each of the compute nodes

            nova_metadata_host = 172.30.0.101
     	metadata_proxy_shared_secret = <SECRET>
     	bridge_mappings = physnet1:br-ex
     	datapath_type = system
     	ovsdb_connection = tcp:127.0.0.1:6640
     	local_ip = 172.30.0.101
    

For the OVN specific containers – ovn-northd, ovn-sb and ovn-nb databases, it was necessary to ensure that they had the correct configuration at startup; specifically, that they knew how to communicate with the relevant dbs. Hence, start commands such as

/usr/sbin/ovsdb-server /var/lib/openvswitch/ovnnb.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/ovnnb_db.sock --remote=ptcp:$ovnnb_port:$ovsdb_ip --unixctl=/run/openvswitch/ovnnb_db.ctl --log-file=/var/log/kolla/openvswitch/ovsdb-server-nb.log

were necessary (for the ovn northbound database) and we had to modify the container start process accordingly.

It was also necessary to update the neutron database to support OVN specific versioning information: this was straightforward using the following command:

docker exec -ti neutron-server-ovn_neutron_server_ovn_1 neutron-db-manage upgrade heads

The last issue which we had to overcome was that Kolla and neutron OVN had slightly different views regarding the naming of the external bridges. Kolla-ansible configured a connection between the br-ex and br-int OVS bridges on the controller node with port names phy-br-ex and int-br-ex respectively. OVN also created ports with the same purpose but with different names patch-provnet-<UUID>-to-br-int and patch-br-int-to-provonet-<UUID>; as these ports had the same purpose, our somewhat hacky solution was to manually remove the the ports created in the first instance by Kolla-ansible.

Having overcome all these steps, it was possible to launch a VM which had external network connectivity and to which a floating IP address could be assigned.

Clearly, this approach is not realistic for supporting a production environment, but it’s an appropriate level of hackery for a testbed.

Other noteworthy issues which arose during this work include the following:

  • Standard Docker apparmor configuration in Ubuntu is such that mount cannot be run inside containers, even if they have the appropriate privileges. This has to be disabled or else it is necessary to ensure that the containers do not use the default docker apparmor profile.
  • A specific issue with mounts inside a container which resulted in the mount table filling up with 65,536 mounts and rendering the host quite unusable (thanks to Stefan for providing a bit more detail on this) – the workaround was to ensure that /run/netns was bind mounted into the container.
  • As we used geneve encapsulation, geneve kernel modules had to be loaded
  • Full datapath NAT support is only available for linux kernel 4.6 and up. We had to upgrade the 4.4 kernel which came with our standard ubuntu 16.04 environment.

This is certainly not a complete guide to how to get OpenStack up and running with OVN, but may be useful to some folks who are toying with this. In future, we’re going to experiment with extending OVN to an edge networking context and will provide more details as this work evolves.

This post first appeared on the blog for the ICCLab (Cloud Computing Lab) and the SPLab (Service Prototyping Lab) of the ZHAW Zurich University of Applied Sciences department.

Superuser is always interested in tutorials about open infrastructure, get in touch at editorATopenstack.org.

The post How to set up container-based OpenStack with Open Virtual Network appeared first on Superuser.

by Superuser at July 20, 2018 03:41 PM

Chris Dent

Placement Update 18-29

This is placement update 18-28, a weekly update of ongoing development related to the OpenStack placement service.

Thanks to Jay for providing one of these last week when I was away: http://lists.openstack.org/pipermail/openstack-dev/2018-July/132252.html

Most Important

Feature freeze is next week. We're racing now to get as much of three styles of work done as possible:

  • Effectively managing nested and shared resource providers when managing allocations (such as in migrations).
  • Correctly handling resource provider and consumer generations in the nova-side report client.
  • Supporting reshaping provider trees.

The latter two are actively in progress. Not sure about the first. Anyone?

As ever, we continue to find bugs with existing features that existing tests are not catching. These are being found by people experimenting. So: experiment please.

What's Changed

Most of the functionality and fixes related to consumer generations is in place on the placement side.

We now enforce that consumer identifiers are uuids.

Bugs

Main Themes

Documentation

This is a section for reminding us to document all the fun stuff we are enabling. Open areas include:

  • "How to deploy / model shared disk. Seems fairly straight-forward, and we could even maybe create a multi-node ceph job that does this - wouldn't that be awesome?!?!", says an enthusiastic Matt Riedemann.

  • The whens and wheres of re-shaping and VGPUs.

Consumer Generations

These are in place on the placement side. There's some pending work on using them properly and addresssing some nits:

Reshape Provider Trees

The work to support a /reshaper URI that allows moving inventory and allocations between resource providers is in progress. The database handling (at the bottom of the stack) is pretty much ready, the HTTP API is close except for a small issue with allocation schema, and the nova side is in active progress.

That's all at: https://review.openstack.org/#/q/topic:bp/reshape-provider-tree

Mirror Host Aggregates

This needs a command line tool:

Extraction

I took some time yesterday to experiment with an alternative to the os-resource-classes that jay created. My version is, thus far, just a simple spike that makes symbols pointing to strings, and that's it. I've made a proof of concept of integrating it with placement.

Other extraction things that continue to need some thought are:

  • infra and co-gating issues that are going to come up
  • copying whatever nova-based test fixture we might like

Other

20 entries two weeks ago. 29 now.

End

Thanks to everyone for all their hard work making this happen.

by Chris Dent at July 20, 2018 12:30 PM

Aija Jauntēva

Outreachy: Redfish Message registry and other

This time I will not act surprised that 2 more weeks have passed because I paid attention to time passing by.

In my previous blog post I mentioned that my last patch was failing CI. It turned out that the function assert_called_once is missing in Python 3.5. (it has assert_called_once_with though, but I can't use it this time). Locally I run Python 3.6 where this function is back, and there were no issues in Python 2.7. I replaced this with asserting call_count for now, but this patch still has to pass code reviews.

With that patch put in code review all green, I returned to @Redfish.Settings that had parts left out previously because of too many things that still required clarification. As it stands now, sushy users can update BIOS attributes, but for now sushy does not expose the status of this update. To get the ball rolling I started to write some code and encountered another dependency - Message Registries. In Redfish response there would be id-s of messages, e.g., Base.1.2.Success, Base.1.2.PropertyValueTypeError and in registry file Base.1.2.0.json that would correspond to section like this[1]:

"PropertyValueTypeError": {
    "Description": "Indicates that a property was given the wrong value type, such as when a number is supplied for a property that requires a string.",
    "Message": "The value %1 for the property %2 is of a different type than the property can accept.",
    "Severity": "Warning",
    "NumberOfArgs": 2,
    "ParamTypes": [
        "string",
        "string"
    ],
    "Resolution": "Correct the value for the property in the request body and resubmit the request if the operation failed."
}

In order to determine if update is successful need to consult the registry and give user some friendly message. In the given sample above the message supports templates and it has placeholders for parameters. sushy would have to build an error message passing the parameters from @Redfish.Settings for specific case. This approach also supports translating and localizing the messages. But for all this to work I need the registries. None of the provided mockup files have sample of these registries included. According to the schema they can be provided via ServiceRoot.Registry property. I remember somewhere I read that they are optional, but then how should sushy handle the case where Redfish service does not provide them? There could be 2 options: download the files programmatically from [2] as necessary or include them in sushy package as fallback. Downloading the files wouldn't be a reliable option because sushy might not have access to the external Internet or the site could be just down. Bundling the files together is the direction to go, but then the mentor queried about the license of these files. These standard registry files provided by DMTF have only copyright statement, but no license. That would mean that this is proprietary and cannot be included in OpenStack projects as they require OSI approved license. No-one was sure and I'm not a lawyer either so it was time to contact OpenStack legal mailing list to clarify this[3]. Before this I talked with the mentors what could be other options if the files couldn't be included - e.g., manually or using a script parse the files, generate a Python dict and store this derived dictionary instead of the original file. In the questions to legal mailing list I also included this approach as possible option. Pretty quickly an answer came back which said: NO, the files cannot be included without a license and the same goes for the derived code. As of this writing this is still on-going and DMTF might apply 3-clause BSD license which would be OK for OpenStack project[4].

On other tasks I did some cleanup patches that emerged from previous code reviews - what usually happens in code reviews is that reviewers notice other thing that need improvement but are not related to the patch in review. Or the necessity for changes is not so big to block the patch but can be done as a followup patch. One of those patches were to clean up sushy-tools documentation to consistently use the same term. Somehow the docs started to have 'simulator' to describe sushy-emulator and sushy-static. It might have been me because I though of 'simulator' to be more general term. Went through some discussions [5][6] to understand which is the right term to use. Turns out it is 'emulator'. Which also means that the title of my previous blog post is incorrect.

Another thing, I took over a patch that emulated Ethernet Interfaces in sushy-emulator. It was rather an old patch from January this year and since it was created sushy-tools had introduced support for openstackdriver and thus changed some structure in the Flask app too. I rebased and updated with the new structure and added support for openstacksdk driver. Which led me to setting up OpenStack cloud locally. A bit funny, but I haven't had a need to have access to OpenStack cloud before. This time I needed a sample to see how openstacksdk returns data for network interfaces, which was not entirely clear from the docs. I used dev-stack[7] on a VM and it worked without any problems. This patch too is in code-review.

by ajya at July 20, 2018 09:00 AM

July 19, 2018

OpenStack Superuser

Zuul case study: BMW

Zuul drives continuous integration, delivery and deployment systems with a focus on project gating and interrelated projects. In a series of interviews, Superuser asks users about why they chose it and how they’re using it.

Here Superuser talks to Tobias Henkel, software engineer at BMW, about benefits, challenges and hopes for future development. Henkel will be sharing more details about the case study in a breakout session at the OpenStack Summit Berlin.

The days of annual, monthly or even weekly releases are long gone. How is CI/CD defining new ways to develop and manage software at BMW?

Software has been an integral part of cars for several decades and has become one of the key enablers for many modern safety and comfort features. The amount of software required to implement all these features, as well as the complexity inducted by the many configuration options of current cars, is constantly rising.

The SW architecture in vehicles has evolved from more-or-less independent electronic control units (ECUs) to a set of highly connected functions spread over many ECUs. Without the right strategies, it’s not possible to manage all the required software projects that must converge on a strict schedule to deliver the BMW experience to customers.

The wide adoption of CI/CD in our internal and external development teams is one of the essential tools to deliver and integrate all software components on time with the required quality, despite the complexity of current and future software projects. Today, most BMW software projects rely on CI/CD for automating use cases of their daily work.

Are there specific features that drew you to Zuul?

After using CI/CD systems for many years for an ever-increasing amount of projects, the limitations of the existing CI solutions were starting to impact our software development efforts. With the increasing size and complexity of today’s software projects such as autonomous driving, the scaling capabilities of our CI/CD solution have become a crucial prerequisite of future development.

While scalability is an absolute must-have for our developers, testers and integrators, there are other important requirements for CI/CD:

1. Support for a centrally hosted instance for many projects

2. Support for complex cross-project CI configurations

3. Compatibility with our existing infrastructure

4. An active open-source community

The Zuul solution, especially after release of version 3.0, fully supports all our requirements to provide a centrally hosted solution that can be shared by many internal software projects. This dramatically reduces operations overhead and frees up valuable developer time to continuously improve all aspects of our CI system setup.

Zuul integrates seamlessly with our in-house OpenStack cloud and our repository systems Gerrit and GitHub. It also has an active community and provides the flexibility that our projects need.

How are you currently using Zuul at BMW?

For several years we’ve been operating several Zuul V2 instances for big software projects that need high CI/CD performance. This, of course, came at the cost of operating many instances with similar configurations.

In recent months, we’ve been preparing a centrally hosted CI/CD instance based on Zuul V3 and many projects using previous CI/CD solutions are already in the progress of migrating to the new Zuul V3 instance.

Hosting many projects on one central platform has many advantages for operation overhead and resource sharing in the cloud, but hosting many projects on one CI/CD instance directly translates to high stability and availability.

To maximize the availability of our central CI/CD service, we’re running Zuul, Nodepool and Zookeeper services in an OpenShift cluster, hosted on OpenStack. In addition to improved availability, we’ve seen several development and operation benefits for our internal CI/CD development team.

What other benefits have you seen so far?

The wide adoption of CI/CD in our software projects is the foundation to deliver high-quality software in time by automating every integral part of the development cycle from simple commit checks to full release processes.

With the introduction of Zuul-based CI/CD instances, projects were able to use the cloud resources for their automation in a very dynamic way. As a result, projects can do much more excessive testing in less time, which directly results in higher quality software, while being faster in development and integration.

With the introduction of Zuul V3 we also see a lot of benefits from the operators perspective by providing a centrally hosted CI/CD instance, as opposed to many small ones that have to be managed individually.

What challenges have you overcome?

A centrally hosted CI/CD platform for many projects faces the challenge to support many different use cases or restrictions, that projects inherently have. A common restriction for our projects is the need for non-Linux build nodes, because some required applications or tool chains are only supported for some operating systems like explicit versions of Microsoft Windows.

From the operator’s perspective, we don’t want special solutions for updating the node images, we want Nodepool to automatically do that for us, just like it does for Linux-based images. This required and still requires some extra effort.

Another interesting challenge is the management of the Zuul and Nodepool configuration, or, to be more precise, the responsibilities for managing the configuration. On one hand, we want to provide the projects as much configuration flexibility as possible, but on the other there are still centralized configuration files that we need to manage centrally. One example is the registration of static nodes at the CI system. We’re still working out how to manage these centralized configuration files effectively.

What can you tell us about future plans?

We’re currently migrating many projects to the centrally provided CI/CD instance based on Zuul V3. This instance will be the go-to solution for many existing and new software projects of BMW. We anticipate a continuous growth of project count and sizes, as well as a massive increase of our user base, which includes internal and external project members.

Given the strategic importance of Zuul and Nodepool for our development infrastructure, our main focus will be stability, availability, as well as scalability. While Zuul is already well prepared for most stability and scalability needs, there are still availability improvements required.

The main issue to solve is the removal of all single points of failure by making all services of the Zuul CI system highly available (HA). The CI/CD service should stay fully operational at all times, even if there are issues in single (virtual) machines or even a whole OpenStack availability zone.

What are you hoping the Zuul community continues to focus on or delivers?

While Zuul V3 provides a solution for most of our software projects out of the box, we still see room for improvement.

Our users of the Zuul CI/CD system would appreciate improvements to the Zuul-Web component, e.g. to provide more information on current and past jobs, cancelling a running job or configuring the status page layout per tenant.

However, the highest priority from our perspective is the removal of any single point of failures to support a configuration with high availability.

Anything else you’d like to add?

The obvious point for using the Zuul CI solution at BMW is the comprehensive feature set of Zuul that supports all major use cases for us.

An equally important part of our decision for Zuul is the active and helpful community that drives the development of Zuul and Nodepool.

Our CI/CD development team at BMW is proud to be part of the Zuul community and will continue to be active contributors of the Zuul OSS project.

We would like to thank all Zuul developers and maintainers for their great work.

 

Superuser wants to hear your open infrastructure story, get in touch: editorATopenstack.org

Cover photo: BMW M5 Competition, courtesy BMW press.

The post Zuul case study: BMW appeared first on Superuser.

by Nicole Martinelli at July 19, 2018 03:41 PM

July 18, 2018

Aptira

The Network is the Computer. Part 1 – Foundations: The Unfulfilled Promise

Aptira: What is Open Networking? The Computer is the Network.

As we saw in parts 1 & 2 of this series, Open Networking has a large footprint covering different components of technologies, skills and practices. To help you understand that footprint we’re starting a series of posts that will look at the evolution of these components and show you how they became Open Networking as we define it today.  We start with the Infrastructure domain.

In 1984 at a legendary company made a bold but, at that time, unfulfillable promise:

The network is the computer

John Gage, Sun Microsystems

Sun was born from the explosion of microelectronics advances in the mid 1970’s that initiated the long growth curve described by Moore’s Law. Sun pioneered a range of computing servers and workstations that brought computing power directly to workgroups and smaller enterprises. 

Previously, computing services were provided by large centralised mainframes, where all the computing work was done. Networks merely provided a pipe to get access to that central resource, using largely passive devices such as terminals and printers. 

This architecture produced a centralised and rigid management structure with control over computing. Centralised control had some benefits e.g. security, resource optimisation and budget control, but it also resulted in inflexibility, huge backlogs, lack of engagement with end users and many well-publicised disaster projects. 

The emergence of workgroup servers and workstations delivered computing power independently of the centralised technical and management architecture. Personal Computers (PCs) opened access to computing power to even larger numbers of people when they became commodity products in the early 1980’s. 

At the time, networking and computing were very distinct technology ecosystems which combined in very limited and fixed ways. These two ecosystems were almost completely separated across the supply chain, organisational structure, architectural principles, implementation, and operation. 

The introduction of departmental servers and PC’s helped to circumvent and undermine this centralised control model but didn’t change the basic paradigm of computer and networks as entrenched and disparate ecosystems. In part this was due to the limited and expensive options available from the telecommunications carriers, who operated in highly regulated telecommunications marketplaces that suppressed competition and innovation. 

A great gap was opening in the industry: between the late 1980’s and the early 1990’s the cost of computing rapidly reduced but the cost of connectivity remained high. At the same time the number of physical devices in distributed computing systems exploded as did end-user need. Innovation created products that solved problems at the margin but did not resolve the fundamental gap long term. For example: PC terminal cards, black box protocol emulators, and products that overlaid departmental networks on top of the mainframe terminal networks.  Solutions were possible but, in many cases, very messy. 

Sun had wanted to promote a paradigm of broad computing availability enabled by network integration, but the promise seemed less deliverable than ever. 

The problems were not only in network-land. The sheer numbers of compute devices were causing their own problems, caused by two aspects of the IT marketplace at that time: 

  • Servers were stand-alone devices with their own cabinetry, power supplies and so forth, in part to operate in uncontrolled environments, they were still housed in many cases in data centre-controlled environments for security and ease of operation.
  • Cheap commodity hardware generated a simplistic architecture that instantiated server components physically dedicated to a solution function. E.g. For a cluster of 10 web-servers, you spun up 10 physical devices in a cluster. Likewise, with database servers, email servers, application servers and so forth.

These two factors produced huge numbers of boxes in “server farms”, and data centres struggled to cope with the exploding demand for power, heat and physical space as more and more organisations sought to implement connected computing solutions, and more and more applications were found to use this computing power within organisations. 

Solutions were found that provided the necessary functionality, but compatibility and interoperability were problematic.  The cost of supporting solutions that integrated disparate components grew rapidly. 

How did these problems get solved?  We will cover that in the next post.  Stay tuned. 

The post The Network is the Computer. Part 1 – Foundations: The Unfulfilled Promise appeared first on Aptira.

by Adam Russell at July 18, 2018 10:28 PM

RDO

Community Blog Round-Up: 18 July

We’ve got three posts this week related to OpenStack – Adam Young’s insight on how to verify if a patch has been tested as a reviewer, while Zane Bitter takes a look at OpenStack’s multiple layers of services, and then Nir Yechiel introduces us to the five things we need to know about networking on Red Hat OpenStack Platform 13. As always, if you know of an article not included in this round up, please comment below or track down leanderthal (that’s me! Rain Leander!) on Freenode irc #rdo.

Testing if a patch has test coverage by Adam Young

When a user requests a code review, the review is responsible for making sure that the code is tested. While the quality of the tests is a subjective matter, their presences is not; either they are there or they are not there. If they are not there, it is on the developer to explain why or why not.

Read more at https://adam.younglogic.com/2018/07/testing-patch-has-test/

Limitations of the Layered Model of OpenStack by Zane Bitter

One model that many people have used for making sense of the multiple services in OpenStack is that of a series of layers, with the ‘compute starter kit’ projects forming the base. Jay Pipes recently wrote what may prove to be the canonical distillation (this post is an edited version of my response):

Read more at https://www.zerobanana.com/archive/2018/07/17#openstack-layer-model-limitations

Red Hat OpenStack Platform 13: five things you need to know about networking by Nir Yechiel, Principal Product Manager, Red Hat

Red Hat OpenStack Platform 13, based on the upstream Queens release, is now Generally Available. Of course this version brings in many improvements and enhancements across the stack, but in this blog post I’m going to focus on the five biggest and most exciting networking features found this latest release.–

Read more at https://redhatstackblog.redhat.com/2018/07/12/red-hat-openstack-platform-13-five-things-you-need-to-know-about-networking/

by Rain Leander at July 18, 2018 09:05 AM

July 17, 2018

OpenStack Superuser

Building new foundations: OpenInfra Days Korea

South Korea, already the world’s top producer of mobile phones, displays and semiconductors, is also helping build key conversations around open infrastructure.

That was in evidence at OpenInfra Days Korea 2018,  a two-day conference with technical sessions, deep-dive sessions and hands-on-labs. About 700 people attended the first day; there were 500 attendees on day two.

Bustling crowd at OpenInfra Days Korea 2018. Photo: Seungjin Han.

Seongsoo Cho, Korea User Group co-leader and Seungkyu Ahn, leader of the Korea Kubernetes User Group, kicked off this year’s event emphasizing collaboration between technologies and user groups. The event featured sessions on various open infrastructure topics as well as many Kubernetes-related contributions from the OpenStack Korea User Group, the AWS Korea User Group, the Korea Azure User Group, the Google Cloud Platform Korea User Group and the Korea Developerworks User Group.  You can check out some of these sessions on YouTube.

Highlights from the event include:

  • SK Telecom talking about its participation in the Airship project with AT&T and introduced TACO (SKT All Container OpenStack), an OpenStack solution offering containers, orchestration and automation technologies that features featured self-healing and continuous upgrades.
  • ManTech, a Korean company focused on high availability and disaster recovery, shared their experience of digital transformation with open infrastructure technologies such as Docker and Kubernetes.
  • Samsung Electronics Korea talked the evolution to cloud native with the requirements of 5G telco network and how OpenStack has evolved.
  • Open Source Consulting, a Korean company that offers OpenStack deployments, emphasized stability and agility in businesses and shared their deployment efforts to space information promotion division in Korea and a Korean cryptocurrency exchange.


In comparison to the previous four OpenStack Days Korea events, there was a larger presence of sponsors and sessions from local companies. More than two-thirds of featured sponsor companies (15 out of 22) were Korean compared to 37.5 percent in 2017 and 23 percent in 2016.

The OSF’s Mark Collier, Lauren Sell and Chris Hoge with OpenInfra Days organizers. Photo: Sung Ki Park.

 

This time around, session topics were both broad and deep, ranging from OpenStack to AI infrastructure, GPU, multi- clouds, SDN/NFV, blockchain infrastructure and open hardware using ARM-based servers.

For the first time at the event, the Upstream Institute was offered by the OSF’s Ildiko Vansca and Kendall Nelson for about 15 participants.

The event is organized by the Korean OpenStack community with the help of many dedicated many volunteers. It would be impossible without sharing and transferring know-how and wisdom through three generations of the Korea user group. Many thanks to Ian Choi, Nalee Jang, Jaesuk Ahn, Seongsoo Cho, Taehee Jang, Hochul Shin, Jungwon Ku and all the other organizers and volunteers.

The volunteer crew from day two.

Find out more about the OpenStack User Group and how to get involved here.

Photo // CC BY NC

The post Building new foundations: OpenInfra Days Korea appeared first on Superuser.

by Superuser at July 17, 2018 11:44 PM

Adam Young

Testing if a patch has test coverage

When a user requests a code review, the review is responsible for making sure that the code is tested.  While the quality of the tests is a subjective matter, their presences is not;  either they are there or they are not there.  If they are not there, it is on the developer to explain why or why not.

Not every line of code is testable.  Not every test is intelligent.  But, at a minimum, a test should ensure that the code in a patch is run at least once, without an unexpected exception.

For Keystone and related projects, we have a tox job called cover that we can run on a git repo at a given revision.  For example, I can code review (even without git review) by pulling down a revision using the checkout link in  gerrit, and then running tox:

 

git fetch git://git.openstack.org/openstack/keystoneauth refs/changes/15/583215/2 && git checkout FETCH_HEAD
git checkout -b netloc-and-version
tox -e cover

I can look at the patch using show –stat to see what files were changed:

$ git show --stat
commit 2ac26b5e1ccdb155a4828e3e2d030b55fb8863b2
Author: wangxiyuan 
Date:   Tue Jul 17 19:43:21 2018 +0800

    Add netloc and version check for version discovery
    
    If the url netloc in the catalog and service's response
    are not the same, we should choose the catalog's and
    add the version info to it if needed.
    
    Change-Id: If78d368bd505156a5416bb9cbfaf988204925c79
    Closes-bug: #1733052

 keystoneauth1/discover.py                                 | 16 +++++++++++++++-
 keystoneauth1/tests/unit/identity/test_identity_common.py |  2 +-

and I want to skip looking at any files in keystoneauth1/tests as those are not production code. So we have 16 lines of new code. What are they?

Modifying someone elses’ code, I got to

 git show | gawk 'match($0,"^@@ -([0-9]+),[0-9]+ [+]([0-9]+),[0-9]+ @@",a){left=a[1];right=a[2];next};\
   /^\+\+\+/{print;next};\
   {line=substr($0,2)};\
   /^-/{left++; next};\
   /^[+]/{print right++;next};\
   {left++; right++}'

Which gives me:

+++ b/keystoneauth1/discover.py
420
421
422
423
424
425
426
427
428
429
430
431
432
433
437
+++ b/keystoneauth1/tests/unit/identity/test_identity_common.py
332

Looking in a the cover directory, I can see if a line is uncovered by its class:

class="stm mis"

For example:

$ grep n432\" cover/keystoneauth1_discover_py.html | grep "class=\"stm mis\""

432

For the lines above, I can use a seq to check them, since they are in order (with none missing)

for LN in `seq 420 437` ; do grep n$LN\" cover/keystoneauth1_discover_py.html ; done

Which produces:

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

I drop the grep “class=\”stm mis\”” to make sure I get something, then add it back in, and get no output.

by Adam Young at July 17, 2018 05:35 PM

Chris Dent

TC Report 18-29

Again a relatively slow week for TC discussion. Several members were travelling for one reason or another.

A theme from the past week is a recurring one: How can OpenStack, the community, highlight gaps where additional contribution may be needed, and what can the TC, specifically, do to help?

Julia relayed that question on Wednesday and it meandered a bit from there. Are the mechanics of open source a bit strange in OpenStack because of continuing boundaries between the people who sell it, package it, build it, deploy it, operate it, and use it? If so, how do we accelerate blurring those boundaries? The combined PTG will help, some.

At Thursday's office hours Alan Clark listened in. He's a welcome presence from the Foundation Board. At the last summit in Vancouver members of the TC and the Board made a commitment to improve communication. Meanwhile, back on Wednesday I expressed a weird sense of jealousy of all the nice visible things one sees the foundation doing for the newer strategic areas in the foundation. The issue here is not that the foundation doesn't do stuff for OpenStack-classic, but that the new stuff is visible and over there.

That office hour included more talk about project-need visibility.

Lately, I've been feeling that it is more important to make the gaps in contribution visible than it is to fill them. If we continue to perform above and beyond, there is no incentive for our corporate value extractors to supplement their investment. That way lies burnout. The health tracker is part of making things more visible. So are OpenStack wide goals. But there is more we can do as a community and as individuals. Don't be a hero: If you're overwhelmed or overworked tell your peers and your management.

In other news: Zane summarized some of his thoughts about Limitations of the Layered Model of OpenStack. This is a continuation of the technical vision discussions that have been happening on an etherpad and email thread.

by Chris Dent at July 17, 2018 04:17 PM

OpenStack Superuser

Airship: Making life cycle management repeatable and predictable

Although the name brings to mind a dirigible, Airship is a actually collection of loosely coupled, interoperable open-source tools that are nautically themed. It’s got a stellar crew working together — AT&T, South Korea Telecom, Intel and the OpenStack Foundation — with the aim of making life cycle management for open infrastructure repeatable and predictable.

Here’s a stem-to-stern rundown of what it is and what it can do for you.

What is Airship?

Airship provides for automated cloud provisioning and life cycle management in a declarative, predictable way, says AT&T’s lead cloud architect Alan Meadows. The telecom giant has been building on the foundation laid by the OpenStack-Helm project since 2017. Airship components help create sites, perform minor updates, make configuration changes and manage major software uplifts like OpenStack upgrades. The focus of the project has been on the implementation of a declarative platform to introduce OpenStack on Kubernetes and the life cycle management of the resulting cloud. “Simply put,” says Meadows, “the goal is to take bare mental infrastructure from data center loading dock to a functioning OpenStack cloud and then care for it beyond.”

Why use Airship?

Airship comes from the perspective that all site definitions should be completely declarative. Teams can manage YAML documents and submit them to the site where they are fully realized by software. End users don’t have to necessarily know what has changed in a YAML document. The full document bundle is submitted, then Airship figures out what needs to be done, whether that’s an initial site deployment or a simple update to the already existing site. Airship uses containers to deliver software, providing a required level of simultaneous coexistence and separation, letting teams handle any number of isolated dependency requirements. It also lets them progress containers that we release through development testing and finally production without changing them so they know what’s being deployed every time.

Airship allows for a single workflow for all life cycle management. That lets teams interact with the system in the exact same way whether it’s during the initial site deployment or future upgrades to live sites. Finally, Airship is a flexible solution. It’s a single platform that can support large cloud installations, small edge sites, and different software-defined-networks. It can also manage more than just OpenStack installations.

What tools make up the Airship project?

It must have been fun to bestow all of these projects with such swashbuckling names: Treasure Map is a documentation project. It outlines the various sub-projects and provides a reference architecture for the platform. Drydock is the bare-metal provisioning engine that works with Promenade to label and join the host to a Kubernetes cluster.

Divingbell is a lightweight solution for bare metal configuration for when elements of an operating system needs some extra non-containerized configuration. Deckhand acts like a centralized site design document storage back-end, keeping revision history of all site designs and changes. That allows teams to reuse common site design settings across different types of sites (and keeps the YAML documents compact with reusable elements). Armada is the enterprise solution, letting teams articulate every piece of software to run in a site, where to pull those charts from, in what order that will to be deployed, how to group them, how to test them, and how to roll them back if there’s a failure — all in a set of documents that teams can life cycle. Airship also uses Berth as a Kubernetes virtual machine runner, Promenade as a resilient, production-ready Kubernetes cluster generator and Pegleg, a tool to manage, build, aggregate, validate and test YAML documents for Airship document sets. You can check out the entire repository here.

What does an Airship-built cloud look like?

The image above shows during the presentation visually explains how Airship manages the entire life cycle of a project. Starting at the bottom and working up, there’s a bare metal host and a host operating system and network and storage configuration that is declared and laid down by Drydock. Promenade handles effectively two aspects of Kubernetes management: the manufacturing of the cluster and its life cycle, then it handles docker and Kubla configurations on new hosts as well as joining them to the cluster and labeling them appropriately so they receive the right workloads.

Airship then instantiates SEF and Calico  using Helm parts orchestrated by Armada to provide storage and network capabilities to Kubernetes and the rest of the Airship components. Finally, OpenStack-Helm is used to provide a functioning OpenStack installation. It also logs, monitors, and handles alerts. “The entire stack is installed and life cycled in the exact same way,” says Meadows, “including Airship itself.”

You can see the entire presentation from the OpenStack Summit Vancouver including Rodolfo Pacheco’s explanatory animations and Seungkyu Ahn’s case study from South Korea, below.

Photo // CC BY NC

The post Airship: Making life cycle management repeatable and predictable appeared first on Superuser.

by Rob LeFebvre at July 17, 2018 03:20 PM

Zane Bitter

Limitations of the Layered Model of OpenStack

One model that many people have used for making sense of the multiple services in OpenStack is that of a series of layers, with the ‘compute starter kit’ projects forming the base. Jay Pipes recently wrote what may prove to be the canonical distillation (this post is an edited version of my response):

Nova, Neutron, Cinder, Keystone and Glance are a definitive lower level of an OpenStack deployment. They represent a set of required integrated services that supply the most basic infrastructure for datacenter resource management when deploying OpenStack. Depending on the particular use cases and workloads the OpenStack deployer wishes to promote, an additional layer of services provides workload orchestration and workflow management capabilities.

I am going to explain why this viewpoint is wrong, but first I want to acknowledge what is attractive about it (even to me). It contains a genuinely useful observation that leads to a real insight.

The insight is that whereas the installation instructions for something like Kubernetes usually contain an implicit assumption that you start with a working datacenter, the same is not true for OpenStack. OpenStack is the only open source project concentrating on the gap between a rack full of unconfigured equipment and somewhere that you could run a higher-level service like Kubernetes. We write the bit where the rubber meets the road, and if we do not there is nobody else to do it! There is an almost infinite variety of different applications and they will all need different parts of the higher layers, but ultimately they must be reified in a physical data center and when they are OpenStack will be there: that is the core of what we are building.

It is only the tiniest of leaps from seeing that idea as attractive, useful, and genuinely insightful to believing it is correct. I cannot really blame anybody who made that leap. But an abyss awaits them nonetheless.

Back in the 1960s and early 1970s there was this idea about Artificial Intelligence: even a 2 year old human can (for example) recognise images with a high degree of accuracy, but doing (say) calculus is extremely hard in comparison and takes years of training. But computers can already do calculus! Ergo, we have solved the hardest part already and building the rest out of that will be trivial, AGI is just around the corner, and so on. The popularity of this idea arguably helped created the AI bubble, and the inevitable collision with the reality of its fundamental wrongness led to the AI Winter. Because, in fact, though you can build logic out of many layers of heuristics (as human brains do), it absolutely does not follow that it is trivial to build other things that also require layers of heuristics out of some basic logic building blocks. (In contrast, the AI technology of the present, which is showing more promise, is called Deep Learning because it consists literally of multiple layers of heuristics. It is also still considerably worse at it than any 2 year old human.)

I see the problem with the OpenStack-as-layers model as being analogous. (I am not suggesting there will be a full-on OpenStack Winter, but we are well past the Peak of Inflated Expectations.) With Nova, Keystone, Glance, Neutron, and Cinder you can build a pretty good Virtual Private Server hosting service. But it is a mistake to think that cloud is something you get by layering stuff on top of VPS hosting. It is relatively easy to build a VPS host on top of a cloud, just like teaching someone calculus. But it is enormously difficult to build a cloud on top of a VPS host (it would involve a lot of expensive layers of abstraction, comparable to building artificial neurons in software).

That is all very abstract, so let me bring in a concrete example. Kubernetes is event-driven at a very fundamental level: when a pod or a whole kubelet dies, Kubernetes gets a notification immediately and that prompts it to reschedule the workload. In contrast, Nova/Cinder/&c. are a black hole. You cannot even build a sane dashboard for your VPS—let alone cloud-style orchestration—over them, because it will have to spend all of its time polling the APIs to find out if anything happened. There is an entire separate project, that almost no deployments include, basically dedicated to spelunking in the compute node without Nova’s knowledge to try to surface this information. It is no criticism of the team in question, who are doing something that desperately needs doing in the only way that is really open to them, but the result is an embarrassingly bad architecture for OpenStack as a whole.

So yes, it is sometimes helpful to think about the fact that there is a group of components that own the low level interaction with outside systems (hardware, or IdM in the case of Keystone), and that almost every application will end up touching those directly or indirectly, while each using different subsets of the other functionality… but only in the awareness that those things also need to be built from the ground up as interlocking pieces in a larger puzzle.

Saying that the compute starter kit projects represent a ‘definitive lower level of an OpenStack deployment’ invites the listener to ignore the bigger picture; to imagine that if those lower level services just take care of their own needs then everything else can just build on top. That is a mistake, unless you believe that OpenStack needs only to provide enough building blocks to build VPS hosting out of, because support for all of those higher-level things does not just fall out for free. You have to consciously work at it.

Imagine for a moment that, knowing everything we know now, we had designed OpenStack around a system of event sources and sinks that are reliable in the face of hardware failures and network partitions, with components connecting into it to provide services to the user and to each other. That is what Kubernetes did. That is the key to its success. We need to enable something similar, because OpenStack is still necessary even in a world where Kubernetes exists.

One reason OpenStack is still necessary is the one we started with above: something needs to own the interaction with the underlying physical infrastructure, and the alternatives are all proprietary. Another place where OpenStack can provide value is by being less opinionated and allowing application developers to choose how the event sources and sinks are connected together. That means that users should, for example, be able to customise their own failover behaviour in ‘userspace’ rather than rely on the one-size-fits-all approach of handling everything automatically inside Kubernetes. This is theoretically an advantage of having separate projects instead of a monolithic design—though the fact that the various agents running on a compute node are more tightly bound to their corresponding services than to each other has the potential to offer the worst of both worlds.

All of these thoughts will be used as fodder for writing a technical vision statement for OpenStack. My hope is that will help align our focus as a community so that we can work together in the same direction instead of at cross-purposes. Along the way, we will need many discussions like this one to get to the root of what can be some quite subtle differences in interpretation that nevertheless lead to divergent assumptions. Please join in if you see one happening!

by Zane Bitter at July 17, 2018 03:17 PM

Stackmasters team

Join us for OpenStack’s 8th Birthday Celebrations!

It’s celebration time! OpenStack is turning 8 years old and if that doesn’t call for a grand get together of “Stackers” then we don’t know what does.

OpenStack: 8th Birthday Celebrations!

Here at Stackmasters, as always, we will be inviting the OpenStack community in Greece to our shard office space at Starttech Ventures HQ in Athens to join in the celebrations.

Here are the juicy details: all-comers are welcome to join us this Wednesday, July 18, from 19:00 – 21:00 at Starttech Ventures. There are still a number of open spots of the meetup, but they are going like hotcakes so RSVP here if you want to join the fun.

OpenStack: onwards and upwards

The Birthday meetup is an event we love to host each July. And it’s not just about celebrating the progress of the OpenStack project with those working with the technology, but to discuss best practices, tricks of the trade and a few zany stories that many of you often have.

Each year we share the latest news, trends and speculations of the OpenStack project with our local community members, as well as all its users. Additionally, we always welcome anyone interested in technology and the development of Open Infrastructure projects with open arms.

There are similar events organized by local OpenStack communities around the world in coordination with the OpenStack Foundation, so wherever you are you are sure to find one near you.

Athens OpenStack MeetUp Agenda

As is usually the case, it will be a laid-back affair. Here is the agenda of what we have in store for you:

18:45: Welcome and Introduction

19:00: 8 Years of OpenStack – Review by Thanassis Parathyras

19:45: Open discussion and call for future meetings

See you there!

As always, it will be our absolute pleasure to see those of you who have worked or want to get to know OpenStack better, as well as those who want to share the experiences you have with OpenStack and its greater community. A big shout out to the sponsors of the OpenStack Foundation, Stackmasters and Starttech Ventures.

RSVP today. And be there or be square!

Join us for OpenStack’s 8th Birthday Celebrations! was last modified: July 17th, 2018 by Stackmasters

The post Join us for OpenStack’s 8th Birthday Celebrations! appeared first on Stackmasters.

by Stackmasters at July 17, 2018 08:10 AM

July 16, 2018

OpenStack Superuser

The power of a moment: Making a difference through mentoring

I’ve been one of the lucky few who has never known a time without a mentor.

I entered the tech community through the Open Cloud Academy, a program in San Antonio run by Rackspace that teaches system administration to anyone who wants to learn regardless of technical background. During my time at the OCA, a handful of volunteers came by to help us learn how to troubleshoot and teach us what it means to be a sys admin. I didn’t know it at the time, but they were my first mentors.

Merriam-Webster defines a mentor as a trusted counselor or guide, but if I’m going to be honest (and I always try to be), for anyone who has had a mentor, the word means so much more. However, I also believe that we’ve been looking at mentorship in the wrong light for far too long. We view mentorships with these formal dynamics and put as much pressure on them to succeed as we do on business deals. That’s why so many mentorships never go beyond the planning stages. I once heard a saying that people come into your life for a reason, a season or a lifetime and that’s the perspective we need to adopt for mentorship.

When Rackers came by the OCA to help, they did so for a reason; they were there to offer guidance to what everyone hoped would be the next crop of Rackspace rookies. There was no formal mentorship program, no official ask of these employees. Some came because they have a volunteer’s heart. Others because they had once been students at the OCA and wanted to repay the favor.

As time passed, it became evident which pairing of students and volunteers worked best. We began to see groups working closer together and exchanging information so they could continue to collaborate outside the OCA. Though some would argue that’s not mentorship, I would argue that it’s time to expand the definition of mentor.

I believe a mentor is someone who can not only help you recognize your strengths, but also help you see the areas where you need to develop to achieve your goals. It’s challenging to create this type of relationship instantly, and because of this, we need to be open to allowing that dynamic to grow organically. Sometimes the first steps to a long-term mentorship can be a few moments where someone gives their time to help foster the growth of someone else. When this happens, the next step requires a moment of vulnerability from the mentee. They have to admit that they could benefit from additional help and then take the risk of asking for it. It can be as simple as saying “I learned so much in our time together. Would you mind if I followed up with you if I have more questions?” This simple comment can help establish the reason.

We view mentorships with these formal dynamics and put as much pressure on them to succeed as we do on business deals. That’s why so many never go beyond the planning stages.

This moment of vulnerability lets the mentor know that their time was well spent and that their effort has value. However, as mentees, our jobs are not done. We need to make an effort to continue our journey on our own, to put what we have learned to work.

I recently had a chance to test my own advice when I decided to learn to program by writing a small Python script to keep track of my grocery list. When I ran the idea by a colleague, I could instantly see the look of panic, accompanied by the comment “I wish I could help but I already have so much on my plate.” I’ve seen this look before; he thought I was asking to take on the responsibility of teaching me. So a few weeks later, when I ran into a few issues where the code didn’t behave how I thought it should, I took the time to prepare my questions before approaching him for help.

Instead of saying, “Can you help me with my code?” or “Ugh, this is broken, and I don’t know why. Could you help?” I narrowed down the area in which the issue was occurring and asked, “The logic in this if-then statement does not seem to be occurring the way I thought it would. If I send it to you, could you look it over?”   The phrasing of the question allowed him to understand that I wasn’t asking for a significant amount of time debugging my entire program or even asking to teach me how to code. I was merely asking for a few minutes of troubleshooting. The result was better than expected. My colleague was enthusiastic about how much I’d managed to accomplish on my own and he explained how my indentations caused a break in the logic of the code. He also offered a few other suggestions on how to improve the script.

A week later, after running into another issue implementing one of his recommendations, I asked if he’d look again. Because I’d previously shown that I was not only willing to put the work in myself but also valued input, his willingness to help was genuine. The mentorship relationship had been seeded without any formal commitment. In the two months since I decided to learn to program, this colleague has found a renewed interest in strengthening his own Python skills along with becoming one of my most prominent advocates and mentors.

The point of sharing my story is to challenge you to not look at mentorship as a formal agreement between two people, but to look for it in the small moments where someone offers to help. Be willing to open up and perhaps feel a moment of vulnerability by admitting how much you could benefit from continued support. The OpenStack and Open source community is full of talented, caring individuals who want to help the community grow; we have to be willing to invest the time to help these relationships evolve.

How to get involved

The OpenStack Mentoring program was recently relaunched.  The new mentoring program will focus on providing mentorship through goal-focused cohorts of mentors. This change will allow mentoring responsibilities to be shared among each group’s mentors. Mentoring cohorts allow for mentees to access a broader knowledge base while enabling them to form connections and help other members of the group. Sign up as a mentor or mentee here.<

About the author

Ell Marquez has been part of the open-source family for a few years now. In this time, she has found the support needed from her mentorship relationships to grow from a Linux Administrator to an OpenStack technical trainer at Rackspace. Recently, she took the leap to join Linux Academy as a technical evangelist.

Have you been a mentor or mentee? Superuser wants to hear your story, get in touch: editorATsuperuser.org

Photo // CC BY NC

The post The power of a moment: Making a difference through mentoring appeared first on Superuser.

by Ell Marquez at July 16, 2018 03:13 PM

Cisco Cloud Blog

Building a User Group Everyone Wants to Join

I’m fairly in awe of Lisa-Marie Namphy. She runs the world’s largest OpenStack user group and has had a hand in starting and/or running at least a dozen other tech meetups—all while holding down a very real full-time job...

by Ali Amagasu at July 16, 2018 01:11 PM

July 13, 2018

OpenStack Superuser

Integrating Tungsten Fabric with OpenStack-Ansible

Tungsten Fabric (formerly OpenContrail) is a multi-cloud, multi-stack software-defined solution sponsored by the Linux Foundation.

In a nutshell, Tungsten Fabric and Contrail, the commercial product based on TF, can replace and augment many of the networking components of a standard OpenStack cloud and provides features such as:

  • Distributed virtual routing
  • DHCP and Metadata services
  • Policy-based access control
  • Compatibility with security groups
  • … and more

The forwarding plane supports MPLS over GRE, VXLAN, L2/L3 unicast and L3 multicast for interconnection between virtual and physical networks. A Tungsten Fabric architecture overview can be found at http://www.opencontrail.org/opencontrail-architecture-documentation/.

I’ve recently undertaken the challenge of integrating Tungsten Fabric into OpenStack-Ansible to ease the deployment of TF and associated OpenStack bits in a production-grade OpenStack cloud. This blog will cover, at a high-level, some patches to the master (Rocky) branch of OpenStack-Ansible as well as some tweaks to the contrail-ansible-deployer playbooks provided by Juniper and the TF community for deploying Tungsten Fabric. The process defined here is by no means meant as the final process, and is probably considered a bit clunky and less than ideal. But, it’s a start.

I will use the terms Tungsten Fabric, OpenContrail, and Contrail interchangably throughout this blog. My apologies in advance!

Requirements

A few weeks ago I deployed a standalone 3-node OpenContrail setup with an OpenStack-Ansible all-in-one node based on Queens. After figuring out some of the tweaks necessary to get things to a semi-working state, I decided to try my hand and deploying an AIO node that contained a single instance of Tungsten Fabric services as well as base OpenStack services.

The following minimum specs are recommended:

  • System: Ubuntu VM
  • OS: 16.04.4 LTS
  • RAM: 48GB
  • Disk: 300GB
  • NIC: Single

A baremetal node may provide less complications, as I will point out later, but a virtual machine on ESXi or some other hypervisor should do the trick.

Getting started w/ OpenStack-Ansible

To get started, clone the OpenStack-Ansible repository. At time of writing, the master branch was tied to Rocky, the 18th release of OpenStack.

# git clone https://git.openstack.org/openstack/openstack-ansible /opt/openstack-ansible
# cd /opt/openstack-ansible
# git checkout master
# export ANSIBLE_ROLE_FETCH_MODE=git-clone

Next, run the bootstrapping scripts:

# scripts/bootstrap-ansible.sh
# scripts/bootstrap-aio.sh

The bootstrapping scripts will download the playbooks to deploy OpenStack, and will also prepare the networking on the machine to meet the OpenStack-Ansible architecture.

Role modifications

Making changes to an OpenStack cloud deployed with OpenStack-Ansible often means making changes to Ansible roles that make up the deployment. This includes changes to tasks, templates, variables, and more.

The roles I needed to modify include:

  • os_neutron
  • os_nova

Whether all of these role changes were necessary remains to be seen, but they’re here for your viewing pleasure.

os_neutron

Some new files include:

root@aio1:/etc/ansible/roles/os_neutron# git diff --staged
diff --git a/tasks/providers/opencontrail_config.yml b/tasks/providers/opencontrail_config.yml
new file mode 100644
index 0000000..8f5fc7d
--- /dev/null
+++ b/tasks/providers/opencontrail_config.yml
@@ -0,0 +1,99 @@
+---
+# Copyright 2018, Rackspace Hosting, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Set the packages to install
+  set_fact:
+    neutron_optional_combined_pip_packages: |-
+      {% set packages = neutron_optional_opencontrail_pip_packages %}
+      {{ packages }}
+
+- name: Install OpenContrail pip packages
+  pip:
+    name: "{{ neutron_optional_combined_pip_packages }}"
+    state: "{{ neutron_pip_package_state }}"
+    virtualenv: "{{ neutron_bin | dirname }}"
+    virtualenv_site_packages: "no"
+    extra_args: >-
+      {{ neutron_developer_mode | ternary(pip_install_developer_constraints | default('--constraint /opt/developer-pip-constraints.txt'), '') }}
+      {{ (pip_install_upper_constraints is defined) | ternary('--constraint ' + pip_install_upper_constraints | default(''),'') }}
+      {{ pip_install_options | default('') }}
+  register: install_packages
+  until: install_packages|success
+  retries: 5
+  delay: 2
+  tags:
+    - opencontrail-install
+    - opencontrail-pip-packages
+
+- name: Install git
+  apt:
+    name: git
+    state: present
+  delegate_to: "{{ item }}"
+  with_items:
+    - "{{ groups['neutron_server'] }}"
+  tags:
+    - opencontrail-install
+
+- name: Clone contrail neutron plugin
+  git:
+    repo: "{{ opencontrail_plugin_git_repo }}"
+    version: "{{ opencontrail_plugin_git_install_branch }}"
+    dest: /opt/contrail-neutron-plugin
+    force: yes
+  register: contrail_plugin_git_clone
+  delegate_to: "{{ item }}"
+  with_items:
+    - "{{ groups['neutron_server'] }}"
+  until: contrail_plugin_git_clone|success
+  retries: 5
+  delay: 2
+  tags:
+    - opencontrail-install
+
+# (jamesdenton) Will need to eventually compile and/or extract from Docker container
+# The tasks marked (temp) should be reworked
+
+- name: Download Contrail python libraries (temp)
+  vars:
+  - dlpath: https://github.com/busterswt/contrail-openstack/raw/master
+  get_url:
+    url: "{{ dlpath }}/{{ item }}"
+    dest: /opt
+    mode: 0440
+  with_items:
+    - contrail-openstack-neutron-init.tar
+  tags:
+    - opencontrail-install
+
+- name: Unpack Contrail python libraries (temp)
+  unarchive:
+    remote_src: yes
+    src: /opt/contrail-openstack-neutron-init.tar
+    dest: /openstack/venvs/neutron-{{ neutron_venv_tag }}/lib/python2.7/site-packages
+  when:
+    inventory_hostname == groups['neutron_server'][0]
+  tags:
+    - opencontrail-install
+
+- name: Install contrail neutron plugin into venv
+  command: "/openstack/venvs/neutron-{{ neutron_venv_tag }}/bin/python setup.py install"
+  args:
+    chdir: /opt/contrail-neutron-plugin
+  delegate_to: "{{ item }}"
+  with_items:
+    - "{{ groups['neutron_server'] }}"
+  tags:
+    - opencontrail-install
diff --git a/templates/plugins/opencontrail/ContrailPlugin.ini.j2 b/templates/plugins/opencontrail/ContrailPlugin.ini.j2
new file mode 100644
index 0000000..9d645b0
--- /dev/null
+++ b/templates/plugins/opencontrail/ContrailPlugin.ini.j2
@@ -0,0 +1,23 @@
+# {{ ansible_managed }}
+
+{% if neutron_plugin_type == 'opencontrail' %}
+[APISERVER]
+api_server_ip = {{ opencontrail_api_vip_address }}
+api_server_port = {{ opencontrail_api_vip_port }}
+multi_tenancy = True
+contrail_extensions = ipam:neutron_plugin_contrail.plugins.opencontrail.contrail_plugin_ipam.NeutronPluginContrailIpam,policy:neutron_plugin_contrail.plugins.opencontrail.contrail_plugin_policy.NeutronPluginContrailPolicy,route-table:neutron_plugin_contrail.plugins.opencontrail.contrail_plugin_vpc.NeutronPluginContrailVpc,contrail:None,service-interface:None,vf-binding:None
+
+[COLLECTOR]
+analytics_api_ip = {{ opencontrail_collector_vip_address }}
+analytics_api_port = {{ opencontrail_collector_vip_port }}
+
+[keystone_authtoken]
+auth_host = {{ internal_lb_vip_address }}
+auth_port = {{ keystone_service_port }}
+auth_protocol = {{ keystone_service_proto }}
+admin_user = {{ keystone_admin_user_name }}
+admin_password = {{ keystone_auth_admin_password }}
+admin_tenant_name = {{ keystone_admin_tenant_name }}
+insecure = True
+region_name = {{ keystone_service_region }}
+{% endif %}

Changes to existing files include:

root@aio1:/etc/ansible/roles/os_neutron# git diff
diff --git a/defaults/main.yml b/defaults/main.yml
index 162e933..7054c96 100644
--- a/defaults/main.yml
+++ b/defaults/main.yml
@@ -63,6 +63,8 @@ networking_bgpvpn_git_repo: https://git.openstack.org/openstack/networking-bgpvp
 networking_bgpvpn_git_install_branch: master
 openstack_ceilometer_git_repo: https://git.openstack.org/openstack/ceilometer
 openstack_ceilometer_git_install_branch: master
+opencontrail_plugin_git_repo: https://github.com/Juniper/contrail-neutron-plugin
+opencontrail_plugin_git_install_branch: master

 # Developer mode
 neutron_developer_mode: false
@@ -164,6 +166,7 @@ neutron_sriov_nic_agent_ini_overrides: {}
 neutron_sriov_nic_agent_init_overrides: {}
 neutron_vpn_agent_init_overrides: {}
 neutron_vpnaas_agent_ini_overrides: {}
+neutron_opencontrail_conf_ini_overrides: {}

 ###
 ### Quotas
@@ -434,3 +437,12 @@ ovs_nsh_support: False

 # Set higher priority to mardim PPA when ovs_nsh_support is True
 ovs_nsh_apt_pinned_packages: [{ package: "*", release: "LP-PPA-mardim-mardim-ppa"}]
+
+###
+### Contrail/OpenContrail/Tungsten Fabric Configuration
+###
+
+opencontrail_api_vip_address: "{{ external_lb_vip_address }}"
+opencontrail_api_vip_port: "8082"
+opencontrail_collector_vip_address: "{{ external_lb_vip_address }}"
+opencontrail_collector_vip_port: "8081"
diff --git a/templates/neutron.conf.j2 b/templates/neutron.conf.j2
index 83d25a7..dd755ca 100644
--- a/templates/neutron.conf.j2
+++ b/templates/neutron.conf.j2
@@ -42,6 +42,10 @@ core_plugin = {{ neutron_plugin_core }}
 {% if neutron_plugin_type.split('.')[0] == 'ml2' %}
 service_plugins = {{ neutron_plugin_loaded_base | join(',') }}
 {% endif %}
+{% if neutron_plugin_type == 'opencontrail' %}
+service_plugins = neutron_plugin_contrail.plugins.opencontrail.loadbalancer.v2.plugin.LoadBalancerPluginV2
+api_extensions_path = /openstack/venvs/neutron-{{ neutron_venv_tag }}/lib/python2.7/site-packages/neutron_plugin_contrail/extensions:/openstack/venvs/neutron-{{ neutron_venv_tag }}/lib/python2.7/site-packages/neutron_lbaas/extensions
+{% endif %}

 # MAC address generation for VIFs
 base_mac = fa:16:3e:00:00:00
@@ -94,8 +98,9 @@ rpc_workers = {{ neutron_rpc_workers }}

 {% set dhcp_agents_max = num_agent if num_agent > 2 else 2 %}
 # DHCP
-{% if neutron_plugin_type == 'ml2.dragonflow' %}
-# In dragonflow, DHCP is fully distributed, and DHCP agents are not used
+{% if neutron_plugin_type == ('ml2.dragonflow' or 'opencontrail') %}
+# In dragonflow and opencontrail, DHCP is fully distributed and DHCP
+# agents are not used
 dhcp_agent_notification = False
 {% else %}
 dhcp_agent_notification = True
diff --git a/vars/main.yml b/vars/main.yml
index cef4ee8..2d1c2a2 100644
--- a/vars/main.yml
+++ b/vars/main.yml
@@ -121,6 +121,10 @@ neutron_plugins:
     plugin_ini: plugins/ml2/ml2_conf.ini
     driver_interface: "openvswitch"
     l3_agent_mode: "legacy"
+  opencontrail:
+    plugin_core: neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2
+    plugin_ini: plugins/opencontrail/ContrailPlugin.ini
+    plugin_conf_ini_overrides: "{{ neutron_opencontrail_conf_ini_overrides }}"

 ###
 ### ML2 Plugin Configuration
diff --git a/vars/source_install.yml b/vars/source_install.yml
index a246a45..24e57ea 100644
--- a/vars/source_install.yml
+++ b/vars/source_install.yml
@@ -96,6 +96,13 @@ neutron_proprietary_nuage_pip_packages:
   - nuage-openstack-neutronclient
   - nuagenetlib

+neutron_optional_opencontrail_pip_packages:
+  - bitarray
+  - bottle
+  - geventhttpclient
+  - psutil>=0.6.0
+  - requests>=1.1.0
+
 neutron_developer_constraints:
   - "git+{{ neutron_git_repo }}@{{ neutron_git_install_branch }}#egg=neutron"
   - "git+{{ neutron_fwaas_git_repo }}@{{ neutron_fwaas_git_install_branch }}#egg=neutron-fwaas"

os_nova

root@aio1:/etc/ansible/roles/os_nova# git diff
diff --git a/defaults/main.yml b/defaults/main.yml
index 67d92e9..bc44511 100644
--- a/defaults/main.yml
+++ b/defaults/main.yml
@@ -325,6 +325,9 @@ nova_network_services:
   calico:
     use_forwarded_for: True
     metadata_proxy_enabled: False
+  opencontrail:
+    use_forwarded_for: True
+    metadata_proxy_enabled: True

 # Nova quota
 nova_quota_cores: 20

HAproxy changes

In order to provide a single TF API and dashboard endpoint, I made a decision to create a VIP that would balance traffic amongst the TF API and analytics services. Whether this is best practice remains to be seen, but the changes to group_vars that facilitate VIP creation are here:

root@aio1:/opt/openstack-ansible# git diff
diff --git a/inventory/group_vars/haproxy/haproxy.yml b/inventory/group_vars/haproxy/haproxy.yml
index b837443..dc53ef4 100644
--- a/inventory/group_vars/haproxy/haproxy.yml
+++ b/inventory/group_vars/haproxy/haproxy.yml
@@ -36,6 +36,7 @@ haproxy_rabbitmq_management_whitelist_networks: "{{ haproxy_whitelist_networks }
 haproxy_repo_git_whitelist_networks: "{{ haproxy_whitelist_networks }}"
 haproxy_repo_cache_whitelist_networks: "{{ haproxy_whitelist_networks }}"
 haproxy_opendaylight_whitelist_networks: "{{ haproxy_whitelist_networks }}"
+haproxy_opencontrail_whitelist_networks: "{{ haproxy_whitelist_networks }}"

 haproxy_default_services:
   - service:
@@ -365,3 +366,23 @@ haproxy_default_services:
       haproxy_backend_httpcheck_options:
         - expect status 405
       haproxy_service_enabled: "{{ (groups['ceph-rgw'] is defined and groups['ceph-rgw'] | length > 0) or (ceph_rgws | length > 0) }}"
+  - service:
+      haproxy_service_name: opencontrail-api
+      haproxy_backend_nodes: "{{ groups['opencontrail-api_hosts'] | default([]) }}"
+      haproxy_bind: "{{ [opencontrail_api_vip_address] }}"
+      haproxy_port: 8082
+      haproxy_balance_type: tcp
+      haproxy_timeout_client: 5000s
+      haproxy_timeout_server: 5000s
+      haproxy_whitelist_networks: "{{ haproxy_opencontrail_whitelist_networks }}"
+      haproxy_service_enabled: "{{ neutron_plugin_type == 'opencontrail' }}"
+  - service:
+      haproxy_service_name: opencontrail-collector
+      haproxy_backend_nodes: "{{ groups['opencontrail-analytics_hosts'] | default([]) }}"
+      haproxy_bind: "{{ [opencontrail_collector_vip_address] }}"
+      haproxy_port: 8081
+      haproxy_balance_type: tcp
+      haproxy_timeout_client: 5000s
+      haproxy_timeout_server: 5000s
+      haproxy_whitelist_networks: "{{ haproxy_opencontrail_whitelist_networks }}"
+      haproxy_service_enabled: "{{ neutron_plugin_type == 'opencontrail' }}"

Some issues I ran into with this approach on an all-in-one include the ability for HAproxy to bind the VIP to port 8081. Turns out that later on in the process, the Contrail playbooks create a listener on 0.0.0.0:8081 that keeps the VIP from binding on the same port. An alternative here would be to comment out that service, or disable it once HAproxy is deployed. For a multi-node installation where HAproxy is on a different node, it can remain. Load balancing these two services may not work out in the end, but I’ll leave it for now.

Environment changes

The default Neutron env.d skeleton will deploy Neutron agent containers that are not necessary as part of the deployment. By overriding the default, we can remove the agent containers and define a few new components:

cat <<'EOF' >> /etc/openstack_deploy/env.d/neutron.yml
component_skel:
  neutron_server:
    belongs_to:
      - neutron_all
  opencontrail_vrouter:
    belongs_to:
    - neutron_all
  opencontrail_api:
    belongs_to:
    - neutron_all
  opencontrail_analytics:
    belongs_to:
    - neutron_all

container_skel:
  neutron_agents_container:
    contains: {}
  opencontail-vrouter_container:
    belongs_to:
      - compute_containers
    contains:
      - opencontrail_vrouter
    properties:
      is_metal: true
  opencontrail-api_container:
    belongs_to:
      - opencontrail-api_containers
    contains:
      - opencontrail_api
    properties:
      is_metal: true
  opencontrail-analytics_container:
    belongs_to:
      - opencontrail-analytics_containers
    contains:
      - opencontrail_analytics
    properties:
      is_metal: true
  neutron_server_container:
      belongs_to:
        - network_containers
      contains:
        - neutron_server

physical_skel:
  opencontrail-api_containers:
    belongs_to:
      - all_containers
  opencontrail-api_hosts:
    belongs_to:
      - hosts
  opencontrail-analytics_containers:
    belongs_to:
      - all_containers
  opencontrail-analytics_hosts:
    belongs_to:
      - hosts
EOF

I am not entirely sure at this point if those will all be necessary and if that’s the proper approach, but it’s there for now.

In the openstack_user_config.yml file I defined two new groups with the intention of being able to split out API and Analytics services across hosts. Since this is an AIO, the same IP is defined:

opencontrail-api_hosts:
  aio1:
    ip: 172.29.236.100

opencontrail-analytics_hosts:
  aio1:
    ip: 172.29.236.100

Overrides

Changes to the os_neutron role resulted in new defaults being added and some overrides being required. In user_variables.yml the following was added:

neutron_plugin_type: opencontrail
neutron_driver_quota: neutron_plugin_contrail.plugins.opencontrail.quota.driver.QuotaDriver
opencontrail_plugin_git_install_branch: R5.0

Some defaults specified in the role include:

opencontrail_api_vip_address: "{{ external_lb_vip_address }}"
opencontrail_api_vip_port: "8082"
opencontrail_collector_vip_address: "{{ external_lb_vip_address }}"
opencontrail_collector_vip_port: "8081"
opencontrail_plugin_git_repo: https://github.com/Juniper/contrail-neutron-plugin
opencontrail_plugin_git_install_branch: master
neutron_opencontrail_conf_ini_overrides: {}

The final requirements are yet to be determined.

Running the OpenStack-Ansible playbooks

At this point, most of the changes required to OpenStack-Ansible playbooks have been made and OpenStack can be deployed:

# cd /opt/openstack-ansible/playbooks
# openstack-ansible setup-hosts.yml
# openstack-ansible setup-infrastructure.yml
# openstack-ansible setup-openstack.yml

While Neutron will be laid down, some additional changes will need to be made and Contrail deployed before networking can be used. Horizon can be used but may not be fully-functional.

Next, we must perform some actions that include cloning the Juniper ansible playbook repo, setting up overrides, and running those playbooks to install Contrail and overlay some Docker containers for Contrail-related services.

Deploying Tungsten Fabric / OpenContrail / Contrail

At this point of the process, OpenStack has been deployed and is aware of the implemented networking backend but the backend doesn’t exist. Juniper provides playbooks that can deploy OpenContrail using Docker containers. Those same playbooks can also deploy an OpenStack release based on Kolla, another OpenStack deployment strategy. We really only need the Contrail-related bits here.

To implement a few changes, I cloned the repo and pushed the required changes needed to deploy on top of an OpenStack-Ansible based cloud.

Clone the repo:

# git clone http://github.com/busterswt/contrail-ansible-deployer /opt/openstack-ansible/playbooks/contrail-ansible-deployer
# cd /opt/openstack-ansible/playbooks/contrail-ansible-deployer
# git checkout osa

Overrides

The Juniper playbooks rely on their own inventory and overrides, so performing these next few steps may feel a bit redundant. I have not yet reworked the playbooks to utilize the inventory in place for OpenStack-Ansible.

These overrides can be defined in a new file at /etc/openstack_deploy/user_opencontrail_vars.yml:

config_file: /etc/openstack_deploy/user_opencontrail_vars.yml
opencontrail_api_vip_address: ""
opencontrail_collector_vip_address: ""
provider_config:
  bms:
    ssh_pwd:
    ssh_user: root
    ssh_public_key: /root/.ssh/id_rsa.pub
    ssh_private_key: /root/.ssh/id_rsa
    ntpserver: 129.6.15.28
instances:
  aio1:
    provider: bms
    ip: 172.29.236.100
    roles:
      config_database:
      config:
      control:
      analytics_database:
      analytics:
      webui:
      vrouter:
	    VROUTER_GATEWAY: 10.50.0.1
		PHYSICAL_INTERFACE: ens160
global_configuration:
  CONTAINER_REGISTRY: opencontrailnightly
#  CONTAINER_REGISTRY: hub.juniper.net/contrail
#  CONTAINER_REGISTRY_USERNAME: 
#  CONTAINER_REGISTRY_PASSWORD: 
contrail_configuration:
  CLOUD_ORCHESTRATOR: openstack
  CONTRAIL_VERSION: latest
#  CONTRAIL_VERSION: 5.0.0-0.40
# UPGRADE_KERNEL: true
  UPGRADE_KERNEL: false
  KEYSTONE_AUTH_HOST: ""
  KEYSTONE_AUTH_PUBLIC_PORT: 5000
  KEYSTONE_AUTH_PROTO: http
  KEYSTONE_AUTH_URL_VERSION:
  KEYSTONE_AUTH_ADMIN_USER: admin
  KEYSTONE_AUTH_ADMIN_PASSWORD: ""
  KEYSTONE_AUTH_URL_VERSION: /v3
  CONTROLLER_NODES: 172.29.236.100
  CONTROL_NODES: 172.29.236.100
  ANALYTICSDB_NODES: 172.29.236.100
  WEBUI_NODES: 172.29.236.100
  ANALYTICS_NODES: 172.29.236.100
  CONFIGDB_NODES: 172.29.236.100
  CONFIG_NODES: 172.29.236.100

Take special note of the following key/value pairs:

VROUTER_GATEWAY: 10.50.0.1
PHYSICAL_INTERFACE: ens160

CONTAINER_REGISTRY: opencontrailnightly
#  CONTAINER_REGISTRY: hub.juniper.net/contrail
#  CONTAINER_REGISTRY_USERNAME: 
#  CONTAINER_REGISTRY_PASSWORD: 
CONTRAIL_VERSION: latest
#  CONTRAIL_VERSION: 5.0.0-0.40

CONTROLLER_NODES: 172.29.236.100
CONTROL_NODES: 172.29.236.100
ANALYTICSDB_NODES: 172.29.236.100
WEBUI_NODES: 172.29.236.100
ANALYTICS_NODES: 172.29.236.100
CONFIGDB_NODES: 172.29.236.100
CONFIG_NODES: 172.29.236.100

The first section defines the “physical” interface that will be repurposed/leveraged by the vhost0 interface required for vRouter. This being an all-in-one node with a single NIC means chances are good the playbooks will automatically determine which interface to use. However, I’ve gone ahead and called it out here. The IP address of this host is 10.50.0.221 and the gateway is 10.50.0.1. For a multi-homed host, it may be possible to dedicate an interface to the vRouter along with its own gateway address. I have yet to do a multi-NIC deploy, but look forward to that being the case.

The second section defines the Docker registry from which the containers will be downloaded. The opencontailnightly registry contains containers that are build every one to two days, and may or may not work on any given day. If you have access to the GA registry through Juniper, you can define that registry instead and provide access credentials. The only version available on the nightly registry is latest, while the Juniper registry may have tagged releases. Use what’s appropriate.

The third section defines address for the respective node types, and are used throughout the playbooks to plug values into templates. These are currently required for the playbooks to complete successfully. Note that I’ve used the ‘CONTAINER_NET’ address here. The idea is that Contrail and OpenStack would communicate on the existing container network used by LXC. In this AIO, 172.29.236.100 is the IP configured on br-mgmt, and happens to also be the internal VIP address. Services deployed in Docker containers will bind ports to this IP for their respective service. How this changes in a multi-node install remains TBD.

OSA repo changes

OpenStack-Ansible includes a repo server for package management that currently inhibits the installation of certain versions of packages needed to deploy OpenContrail. Here I’ve disabled the use of the internal repo server by modifying /root/.pip/pip.conf:

[global]
disable-pip-version-check = true
timeout = 120
#index-url = http://172.29.236.100:8181/simple
#trusted-host =
#        172.29.236.100

[install]
upgrade = true
upgrade-strategy = only-if-needed
pre = true

Running the playbooks

At this point, we should be in a good spot to begin running the Contrail playbooks.

First, the node(s) must be bootstrapped:

# cd /opt/openstack-ansible/playbooks/
# openstack-ansible -e orchestrator=openstack contrail-ansible-deployer/playbooks/configure_instances.yml

When UPGRADE_KERNEL is true, the host may reboot. If this happens, just rerun the playbook again. When the Contrail nodes are not the deploy host, the playbooks have a timer that waits for the host(s) to return.

Next, run the install_contrail.yml playbook to deploy OpenContrail:

# cd /opt/openstack-ansible/playbooks/
# openstack-ansible -e orchestrator=openstack contrail-ansible-deployer/playbooks/install_contrail.yml

Install additional components

Because the Juniper playbooks expect a Kolla-based deployment, some of the components are not installed on top of our OpenStack-Ansible based cloud. I’ve taken the liberty of extracting some utilities, modules, and more from Docker containers and wrote a playbook to lay those down:

# cd /opt/openstack-ansible/playbooks/
# openstack-ansible contrail-ansible-deployer/playbooks/install_contrailtools.yml

Reboot

Once OpenContrail has been deployed and the vRouter kernel module compiled and inserted, it may be necessary to reboot the host to clear up some issues encounted inside the vRouter agent container, including:

contrail-vrouter-agent: controller/src/vnsw/agent/vrouter/ksync/ksync_memory.cc:107: void KSyncMemory::Mmap(bool): Assertion `0' failed.
/entrypoint.sh: line 304: 28142 Aborted                 (core dumped) $@

I found that rebooting the host post-install was enough for the issue to go away.

Post-Deploy

Now that OpenContrail is (hopefully) installed, use the contrail-status command to check the status of the services:

root@aio1:/opt/openstack-ansible/playbooks# contrail-status
Pod        Service         Original Name                          State    Status
analytics  api             contrail-analytics-api                 running  Up 21 minutes
analytics  collector       contrail-analytics-collector           running  Up 21 minutes
analytics  nodemgr         contrail-nodemgr                       running  Up 21 minutes
analytics  query-engine    contrail-analytics-query-engine        running  Up 21 minutes
config     api             contrail-controller-config-api         running  Up 24 minutes
config     cassandra       contrail-external-cassandra            running  Up 27 minutes
config     device-manager  contrail-controller-config-devicemgr   running  Up 24 minutes
config     nodemgr         contrail-nodemgr                       running  Up 24 minutes
config     rabbitmq        contrail-external-rabbitmq             running  Up 27 minutes
config     schema          contrail-controller-config-schema      running  Up 24 minutes
config     svc-monitor     contrail-controller-config-svcmonitor  running  Up 24 minutes
config     zookeeper       contrail-external-zookeeper            running  Up 27 minutes
control    control         contrail-controller-control-control    running  Up 23 minutes
control    dns             contrail-controller-control-dns        running  Up 23 minutes
control    named           contrail-controller-control-named      running  Up 23 minutes
control    nodemgr         contrail-nodemgr                       running  Up 23 minutes
database   cassandra       contrail-external-cassandra            running  Up 22 minutes
database   nodemgr         contrail-nodemgr                       running  Up 22 minutes
database   zookeeper       contrail-external-zookeeper            running  Up 22 minutes
vrouter    agent           contrail-vrouter-agent                 running  Up 19 minutes
vrouter    nodemgr         contrail-nodemgr                       running  Up 19 minutes
webui      job             contrail-controller-webui-job          running  Up 23 minutes
webui      web             contrail-controller-webui-web          running  Up 23 minutes

vrouter kernel module is PRESENT
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active

== Contrail database ==
nodemgr: active
zookeeper: active
cassandra: active

== Contrail analytics ==
nodemgr: active
api: initializing (UvePartitions:UVE-Aggregation[Partitions:0] connection down)
collector: initializing (KafkaPub:172.29.236.100:9092 connection down)
query-engine: active

== Contrail webui ==
web: active
job: active

== Contrail vrouter ==
nodemgr: active
agent: active

== Contrail config ==
api: active
zookeeper: active
svc-monitor: active
nodemgr: active
device-manager: active
cassandra: active
rabbitmq: active
schema: active

If you see nodemgr services in an Initializing state with (NTP state unsynchronized.), try restarting NTP on the host. It may be necessary to define multiple NTP servers. I have not yet worked out the Analytics issues, but hope to resolve that soon.

Issues

I’d like to say that things are completely working at this point, but they’re not! Using the openstack or neutron clients may fail to work, as the neutron-server service is likely failing with the following error:

Unrecoverable error: please check log for details.: ExtensionsNotFound: Extensions not found: ['route-table']. Related to WARNING neutron.api.extensions [-] Extension file vpcroutetable.py wasn't loaded due to cannot import name attributes.

In Rocky, one of the deprecated modules that the Contrail plugin replies has finally been removed. Fret not, as it was rolled into another module/file.

In the neutron_server container, I modified the /openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron_plugin_contrail/extensions/vpcroutetable.py file like so:

-from neutron.api.v2 import attributes as attr
+from neutron_lib.api import attributes as attr

A restart of the neutron-server service is needed:

# systemctl restart neutron-server

Testing

To validate that OpenContrail has been configured to utilize Keystone authentication, open up the Contrail UI in a browser using the external VIP address and port 8143:

Specify the admin user and the password defined in the openrc file. The domain is default. If authentication is successful, the dashboard should appear:

Switching back to OpenStack, we can perform an openstack network list to see the networks:

root@aio1-utility-container-ee37a935:~# openstack network list
+--------------------------------------+-------------------------+---------+
| ID                                   | Name                    | Subnets |
+--------------------------------------+-------------------------+---------+
| 723e67c1-8ccd-43ba-a6f4-8b2399c1b8d2 | __link_local__          |         |
| 5a2947ce-0030-4d2a-a06a-76b0d6934102 | ip-fabric               |         |
| 5f4b0153-8146-4e9c-91d4-c60364ece6bc | default-virtual-network |         |
+--------------------------------------+-------------------------+---------+

Those networks were all created by the Contrail plugin/driver and should not be removed.

Create a test network:

root@aio1-utility-container-ee37a935:~# openstack network create test_network_green
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   | None                                 |
| availability_zones        | None                                 |
| created_at                | None                                 |
| description               | None                                 |
| dns_domain                | None                                 |
| id                        | d9e0507f-5ef4-4b62-bf69-176340095053 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | None                                 |
| is_vlan_transparent       | None                                 |
| mtu                       | None                                 |
| name                      | test_network_green                   |
| port_security_enabled     | True                                 |
| project_id                | e565909917a5463b867c5a7594a7612f     |
| provider:network_type     | None                                 |
| provider:physical_network | None                                 |
| provider:segmentation_id  | None                                 |
| qos_policy_id             | None                                 |
| revision_number           | None                                 |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | None                                 |
+---------------------------+--------------------------------------+

Notice the provider attributes are not specified. These are no longer important to us. Other attributes may not be supported by the Contrail plugin.

Create the subnet:

root@aio1-utility-container-ee37a935:~# openstack subnet create --subnet-range 172.23.0.0/24 --network test_network_green test_subnet_green
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 172.23.0.2-172.23.0.254              |
| cidr              | 172.23.0.0/24                        |
| created_at        | None                                 |
| description       | None                                 |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 172.23.0.1                           |
| host_routes       |                                      |
| id                | cc2d2f56-5c87-49fb-afd5-14e32feccd6a |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | test_subnet_green                    |
| network_id        | d9e0507f-5ef4-4b62-bf69-176340095053 |
| project_id        | e565909917a5463b867c5a7594a7612f     |
| revision_number   | None                                 |
| segment_id        | None                                 |
| service_types     | None                                 |
| subnetpool_id     | None                                 |
| tags              |                                      |
| updated_at        | None                                 |
+-------------------+--------------------------------------+

IPv6 should be supported, but I ran into issues trying to create an IPv6 subnet. At this point, our network is ready for VMs. For good measure, I created a security group that could be applied to our instance that would allow SSH:

root@aio1-utility-container-ee37a935:~# openstack security group create allow_ssh
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| created_at      | None                                 |
| description     | allow_ssh                            |
| id              | 39a9e241-27c3-452a-b37a-80b6dcbbf783 |
| name            | allow_ssh                            |
| project_id      | e565909917a5463b867c5a7594a7612f     |
| revision_number | None                                 |
| rules           |                                      |
| tags            | []                                   |
| updated_at      | None                                 |
+-----------------+--------------------------------------+

root@aio1-utility-container-ee37a935:~# openstack security group rule create --dst-port 22 allow_ssh

+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| created_at        | None                                 |
| description       | None                                 |
| direction         | ingress                              |
| ether_type        | IPv4                                 |
| id                | b8393e4d-1d9d-47e9-877e-86374f38dca1 |
| name              | None                                 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| project_id        | e565909917a5463b867c5a7594a7612f     |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| revision_number   | None                                 |
| security_group_id | 39a9e241-27c3-452a-b37a-80b6dcbbf783 |
| updated_at        | None                                 |
+-------------------+--------------------------------------+

I then booted the instance using a tiny flavor and a CirrOS image:

root@aio1-utility-container-ee37a935:~# openstack server create --image cirros --flavor test_flavor --nic net-id=test_network_green --security-group allow_ssh test1
+-------------------------------------+----------------------------------------------------+
| Field                               | Value                                              |
+-------------------------------------+----------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                             |
| OS-EXT-AZ:availability_zone         |                                                    |
| OS-EXT-SRV-ATTR:host                | None                                               |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                               |
| OS-EXT-SRV-ATTR:instance_name       |                                                    |
| OS-EXT-STS:power_state              | NOSTATE                                            |
| OS-EXT-STS:task_state               | scheduling                                         |
| OS-EXT-STS:vm_state                 | building                                           |
| OS-SRV-USG:launched_at              | None                                               |
| OS-SRV-USG:terminated_at            | None                                               |
| accessIPv4                          |                                                    |
| accessIPv6                          |                                                    |
| addresses                           |                                                    |
| adminPass                           | a8tghwSoTWZP                                       |
| config_drive                        |                                                    |
| created                             | 2018-06-18T14:34:49Z                               |
| flavor                              | test_flavor (5c0600b7-f9fe-46f3-8af5-f8390ee5c6f3) |
| hostId                              |                                                    |
| id                                  | b14d1861-8855-4d17-a2d3-87eb67a3d81c               |
| image                               | cirros (4006fd58-cdc5-4bd8-bc25-ef73be1cd429)      |
| key_name                            | None                                               |
| name                                | test1                                              |
| progress                            | 0                                                  |
| project_id                          | e565909917a5463b867c5a7594a7612f                   |
| properties                          |                                                    |
| security_groups                     | name='39a9e241-27c3-452a-b37a-80b6dcbbf783'        |
| status                              | BUILD                                              |
| updated                             | 2018-06-18T14:34:49Z                               |
| user_id                             | f6aac1aa53294659998aa71838133a1d                   |
| volumes_attached                    |                                                    |
+-------------------------------------+----------------------------------------------------+

root@aio1-utility-container-ee37a935:~# openstack server list
+--------------------------------------+-------+--------+-------------------------------+--------+-------------+
| ID                                   | Name  | Status | Networks                      | Image  | Flavor      |
+--------------------------------------+-------+--------+-------------------------------+--------+-------------+
| b14d1861-8855-4d17-a2d3-87eb67a3d81c | test1 | ACTIVE | test_network_green=172.23.0.3 | cirros | test_flavor |
+--------------------------------------+-------+--------+-------------------------------+--------+-------------+

Now, I could attach to the instance’s console and attempt outbound connectivity:

Within the Contrail UI, I was able to enable snat on the network to allow the vRouter to snat outbound connections from the VM:

A quick test showed ping working:

Inbound connections to the VM are also possible, but require some additional work within Contrail to advertise the VMs address. I have a Cisco ASA 1001 in my lab that has been configured to peer with the Contrail controller, but will have to save that configuration demonstration for another day.

Summary

There is still a lot of work to do to not only learn how Tungsten Fabric / OpenContrail / Contrail operates, but to also construct best practices around how it should be deployed within an OpenStack-Ansible based cloud. Some of the components used for the installation are heavily wrapped up in Docker containers and had to be extracted before deploying in the LXC container and/or the host(s). This isn’t scalable, but is good enough for now.

I ran into issues with the opencontrailnightly build recently, in that the vRouter was dropping outbound or response traffic from my VM. Resorting to a GA build from Juniper’s repo solved the issue, but not everyone may have that access.

Another problem I encountered was that while pings to/from the VM worked (with the ASR bits in place), SSH connections failed. In fact, any TCP connection failed. The SYNs were seen at the instance, and SYN/ACK was observed going out. However, the SYN/ACK never made it past the vRouter. A packet capture showed that the SYN/ACK had an invalid checksum. Disabling generic IP checksums on the “physical” interface of the host, ens160 in this case, allowed things to work. This article was super helpful:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB30500

As I get more reps with this I hope to simplify the process and one day get this moved upstream for inclusion into OpenStack-Ansible. Until then, good luck!

Photo // CC BY NC

The post Integrating Tungsten Fabric with OpenStack-Ansible appeared first on Superuser.

by James Denton at July 13, 2018 04:52 PM

Red Hat Stack

Red Hat OpenStack Platform 13: five things you need to know about networking

Red Hat OpenStack Platform 13, based on the upstream Queens release, is now Generally Available. Of course this version brings in many improvements and enhancements across the stack, but in this blog post I’m going to focus on the five biggest and most exciting networking features found this latest release.

franck-v-705445-unsplash
Photo by Franck V. on Unsplash

ONE: Overlay network management – bringing consistency and better operational experience

Offering solid support for network virtualization was always a priority of ours. Like many other OpenStack components, the networking subsystem (Neutron) is pluggable so that customers can choose the solution that best fits their business and technological requirements. Red Hat OpenStack Platform 13 adds support for Open Virtual Network (OVN), a network virtualization solution which is built into the Open vSwitch (OVS) project. OVN supports the Neutron API, and offers a clean and distributed implementation of the most common networking capabilities such as bridging, routing, security groups, NAT, and floating IPs. In addition to OpenStack, OVN is also supported in Red Hat Virtualization (available with Red Hat Virtualization 4.2 which was announced earlier this year), with support for Red Hat OpenShift Container Platform expected down the road. This marks our efforts to create consistency and a more unified operational experience between Red Hat OpenStack Platform, Red Hat OpenShift, and Red Hat Virtualization.     

OVN was available as a technology preview feature with Red Hat OpenStack Platform 12, and is now fully supported with Red Hat OpenStack Platform 13. OVN must be enabled as the overcloud Neutron backend from Red Hat OpenStack Platform director during deployment time, as the default Neutron backend is still ML2/OVS. Also note that migration tooling from ML2/OVS to OVN is not supported with Red Hat OpenStack Platform 13, and is expected to be offered in a future release, and so OVN is only recommended for new deployments.

TWO: Open source SDN Controller

OpenDaylight is a flexible, modular, and open software-defined networking (SDN) platform, which is now fully integrated and supported with Red Hat OpenStack Platform 13. The Red Hat offering combines carefully selected OpenDaylight components that are designed to enable the OpenDaylight SDN controller as a networking backend for OpenStack, giving it visibility into, and control over, OpenStack networking, utilization, and policies.

OpenDaylight is co-engineered and integrated with Red Hat OpenStack Platform, including Red Hat OpenStack Platform director for automated deployment, configuration and lifecycle management.

The key OpenDaylight project used in this solution is NetVirt, offering support for the OpenStack Neutron API on top of OVS. For telecommunication customers this support extends to OVS-DPDK implementations. Also available in technology preview, customers can leverage OpenDaylight with OVS hardware offload on capable network adapters to offload the virtual switch data path processing to the network card, further optimizing the server footprint.

 

2OpenStack_OpenDaylight-Product-Guide_437720_0217_ECE_Architecture

THREE: Cloud ready load balancing as a service

Load balancing is a fundamental service of any cloud. It is a key element essential for enabling automatic scaling and availability of applications hosted in the cloud, and is required for both “three tier” apps, as well as for emerging cloud native, microservices based, app architectures.

During the last few development cycles, the community has worked on a new load balancing as a service (LBaaS) solution based on the Octavia project. Octavia provides tenants with a load balancing API, as well as implements the delivery of load balancing services via a fleet of service virtual machine instances, which it spins up on demand. With Red Hat OpenStack Platform 13, customers can use the OpenStack Platform director to easily deploy and setup Octavia and expose it to the overcloud tenants, including setting up a pre-created, supported and secured Red Hat Enterprise Linux based service VM image.

OpenStack_Networking-Guide_450456_0617_ECE_LBaaS
Figure 2. Octavia HTTPS traffic flow through to a pool member

FOUR: Integrated networking for OpenStack and OpenShift

OpenShift Container Platform, Red Hat’s enterprise distribution of Kubernetes optimized for continuous application development, is infrastructure independent. You can run it on public cloud, virtualization, OpenStack or anything that can boot Red Hat Enterprise Linux. But in order to run Kubernetes and application containers, you need control and flexibility at scale on the infrastructure level. Many of our customers are looking into OpenStack as a platform to expose VM and bare metal resources for OpenShift to provide Kubernetes clusters to different parts of the organization – nicely aligning with the strong multi-tenancy and isolation capabilities of OpenStack as well as its rich APIs.     

As a key contributor to both OpenStack and Kubernetes, Red Hat is shaping this powerful combination so that enterprises can not only deploy OpenShift on top of OpenStack, but also take advantage of the underlying infrastructure services exposed by OpenStack. A good example of this is through networking integration. Out of the box, OpenStack provides overlay networks managed by Neutron. However, OpenShift, based on Kubernetes and the Container Network Interface (CNI) project, also provides overlay networking between container pods. This results in two, unrelated, network virtualization stacks that run on top of each other and make the operational experience, as well as the overall performance of the solution, not optimal. With Red Hat OpenStack Platform 13, Neutron was enhanced so that it can serve as the networking layer for both OpenStack and OpenShift, allowing a single network solution to serve both container and non-container workloads. This is done through project Kuryr and kuryr-kubernetes, a CNI plugin that provides OpenStack networking to Kubernetes objects.

Customers will be able to take advantage of Kuryr with an upcoming Red Hat OpenShift Container Platform release, where we will also release openshift-ansible support for automated deployment of Kuryr components (kuryr-controller, kuryr-cni) on OpenShift Master and Worker nodes.   

Screen Shot 2018-07-12 at 3.13.30 pm
Figure 3. OpenShift and OpenStack

FIVE: Deployment on top of routed networks

As data center network architectures evolve, we are seeing a shift away from L2-based network designs towards fully L3 routed fabrics in an effort to create more efficient, predictable, and scalable communication between end-points in the network. One such trend is the adoption of leaf/spine (Clos) network topology where the fabric is composed of leaf and spine network switches: the leaf layer consists of access switches that connect to devices like servers, and the spine layer is the backbone of the network. In this architecture, every leaf switch is interconnected with each and every spine switch using routed links. Dynamic routing is typically enabled throughout the fabric and allows the best path to be determined and adjusted automatically. Modern routing protocol implementations also offers Equal-Cost Multipathing (ECMP) for load sharing of traffic between all available links simultaneously.

Originally, Red Hat OpenStack Platform director was designed to use shared L2 networks between nodes. This significantly reduces the complexity required to deploy OpenStack, since DHCP and PXE booting are simply done over a shared broadcast domain. This also makes the network switch configuration straightforward, since typically there is only a need to configure VLANs and ports, but no need to enable routing between all switches. This design, however, is not compatible with L3 routed network solutions such as the leaf/spine network architecture described above.

With Red Hat OpenStack Platform 13, director can now deploy OpenStack on top of fully routed topologies, utilizing its composable network and roles architecture, as well as a DHCP relay to support provisioning across multiple subnets. This provides customers with the flexibility to deploy on top of L2 or L3 routed networks from a single tool.

OpenStack_NFV_Mobile_Networks_438707_0317_ECE_Figure12

Learn more

Learn more about Red Hat OpenStack Platform:


For more information on Red Hat OpenStack Platform and Red Hat Virtualization contact your local Red Hat office today!

by Nir Yechiel, Principal Product Manager, Red Hat at July 13, 2018 01:28 AM

July 12, 2018

StackHPC Team Blog

Pierre Riteau Joins our Team

We are very happy to announce that our team has grown further, with the addition of Pierre Riteau. Pierre is a core team member of the OpenStack Blazar project, has been highly active in the OpenStack Scientific SIG and joins us from the Chameleon testbed, where he was lead DevOps engineer.

StackHPC sees the huge potential of the Blazar reservation service, particularly if it could be combined with the recent work on preemptible instances, and we hope Pierre can help us to deliver this vision for our clients and the Scientific OpenStack community.

We will also draw on Pierre's expertise gained from designing, deploying and managing a bare metal research cloud that has pushed OpenStack's boundaries in some disruptive ways.

Pierre adds "StackHPC has been doing amazing work building research computing platforms with OpenStack. I am honoured and delighted to be joining this talented team to help deliver even more powerful capabilities."

Follow Pierre on Twitter @priteau.

Pierre Riteau

by Stig Telfer at July 12, 2018 09:00 PM

OpenStack Superuser

How to create an effective developer experience with Kubernetes

Hiring and keeping tech staff with in-demand container chops is a challenge for many companies.

Daniel Bryant, an independent tech consultant with 15 years of experience focused on dev-ops, cloud container platforms and microservice implementations, has a few ideas on how to set up a developer experience that will keep the both company and devs happy. He led a recent hour-long webinar packed with tips hosted by the Cloud Native Computing Foundation (CCNF).

“Developer experience is all about minimizing the friction from the idea through to the code in the config to delivering business value,” he says.  “how you construct your platform influences the developer experience, high productivity and bottom line fun as an engineer comes from intentionally designing the experience of things like local development, and debugging packaging apps, CI/CD deployment control and observability.”

Infrastructure first

“It’s kind of like Maslow’s hierarchy of needs, you definitely need to get the infrastructure right, build the platform on top and then correspondingly create a workflow,” Bryant says, adding that the workflow can have a bigger impact on velocity and business value but must be built on firm foundations.

Find your workflow

The ideal workflow has definitely shifted over the years. When Bryant started, waterfall was the way to go and while it’s not necessarily a bad method, it assumes you know all the requirements before starting, which might be the case in, say building a bridge, but doesn’t always work in tech.

“With software we have hypotheses, we have ideas but we need to validate them and therefore agile become the de-facto way of doing things, because it’s kind of waterfall but in smaller iterations,” Bryant says. He cites Netflix as an example: with cross-functional teams, each with its own dev, QAs and product managers, “They’ve got everything you need to basically offer a product, they’re moving away from projects to products,” he says, whether that’s streaming play back or analytics. The cycle goes from idea, coded locally, deployed to Kubernetes, then Canary tested, and then measure the impact (with a tool like Prometheus) on users.

DIY platform on Kubernetes

If you’re thinking it would make sense to roll your own platform-as-a-service on Kubernetes, Bryant says there are a few key questions to consider:

  • Do you want to do everything locally, or do you want to do more more in the cluster?
  • Do you want to have a Cloud Foundry-like an experience (where Docker/containers aren’t installed on local machines).
  • How quickly do you need user feedback?
  • Do you have a strong opinion on code repository structure?
  • Do you want “guide rails” for dev teams?

Whether you go with remote or local development, “it’s all a trade-off, we’re not saying that one ways better than the other, though if you run everything locally it’s probably not realistic to production,” Bryant says. More on these options and consequences in the chart:

Find the right pattern

There’s definitely a pattern emerging for Kubernetes for for ops, it’s pretty much becoming the de facto container orchestrator-as-a-service,” Bryant says, adding that he has previously worked with Amazon ECS and Mesos. “All the cloud vendors are basically coming up with a hosted Kubernetes offering, it’s kind of becoming the new cloud brokerage platform.” The extensibility of Kubertetes means that there’s plenty of room to impact the developer experience, whether its building custom controllers. “As platform operator has a platform engineer you can really add a lot of value to developers by potentially creating custom resources custom controllers.” For development and deployment, Bryant goes into the advantages of using Draft, GitKube, Skaffold, Helm, KSonnet and Forge. Other key patterns to put in place include those for  overall development and deployment, CI/CD and observability.

You can catch the entire hour-long webinar below or on the CCNF’s YouTube channel.

Photo // CC BY NC

The post How to create an effective developer experience with Kubernetes appeared first on Superuser.

by Nicole Martinelli at July 12, 2018 04:03 PM

Aptira

Selection Criteria for Adopting Open Source Projects

Aptira: Selecting a Well Engaged Open Source Vendor

I wrote recently about how to select a partner when deploying Open Source projects within an enterprise — basically you want someone who is engaged with the community and can represent your interests and needs to that community. Now that we’ve covered how to find someone to help you along your journey, how do you decide what Open Source projects are ready for use in your organization?

First off, unless you’re a super agile software user, you need stability in the projects you deploy. It is cool that some organizations are willing to deploy code from upstream weekly (or even daily), but that’s not everyone. It is likely that you’ll want to run whatever code you deploy today for a fair while — perhaps years. You need the projects you deploy to understand and support that. This doesn’t mean you never deploy, but it means you want to deploy small incremental bug fix releases, not something with major changes in it. When you do get around to upgrading between major releases, you also want projects that understand that compatibility between major releases is super important — redeveloping your apps because a component shipped a major release isn’t desirable for most organizations.

Some things to look for:

  • Breaking changes to user interfaces and APIs are minimized between releases.
  • Large changes have a clear rationale as to how they are in the best interests of the community as a whole, and provide a clear and well tested transition plan. So for example, if the project’s main corporate sponsor makes a strategic decision about what’s right for their product and changes things despite the wishes of the community, then that’s a huge warning sign.
  • The project must maintain a stable release branch, have a clear policy about what is backported to that branch, and follow that policy even when it is inconvenient for the developers. I would recommend stable branches receive bug fix backports for at least six months, and security fix backports for at least twelve months. These periods need to be in line with your organization’s preferred rate of change however.
  • The project should be well tested. For modern projects that probably means Continuous Integration testing is enforced before changes are merged into the source code, but there are other valid ways to test projects.
  • In this day and age it is also mandatory that code changes should be reviewed and approved by other developers before they are merged.

You should also look for projects with a regular release cadence. You want to be able to predict when major releases will ship, as well as when they will be stable, well before you deploy them. This helps with forward planning. Regular releases also mean that there isn’t a ton of new features sitting unreleased all the time. Unreleased code is generally less tested, and introducing a lot of it at once is more risky than smaller incremental changes. Similarly, checkout the project’s bug tracker and see if issues that at reported are regularly being addressed. You need to know that the project cares about the experiences of their users and is listening.  

Diversity of contributions is really important too. It is obvious that there are forms of contribution to an Open Source project other than code, especially in terms of a community of users for the project and a solid documentation team. That said, the project dies if its developers all wander off. Having most of the developers work for a single employer increases the risk of project abandonment massively.  

One of the ways that you attract other developers to your project is to make the project welcoming. Another list of things to look for to judge if the project is trying to be welcoming to other developers: 

  • Code is developed in public and anyone can leave review comments.
  • There is a public bug tracker that anyone can file bugs in.
  • The project has a reasonable code submission policy that welcomes public contributions. Sometimes projects require copyright assignment, which isn’t an unreasonable requirement although it is unpopular with a large number of Open Source developers.
  • Upstream code and reviews are coming from more than one or two sources.
  • The code base has existed for long enough that other people have deployed it in production (those people can be from the sources of the code though).

An active community of users and a documentation team which is keeping up with the overall level of change in the project is just as important as the code itself. If new code is landing, but no one can tell you how it works or how to use it, then we have a problem. User communities take different forms, but I’d be looking for active mailing lists, chat forums (IRC, slack, whatever), and blogs from larger deployers discussing what problems they’ve seen and how they’ve solved them.

Some evaluation criteria:

  • There is an active user community separate to the developer community (i.e. people other than the devs are able to deploy the code and have formed into a self-supporting community)
  • There is solid operations documentation — you can install and operate the project without having to engage the services of the software developers to make it go. While freemium support models exist and are sometimes the way to go, in general being held hostage to get help from the developers isn’t something that makes me comfortable.

An inclusive governance model is next on our shopping list. The governance model of a project is how big decisions about the overall direction of the project are made. There are a few common models seen in the wild — some sort of elected Technical Committee structure, a “Benevolent Dictator For Life” (BFDL), as well as others. You could argue that this criteria is very similar to the diversity criteria above, but this is more explicit — the project should have enshrined a clear process to deal with technical disputes as well as producing an overall roadmap so you can predict where the project will be in a few years.

  • While BDFL projects might be the stereotype, it is generally better to have a community where the contributors (developers, operators, users, supporters, et cetera) have some say in the future direction of the project.
  • That said, don’t confuse a lack of direction with being “inclusive”. You still need some set of people that is defining the overall long term direction of the project so that its still meeting your needs in a few years time.

There’s no clear answers here, you’re going to have to assess each project on their merits — for example some projects get their governance models partially “right”. OpenStack for example has a good elected structure for their Technical Committee, but they have a Board of Directors stacked with “pay for play” seats. Worse still, there hasn’t been much success with defining a long term road map for OpenStack as a whole. Despite this, I should be clear that OpenStack is still probably one of the best projects when rated against all the criteria in this post, it just doesn’t rate well for corporate governance.

Finally, a personal bugbear of mine. Look for projects that make technical decisions based on technical merit. For example, one of the real strengths of OpenStack is that they didn’t implement another hypervisor, but instead chose to orchestrate existing hypervisors. A project needs to know where their unique value add is, and avoid stepping outside the boundaries of that.

There’s been a lot to digest in this post, because these are complicated issues. Need a hand assessing if an Open Source project is ready for your organization? Why not drop us a line and have a chat?

One final personal note — this post and its predecessor are based on extensive conversations with many colleagues with Open Source experience adding up to hundreds of years. There are too many of them to list here individually, but I appreciate all the wisdom they were so happy to share. It is a great example of the Open Source community at work that people are willing to help out on non-technical exercises such as this post as well as just code or documentation.

The post Selection Criteria for Adopting Open Source Projects appeared first on Aptira.

by Michael Still at July 12, 2018 09:56 AM

July 11, 2018

OpenStack Superuser

How to get your talk accepted for the next OpenStack Summit

If you’re interested in sharing knowledge with fellow Stackers, you’re in good company: there are usually more than 1,500 submissions for the OpenStack Summit. All community members are encouraged to submit, here are a few tips to improve the chances that your pitch is picked.

Here’s what’s new along with some time-tested tips on what the selection members are looking for. The deadline to submit your presentation for the Berlin Summit is July 17.

At the Vancouver Summit, we focused on open infrastructure integration as the Summit has evolved over the years to cover more than just OpenStack. Over 30 different projects from the open infrastructure community joined, including Kubernetes, Docker, Ansible, OpenShift and many more.

The tracks were organized around specific use cases and will remain the same for Berlin, with the addition of hands-on workshops as a dedicated track. Submit presentations about the open infrastructure tools you’re using, as well as the integration work needed to address these use cases. The more the merrier: you’re also encouraged to invite peers from other open source communities to speak and collaborate.

The tracks:

• CI/CD
• Container infrastructure
• Edge computing
• HPC / GPU / AI
• Open source community
• Private and hybrid cloud
• Public cloud
• Telecom and NFV
• Hands-on workshops

Voting process

Community voting opens shortly after the deadline.  After that, a Programming Committee for each track will build the schedule. Programming Committees are made up of individuals from a variety of open-source communities working in open infrastructure, in addition to people who’ve participated in the past.

Landing a talk

Here are a few tips to make your proposal stand out from the pack from veteran speaker Beth Cohen, cloud networking product manager at Verizon.

  • No matter how wonderful you think your innovation is,  product pitch presentations are highly discouraged.
  • Speakers are encouraged to demonstrate past experience with making presentations and speaking to large audiences. If you do not have prior speaking experience, consider participating on a panel.
  • Make the proposal compelling. The community is always interested in real user stories – good and bad, new innovations and details about technical contributions to OpenStack components and of course, the platform itself.
  • Provide any relevant links to additional information, such as code repositories, case studies, papers, blog posts, etc.

More details on designing a successful OpenStack Summit submission can be found in this post from a Women of OpenStack video webinar hosted by Niki Acosta, Anne Gentle and Diane Mueller.

Or you can check out this podcast from the Women of OpenStack.

The deadline for submissions is July 17.

The post How to get your talk accepted for the next OpenStack Summit appeared first on Superuser.

by Superuser at July 11, 2018 04:13 PM

RDO

Community Blog Round-Up: 11 July

I know what you’re thinking – another blog round up SO SOON?!? Is it MY BIRTHDAY?!? Maybe! But it’s definitely OpenStack’s birthday this month – eight years old – and there are an absolute TON of blog posts as a result. Well, maybe not a ton, but definitely a lot to write about and therefore, there are a lot more community blog round ups. Expect more of the same as content allows! So, sit back and enjoy the latest RDO community blog round-up while you eat a piece of cake and wish a very happy birthday to OpenStack.

Virtualize your OpenStack control plane with Red Hat Virtualization and Red Hat OpenStack Platform 13 by Ramon Acedo Rodriguez, Product Manager, OpenStack

With the release of-Red Hat OpenStack Platform 13 (Queens)-we’ve added support to Red Hat OpenStack Platform director to deploy the overcloud controllers as virtual machines in a Red Hat Virtualization cluster. This allows you to have your controllers, along with other supporting services such as Red Hat Satellite, Red Hat CloudForms, Red Hat Ansible Tower, DNS servers, monitoring servers, and of course, the undercloud node (which hosts director), all within a Red Hat Virtualization cluster. This can reduce the physical server footprint of your architecture and provide an extra layer of availability.

Read more at https://redhatstackblog.redhat.com/2018/07/10/virtualize-your-openstack-control-plane-with-red-hat-virtualization-and-red-hat-openstack-platform-13/

Red Hat OpenStack Platform: Making innovation accessible for production by Maria Bracho, Principal Product Manager OpenStack

An OpenStack®️-based cloud environment can help you digitally transform to succeed in fast-paced, competitive markets. However, for many organizations, deploying open source software supported only by the community can be intimidating. Red Hat®️ OpenStack Platform combines community-powered innovation with enterprise-grade features and support to help your organization build a production-ready private cloud.

Read more at https://redhatstackblog.redhat.com/2018/07/09/red-hat-openstack-platform-making-innovation-accessible-for-production/

Converting policy.yaml to a list of dictionaries by Adam Young

The policy .yaml file generated from oslo is not very useful for anything other than feeding to oslo-policy to enforce. If you want to use these values for anything else, it would be much more useful to have each rule as a dictionary, and all of the rules in a list. Here is a little bit of awk to help out:

Read more at https://adam.younglogic.com/2018/07/policy-yaml-dictionary/

A Git Style change management for a Database driven app. by Adam Young

The Policy management tool I’m working on really needs revision and change management.- Since I’ve spent so much time with Git, it affects my thinking about change management things.- So, here is my attempt to lay out my current thinking for implementing a git-like scheme for managing policy rules.

Read more at https://adam.younglogic.com/2018/07/a-git-style-change-management-for-a-database-driven-app/

by Rain Leander at July 11, 2018 02:58 PM

SUSE Conversations

10 Top Tech Trends: Why Open Source is Center Stage – Part 1

The image of open source software has changed. No longer seen as the geeky, slightly awkward alternative to mainstream software, it’s now trendy, fashionable and super-cool. Well, OK, it still has the geeky edge and it’s still all about innovation. However, it has grown and matured to the point where everyone from small businesses to […]

The post 10 Top Tech Trends: Why Open Source is Center Stage – Part 1 appeared first on SUSE Communities.

by Terri Schlosser at July 11, 2018 01:00 PM

July 10, 2018

Red Hat Stack

Virtualize your OpenStack control plane with Red Hat Virtualization and Red Hat OpenStack Platform 13

With the release of Red Hat OpenStack Platform 13 (Queens) we’ve added support to Red Hat OpenStack Platform director to deploy the overcloud controllers as virtual machines in a Red Hat Virtualization cluster. This allows you to have your controllers, along with other supporting services such as Red Hat Satellite, Red Hat CloudForms, Red Hat Ansible Tower, DNS servers, monitoring servers, and of course, the undercloud node (which hosts director), all within a Red Hat Virtualization cluster. This can reduce the physical server footprint of your architecture and provide an extra layer of availability.

Please note: this is not using Red Hat Virtualization as an OpenStack hypervisor (i.e. the compute service, which is already nicely done with nova via libvirt and KVM) nor is this about hosting the OpenStack control plane on OpenStack compute nodes.

Video courtesy: Rhys Oxenham, Manager, Field & Customer Engagement

Benefits of virtualization

Red Hat Virtualization (RHV) is an open, software-defined platform built on Red Hat Enterprise Linux and the Kernel-based Virtual Machine (KVM) featuring advanced management tools.  RHV gives you a stable foundation for your virtualized OpenStack control plane.

By virtualizing the control plane you gain instant benefits, such as:

  • Dynamic resource allocation to the virtualized controllers: scale up and scale down as required, including CPU and memory hot-add and hot-remove to prevent downtime and allow for increased capacity as the platform grows.
  • Native high availability for Red Hat OpenStack Platform director and the control plane nodes.
  • Additional infrastructure services can be deployed as VMs on the same RHV cluster, minimizing the server footprint in the datacenter and making an efficient use of the physical nodes.
  • Ability to define more complex OpenStack control planes based on composable roles. This capability allows operators to allocate resources to specific components of the control plane, for example, an operator may decide to split out networking services (Neutron) and allocate more resources to them as required. 
  • Maintenance without service interruption: RHV supports VM live migration, which can be used to relocate the OSP control plane VMs to a different hypervisor during their maintenance.
  • Integration with third party and/or custom tools engineered to work specifically with RHV, such as backup solutions.

Benefits of subscription

There are many ways to purchase Red Hat Virtualization, but many Red Hat OpenStack Platform customers already have it since it’s included in our most popular OpenStack subscription bundles, Red Hat Cloud Infrastructure and Red Hat Cloud Suite. If you have purchased OpenStack through either of these, you already own RHV subscriptions!

Logical Architecture

This is how the architecture looks when splitting the overcloud between Red Hat Virtualization for the control plane and utilizing bare metal for the tenants’ workloads via the compute nodes.

Screen Shot 2018-07-10 at 1.22.13 pm

Installation workflow

A typical installation workflow looks like this:

RHVOSP integration Blog post

Preparation of the Cluster/Host networks

In order to use multiple networks (referred to as “network isolation” in OpenStack deployments), each VLAN (Tenant, Internal, Storage, …) will be mapped to a separate logical network and allocated to the hosts’ physical nics. Full details are in the official documentation.

Preparation of the VMs

The Red Hat OpenStack Platform control plane usually consists of one director node and (at least) three controller nodes. When these VMs are created in RHV, the same requirements we have for these nodes on bare metal apply.

The director VM should have a minimum of 8 cores (or vCPUs), 16 GB of RAM and 100 GB of storage. More information can be found in the official documentation.

The controllers should have at least 32 GB of RAM and 16 vCPUs. While the same amount of resources are required for virtualized controllers, by using RHV we gain the ability to better optimize that resource consumption across our underlying hypervisors

Red Hat Virtualization Considerations

Red Hat Virtualization needs to be configured with some specific settings to host the VMs for the controllers:

Anti-affinity for the controller VMs

We want to ensure there is only one OpenStack controller per hypervisor so that in case of a hypervisor failure, the service level disruption minimalized to a single controller. This allows for HA to be taken care of using the different levels of high availability mechanisms already built in to the system. For this to work we use RHV to configure an affinity group with “soft negative affinity,” effectively giving us “anti-affinity!” Additionally it provides the flexibility to override this rule in case of system constraints.

VM network configuration

One vNIC per VLAN

In order to use multiple networks (referred to as “network isolation” in OpenStack deployments), each VLAN (Tenant, Internal, Storage, …) will be mapped to a separate virtual NIC (vNIC) in the controller VMs and VLAN “untagging” will be done at the hypervisor (cluster) and VM level.

Full details can be found in the official documentation.

Screen Shot 2018-07-10 at 11.35.44 (1)

Allow MAC Spoofing

For the virtualized controllers to allow the network traffic in and out correctly, the MAC spoofing filter must be disabled on the networks that are attached to the controller VMs. To do this we set no_filter in the vNIC of the director and controller VMs, then restart the VMs and disable the MAC anti-spoofing filter.

Important Note: If this is not done DHCP and PXE booting of the VMs from director won’t work.

Implementation in director

Red Hat OpenStack Platform director (TripleO’s downstream release) uses the Ironic Bare Metal provisioning component of OpenStack to deploy the OpenStack components on physical nodes. In order to add support for deploying the controllers on Red Hat Virtualization VMs, we enabled support in Ironic with a new driver named staging-ovirt.

This new driver manages the VMs hosted in RHV similar to how other drivers manage physical nodes using BMCs supported by Ironic, such as iRMC, iDrac or iLO. For RHV this is done by interacting with the RHV manager directly to trigger power management actions on the VMs.

Enabling the staging-ovirt driver in director

Director needs to enable support for the new driver in Ironic. This is done as you would do it for any other Ironic driver by simply specifying it in the undercloud.conf configuration file:

enabled_hardware_types = ipmi,redfish,ilo,idrac,staging-ovirt

After adding the new entry and running openstack undercloud install we can see the staging-ovirt driver listed in the output:

(undercloud) [stack@undercloud-0 ~]$ openstack baremetal driver list
+---------------------+-----------------------+
| Supported driver(s) | Active host(s)        |
+---------------------+-----------------------+
| idrac               | localhost.localdomain |
| ilo                 | localhost.localdomain |
| ipmi                | localhost.localdomain |
| pxe_drac            | localhost.localdomain |
| pxe_ilo             | localhost.localdomain |
| pxe_ipmitool        | localhost.localdomain |
| redfish             | localhost.localdomain |
| staging-ovirt       | localhost.localdomain |

Register the RHV-hosted VMs with director

When defining a RHV-hosted node in director’s instackenv.json file we simply set the power management type (pm_type) to the “staging-ovirt” driver, provide the relevant RHV manager host name, and include the username and password for the RHV account that can control power functions for the VMs.

{
    "nodes": [
        {
            "name":"osp13-controller-1",
            "pm_type":"staging-ovirt",
            "mac":[
                "00:1a:4a:16:01:39"
            ],
            "cpu":"2",
            "memory":"4096",
            "disk":"40",
            "arch":"x86_64",
            "pm_user":"admin@internal",
            "pm_password":"secretpassword",
            "pm_addr":"rhvm.lab.redhat.com",
            "pm_vm_name":"osp13-controller-1",
            "capabilities": "profile:control,boot_option:local"
        },
        {
            "name":"osp13-controller-2",
            "pm_type":"staging-ovirt",
            "mac":[
                "00:1a:4a:16:01:3a"
            ],
            "cpu":"2",
            "memory":"4096",
            "disk":"40",
            "arch":"x86_64",
            "pm_user":"admin@internal",
            "pm_password":"secretpassword",
            "pm_addr":"rhvm.lab.redhat.com",
            "pm_vm_name":"osp13-controller-2",
            "capabilities": "profile:control,boot_option:local"
        },
        {
            "name":"osp13-controller-3",
            "pm_type":"staging-ovirt",
            "mac":[
                "00:1a:4a:16:01:3b"
            ],
            "cpu":"2",
            "memory":"4096",
            "disk":"40",
            "arch":"x86_64",
            "pm_user":"admin@internal",
            "pm_password":"secretpassword",
            "pm_addr":"rhvm.lab.redhat.com",
            "pm_vm_name":"osp13-controller-3",
            "capabilities": "profile:control,boot_option:local"
        }
    ]
}

A summary of the relevant parameters required for RHV are as follows:

  • pm_user: RHV-M username.
  • pm_password: RHV-M password.
  • pm_addr: hostname or IP of the RHV-M server.
  • pm_vm_name: Name of the virtual machine in RHV-M where the controller will be created.

For more information on Red Hat OpenStack Platform and Red Hat Virtualization contact your local Red Hat office today!

by Ramon Acedo Rodriguez, Product Manager, OpenStack at July 10, 2018 08:31 PM

OpenStack Superuser

Driving innovation through data: The Progressive Insurance journey

Even in a sector ruled by data, Progressive Insurance stands out. One of the largest car insurers in the United States, they were already pushing the envelope back in 1987 by holding the first large-scale security analyst’s meeting.  These days, Progressive has a product called Snapshot, offered as an app or a plug-in device, that bases customer’s insurance rates on their actual driving habits.

Devices used to measure driving have undergone an evolution over the last 20 years, says Progressive’s Rick Bliss. The original 1998 device plugged into the ODB port in your car then shipped back to Progressive for analysis. In 2010, the device went wireless and later became an app. The large amount of data collected (2,827,247,891 trips!) means that these devices have traveled farther than the Voyager spacecraft.

Progressive employs thousands of IT professionals working on data solutions. The analysis of all that driving data meant that they needed to try things out and fail fast. The team needed to build its infrastructure as a service to enable that and get successful models out to its customers quickly.

In addition, said Bliss, Progressive wanted to honor that data by protecting it. “We need to care about customer data, we need to care about that security, and we need to make sure that data is secure and never gets in the wrong hands,” he said.

The cloud you experiment on often becomes the cloud you go to production with. That’s why the Progressive team chose OpenStack. However, they’re also looking to Kubernetes as a way to abstract that even further, allowing them to move workloads around as the enterprise needs and not having that affect their customers or IT workers.

Further, the team chose VMware-integrated OpenStack as its on-premise cloud, since it helped the company integrate its already existing vSphere environment, said Jeff Maruschek, speaking with Bliss at the recent OpenStack Summit Vancouver. “We wanted to utilize that support structure and also experiment with OpenStack at the same time and we felt that VIO kind of solved that for us,” he said.

Ultimately, the Progressive team went from 100 percent physical servers in 2004 to a production environment with more virtual machines than physical in 2009. Now Progressive uses data centers full of Cisco UCS blade servers and is evaluating hyper conversion for structures.

They also choose Red Hat OpenShift as their application platform-as-a-service, or aPaaS. It helps manage some of the issues Kubernetes can have, like a weaker RBAC authorization system, said Bliss. “OpenShift gave us many levels of interactions,” he said, “[including] a product called Source to Image.” This lets Bliss and his colleagues web-hook their GitHub code as a node app, put it in a container and then build and spin it up all in one go.

Ultimately, said Bliss, OpenStack is an important part of the data puzzle at Progressive. “We see OpenStack as a way to actually bridge all that together [and] automate it. Automate it all and self-service when you can’t automate it.”

It’s not about just specific tools, he concluded, but about culture and embracing change. “That’s been where a lot of our work has been done — educate, educate, educate. Say why these tools actually help us do things faster [and] why they are valuable to our business.”

To see the whole 30-minute presentation, check out the video below:

Photo // CC BY NC

The post Driving innovation through data: The Progressive Insurance journey appeared first on Superuser.

by Rob LeFebvre at July 10, 2018 04:21 PM

Chris Dent

TC Report 18-28

With feature freeze approaching at the end of this month, it seems that people are busily working on getting-stuff-done so there is not vast amounts of TC discussion to report this week.

Actually that's not entirely true. There's quite a bit of interesting discussion in the logs but it ranges widely and resists summary. If you're a fast reader, it can be pretty straightforward to read the whole week.

Some highlights:

Contextualizing Change

The topics of sharing personal context, creating a new technical vision for OpenStack, and trying to breach the boundaries between the various OpenStack sub-projects flowed in amongst one another.

In a vast bit of background and perspective sharing, Zane provided his feelings on what OpenStack ought to be. While long, such things help provide much more context to understanding some of the issues. Reading such things can be effort, but they fill in blanks in understanding, even if you don't agree.

Meanwhile, and related, there are continued requests for nova to engage in orchestration, in large part because there's nothing else commonly available to do it and while that's true we can't serve people's needs well.

Some have said that the need for orchestration could in part be addressed by breaking down some of the boundaries between projects but which boundaries is unclear. Thierry says we should organize work based on objectives.

Goals of Health Tracking

In last week's report I drew a connection between the removal of diversity tags and the health tracker. This created some concern that there were going to be renewed evaluations of projects that would impact their standing in the community and that these evaluations were going to be too subjective.

While it is true that the health tracker is a subjective review of how a project is doing, the evaluation is a way to discover and act on opportunities to help a project, not punish it or give it a black mark.

It is important, however, that the TC is making an independent evaluation.

by Chris Dent at July 10, 2018 01:51 PM

July 09, 2018

Red Hat Stack

Red Hat OpenStack Platform: Making innovation accessible for production

An OpenStack®️-based cloud environment can help you digitally transform to succeed in fast-paced, competitive markets. However, for many organizations, deploying open source software supported only by the community can be intimidating. Red Hat®️ OpenStack Platform combines community-powered innovation with enterprise-grade features and support to help your organization build a production-ready private cloud.

Through an open source development model, community leadership, and production-grade life-cycle options, Red Hat makes open source software more accessible for production use across industries and organizations of any size and type.

omar-albeik-589641-unsplash
Photo by Omar Albeik on Unsplash

Open source development model

In order for open source technologies to be effective in production, they must provide stability and performance while also delivering the latest features and advances. Our open source development model combines fast-paced, cross-industry community innovation with production-grade hardening, integrations, support, and services. We take an upstream-first approach by contributing all developments back to the upstream community. This makes new features immediately available and helps to drive the interoperability of Red Hat products with upstream releases. Based on community OpenStack releases, Red Hat OpenStack Platform is intensively tested and hardened to meet the rigors of production environments. Ongoing patching, bug fixes, and certification keep your environment up and running.

Community leadership

We know that open source technologies can be of the highest quality and work with communities to deliver robust code. Red Hat is the top code contributor to the OpenStack community. We are responsible for 28% of the code in the Queens release and 18% of the code across all releases. We collaborate with our customers, partners, and industry organizations to identify the features they need to be successful. We then work to add that functionality into OpenStack. Over time, these efforts have resulted in enhancements in OpenStack’s availability, manageability, and performance, as well as industry-specific additions like OpenDaylight support for telecommunications.

Production-grade life-cycle options

The OpenStack community delivers new releases every six months, which can be challenging for many organizations looking to deploy OpenStack-based production environments. We provide stable branch releases of OpenStack that are supported for an enterprise production life cycle—beyond the six-month release cycle of the OpenStack community. With Red Hat OpenStack Platform, we give you two life-cycle options that let you choose when to upgrade and add new features to your cloud environment.

  • Standard release cadence. Upgrade every six to twelve months between standard releases to stay aligned with the latest features as they become available. Standard releases include one year of support.
  • Long-life release cadence. Standardize on long-life releases for up to five years. Long-life releases include three years of support, with the option to extend support for an additional two years with extended life-cycle support (ELS), for up to five years of support total. All new features are included with each long-life release.

Red Hat OpenStack Platform director—an integrated deployment and life-cycle management tool—streamlines upgrades between standard releases. And, the new fast forward upgrade feature in director lets you easily transition between long-life releases, without the need to upgrade to each in-between release. So, if you are currently using Red Hat OpenStack Platform 10, you now have an easy upgrade path to Red Hat OpenStack Platform 13—with fewer interruptions, no need for additional hardware, and simpler implementation of containerized OpenStack services.

Fast forward upgrade diagram v1

Learn more

Red Hat OpenStack Platform can help you overcome the challenges of deploying OpenStack into production use. And, if you aren’t sure about how to build your cloud environment, don’t have the time or resources to do so, or just want some help on your cloud journey, we provide a variety of expert services and training.

Learn more about Red Hat OpenStack Platform:

by Maria Bracho, Principal Product Manager OpenStack at July 09, 2018 08:19 PM

OpenStack Superuser

Five fundamentals for leveraging multi-cloud in Kubernetes

If it’s time to construct your container strategy, a few key decisions will provide the blueprint to make it a success.

The tips excerpted here come from the “DZone Guide to Containers: Development and Management.” They were written by Jim Bugwadia, founder at Nirmata, a San Francisco bay area startup providing multi-cloud container services.

Developed in partnership with companies including Oracle, Platform9, Chef, Cloud Foundry the 58-page ebook is free with registration. In addition to a survey of over 500 tech professionals about how they’re using containers today, the book features chapters on Istio service mesh for containers, a deep dive into deployments and a comparison of Kubernetes, Docker Swarm and Amazon ECS.

Leverage best-of-breed infrastructure

The first consideration is whether to use the storage and networking services offered by your current cloud provider. Bugwadia says it’s a good idea to prioritize a vendor-native approach. “Rather than run an overlay network in AWS, it may be best to use the Kubernetes CNI (Container Network Interface) plugin from AWS that offers native networking capabilities to Kubernetes.”

Manage your own upstream Kubernetes versions

When trying to stay on board a rapidly changing open source project, another important decision is whether to team up with a vendor or drink directly from upstream. “The best bet is to provide teams with the flexibility of choosing multiple validated upstream releases, or trying newer versions as needed at their own risk,” Bugwadia says.

Standardize cluster deployments via policies

There are a several important decisions to make when in-stalling a Kubernetes cluster, he says, including which version of Kubernetes, networking, storage, ingress, monitoring and logging.

Provide end-to-end security

Again, there are a number of important decisions to make including: image scanning and provenance, host and cluster scanning, segementation and isolation and identity management and access controls.

Centralize application management

“As with security, managing applications on Kubernetes clusters requires a centralized and consistent approach. While Kubernetes offers a comprehensive set of constructs that can be used to define and operate applications, it does have a built-in concept of an application,” Bugwadia notes.

You can check out the whole chapter online and download the book here.

Photo // CC BY NC

The post Five fundamentals for leveraging multi-cloud in Kubernetes appeared first on Superuser.

by Nicole Martinelli at July 09, 2018 04:06 PM

ICCLab

Setting up container based Openstack with OVN networking

OVN is a relatively new networking technology which provides a powerful and flexible software implementation of standard networking functionalities such as switches, routers, firewalls, etc. Importantly, OVN is distributed in the sense that the aforementioned network entities can be realized over a distributed set of compute/networking resources. OVN is tightly coupled with OVS, essentially being a layer of abstraction which sits above a set of OVS switches and realizes the above networking components across these switches in a distributed manner.

A number of cloud computing platforms and more general compute resource management frameworks are working on OVN support, including oVirt, Openstack, Kubernetes and Openshift – progress on this front is quite advanced. Interestingly and importantly, one dimension of the OVN vision is that it can act as a common networking substrate which could facilitate integration of more than one of the above systems, although the realization of that vision remains future work.

In the context of our work on developing an edge computing testbed, we set up a modest Openstack cluster, to emulate functionality deployed within an Enterprise Data Centre with OVN providing network capabilities to the cluster. This blog post provides a brief overview of the system architecture and notes some issues we had getting it up and running.

As our system is not a production system, providing High Availability (HA) support was not one of the requirements; consequently, it was not necessary to consider HA OVN mode. As such, it was natural to host the OVN control services, including the Northbound and Southbound DBs and the Northbound daemon (ovn-northd) on the Openstack controller node. As this is the node through which external traffic goes, we also needed to run an external facing OVS on this node which required its own OVN controller and local OVS database. Further, as this OVS chassis is intended for external traffic, it needed to be configured with ‘enable-chassis-as-gw‘.

We configured our system to use DHCP provided by OVN; consequently the neutron DHCP agent was no longer necessary, we removed this process from our controller node. Similarly, L3 routing was done within OVN meaning that the neutron L3 agent was no longer necessary. Openstack metadata support is implemented differently when OVN is used: instead of having a single metadata process running on a controller serving all metadata requests, the metadata service is deployed on each node and the OVS switch on each node routes requests to 169.254.169.254 to the local metadata agent; this then queries the nova metadata service to obtain the metadata for the specific VM.

The services deployed on the controller and compute nodes are shown in Figure 1 below.

Figure 1: Neutron containers with and without OVN

We used Kolla to deploy the system. Kolla does not currently have full support for OVN; however specific Kolla containers for OVN have been created (e.g. kolla/ubuntu-binary-ovn-controller:queens, kolla/ubuntu-binary-neutron-server-ovn:queens). Hence, we used an approach which augments the standard Kolla-ansible deployment with manual configuration of the extra containers necessary to get the system running on OVN.

As always, many smaller issues were encountered while getting the system working – we will not detail all these issues here, but rather focus on the more substantive issues. We divide these into three specific categories: OVN parameters which need to be configured, configuration specifics for the Kolla OVN containers and finally a point which arose due to assumptions made within Kolla that do not necessarily hold for OVN.

To enable OVN, it was necessary to modify the configuration of the OVS switches operating on all the nodes; the existing OVS containers and OVSDB could be used for this – the OVS version shipped with Kolla/Queens is v2.9.0 – but it was necessary to modify some settings. First, it was necessary to configure system-ids for all of the OVS chassis’ – we chose to select fixed UUIDs a priori and use these for each deployment such that we had a more systematic process for setting up the system but it’s possible to use a randomly generated UUID.

docker exec -ti openvswitch_vswitchd ovs-vsctl set open_vswitch . external-ids:system-id="$SYSTEM_ID"

On the controller node, it was also necessary to set the following parameters:

docker exec -ti openvswitch_vswitchd ovs-vsctl set Open_vSwitch . \
    external_ids:ovn-remote="tcp:$HOST_IP:6642" \
    external_ids:ovn-nb="tcp:$HOST_IP:6641" \
    external_ids:ovn-encap-ip=$HOST_IP external_ids:ovn-encap type="geneve" \
    external-ids:ovn-cms-options="enable-chassis-as-gw"

docker exec openvswitch_vswitchd ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-ex

On the compute nodes this was necessary:

docker exec -ti openvswitch_vswitchd ovs-vsctl set Open_vSwitch . \
    external_ids:ovn-remote="tcp:$OVN_SB_HOST_IP:6642" \
    external_ids:ovn-nb="tcp:$OVN_NB_HOST_IP:6641" \
    external_ids:ovn-encap-ip=$HOST_IP \
    external_ids:ovn-encap-type="geneve"

Having changed the OVS configuration on all the nodes, it was then necessary to get the services operational on the nodes. There are two specific aspects to this: modifying the service configuration files as necessary and starting the new services in the correct way.

Not many changes to the service configurations were required. The primary changes related to ensuring the the OVN mechanism driver was used and letting neutron know how to communicate with OVN. We also used the geneve tunnelling protocol in our deployment and this required the following configuration settings:

  • For the neutron server OVN container
    • ml2_conf.ini
              mechanism_drivers = ovn
       	type_drivers = local,flat,vlan,geneve
       	tenant_network_types = geneve
      
       	[ml2_type_geneve]
       	vni_ranges = 1:65536
       	max_header_size = 38
      
       	[ovn]
       	ovn_nb_connection = tcp:172.30.0.101:6641
       	ovn_sb_connection = tcp:172.30.0.101:6642
       	ovn_l3_scheduler = leastloaded
       	ovn_metadata_enabled = true
      
    • neutron.conf
              core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
       	service_plugins = networking_ovn.l3.l3_ovn.OVNL3RouterPlugin
      
  • For the metadata agent container (running on the compute nodes) it was necessary to configure it to point at the nova metadata service with the appropriate shared key as well as how to communicate with OVS running on each of the compute nodes
            nova_metadata_host = 172.30.0.101
     	metadata_proxy_shared_secret = <SECRET>
     	bridge_mappings = physnet1:br-ex
     	datapath_type = system
     	ovsdb_connection = tcp:127.0.0.1:6640
     	local_ip = 172.30.0.101
    

For the OVN specific containers – ovn-northd, ovn-sb and ovn-nb databases, it was necessary to ensure that they had the correct configuration at startup; specifically, that they knew how to communicate with the relevant dbs. Hence, start commands such as

/usr/sbin/ovsdb-server /var/lib/openvswitch/ovnnb.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/ovnnb_db.sock --remote=ptcp:$ovnnb_port:$ovsdb_ip --unixctl=/run/openvswitch/ovnnb_db.ctl --log-file=/var/log/kolla/openvswitch/ovsdb-server-nb.log

were necessary (for the ovn northbound database) and we had to modify the container start process accordingly.

It was also necessary to update the neutron database to support OVN specific versioning information: this was straightforward using the following command:

docker exec -ti neutron-server-ovn_neutron_server_ovn_1 neutron-db-manage upgrade heads

The last issue which we had to overcome was that Kolla and neutron OVN had slightly different views regarding the naming of the external bridges. Kolla-ansible configured a connection between the br-ex and br-int OVS bridges on the controller node with port names phy-br-ex and int-br-ex respectively. OVN also created ports with the same purpose but with different names patch-provnet-<UUID>-to-br-int and patch-br-int-to-provonet-<UUID>; as these ports had the same purpose, our somewhat hacky solution was to manually remove the the ports created in the first instance by Kolla-ansible.

Having overcome all these steps, it was possible to launch a VM which had external network connectivity and to which a floating IP address could be assigned.

Clearly, this approach is not realistic for supporting a production environment, but it’s an appropriate level of hackery for a testbed.

Other noteworthy issues which arose during this work include the following:

  • Standard docker apparmor configuration in ubuntu is such that mount cannot be run inside containers, even if they have the appropriate privileges. This has to be disabled or else it is necessary to ensure that the containers do not use the default docker apparmor profile.
  • A specific issue with mounts inside a container which resulted in the mount table filling up with 65536 mounts and rendering the host quite unusable (thanks to Stefan for providing a bit more detail on this) – the workaround was to ensure that /run/netns was bind mounted into the container.
  • As we used geneve encapsulation, geneve kernel modules had to be loaded
  • Full datapath NAT support is only available for linux kernel 4.6 and up. We had to upgrade the 4.4 kernel which came with our standard ubuntu 16.04 environment.

This is certainly not a complete guide to how to get Openstack up and running with OVN, but may be useful to some folks who are toying with this. In future, we’re going to experiment with extending OVN to an edge networking context and will provide more details as this work evolves.

 

by murp at July 09, 2018 09:03 AM

July 08, 2018

Aija Jauntēva

Outreachy: Simulating BIOS for sushy-tools

I can't believe that another 2 weeks have passed since the last blog post. Shortly after my previous blog post about adding support for BIOS resource in sushy library it got merged \o/. The next thing was to add BIOS support for sushy-tools so that developers can use it during development to test against as bare metals with Redfish are not readily available. Also sushy-tools could be used in CI when running automated tests. There is another developer working on adding BIOS-Redfish support in Ironic.

To re-iterate on sushy-tools, it consists of two simulators sushy-static and sushy-emulator, Sushy static is straightforward - it serves JSON files from provided mockup folder where mockup files can be downloaded from DMTF Redfish page. Whatever is in those JSON files, is returned back to client. sushy-emulator is more dynamic and uses actual systems to emulate what is necessary for Redfish API. Now it supports 2 drivers - libvirt[1] and openstacksdk[2] (interchangeably also called nova driver).

And to re-iterate on Redfish BIOS resource - it provides BIOS attributes, updating those attributes, resetting the attributes to default and changing BIOS password.

I started to look at implementing BIOS support and writing up a story for storyboard, but stopped when decided to check what there is for libvirt and openstacksdk in relation to BIOS. Because at first I made assumption that there would be some API functions in libvirt and openstack libraries same as there were for parts already implemented. tl;dr; There is none.

Browsing through libvirt and openstack sdk I couldn't see anything that stand out as BIOS API. Ok, so maybe there is nothing for BIOS, but maybe I don't know what I'm looking for (I don't have much experience with BIOS - as a computer user I have accessed BIOS to change boot order and enable hardware virtualization, if there was anything else I don't remember it). What are these "BIOS attributes"? Looking at the samples provided by DMTF, there are attributes like 'AdminPhone', 'BootMode', 'PowerProfile', 'UsbControl'. I take a look at my BIOS and cannot map these, probably there is something like that, but named slightly different (not expecting to see AdminPhone though). Still I don't want this let go, and I want to find out how this should work on real servers that support Redfish.

This presentation[3] comes handy and introduces me to UEFI HII (Human Interface Infrastructure) which would be the source for UEFI variables mapped to Redfish BIOS attributes. I read more about it at[4] and then go back to libvirt. For openstacksdk it is already clear that there is nothing for BIOS.

A side note, here I'm not looking at legacy BIOS, but UEFI. BIOS is being replaced by UEFI and one day will be gone, however its name still will live in Redfish, Ironic and elsewhere, because at least for Ironic this name is chosen as better cannot be identified[5].

Back to libvirt, I find that for virtual machines' UEFI, the term OVMF or AAVMF is used[6]. Start searching using OVMF, find a doc about it, go through it[7], and there it mentions NVRAM config file that contains something and my mentor had mentioned it. I need to take a look what is there. I create a VM with UEFI/OVMF[8] and get the sample file. Its extension is '.fd' and it's binary file. I try to find if there is some utility to read just to see if something useful is there. Couldn't find it, and even if there is such, it is not part of libvirt API - so sushy-emulator would bypass libvirt API and would have to work with it directly which is not ideal.

OK, here I give up and look at backup plan - have a set of string mimicking BIOS attributes with no effect on VM and stored at libvirt domain XML so that it is persisted between VM and sushy-tools reboots. Where would that be? I cannot find a place for it, but there is section named SMBIOS[9] in libvirt.

System Management BIOS (SMBIOS) specification defines data structures (and access methods) that can be used to read management information produced by the BIOS of a computer

Sounds close enough? SBMBIOS has section oemStrings where it can store custom information. But it is added in the latest version released this spring, even I don't have it on my a little bit outdated Fedora (I will upgrade when Outreachy is over :)). And still this is not ideal place for custom data. Talk with mentors and then we decide that we will start with storing this in emulator's memory that would not be persisted accross reboots. I promise to take another look if really, really there is nothing better in libvirt. Next time I look and start reading the XML spec from the very beginning (not looking at table of contents which I did before) and here it is - under General data section, there is metadata section that allows to store whatever is necessary using own XML namespace. That is the perfect place for what we need.

There were some other obstacles before I could get to implementing this - due to Flask upgrade tests started to fail in sushy-tools and I tried to fix it (that would be another paragraph about this, but I'm leaving this out so the blog post does not get too long). Once I got back to implementation I did it in less than a day, but there were and still are other patches for sushy-tools which introduce new and restructure existing tests with which I need to align. After the direction to go was clarified I updated my tests.
As of writing this, the patch is submitted and fails CI jobs.
I fixed the first time it failed CI - it was concurrency issue which was not visible locally because I had libvirtd running. I stopped the service and was able to reproduce the test failures. But only when I run full test suite. When I tried to isolate the issue, the tests passed. I found the missing piece and fixed my test setup (had to nullify global driver) and it passed again, but then CI failed again with different errors. Perhaps more concurrency issues, suspecting global variable (everything fine locally).

Overall, it does not feel like already 2 weeks have passed but when I start writing about it a lot of interruptive things come up which I will leave out for brevity. One thing that stands out for me is that I was not noticing some things that I feel I should have. Better attention next time. The good thing - when I was researching UEFI BIOS thing I remembered about and used the Pomodoro technique[10] with Gnome app[11] to keep myself focused. Worked flawlessly.

by ajya at July 08, 2018 03:00 PM

Adam Young

Converting policy.yaml to a list of dictionaries

The policy .yaml file generated from oslo has the following format:

# Intended scope(s): system
#"identity:update_endpoint_group": "rule:admin_required"

# Delete endpoint group.
# DELETE /v3/OS-EP-FILTER/endpoint_groups/{endpoint_group_id}
# Intended scope(s): system
#"identity:delete_endpoint_group": "rule:admin_required"

This is not very useful for anything other than feeding to oslo-policy to enforce. If you want to use these values for anything else, it would be much more useful to have each rule as a dictionary, and all of the rules in a list. Here is a little bit of awk to help out:

#!/usr/bin/awk -f
BEGIN {apilines=0; print("---")}
/#"/ {
    if (api == 1){
	printf("  ")
    }else{
	printf("- ")
    }
  split ($0,array,"\"")
  print ("rule:", array[2]);
  print ("  check:", array[4]);
  rule=0
}    
/# / {api=1;}
/^$/ {api=0; apilines=0;}
api == 1 && apilines == 0 {print ("- description:" substr($0,2))}
/# GET/  || /# DELETE/ || /# PUT/ || /# POST/ || /# HEAD/ || /# PATCH/ {
     print ("  " $2 ": " $3)
}
api == 1 { apilines = apilines +1 }

I have it saved in mungepolicy.awk. I ran it like this:

cat etc/keystone.policy.yaml.sample | ./mungepolicy.awk > /tmp/keystone.access.yaml

And the output looks like this:

---
- rule: admin_required
  check: role:admin or is_admin:1
- rule: service_role
  check: role:service
- rule: service_or_admin
  check: rule:admin_required or rule:service_role
- rule: owner
  check: user_id:%(user_id)s
- rule: admin_or_owner
  check: rule:admin_required or rule:owner
- rule: token_subject
  check: user_id:%(target.token.user_id)s
- rule: admin_or_token_subject
  check: rule:admin_required or rule:token_subject
- rule: service_admin_or_token_subject
  check: rule:service_or_admin or rule:token_subject
- description: Show application credential details.
  GET: /v3/users/{user_id}/application_credentials/{application_credential_id}
  HEAD: /v3/users/{user_id}/application_credentials/{application_credential_id}
  rule: identity:get_application_credential
  check: rule:admin_or_owner
- description: List application credentials for a user.
  GET: /v3/users/{user_id}/application_credentials
  HEAD: /v3/users/{user_id}/application_credentials
  rule: identity:list_application_credentials
  check: rule:admin_or_owner

Which is valid yaml. It might be a pain to deal with the verbs in separate keys. Ideally, that would be a list, too, but this will work for starters.

by Adam Young at July 08, 2018 03:38 AM

July 06, 2018

SUSE Conversations

Test drive managing z/VM with SUSE OpenStack Cloud 7

          SUSE is introducing a technology preview for managing IBM z/VM with SUSE OpenStack Cloud 7. Historically, SUSE OpenStack Cloud has focused on Intel based hypervisors like XEN, KVM, VMWare and Hyper-V.  We are looking to improve SUSE OpenStack’s support of IBM Z and LinuxONE but we need your feedback. What’s new about this? You are now […]

The post Test drive managing z/VM with SUSE OpenStack Cloud 7 appeared first on SUSE Communities.

by Michael Friesenegger at July 06, 2018 09:58 PM

Adam Young

A Git Style change management for a Database driven app.

The Policy management tool I’m working on really needs revision and change management.  Since I’ve spent so much time with Git, it affects my thinking about change management things.  So, here is my attempt to lay out my current thinking for implementing a git-like scheme for managing policy rules.

A policy line is composed of two chunks of data.  A Key and a Value.  The keys are in the form

  identity:create_user.

Additionally, the keys are scoped to a specific service (Keystone, Nova, etc).

The value is the check string.  These are of the form

role:admin and project_id=target.project_id

It is the check string that is most important to revision control. This lends itself to an entity diagram like this:

Whether each of these gets its own table remains to be seen.  The interesting part is the rule_name to policy_rule mapping.

Lets state that the policy_rule table entries are immutable.  If we want to change policy, we add a new entry, and leave the old ones in there.  The new entry will have a new revision value.  For now, lets assume revisions are integers and are monotonically increasing.  So, when I first upload the Keystone policy.json file, each entry gets a revision ID of 1.  In this example, all check_strings start off as are “is_admin:True”

Now lets assume I modify the identity:create_user rule.  I’m going to arbitrarily say that the id for this record is 68.  I want to Change it to:

role:admin and domain_id:target.domain_id

So we can do some scope checking.  This entry goes into the policy_rule table like so:

 

rule_name_id check_string revision
68 is_admin:True 1
68 role:admin and domain_id:target.domain_id 2

From a storage perspective this is quite nice, but from a “what does my final policy look like” perspective it is a mess.

In order to build the new view, we need sql along the lines of

select * from policy_rule where revision = ?

Lets call this line_query and assume that when we call it, the parameter is substituted for the question mark.  We would then need code like this pseudo-code:

doc = dict()
for revision in 1 to max:
    for result in line_query.execute(revision):
        index = result['rule_name_id']
        doc[index] = result.check_string()

 

This would build a dictionary layer by layer through all the revisions.

So far so good, but what happens if we decided to revert, and then to go a different direction? Right now, we have a revision chain like this:

And if we keep going, we have,

But what happens if 4 was a mistake? We need to revert to 6 and create a new new branch.

We have two choices. First, we could be destructive and delete all of the lines in revision 4, 5, and 6. This means we can never recreate the state of 6 again.

What if we don’t know that 4 is a mistake? What if we just want to try another route, but come back to 4,5, and 6 in the future?

We want this:

 

But how will we know to take the branch when we create the new doc?

Its a database! We put it in another table.

revision_id revision_parent_id
2 1
3 2
4 3
5 4
6 5
7 3
8 7
9 8

In order to recreate revision 9, we use a stack. Push 9 on the stack, then find the row with revision_id 9 in the table, push the revision_parent_id on the stack, and continue until there are no more rows.  Then, pop each revision_id off the stack and execute the same kind of pseudo code I posted above.

It is a lot.  It is kind of complicated, but it is the type of complicated that Python does well.  However, database do not do this kind of iterative querying well.  It would take a stored procedure to perform this via a single database query.

Talking through this has encouraged me decide to take another look at using git as the backing store instead of a relational database.

by Adam Young at July 06, 2018 07:38 PM

OpenStack Superuser

Check out these OpenStack project updates

If you’re interested in getting up to speed on what’s next for OpenStack software in the current development cycle, the project update videos from the recent OpenStack Summit Vancouver are available now.

In them you’ll hear from the project team leaders (PTLs) and core contributors about what they’ve accomplished, where they’re heading for future releases plus how you can get involved and influence the roadmap.

You can find the complete list of them on the OpenStack video page. You can also get a complete overview of them on the project navigator.

Some project updates that you won’t want to miss include:

Heat orchestration

Ironic bare metal provisioning service

Keystone identity service

Octavia load balancer

Swift object storage

Tacker  generic VNF Manager

Vitrage root cause analysis (RCA)

And check out these more detailed write-ups of Cyborg, Nova, Kata Containers and Neutron.

The post Check out these OpenStack project updates appeared first on Superuser.

by Superuser at July 06, 2018 03:01 PM

Chris Dent

Placement Update 18-27

This is placement update 18-27, a weekly update of ongoing development related to the OpenStack placement service. This is a contract version.

Most Important

In the past week or so we've found a two suites of bugs that are holding up other work. One set is related to consumers and the handling of consumer generations (linked in that theme, below). The other is related to various ways in which managing parents of nested providers is not correct. Those are:

The first is already fixed, the second was discovered as a result of thinking about the first.

We also have some open questions about which of consumer id, project id, and user id are definitely going to be a valid UUID and what that means in relation to enforcement and our definition of "what's a good uuid":

As usual, this is more support for the fact that we need to be doing increased manual testing to find where our automated tests have gaps, and fill them.

On themes, we have several things rushing for attention before the end of the cycle (reminder: Feature Freeze is the end of this month). We need to get the various facets of consumers fixed up in a way that we all agree is correct. We need to get the Reshaped Providers implemented. And there's some hope (maybe vain?) that we can get the report client and virt drivers talking in a more nested and shared form.

What's Changed

The microversion for nested allocation candidates has merged as 1.29.

The huge pile of osc-placement changes at https://review.openstack.org/#/q/topic:bp/placement-osc-plugin-rocky has merged. Yay!

Bugs

Questions

  • Will consumer id, project and user id always be a UUID? We've established for certain that user id will not, but things are less clear for the other two. This issue is compounded by the fact that these two strings are different but the same UUID: 5eb033fd-c550-420e-a31c-3ec2703a403c, 5eb033fdc550420ea31c3ec2703a403c (bug 1758057 mentioned above) but we treat them differently in our code.

Main Themes

Documentation

This is a section for reminding us to document all the fun stuff we are enabling. Open areas include:

  • Documenting optional placement database. A bug, 1778227 has been created to track this. This has started, for the install docs, but there are more places that need to be touched.

  • "How to deploy / model shared disk. Seems fairly straight-forward, and we could even maybe create a multi-node ceph job that does this - wouldn't that be awesome?!?!", says an enthusiastic Matt Riedemann.

  • The whens and wheres of re-shaping and VGPUs.

Nested providers in allocation candidates

The main code of this has merged. What's left are dealing with things like the parenting bugs mentioned above, and actually reporting any nested providers and inventory so we can make use of them.

Consumer Generations

A fair bit of open bugs fixes and debate on this stuff.

Note that once this is correct we'll still have work to do in the report client to handle consumer generations before nova can do anything with it.

Reshape Provider Trees

This allows moving inventory and allocations that were on resource provider A to resource provider B in an atomic fashion. The blueprint topic is:

There are WIPs for the HTTP parts and the resource tracker parts, on that topic.

Mirror Host Aggregates

This needs a command line tool:

Extraction

Extraction is mostly taking a back seat at the moment while we find and fix bugs in existing features. We've also done quite a lot of the preparatory work. The main things to be thinking about are:

  • os-resource-classes
  • infra and co-gating issues that are going to come up
  • copying whatever nova-based test fixture we might like

Other

24 entries last week. 20 now (this is a contract week, there's plenty of new reviews not listed here).

End

You are the key to the coming revolution.

by Chris Dent at July 06, 2018 02:00 PM

July 05, 2018

Chris Dent

Placement Container Playground 7

This is an appendix to my Placement Container Playground series. The last one was Playground 6. That was going to be the end of the series because "woot, it works" and that was enough.

Since then the container has proven useful in a variety of ways, mostly as the host for a collection of gabbi-based set of tests and demonstrations of the API. These are kept in the placecat.

Some of those demonstrations use placement to represent the food in a refrigerator and build some sandwiches. That model was sufficiently comprehensible that it helped drive a presentation given by fellow placement developers Ed Leafe and Eric Fried at Summit:

Since then I've been using the system to exercise the more esoteric features in placement, including nested resource providers and consumer generations, sometimes finding important bugs. Bugs are most often revealed when I try to use multiple features in the same series of tests, outside of their normal intentions.

Because of all that, the files in placecat have now become a sort of evolving third party integration test of placement so I thought I better automate it.

Now any time the placecat repo is updated or any time the associated container on dockerhub that is built from the placedock repository is refreshed, a travis ci job is automatically triggered. That fetches, builds, and runs the latest container and then runs all placecat tests against it.

As of today it fails in ways that are well known and fixes are in progress.

Build Status

The next step is to build a new container any time new code is merged to placement.

by Chris Dent at July 05, 2018 06:00 PM

OpenStack Superuser

Revamping retail for the online age at Target

The retail market is getting an extreme makeover, like it or not.  At a time when online sales have upended brick-and-mortar shopping habits, Target Corporation has upgraded its back-end infrastructure to improve how it does business and serves customers.

It’s a big job: Target has 1,829 stores in the United States with about 240 million square feet of retail space. It operates 39 distribution centers across the country and employs about 350,000 employees. Target.com is the fourth-most visited website in the US, with more than 26 million unique visitors each month.

The company has moved from the traditional suburban large-format store to more urban, smaller, college campus-style stores. They have introduced services like Restock, which lets customers order from home and have household items delivered the next day for a small fee. Target has also made a few key buys this year with Shipt, a same-day grocery delivery service and Grand Junction, a shipping logistics company.

In order to make all these changes, innovate in their stores, online and mobile, the company has gone through some transformative changes, including a broad adoption of open source and enabling agile for all its software teams. In addition, everything the company does now has an API. “OpenStack is an important enabler for that,” said principal engineer Marcus Jensen, speaking at the OpenStack Summit about Target’s use of OpenStack and its deployment at the massive retailer.

Before 2014, Target had two production data centers. Everything was centrally managed on a mix of blade and rack mount infrastructure, with Intel 2630L with 384 GB of RAM and traditional Layer 2 switching broadly across the data centers. It could take weeks to get a virtual machine (VM) assigned, with a manual approval and provisioning process in place.

Target deployed OpenStack for the first time in 2015. The team grew quickly, adding about half a dozen engineers in the first year. The team ran development mostly on Linux, but also some Windows workloads to support the company’s point-of-sale development efforts. Most of it was still internally-facing projects like inventory management, however. Now Target has about 150 hypervisors in each data center and another facility on campus that developers use to do their initial deployment work before moving it to production. That adds up to an eye-popping total of about 10,000 cores and 350,000 virtual machines in production and slightly more than 13,000 active VMs.

The team upgraded to a Layer 3 Spine/Leaf topology they deploy in the fabric as a private operations network. There are three separate VLANs and three separate IP networks — the control plane talks to the hypervisors that takes care of the storage. There’s a routable service network that supports incoming APIs, and then a SNAP network where the network address translation for the VMs occurs on its own VLAN.

Target’s team put together new hardware, too, standardizing on rack-mount servers. Each rack consists of two blade chassis, so the data center now has 32 Intel 2690 blades per rack with half a terabyte of RAM each.

The second generation deployment was powered by Ironic, which uses the same OpenStack APIs Target was already comfortable with. They used Ironic Baremetal provisioning, starting with the vendor implementation of TripleO with Mitaka. They eventually replaced TripleO with an internally-developed stack-per-node model to help with future updates. Jensen and his team also build their own DIB images, modifying the disk image builder to take advantage of Ansible in the process.

They use Chef for config management, mostly because it was already pretty well entrenched at Target, and have been moving to containerize everything in the future. They transitioned from vendor OS/Openstack to CentOS/RDo, so that Target is now completely open source. They utilize Hashey Consul and Vault, with a reliance on Vault for secret management, while planning to implement Consul more for service discovery and some configuration settings and management. They use Jenkins and Drone for CI, and other tools like GitHub, Enterprise ARtifactory, and more. The team mostly uses Telegraf to collect metrics and sends them, along with their logs, to Kafka topic, which are picked up by internal aggregation services that use Grafana and Cabana.

The third generation is currently in the planning stages, and includes an internally developed Target Application Platform (TAP), which lets applications run on a variety of infrastructure, including OpenStack, Kubernetes, or even public clouds.

You can catch the entire talk below.

Photo // CC BY NC

The post Revamping retail for the online age at Target appeared first on Superuser.

by Rob LeFebvre at July 05, 2018 04:36 PM

Aptira

What is Open Networking? Part 2 – The Three Domains of Open Networking

Aptira What is Open Networking

In our first post in this series on Open Networking What is Open Networking?, we outlined a set of nine (9) key areas that Aptira views are essential paradigms, technologies, or practices that we must employ for the realisation and management of highly performant, high-quality and high-value computing systems. 

  • Open Source Software
  • Open Network Devices
  • Open Compute Hardware
  • Software Defined Networks
  • Network Function Virtualisation
  • Cloud Computing
  • Automation
  • Agile Methods & Processes
  • Server Virtualisation

In this post I want to complete the top-level presentation of Aptira’s holistic view of Open Networking, and to prepare the ground for the rest of this series of articles. 

We like to think of these nine components of Open Networking as fitting into three groups or “domains” of Infrastructure, Software and Integration. From one perspective, these domains are simply convenience groupings that represent several common attributes of the Open Networking components that fall into them. 

But if we think about Open Networking from a value-creation perspective, its objective is to take technology components and shape them optimally to deliver the maximum alignment with end customer requirements.  From this perspective the domains have very different qualities and we see a hierarchy like this: 

These domains are supported by the generic practises common to all solution realisation projects: Software development, Project Management, Network and Systems engineering, procurement and so forth.  These important capabilities are beyond the scope of this series of posts. 

These domains are organised in this hierarchy to show the relative ability of to be shaped and moulded in order to meet the specific needs of the customer, sustainably and over the entire solutions lifecycle.  In other words, for a solutions integrator, to what extent can these components be modified to ensure precise fit between need and solution.  The integration domain sits at the top with the greatest level of flexibility, working down to the infrastructure level. 

Now, how do we group our nine Open Networking key areas within these domains? 

Open Network Integration:

  • Agile Methods & Processes (for Integration & Software Development)
  • Automation
Open Network Software:

  • Open Source Software
  • Software Defined Networks
  • Network Function Virtualisation
Open Network Infrastructure:

  • Open Network Devices
  • Open Compute Hardware
  • Cloud Computing
  • Server Virtualisation

This hierarchy is not meant to imply that any one technology or practice is “better” or “more important” than any other.  Solution components from each of the nine key areas will most likely play a part in an integrated Open Networking solution.   And furthermore, stability of the infrastructure level: the ability to rely on standardised, interoperable and proven solution components is the foundation of solution design for these types of solutions. 

This hierarchy isn’t about a simple hardware/software split.  Now, and increasingly more so in the Open networking marketplace, there is software at the core of every one of these components. As we will examine in later articles, Open Networking is critically dependent on a top-to-bottom understanding of the nature of software and software development. 

Our civilization depends critically on software;
it had better be quality software

Bjarne Stroustrup

What we have found during the many projects we’ve managed at Aptira is that these three domains require different approaches in terms of the paradigms, management techniques, practices and tools when integrating them into a coherent solution that meets customer needs. 

Many Open Networking projects run into trouble because they ignore this fundamental separation and try a “one-size fits all” approach. Or even worse, projects fail to recognise the necessity of applying the hard-won lessons of systems engineering. To be a successful Open Networking integrator like Aptira, a deep understanding of the similarities and differences between each of these three domains is mandatory.  And, as we will see, many organisations are not set up to manage across these domains, especially in a field as young as Open Networking. 

There is no silver bullet

Fred Brooks

Developing software is a difficult and troublesome process for many organisations, despite the many tools, techniques and methodologies available.  The only thing harder and more troublesome than software development is systems integration involving Software and infrastructure components. 

At its most simplistic, Open Networking is about the Integration of infrastructure and software into performant solutions to meet customer needs.  Put simply, you could look at it this way:  

As expert practitioners in the field of “Open Networking”, including SDN/NFV and ONAP Aptira can help guide you through these considerations to a desired outcome. Just reach out to us so we can have a chat.

And we will expand on these topics and more in future articles.  Stay tuned.

The post What is Open Networking? Part 2 – The Three Domains of Open Networking appeared first on Aptira.

by Adam Russell at July 05, 2018 09:33 AM

July 04, 2018

RDO

Community Blog Round-Up: 04 July

So much happened over the past month that it’s definitely time to set off the fireworks! To start, Steve Hardy shares his tips and tricks for TripleO containerized deployments, then Zane Bitter talks discusses the ever expanding OpenStack Foundation, while Maria Bracho introduces us to Red Hat OpenStack Platform’s fast forward upgrades in a step-by-step overview, and so very much more. Obviously, prep the barbecue, it’s time for the fourth of July community blog round-up!

Red Hat OpenStack Platform: Two life-cycle choices to fit your organization by Maria Bracho, Principal Product Manager OpenStack

OpenStack®️ is a powerful platform for building private cloud environments that support modern, digital business operations. However, the OpenStack community’s six-month release cadence can pose challenges for enterprise organizations that want to deploy OpenStack in production. Red Hat can help.

Read more at https://redhatstackblog.redhat.com/2018/07/02/red-hat-openstack-platform-two-life-cycle-choices-to-fit-your-organization/

CPU model configuration for QEMU/KVM on x86 hosts by Daniel Berrange

With the various CPU hardware vulnerabilities reported this year, guest CPU configuration is now a security critical task. This blog post contains content I’ve written that is on its way to become part of the QEMU documentation.

Read more at https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/

Requirements for an OpenStack Access Control Policy Management Tool by Adam Young

“We need a read only role.”

Read more at https://adam.younglogic.com/2018/06/requirements-for-an-openstack-access-control-policy-management-tool/

Red Hat OpenStack Platform 13 is here! by Rosa Guntrip

Accelerate. Innovate. Empower. In the digital economy, IT organizations can be expected to deliver services anytime, anywhere, and to any device. IT speed, agility, and innovation can be critical to help stay ahead of your competition. Red Hat OpenStack Platform lets you build an on-premise cloud environment designed to accelerate your business, innovate faster, and empower your IT teams.

Read more at https://redhatstackblog.redhat.com/2018/06/27/red-hat-openstack-platform-13-is-here/

Red Hat Certified Cloud Architect – An OpenStack Perspective – Part Two by Chris Janiszewski – Senior OpenStack Solutions Architect – Red Hat Tiger Team

Previously we learned about what the Red Hat Certified Architect certification is and what exams are included in the “OpenStack-focused” version of the certification. This week we want to focus on personal experience and benefits from achieving this milestone.

Read more at https://redhatstackblog.redhat.com/2018/06/24/red-hat-certified-cloud-architect-an-openstack-perspective-part-two/

Red Hat OpenStack Platform fast forward upgrades: A step-by-step overview by Maria Bracho, Principal Product Manager OpenStack

New in Red Hat®️ OpenStack®️ Platform 13, the fast forward upgrade feature lets you easily move between long-life releases, without the need to upgrade to each in-between release. Fast forward upgrades fully containerize Red Hat OpenStack Platform deployment to simplify and speed the upgrade process while reducing interruptions and eliminating the need for additional hardware. Today, we’ll take a look at what the fast forward upgrade process from Red Hat OpenStack Platform 10 to Red Hat OpenStack Platform 13 looks like in practice.

Read more at https://redhatstackblog.redhat.com/2018/06/22/red-hat-openstack-platform-fast-forward-upgrades-a-step-by-step-overview/

Red Hat Certified Cloud Architect – An OpenStack Perspective – Part One by Chris Janiszewski – Senior OpenStack Solutions Architect – Red Hat Tiger Team

The Red Hat Certified Architect (RHCA) is the highest certification provided by Red Hat. To many, it can be looked at as a “holy grail” of sorts in open source software certifications. It’s not easy to get. In order to receive it, you not only need to already be a Red Hat Certified Engineer -(RHCE) for Red Hat Enterprise Linux (with the Red Hat Certified System Administrator, (RHCSA) as pre-requisite) but also pass additional exams from various technology categories.—

Read more at https://redhatstackblog.redhat.com/2018/06/21/red-hat-certified-cloud-architect-an-openstack-perspective-part-one/

Tips on searching ceph-install-workflow.log on TripleO by John

  1. Only look at the logs relevant to the last run

Read more at http://blog.johnlikesopenstack.com/2018/06/tips-on-searching-ceph-install.html

TripleO Ceph Integration on the Road in June by John

The first week of June I went to an upstream TripleO workshop in Brno. The labs we used are at https://github.com/redhat-openstack/tripleo-workshop

Read more at http://blog.johnlikesopenstack.com/2018/06/tripleo-ceph-integration-on-road-in-june.html

The Expanding OpenStack Foundation by Zane Bitter

The OpenStack Foundation has begun the process of becoming an umbrella organisation for open source projects adjacent to but outside of OpenStack itself. However, there is no clear roadmap for the transformation, which has resulted in some confusion. After attending the joint leadership meeting with the Foundation Board of Directors and various Forum sessions that included some members of the board at the (2018) OpenStack Summit in Vancouver, I believe I can help shed some light on the situation. (Of course this is my subjective take on the topic, and I am not speaking for the Technical Committee.)–

Read more at https://www.zerobanana.com/archive/2018/06/14#osf-expansion

Configuring a static address for wlan0 on Raspbian Stretch by Lars Kellogg-Stedman

Recent releases of Raspbian have adopted the use of dhcpcd to manage both dynamic and static interface configuration. If you would prefer to use the traditional /etc/network/interfaces mechanism instead, follow these steps.

Read more at https://blog.oddbit.com/2018/06/14/configuring-a-static-address-f/

Configuring collectd plugins with TripleO by mrunge

A way of deploying OpenStack is to use TripleO. This takes the an approach to deploy a small OpenStack environment, and then to take OpenStack provided infrastructure and tools to deploy the actual production environment.

Read more at http://www.matthias-runge.de/2018/06/08/tripleo-collectd/

TripleO Containerized deployments, debugging basics by Steve Hardy

Since the Pike release, TripleO has supported deployments with OpenStack services running in containers.- Currently we use docker to run images based on those maintained by the Kolla project.We already have some tips and tricks for container deployment debugging in tripleo-docs, but below are some more notes on my typical debug workflows.

Read more at https://hardysteven.blogspot.com/2018/06/tripleo-containerized-deployments.html

by Rain Leander at July 04, 2018 02:00 PM

July 03, 2018

Chris Dent

TC Report 18-27

This week's TC Report will be relatively short. I wrote a lot of OpenStack related words yesterday in Some Opinions On Openstack.

That post was related to one of the main themes that has shown up in IRC and email discussions recently: creating a technical vision for the near future of OpenStack. One idea has been to separate plumbing from porcelain. There's also a long email thread considering many ideas. One idea from that which I particularly like is unifying all the various agents that live on a compute node into one agent, one that likely talks to etcd. nodelet like a kubelet. None of this is something that will happen overnight. I hope at least some if it does, eventually.

Some change that's actually in progress now: For a long time OpenStack has tracked the organizational diversity of contributors to the various sub-projects. There's been a fair bit of talk that the tracking doesn't map to reality in a useful way and we need to figure out what to do about it. That has resulted in a plan to remove team diversity tags and instead use a more holistic approach to being aware of and dealing with what's now being called "fragility" in teams. One aspect of this is the human-managed health tracker.

Julia went to China for an OpenStack event and her eyes were opened about the different context contributors there experience. She wrote a superuser post and there's been subsequent related IRC discussion about the challenges that people in China experience trying to participate in OpenStack. More generally there is a need to figure out some ways to build a shared context that involves people who are not a part of our usual circles.

As usual, one of the main outcomes of that was that we need to make the time to share and talk more and in a more accessible fashion. We see bursts of that (we're seeing one now) but how do we sustain it and how do we extract some agreement that can lead to concerted action?

by Chris Dent at July 03, 2018 09:56 PM

SUSE Conversations

Calling All SUSE ISVs! Don’t Miss SUSECON in April 2019.

Attention ISVs! There’s a reason the Chameleon and his buddies are smiling: SUSE is on a roll. In just the last 60 days: SUSE announced plans to partner with EQT, a development-focused growth investor with extensive experience in the software industry, in order to launch SUSE as a fully independent company (to be completed by […]

The post Calling All SUSE ISVs! Don’t Miss SUSECON in April 2019. appeared first on SUSE Communities.

by frego at July 03, 2018 09:42 PM

OpenStack Superuser

The open source job market: Cloudy with no signs of containment, report says

A recent report finds an ongoing abundance of jobs and intense competition for professionals with open-source skills.

Calling it a “fierce market” that shows no signs of abating, the “2018 Open Source Jobs Report” from the Linux Foundation and job website Dice sounded out 750 hiring managers at corporations, small and medium businesses, government agencies and staffing firms worldwide as well as over 6,500 open-source professionals. The 13-page report is available as a .PDF at no cost but requires email registration.

Cloud/virtualization is still the most-sought after skill set by hiring managers at 66 percent, with containers coming up fast at 57 percent, more than doubling from 27 percent in 2016. Rounding out the top five technologies are security, sought by 49 percent of hiring managers, networking (47 percent) and cloud native computing (39 percent.)

Of the roles, recruiters are most actively seeking developers. These “most wanted” professionals say that their biggest challenges are ongoing training opportunities, missing documentation, buy-in from management and clear policies for contributing to open-source projects. Professionals in open source are steady with ongoing education, with 64 percent requesting or taking courses on their own, the same percentage as 2017.

What keeps these in-demand professionals from happy and from job hopping hasn’t changed over the last two years: only 3 percent  said money and perks were the best thing about their jobs, the same as 2017. Thirty percent cited the ability to work on interesting projects, collaborate with a global community (19 percent) and the opportunity to work on the most cutting edge technology challenges (16 percent) as what keeps them happy – the same percentages since 2016.

Interestingly, the outlook on diversity isn’t as rosy as the job market itself. While some 70 percent of hiring managers rate their efforts on this front as “good,” that opinion is shared by only 52 percent of open-source professionals. Sixteen percent of professionals rate these efforts at their company as “poor” or “extremely poor,” while only eight percent of recruiters did. The silver lining: some 79 percent of recruiters are actively seeking diversity now.

Download the report here.

Photo // CC BY NC

The post The open source job market: Cloudy with no signs of containment, report says appeared first on Superuser.

by Superuser at July 03, 2018 04:32 PM

July 02, 2018

OpenStack Superuser

Why free software needs free tools

History often repeats itself and the open-source world is no exception.

University of Washington’s Benjamin Mako Hill believes that the Linux community is having  a “Groundhog Day” moment. During his talk at the OpenDev Conference, Hill explained how the current reliance on proprietary software continues to be problematic. His talk was prescient; a virtual Greek chorus piped up after Microsoft  bought GitHub.

Thirteen years ago, BitKeeper became the standard way to manage versioning. It replaced CVS, which was groaning under the weight of a more modern Linux. While the community version of BitKeeper allowed Linux developers use it for free, it also signed into a sever that kept tabs on what they were using it for, and how they were using it. Several notable developers, including Alan Cox, who maintained large chunks of the Linux kernel at the time, refused to use BitKeeper. Instead, they had to rely on workarounds to keep their software up to date.

Andrew Tridgell came up with an alternative to BitKeeper, which he reverse engineered from the help files of BitKeeper itself. The owner of BitKeeper, Larry McCoy, found out and threatened to withdraw the BitKeeper license from all other developers at the company, which included Linus Torvalds, the father of Linux. Torvalds knew that Linux couldn’t return to CVS, so he and some other kernel developers spent the time that they could have been developing Linux itself to create brand new free software to do what BitKeeper did. The result became Git, which was eventually adopted by developers as the main way to manage their software.

GitHub soon became the way most developers interact with Git. Ironically, this proprietary system now has the same issues as BitKeeper (now open source) did. “GitHub, like BitKeeper, relies on a central server that allows [it] to monitor users,” said Hill. “GitHub, like BitKeeper, doesn’t release its server-side source code or show us what’s running on it (at least not most of it).”

Similarly, GitHub is a for-profit company that allows the developer community to use its tools for no cost as long as it’s used to build free and open source software, just like, you guessed it, BitKeeper. “Finally, GitHub, like BitKeeper, can turn any of us off without any warning without any notice and for any reason they want,” said Hill as he showed a slide with a relevant passage from GitHub’s terms of service.

This isn’t to say GitHub is doing anything inherently malicious. This is just what happens when a for-profit company offers proprietary tools for open source development. “I think it behooves us to learn from our mistakes in the past,” Hill said, “and I can’t help but think that our contemporary free and open source software world’s continued reliance on proprietary tools and proprietary services suggests that we failed to learn from our experience with Linux and BitKeeper.”

The lessons we’ve learned, said Hill, are fairly clear. While the makers of proprietary software legitimately want to help and their proprietary tools can indeed help develop open source software, the community must reject these tools from the start. To reiterate: “When we’re building free software or open source tools, we must reject proprietary software development tools,” said Hill.

Using proprietary tools undermines the whole concept behind free and open source software. “It makes us hypocrites,” Hill said. In addition, the software created is only as free as the software it depends on for its continued use, distribution and evolution.

Ultimately, Hill encourages open source developers to use free tools. He believes developers should use open alternatives like GitLab instead of GitHub. As many of these alternatives are still not quite as “good” as proprietary solutions, Hill recommends fighting for, improving and creating these free tools in order to avoid these lessons of the past.

Check out the full talk below.

Photo // CC BY NC

The post Why free software needs free tools appeared first on Superuser.

by Rob LeFebvre at July 02, 2018 04:29 PM

Chris Dent

Some Opinions on OpenStack

In a recent TC Report I said we need more 'written works...that layout a vision of how things could or should be that community members can react and refer to' and 'Many of the things that get written will start off wrong but the only way they have a chance of becoming right is if they are written in the first place. Providing ideas allows people to say "that's right" or "that's wrong" or "that's right, except...'.

The point of writing stuff like this is so there can be a reaction and a process of refinement where we can reach some ideas that we all can live with. When I write, I hope that when you read this you don't think of ways to shoot me down and tell me I'm wrong, but instead think of ways to make the ideas more right.

This is a collection of opinions about the state of OpenStack that I wrote down this morning while I had no connection to the internet. There are some ideas (many wrong) that I hope can provide some grist for discussions surrounding building a tech vision and working through some of the pains we're experiencing while evolving as a community.

This only way in which this post is actionable is as a way to have further discussion. It's not really an accurate description of the current reality, nor a prescription for how to get to a new one from here. There are an enormous number of people and activities with whom I have no experience. These ideas take no account of those. I'm not leaving something out because I don't care about it, but because I'm ignorant. Want me to know something? Tell me about it.

These are vast glittering generalities with sweeping gestures. To me, these provide the foundations upon which to think about stuff and understand it. It's how I see the world: vast pattern-based analogies. For others it is annoying because there's no concrete advice. We probably need both, and each feeds the other, so I hope this is useful in what amounts to an aspirational fashion: It would be nice if we could get closer to $something even if only by a little bit.

With all that out of the way, let's begin.

The current work on the technical vision is mostly oriented around describing what OpenStack is and what OpenStack can become. A lot of that is trying to discern what different groups have thought it to be. A thing which provides the building blocks for enabling a market. Or the blocks for building a datacentre. Or not solely blocks, but a full service cloud.

Each of these is a reasonable thing but none of it is a description for how to build or how to make decisions in the face of ambiguity. We need answers to questions related to the boundaries between things and how they are composed. Descriptions of not only vertical layers but also horizontal groupings. Common wisdom to live by.

And in all these areas the questions are not limited to technical. OpenStack is a complex system of people, companies, and technology, all making competing and conflicting demands.

Because of the way OpenStack chose to model itself, the system (technical and social) is simultaneously centralized and siloed, leading to the now well-known problems of fragility and unreliable views and understanding (of social and technical state).

Random Ideas on Maintainership

As the number of people available to provide large amounts of time to keeping OpenStack healthy shrinks, the need to focus on maintenance grows. Those people who are most present must focus their energy on enabling other people. They should be maintainers, foremost, and focus on:

  • bugs before features
  • code maintainability before features
  • refactoring to enhance code scrutability before features
  • code reviews before features
  • enabling other people to write features

If no one shows up to write features, that presumably means the features aren't all that important.

This kind of focus requires that corporate contributors appreciate and acknowledge the importance of providing maintainers. This feels like a hard sell, but we can make its importance more obvious by not squeezing these maintenance tasks in between other "more important" stuff or after hours. Do the maintenance tasks first, go home at the end of the day, and let the other things lag. Don't be a hero. Otherwise, everyone else is forced to be one too and the true cost of doing business is obscured.

Anyone who self-identifies as a "maintainer" should consider themselves a maintainer of OpenStack, not of an individual project (e.g., nova, cinder, etc). If we can break down our own mental barriers between projects we can break down some of the classism in OpenStack society and pay more effective attention to managing the technical boundaries between projects.

Random Ideas on State and State Management

  • In an OpenSource and multi-purpose system like OpenStack, the purpose of an HTTP API is to both enable and enforce boundaries and to not share state. They are for exposing state and manipulating state which can be used for both planned and unplanned uses. If you don't want unplanned uses, don't use an HTTP API.
  • You want unplanned uses. That's the engine of innovation.
  • If the real world can provide information with sufficient reliability there's little point modelling and mirroring that world in the artifact world of a database.
  • What are the reasons for why each sub-project uses its own database? Managing migrations? Scaling and distribution opportunities? What else? Is it worth it?

Random Ideas on Architecture and Design

Discussing OpenStack architecture can easily become a dark depressing hole as we get trapped in a state of "how can we get there from here?". We can work on that. We have to. It may be necessary to break some of our own rules to achieve it.

These range all over the place. Some of them are obvious or repeats of things we've heard many times.

  • Single responsibility principle is a good one.
  • Separate specific actions (launch a VM with these resources) from orchestration (give me a vm of flavor X, affined to group Z, using block storage A, with network B, running application C).
  • Make the actions as granular and composable as possible, in both use and code.
  • Put UI/UX at the level of orchestration. Thus the "public" HTTP APIs too.
  • At each layer produce enough information about activity so the layer above can consume it usefully and make some decisions. This is akin to backpressure.
  • Anything can be a layer to something else. One person's top is another person's bottom.
  • Don't push lots of info to things. Instead push typed events with identifiers that allow the recipient to choose what and how much information they retrieve. For example: What would it take and what would it achieve if we were to stop doing RPC with fat objects and instead pass identifiers which allow agents to retrieve (only) the necessary information?
  • If there is a already a type of tool that provides useful and mature functionality, don't reinvent tools in that class and don't use something weird. Use standard and boring stuff that can work and be forgotten. The classic misstep in OpenStack was using the eventlet-based web server. If you are making a fairly simply CRUD-style web service, then make a WSGI application that can be hosted by any WSGI server and don't delve into the complexity of using eventlet.

Random Ideas on Project Inclusion

  • It should not be the job of OpenStack to audit, bill, monitor or archive. Instead OpenStack should be auditable, billable, monitorable, and archivable. That is, it should be observable and eventing.
  • Similarly, if it is (is it?) to be OpenStack's job to install itself, then there should be one good way to do that which OpenStack (the community) owns, a way that enables realistic and quick trials and experimentation. If we're not going to do that, then OpenStack should enable itself to be installed in various ways, but tools to do it are not OpenStack itself. Instead they are very good friends of OpenStack.

Random Ideas on Code Style

  • Using globals to avoid a bit of boilerplate is often not a good trade. I'm looking at you oslo.config.
  • If you're writing code that requires mocks to test it, you have just discovered an opportunity for refactoring. If you're writing a new module or class, write your tests first, because then you'll be inclined to avoid mocks and as a result will be writing more maintainable, scrutable, and reusable code.
  • We continue to use Python because it has many advantages with regards to maintainability and readability. Don't let Python down by building classes, methods and files that are 100s or even 1000s of lines long. Try to be idiomatic in your use of Python. import this and other clichés.

Random Ideas on Random Ideas

  • Don't use IRC to share information on the fly or build decisions and expect them to stick. Doing so leaves out vast swathes of the community. Either because they are asleep or because they can't use IRC. The only reliable messaging platforms we have are email lists and gerrit reviews.
  • If you're a long term member of the community, one of your main jobs is making more members of the community. One way to do this is by sharing random ideas in a way that people can see them. If you're spending too much time hanging out in your echo chamber with your chums, you're building confirmation and survivorship bias.
  • For reasons I can't quite discern, there are karmic social forces at work in OpenStack that work against spouting random ideas and engaging people in conversation that questions the current state of things. This is a great way to never change anything. I hope we've finally learned we need to change.

All of the above statements could very well be wrong (some assuredly are), but they are related to topics that keep coming up. Some kind of resolution needs to happen, somehow. We've managed reasonably well so far, and we can continue to manage, even in the face of the changes we are witnessing. But wouldn't it be nice to do more than manage?

by Chris Dent at July 02, 2018 01:15 PM

Red Hat Stack

Red Hat OpenStack Platform: Two life-cycle choices to fit your organization

OpenStack®️ is a powerful platform for building private cloud environments that support modern, digital business operations. However, the OpenStack community’s six-month release cadence can pose challenges for enterprise organizations that want to deploy OpenStack in production. Red Hat can help.

elizabeth-lies-20237-unsplash
Photo by elizabeth lies on Unsplash

Red Hat®️ OpenStack Platform is an intensely tested, hardened, and supported distribution of OpenStack based on community releases. In addition to production-grade features and functionality, it gives you two life-cycle choices to align with the way your organization operates:

  • Standard releases. These releases follow the six-month community release cadence and include one year of support.
  • Long-life releases. Starting with Red Hat OpenStack Platform 10, every third release is a long-life release. These include three years of support, with option to extend support for an additional two years with extended life-cycle support (ELS), for up to five years of support total.

Why does this matter? Different organizations have different needs when it comes to infrastructure life cycles and management. Some need to implement the latest innovations as soon as they are available, and have the processes in place to continuously upgrade and adapt their IT environment. For others, the ability to standardize and stabilize operations for long durations of time is paramount. These organizations may not need the newest features right away—periodic updates are fine.

tristan-colangelo-39719-unsplash
Photo by Tristan Colangelo on Unsplash

Red Hat OpenStack Platform life-cycle options accommodate both of these approaches. Organizations that need constant innovation can upgrade to the latest Red Hat OpenStack Platform release every six months to take advantage of new features as they become available. Organizations that prefer to use a given release for a longer time can skip standard releases and simply upgrade between long-life releases every 18 to 60 months.

Here’s a deeper look into each option and why you might choose one over the other.

Standard upgrade path

With this approach, you upgrade every six to twelve months as a new release of Red Hat OpenStack Platform is made available. Red Hat OpenStack Platform director provides upgrade tooling to simplify the upgrade process. As a result, you can adopt the latest features and innovations as soon as possible. This keeps your cloud infrastructure aligned closely with the upstream community releases, so if you’re active in the OpenStack community, you’ll be able to take advantage of your contributions sooner.

This upgrade path typically requires organizations to have processes in place to efficiently manage continuously changing infrastructure. If you have mature, programmatic build and test processes, you’re in good shape.

The standard upgrade path is ideal for organizations involved in science and research, financial services, and other fields that innovate fast and change quickly.

jordan-ladikos-62738-unsplash
Photo by Jordan Ladikos on Unsplash 

 

Long-life upgrade path

With this approach, you upgrade every 18 to 60 months between long-life releases of Red Hat OpenStack Platform, skipping two standard releases at a time. Starting with Red Hat OpenStack Platform 13, the fast forward upgrade feature in director simplifies the upgrade process by fully containerizing Red Hat OpenStack Platform deployment. This minimizes interruptions due to upgrading and eliminates the need for additional hardware to support the upgrade process. As a result, you can use a long-life release, like Red Hat OpenStack Platform 10 or 13, for an extended time to stabilize operations. Based on customer requests and feasibility reviews, select features in later standard releases may be backported to the last long-life release (Full Support phase only), so you can still gain access to some new features between upgrades.

The long-life upgrade path works well for organizations that are more familiar and comfortable with traditional virtualization and may still be adopting a programmatic approach to IT operations.

This path is ideal for organizations that prefer to standardize on infrastructure and don’t necessarily need access to the latest features right away. Organizations involved in telecommunications and other regulated fields often choose the long-life upgrade path.

Wrapping up

With two life-cycle options for Red Hat OpenStack Platform, Red Hat supports you no matter where you are in your cloud journey. If you have questions about which path is best for your organization, contact us and we’ll help you get started.

Learn more about Red Hat OpenStack Platform:

by Maria Bracho, Principal Product Manager OpenStack at July 02, 2018 12:36 PM

June 29, 2018

OpenStack Superuser

Project update: What’s next for Neutron

At the recent Vancouver Summit OpenStack project team leads (PTLs) and core team members offered updates for the OpenStack projects they manage, plus how you can get involved and influence the roadmap.

Superuser features summaries of the videos; you can also catch them on the OpenStack Foundation YouTube channel.


What
Neutron’s goal is to implement services and associated libraries to provide on-demand, scalable and technology-agnostic network abstraction. Neutron provides networking-as-a-service between interface devices (e.g., vNICs) managed by other Openstack services (e.g., Nova, the compute service). There were 240 contributors for the current release for a project that 86 percent of OpenStack users deploy, according to the last User Survey.

Who Miguel Lavalle, PTL for the Queens and Rocky releases.

What’s new

What’s next

Here are some of the features, improvements and fixes to expect from the upcoming Rocky release.

Get involved!
Use Ask OpenStack for general questions
For roadmap or development issues, subscribe to the OpenStack development mailing list, and use the tag [neturon]
To get code, ask questions, view blueprints, etc, see: Neutron Launchpad Page
Check out Neutron’s regular IRC meetings on the #openstack-meeting channel: http://wiki.openstack.org/Network/Meetings

Catch the full 40-minute talk below and download the slides here.

 

The post Project update: What’s next for Neutron appeared first on Superuser.

by Superuser at June 29, 2018 04:03 PM

Chris Dent

Placement Update 18-26

This is placement update 18-26, a weekly update of ongoing development related to the OpenStack placement service. This is an expand version.

For the next few weeks the "Specs" section will not be present. When we start reviewing specs for Stein, I'll add it back in.

Most Important

Nested allocation candidates are getting very close, but remain a critical piece of functionality. After that is making sure that we are progressing on the /reshapher functionality and bringing the various virt drivers into line with all this nice new functionality (which mostly means ProviderTrees).

All that nice new functionality means bugs. Experiment. Break stuff. Find bugs. Fix them.

Speaking of bugs: a collection of problems—not covered by tests— with consumer generations was discovered this week. Also a problem with the limit functionality on GET /allocation_candidates and how that works when force_hosts is being used. Fixes are in progress but these issues are a strong indicator of our need to make sure that we are experimenting with things: it's where features integrate with each other that problems are showing up.

What's Changed

Quite a lot of bug fixes and bug demonstrations have merged in this week, but features mostly stable.

Bugs

Questions

As far as I can tell there was no discussion on last week's question, so here it is again:

In IRC [last week] we had an extensive discussion about being able to set custom resource classes on the resource provider representing a compute node, outside the virt driver. At the moment the virt driver will clobber it. Is this what we always want?

Main Themes

Documentation

This is a section for reminding us to document all the fun stuff we are enabling. Open areas include:

  • Documenting optional placement database. A bug, 1778227 has been created to track this. This has started, for the install docs, but there are more places that need to be touched.

  • "How to deploy / model shared disk. Seems fairly straight-forward, and we could even maybe create a multi-node ceph job that does this - wouldn't that be awesome?!?!", says an enthusiastic Matt Riedemann.

  • The when's and where's of re-shaping and VGPUs.

Nested providers in allocation candidates

As far as I can tell the main thing left here is to turn it on in a microversion. That code is at:

Consumer Generations

There are new patches in progress on this, related to the bugs that were discovered:

There are a patches left on the consumer generation topic, some tidy ups, and some stuff related to healing allocations:

Is someone already working on code for making use of this in the resource tracker?

Reshape Provider Trees

This allows moving inventory and allocations that were on resource provider A to resource provider B in an atomic fashion. The blueprint topic is:

There are WIPs for the HTTP parts and the resource tracker parts, on that topic.

Mirror Host Aggregates

This needs a command line tool:

Extraction

A while back, Jay made a first pass at an os-resource-classes, which needs some additional eyes on it. I personally thought it might be heavier than required. If you have ideas please share them.

An area we will need to prepare for is dealing with the various infra and co-gating issues that will come up once placement is extracted.

We also need to think about how to manage the fixtures currently made available by nova that we might need or want to use in placement. Some of them might be worth sharing. How should we do that?

Other

18 entries last week. 24 now.

End

A butterfly just used my house as a south to north shortcut. That'll do.

by Chris Dent at June 29, 2018 01:00 PM

Daniel P. Berrangé

CPU model configuration for QEMU/KVM on x86 hosts

With the various CPU hardware vulnerabilities reported this year, guest CPU configuration is now a security critical task. This blog post contains content I’ve written that is on its way to become part of the QEMU documentation.

QEMU / KVM virtualization supports two ways to configure CPU models

Host passthrough
This passes the host CPU model features, model, stepping, exactly to the guest. Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization. Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts. This is the recommended CPU to use, provided live migration is not required.
Named model
QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD. These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.

In both cases, it is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.

Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with “Host passthrough”, but gives much of the benefit of passthrough, while making live migration safe.

Recommendations for KVM CPU model configuration on x86 hosts

The information that follows provides recommendations for configuring CPU models on x86 hosts. The goals are to maximise performance, while protecting guest OS against various CPU hardware flaws, and optionally enabling live migration between hosts with hetergeneous CPU models.

Preferred CPU models for Intel x86 hosts

The following CPU models are preferred for use on Intel hosts. Administrators / applications are recommended to use the CPU model that matches the generation of the host CPUs in use. In a deployment with a mixture of host CPU models between machines, if live migration compatibility is required, use the newest CPU model that is compatible across all desired hosts.

Skylake-Server
Skylake-Server-IBRS
Intel Xeon Processor (Skylake, 2016)
Skylake-Client
Skylake-Client-IBRS
Intel Core Processor (Skylake, 2015)
Broadwell
Broadwell-IBRS
Broadwell-noTSX
Broadwell-noTSX-IBRS
Intel Core Processor (Broadwell, 2014)
Haswell
Haswell-IBRS
Haswell-noTSX
Haswell-noTSX-IBRS
Intel Core Processor (Haswell, 2013)
IvyBridge
IvyBridge-IBRS
Intel Xeon E3-12xx v2 (Ivy Bridge, 2012)
SandyBridge
SandyBridge-IBRS
Intel Xeon E312xx (Sandy Bridge, 2011)
Westmere
Westmere-IBRS
Westmere E56xx/L56xx/X56xx (Nehalem-C, 2010)
Nehalem
Nehalem-IBRS
Intel Core i7 9xx (Nehalem Class Core i7, 2008)
Penryn
Intel Core 2 Duo P9xxx (Penryn Class Core 2, 2007)
Conroe
Intel Celeron_4x0 (Conroe/Merom Class Core 2, 2006)

Important CPU features for Intel x86 hosts

The following are important CPU features that should be used on Intel x86 hosts, when available in the host CPU. Some of them require explicit configuration to enable, as they are not included by default in some, or all, of the named CPU models listed above. In general all of these features are included if using “Host passthrough” or “Host model”.

pcid
Recommended to mitigate the cost of the Meltdown (CVE-2017-5754) fix. Included by default in Haswell, Broadwell & Skylake Intel CPU models. Should be explicitly turned on for Westmere, SandyBridge, and IvyBridge Intel CPU models. Note that some desktop/mobile Westmere CPUs cannot support this feature.
spec-ctrl
Required to enable the Spectre (CVE-2017-5753 and CVE-2017-5715) fix, in cases where retpolines are not sufficient. Included by default in Intel CPU models with -IBRS suffix. Must be explicitly turned on for Intel CPU models without -IBRS suffix. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.
ssbd
Required to enable the CVE-2018-3639 fix. Not included by default in any Intel CPU model. Must be explicitly turned on for all Intel CPU models. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.
pdpe1gb
Recommended to allow guest OS to use 1GB size pages.Not included by default in any Intel CPU model. Should be explicitly turned on for all Intel CPU models. Note that not all CPU hardware will support this feature.

Preferred CPU models for AMD x86 hosts

The following CPU models are preferred for use on Intel hosts. Administrators / applications are recommended to use the CPU model that matches the generation of the host CPUs in use. In a deployment with a mixture of host CPU models between machines, if live migration compatibility is required, use the newest CPU model that is compatible across all desired hosts.

EPYC
EPYC-IBPB
AMD EPYC Processor (2017)
Opteron_G5
AMD Opteron 63xx class CPU (2012)
Opteron_G4
AMD Opteron 62xx class CPU (2011)
Opteron_G3
AMD Opteron 23xx (Gen 3 Class Opteron, 2009)
Opteron_G2
AMD Opteron 22xx (Gen 2 Class Opteron, 2006)
Opteron_G1
AMD Opteron 240 (Gen 1 Class Opteron, 2004)

Important CPU features for AMD x86 hosts

The following are important CPU features that should be used on AMD x86 hosts, when available in the host CPU. Some of them require explicit configuration to enable, as they are not included by default in some, or all, of the named CPU models listed above. In general all of these features are included if using “Host passthrough” or “Host model”.

ibpb
Required to enable the Spectre (CVE-2017-5753 and CVE-2017-5715) fix, in cases where retpolines are not sufficient. Included by default in AMD CPU models with -IBPB suffix. Must be explicitly turned on for AMD CPU models without -IBPB suffix. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.
virt-ssbd
Required to enable the CVE-2018-3639 fix. Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models. This should be provided to guests, even if amd-ssbd is also provided, for maximum guest compatibility. Note for some QEMU / libvirt versions, this must be force enabled when when using “Host model”, because this is a virtual feature that doesn’t exist in the physical host CPUs.
amd-ssbd
Required to enable the CVE-2018-3639 fix. Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models. This provides higher performance than virt-ssbd so should be exposed to guests whenever available in the host. virt-ssbd should none the less also be exposed for maximum guest compatability as some kernels only know about virt-ssbd.
amd-no-ssb
Recommended to indicate the host is not vulnerable CVE-2018-3639. Not included by default in any AMD CPU model. Future hardware genarations of CPU will not be vulnerable to CVE-2018-3639, and thus the guest should be told not to enable its mitigations, by exposing amd-no-ssb. This is mutually exclusive with virt-ssbd and amd-ssbd.
pdpe1gb
Recommended to allow guest OS to use 1GB size pages. Not included by default in any AMD CPU model. Should be explicitly turned on for all AMD CPU models. Note that not all CPU hardware will support this feature.

Default x86 CPU models

The default QEMU CPU models are designed such that they can run on all hosts. If an application does not wish to do perform any host compatibility checks before launching guests, the default is guaranteed to work.

The default CPU models will, however, leave the guest OS vulnerable to various CPU hardware flaws, so their use is strongly discouraged. Applications should follow the earlier guidance to setup a better CPU configuration, with host passthrough recommended if live migration is not needed.

qemu32
qemu64
QEMU Virtual CPU version 2.5+ (32 & 64 bit variants). qemu64 is used for x86_64 guests and qemu32 is used for i686 guests, when no -cpu argument is given to QEMU, or no <cpu> is provided in libvirt XML.

Other non-recommended x86 CPUs

The following CPUs models are compatible with most AMD and Intel x86 hosts, but their usage is discouraged, as they expose a very limited featureset, which prevents guests having optimal performance.

kvm32
kvm64
Common KVM processor (32 & 64 bit variants). Legacy models just for historical compatibility with ancient QEMU versions.
486
athlon
phenom
coreduo
core2duo
n270
pentium
pentium2
pentium3
Various very old x86 CPU models, mostly predating the introduction of hardware assisted virtualization, that should thus not be required for running virtual machines.

Syntax for configuring CPU models

The example below illustrate the approach to configuring the various CPU models / features in QEMU and libvirt

QEMU command line

Host passthrough
   $ qemu-system-x86_64 -cpu host

With feature customization:

   $ qemu-system-x86_64 -cpu host,-vmx,...
Named CPU models
   $ qemu-system-x86_64 -cpu Westmere

With feature customization:

   $ qemu-system-x86_64 -cpu Westmere,+pcid,...

Libvirt guest XML

Host passthrough
   <cpu mode='host-passthrough'/>

With feature customization:

   <cpu mode='host-passthrough'>
       <feature name="vmx" policy="disable"/>
       ...
   </cpu>
Host model
   <cpu mode='host-model'/>

With feature customization:

   <cpu mode='host-model'>
       <feature name="vmx" policy="disable"/>
       ...
   </cpu>
Named model
   <cpu mode='custom'>
       <model name="Westmere"/>
   </cpu>

With feature customization:

   <cpu mode='custom'>
       <model name="Westmere"/>
       <feature name="pcid" policy="require"/>
       ...
   </cpu>

 

by Daniel Berrange at June 29, 2018 12:49 PM

June 28, 2018

Adam Young

Requirements for an OpenStack Access Control Policy Management Tool

“We need a read only role.”

It seems like such a simple requirement.  Users have been requesting a read-only role for several years now.  Why is it so tough to implement?   Because it calls for  modifying access control policy across multiple, disjoint services deployed at innumerable distinct locations.

“We need help in modifying policy to implement our own read only role.”

This one is a little bit more attainable.  We should be able to provide better tools to help people customize their policy.  What should that look like?

We gathered some information at the last summit, and I am going to try and distill it to a requirements document here.

Definitions

  • Verb and Path:  the combination of the HTTP verb and the templated sub path that is used by the mapping engines.  If I were to use Curl to call https://hostname:5000/v3/users/a0123ab6, the verb would be the implicit GET, and the path would be /v3/users/{user_id}.
  • policy key:  the key in the policy.json and policy.yaml file that is used to match the python code to the policy.  For example, the Keystone GET /v3/user/{user_id} verb and path tests against the policy key identity:get_user.
  • API Policy Mapping:  the mapping from Verb and Path to Policy key.

The tool needs to be run from the installer. While that means Tripleo for my team, it should be a tool that can be enlisted into any of the installers.  It should also be able to run for day 2 operations from numerous tools.

It should not be deployed as a standard service, at least not one tied in with the active OpenStack install, as modifying policy is a tricky and potentially destructive and dangerous operation.

Input

Policy files need to be gathered from the various services, but this tool does not need to do that; the variations in how to generate, collect, and distribute policy files are too numerous to solve in a single, focused tool.  The collection and distribution fits more into Ansible playbooks than a tool for modifying policy.

External API definitions

End users need to be able to test their policy.  While the existing oslo-policy command line can tell whether a token would or would not pass the checks, those are done at the policy key level.  All integration is done at the URL level, even if it then passes through libraries or the CLI.  The Verb and URL  can be retrieved from network tools or debug mode of the CLI, and matched against the tuple of (service,verb,template path) to link back to the policy key, and the thus the policy rule that oslo-policy will enforce.  Deducing this mapping must be easy.  With this mapping, additional tools can mock a request/response to test whether a given set of auth-data would pass or fail a request.  Thus, the tool should accept a simple format for uploading the mappings of Verb and Path to policy key.

Policy.json

Policy files have several implementations.  The old Policy.json structure provides the least amount of information. Here is a sample:

"context_is_admin": "role:admin",
"default": "role:admin",

"add_image": "",
"delete_image": "",
"get_image": "",
"get_images": "",
"modify_image": "",
"publicize_image": "role:admin",
"copy_from": "",

policy.yaml

The policy in code structure provides the most, including the HTTP Verbs and templated Paths that map to the rules that are the keys in the policy files. The Python code that is used by oslo-policy to generate the sample YAML files uses, but does not expose, all that data.  Here is an example:

# This policy only checks if the user has access to the requested
# project limits. And this check is performed only after the check
# os_compute_api:limits passes
# GET /limits
# "os_compute_api:os-used-limits": "rule:admin_api"

A secondary tool should expose all this data as YAML , probably a modification of the oslo-policy CLI.  The management tool should be able to consume this format.  It should also be able to consume a document that maps the policy keys  to the Verb and Path separate from the policy

Upgrades

A new version of an OpenStack service will likely have new APIs.  These APIs will not be covered by existing policy.  However, if a site has made major efforts into customizing policy in the past, they will not want to lose and redo all of their changes.  Thus, it should be possible to upload a new file indicating the over all or just changes to the API mapping from a previous version.  If an updated policy-in-code format is available, that file should merge in with the existing policy modifications.  The user needs to be able to identify

  • Any new APIs that require application of the transformations listed below
  • Any changes to base policy that the user has customized and now conflict with the assumptions.  The tool user should be able to accept the old version, the new version, or come up with a modified new, manually merged version.

Transformations

End users need to be able to describe the transformations that then need to perform in simple terms.  Here are some that have been identified so far:

  • ensure that all APIs match against some role
  • ensure that APIs that require an role (especially admin) also perform a scope check
  • switch the role used for a given operation or set of operations
  • standardize the meaning of interim rules such as “owner.”
  • Inline an interim rule into the rules that use it
  • Extract an interim rule from all the rules that have a common fragment

Implied Roles

The Implied Roles mechanism provides support for policy,  The tool should be able to help the tool users to take advantage of implied roles.

  • Make use of implied roles to simplify complex matching rules
  • Make use of implied roles to provide additional granularity for an API:
  • Make it possible to expand implied rules in the policy file based on a data model

Change sets

The operations to transform the rules are complex enough that users will need to be able to role them forward and back, much like a set of changes to a git repository.

User Interface

While the tool should be visible, the majority of the business logic should reside in an API that is callable from other systems.  This seems to imply a pattern of REST API + A visible UI toolkit.

The User Interface should make working with large sets of rules possible and convenient.  Appropriate information hiding and selection should be coupled with the transformations to select the set of rules to be transformed.

Datastore

The data store for the application should be light enough to run during the install process. For example, SQLite would be preferred over MySQL.

Output

The tool should be able to produce the individual policy files consumed by the APIs.

It is possible to have a deployment where different policy is in place for different endpoints of the same service.  The tools should support endpoint specific overrides.  However, the main assumption is that these will be small changes from the core service definitions.  As such, they should be treated as “service X plus these changes” as opposed to a completely separate set of policy rules.

 

by Adam Young at June 28, 2018 06:57 PM

OpenStack Superuser

Translating context: Understanding the global open source community

Recently I had the opportunity to attend OpenInfra Days in Beijing. This was the first time I’ve ever been to China and perhaps one of the most eye opening experiences for me in my OpenStack career.

At 9 a.m. sharp, the conference hall filled towards standing-room only and the keynote started. It was a little loud but it was a very large room with what seemed like at least a thousand attendees. One of the organizers mentioned that attendees typically live streamed the keynote sessions as opposed to attending in person. Live translation was being performed and transmitted in the room because only some portions of the keynote were delivered in English.

The communication style was not what I would have expected at an OpenStack summit. In a sense, the focus of the speakers was to convey their message. The focus of the attendees was to take in the message from the speaker.

The second day is was when my perceptions were completely challenged.

One of the reasons I made the trip was to be able to talk to several companies that are using Ironic, obtain feedback and hopefully use that to better inform our community direction. I was unprepared for some of the discussions of scale. Problems that seemed to be fairly unique to one or two deployments around the world suddenly existed in multiple large scale deployments.

Some of the problems that experienced by these deployments were things that had already been solved, but in other cases the uses of the software were just different enough that without a solid understanding of the context of the problem and use case, the larger community would be unable to adequately discuss.

Each discussion shifted from “How can we implement x? How can we contribute back to the community?” and led to “How can we contribute?! We’re small! Not a super large well-known name!”

It immediately became clear that the larger context of not just of the status of contributions but also underlying reasons that suffered from missing context. In the end, we all look at problems the same way. We try and take what we believe to be the logical path forward.

Those of us who are more active day-to-day in our corner of the community operate with that as our community context. We need to remember to listen and that the people we talk to day in and day out are not the the whole community, but only a fraction of it. Communities are always larger than we can possibly perceive and we need to keep that in mind at all times.

And so, moving forward, I will challenge my preconceptions for I now know there’s an even larger world.

We need to better engage and not expect contributors to have our context. We need to share our context and keep in mind that the ways to communicate can be different. No single way is going to ever work.

And now I can be found on WeChat as juliaashleykreger.

The post Translating context: Understanding the global open source community appeared first on Superuser.

by Julia Kreger at June 28, 2018 04:12 PM

Red Hat Stack

Red Hat OpenStack Platform 13 is here!

Accelerate. Innovate. Empower.

In the digital economy, IT organizations can be expected to deliver services anytime, anywhere, and to any device. IT speed, agility, and innovation can be critical to help stay ahead of your competition. Red Hat OpenStack Platform lets you build an on-premise cloud environment designed to accelerate your business, innovate faster, and empower your IT teams.

Logotype_RH_OpenStackPlatform_RGB_Black (2)

Accelerate. Red Hat OpenStack Platform can help you accelerate IT activities and speed time to market for new products and services. Red Hat OpenStack Platform helps simplify application and service delivery using an automated self-service IT operating model, so you can provide users with more rapid access to resources. Using Red Hat OpenStack Platform, you can build an on-premises cloud architecture that can provide resource elasticity, scalability, and increased efficiency to launch new offerings faster.

Innovate. Red Hat OpenStack Platform enables you differentiate your business by helping to make new technologies more accessible without sacrificing current assets and operations. Red Hat’s open source development model combines faster-paced, cross-industry community innovation with production-grade hardening, integrations, support, and services. Red Hat OpenStack Platform is designed to provide an open and flexible cloud infrastructure ready for modern, containerized application operations while still supporting the traditional workloads your business relies on.

Empower. Red Hat OpenStack Platform helps your IT organization deliver new services with greater ease. Integrations with Red Hat’s open software stack let you build a more flexible and extensible foundation for modernization and digital operations. A large partner ecosystem helps you customize your environment with third-party products, with greater confidence that they will be interoperable and stable.

With Red Hat OpenStack Platform 13, Red Hat continues to bring together community-powered innovation with the stability, support, and services needed for production deployment. Red Hat OpenStack Platform 13 is a long-life release with up to three years of standard support and an additional, optional two years of extended life-cycle support (ELS). This release includes many features to help you adopt cloud technologies more easily and support digital transformation initiatives.

Fast forward upgrades

With both standard and long-life releases, Red Hat OpenStack Platform lets you choose when to implement new features in your cloud environment:

  • Upgrade every six months and benefit from one year of support on each release.
  • Upgrade every 18 months with long-life releases and benefit from 3 years of support on that release, with an optional ELS totalling to up to 5 years of support. Long life releases include innovations from all previous releases.

Now, with the fast forward upgrade feature, you can skip between long-life releases on an 18-month upgrade cadence. Fast forward upgrades fully containerize Red Hat OpenStack Platform deployment to simplify the process of upgrading between long-life releases. This means that customers who are currently using Red Hat OpenStack Platform 10 have an easier upgrade path to Red Hat OpenStack Platform 13—with fewer interruptions and no need for additional hardware.

Fast forward upgrade diagram v1
Red Hat OpenStack Platform life cycle by version

Containerized OpenStack services

Red Hat OpenStack Platform now supports containerization of all OpenStack services. This means that OpenStack services can be independently managed, scaled, and maintained throughout their life cycle, giving you more control and flexibility. As a result, you can simplify service deployment and upgrades and allocate resources more quickly, efficiently, and at scale.

Red Hat stack integrations

The combination of Red Hat OpenStack Platform with Red Hat OpenShift provides a modern, container-based application development and deployment platform with a scalable hybrid cloud foundation. Kubernetes-based orchestration simplifies application portability across scalable hybrid environments, designed to provide a consistent, more seamless experience for developers, operations, and users.

Red Hat OpenStack Platform 13 delivers several new integrations with Red Hat OpenShift Container Platform:

  • Integration of openshift-ansible into Red Hat OpenStack Platform director eases troubleshooting and deployment.
  • Network integration using the Kuryr OpenStack project unifies network services between the two platforms, designed to eliminate the need for multiple network overlays and reduce performance and interoperability issues.  
  • Load Balancing-as-a-Service with Octavia provides highly available cloud-scale load balancing for traditional or containerized workloads.

Additionally, support for the Open Virtual Networking (OVN) networking stack supplies consistency between Red Hat OpenStack Platform, Red Hat OpenShift, and Red Hat Virtualization.

Security features and compliance focus

Security and compliance are top concerns for organizations deploying clouds. Red Hat OpenStack Platform includes integrated security features to help protect your cloud environment. It encrypts control flows and, optionally, data stores and flows, enhancing the privacy and integrity of your data both at rest and in motion.

Red Hat OpenStack Platform 13 introduces several new, hardened security services designed to help further safeguard enterprise workloads:

  • Programmatic, API-driven secrets management through Barbican
  • Encrypted communications between OpenStack services using Transport Layer Security (TLS) and Secure Sockets Layer (SSL)
  • Cinder volume encryption and Glance image signing and verification

Additionally, Red Hat OpenStack Platform 13 can help your organization meet relevant technical and operational controls found in risk management frameworks globally. Red Hat can help support compliance guidance provided by government standards organizations, including:

  • The Federal Risk and Authorization Management Program (FedRAMP) is a U.S. government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services.
  • Agence nationale de la sécurité des systèmes d’information (ANSSI) is the French national authority for cyber-defense and network and information security (NIS).

A updated security guide is also available to help you when deploying a cloud environment.

Storage and hyperconverged infrastructure options

Red Hat Ceph Storage provides unified, highly scalable, software-defined block, object, and file storage for Red Hat OpenStack Platform deployments and services. Integration between the two enables you to deploy, scale, and manage your storage back end just like your cloud infrastructure. New storage integrations included in Red Hat OpenStack Platform 13 give you more choice and flexibility. With support for the OpenStack Manila project, you can use the CephFS NFS file share as a service to better support applications using file storage. As a result, you can choose the type of storage for each workload, from a unified storage platform.

Red Hat Hyperconverged Infrastructure for Cloud combines Red Hat OpenStack Platform and Red Hat Ceph Storage into a single offering with a common life cycle and support. Both Red Hat OpenStack Platform compute and Red Hat Ceph Storage functions are run on the same host, enabling consolidation and efficiency gains. NFV use cases for Red Hat Hyperconverged Infrastructure for Cloud include:

  • Core datacenters
  • Central office datacenters
  • Edge and remote point of presence (POP) environments
  • Virtual radio access networks (vRAN)
  • Content delivery networks (CDN)

You can also add hyperconverged capabilities to your current Red Hat OpenStack Platform subscriptions using an add-on SKU.

RHHCI use cases v0
Red Hat Hyperconverged Infrastructure for Cloud use cases

Telecommunications optimizations

Red Hat OpenStack Platform 13 delivers new telecommunications-specific features that allow CSPs to build innovative, cloud-based network infrastructure more easily:

  • OpenDaylight integration lets you connect your OpenStack environment with the OpenDaylight software-defined networking (SDN) controller, giving it greater visibility into and control over OpenStack networking, utilization, and policies.
  • Real-time Kernel-based Virtual Machine (KVM) support designed to deliver ultra-low latency for performance-sensitive environments.
  • Open vSwitch (OVS) offload support (tech preview) lets you implement single root input/output virtualization (SR-IOV) to help reduce the performance impact of virtualization and deliver better performance for high IOPS applications.
OpenStack_OpenDaylight-NetVirt_437720_0317-illustrated
Red Hat OpenStack Platform and OpenDaylight cooperation

Learn more

Red Hat OpenStack Platform combines community-powered innovation with enterprise-grade features and support to help your organization build a production-ready private cloud. With it, you can accelerate application and service delivery, innovate faster to differentiate your business, and empower your IT teams to support digital initiatives.

Learn more about Red Hat OpenStack Platform:

by Rosa Guntrip, Senior Principal Product Marketing Manager at June 28, 2018 12:53 AM

June 27, 2018

OpenStack Superuser

Project update: What’s next for Cyborg

At the recent Vancouver Summit, OpenStack project team leads (PTLs) and core team members offered updates for the OpenStack projects they manage, what’s new for this release and what to expect for the next one, plus how you can get involved and influence the roadmap.

Superuser features summaries of the videos; you can also catch them on the OpenStack Foundation YouTube channel.

In this one, Zhipeng Huang, open source operation manager for Huawei and Cyborg project team lead (PTL) walks you through what’s happening and what’s next.

What

Cyborg (previously known as Nomad) is an OpenStack project that aims to provide a general purpose management framework for acceleration resources (i.e. various types of accelerators such as GPU, FPGA, ASIC, NP, SoCs, NVMe/NOF SSDs, ODP, DPDK/SPDK etc.) “Nomad was started a few years ago when the industry began to realize that accelerator hardware has become a necessity rather than a interesting option or an interesting tool,” Huang says.

What’s new & what’s next

Huang ran through the history and timeline of the project, noting that the first release offering a basic framework coincided with the Pike cycle, showed how the Queens version built on that and what’s in store for Rocky:

A couple of items worth noting: support for FPGAs and quota controls. “We’ll have a programming support for FPGAs. It’s a set of APIs that allow admins or users to be able to change the functionality on the FPGA dynamically without powering down the machine.” There are also plans to connect with the Kubernetes community, Huang says these plans are part of the roadmap but not finalized yet.

How to get involved

For newcomers, there are first contact meetings and the team also holds public meetings in #openstack-cyborg Wednesdays at UTC 14:00.

How to contribute

  1. Git clone https://git.openstack.org/openstack/cyborg
  2. Make the changes to your entry, be sure to include what’s changed and why
  3. Commit the change for review
  4. The changes will be reviewed, merged within a day or so.

Cyborg is designed to use the same tools for submission and review as other OpenStack projects, meaning it follows the OpenStack development workflow. New contributors should follow the getting started steps before proceeding, as a Launchpad ID and signed contributor license are required to add new entries.

New contributors can follow this guide — DevStack Quick Start — to setup Cyborg environment.

The Cyborg Storyboard page can be found at https://storyboard.openstack.org/#!/project_group/77.

Catch the whole 10-minute talk below.

Photo // CC BY NC

The post Project update: What’s next for Cyborg appeared first on Superuser.

by Superuser at June 27, 2018 04:31 PM

June 26, 2018

OpenStack Superuser

How open source projects start, grow and flourish

While there are plenty of models for traditional businesses, if you start to look at open source the picture gets cloudy, fast.

A team at Mozilla went on an internal mission to clarify what successful models looked like and ended up with a 37-page report titled “Open source archetypes: A framework for purposeful open source.”  Published as a booklet, available online and CC-licensed, it dissects topics including benefits, the basics of licensing and practical questions to ask about open-source projects.

Archetypes are at the heart of the document. The team (unnamed to reflect the collaborative nature of the initiative) states that while projects can change from one archetype to another — either through deliberate and guided transformation, or a gradual and incremental evolution —they wanted to chart how open-source projects begin and evolve.

The report breaks it down into 10 of them, offering examples as well as pros and cons: Business-to-business, multi-vendor infrastructure, rocket ship to mars, controlled ecosystem, wide open, mass market, specialty library, trusted vendor, upstream dependency and bathwater.

Clearly these cover a lot of ground, here are some examples of the less self-explanatory types:

Rocket ship to mars

Examples: Meteor, Signal
These feature a small full-time core team completely focused on a well-articulated and highly specific goal. Their open source strategy is often rooted in a commitment to transparency and providing insurance…they want to instill confidence among developers and users to promote adoption. This is a frequent type for a starting point for projects that transition, for example, to B2B.

Wide open

Examples: Rust (present day), Apache HTTPD
These projects actively welcome contributions from any source and tend to get contributions at all levels: from individual developers providing one-off bug fixes to sustained organizational contributors. The authors say this is a good model for projects that aim to serve the same technical purpose over a long period of time and for which stability and reliability are more important than rapid development of the code base or fast adoption of field-specific innovations.

Bathwater

No examples provided
Described as “code dumped over the wall,” purely about distribution, not about any particular mode of production. “Think of it as the ground state of open source projects: someone publishes code under free license but invests no follow up” to build an open-source community. “Code gets dumped this way quite often and ‘throwing code over the wall’ has a bad reputation among open-source practitioners.” Despite this, the report authors note it can be useful – as an initial foray into open source for companies, for more experienced companies it can be a “reasonable state” for an abandoned initiative and because it leaves forkable code, anyone can try to move it to another archetype.

Download the report or check out the full chart below:

Click for full-size version.

The post How open source projects start, grow and flourish appeared first on Superuser.

by Nicole Martinelli at June 26, 2018 04:27 PM

Chris Dent

TC Report 18-26

All the bits and pieces of OpenStack are interconnected and interdependent across the many groupings of technology and people. When we plan or make changes, wiggling something here has consequences over there. Some intended, some unintended.

This is such commonly accepted wisdom that to say it risks being a cliche but acting accordingly remains hard.

This morning Thierry and I had a useful conversation about the Tech Vision 2018 etherpad. One of the issues there is agreeing on what we're even talking about. How can we have a vision for a "cloud" if we don't agree what that is? There's hope that clarifying the vision will help unify and direct energy, but as the discussion and the etherpad show, there's work to do.

The lack of clarity on the vision is one of the reasons why Adjutant's application to be official still has no clear outcome.

Meanwhile, to continue last week's theme, the TC's role as listener, mediator, and influencer lacks definition.

Zane wrote up a blog post explaining the various ways in which the OpenStack Foundation is expanding. But this raises questions about what, if any, role the TC has in that expansion. It appears that the board has decided to not to do a joint leadership meeting at the PTG, which means discussions about such things will need to happen in other media, or be delayed until the next summit in Berlin.

To make up for the gap, the TC is planning to hold a gathering to work on some of the much needed big-picture and shared-understanding building.

While that shared understanding is critical, we have to be sure that it incorporates what we can hear from people who are not long-term members of the community. In a long discussion asking if our tooling makes things harder for new contributors several of us tried to make it clear that we have an incomplete understanding about the barriers people experience, that we often assume rather than verify, and that sometimes our interest in and enthusiasm for making incremental progress (because if iterating in code is good and just, perhaps it is in social groups too?) can mean that we avoid the deeper analysis required for paradigm shifts.

by Chris Dent at June 26, 2018 12:39 PM

June 25, 2018

OpenStack Superuser

Rebooting the OpenStack Mentorship program

In the recently published gender diversity analysis in the OpenStack Community, Intel’s Imad Sousou says: “One characteristic of a strong and thriving open source community is an active effort to attract new talent, diverse skill sets and fresh perspectives. Mentorship programs are especially important to help newcomers — as well as those working in new areas — to engage and contribute successfully.”

We saw the response to this call to action play a significant part at the Vancouver Summit. From a panel discussion on mentorship to speed mentoring sessions, our community had shown its willingness to invest their time. During these events, we also began to see a change from our traditional view of one-on-one long term mentorship to a group mentorship mindset.

Having been on both sides of the one-on-one mentorship relationship, I’ve learned that these relationships don’t always live up to their fullest potential. Often this occurs due to mismatches between assigned mentors and mentees; other times unexpected circumstances arise in the participant’s personal lives, or careers that make it difficult for either party to continue investing time in the relationship.

The OpenStack Community recognizes the difficulties and stresses that these situations can cause and has worked to find a solution. During the past few Summits, we’ve held speed mentoring workshops, bringing together experienced contributors with newcomers who are interested in growing their careers, their technical skills or looking for how to become involved in the community. These speed mentoring sessions allow attendees to make contact with individuals willing to answer their questions long past the end of the session.

How to get involved

To expand this important opportunity further, we’re happy to announce the relaunch of the OpenStack Mentoring program. Our new mentoring program will focus on providing mentorship through goal-focused cohorts of mentors. This change will allow mentoring responsibilities to be shared among each group’s mentors. Mentoring cohorts allow for mentees to access a broader knowledge base while enabling them to form connections and help other members of the group.

We hope that this change can help drive a diverse group of experienced OpenStack users and developers in assisting newcomers to the OpenStack Community.

The initial cohorts will be:

  • Get your first patch merged
  • First CFP submission / Give your first talk
  • Become COA certified / study for COA
  • Deploy your first cloud

If you’re interested in joining as a mentor or mentee, please sign up at:

Mentors: https://openstackfoundation.formstack.com/forms/mentoring_cohorts_mentors
Mentees: https://openstackfoundation.formstack.com/forms/mentoring_cohorts_mentees

 

Superuser wants to hear about your experience as a mentor or mentee, get in touch at editorATopenstack.org

The post Rebooting the OpenStack Mentorship program appeared first on Superuser.

by Ell Marquez at June 25, 2018 05:01 PM

Thierry Carrez

A framework for lightweight open source governance

Any group of humans needs some form of governance. It’s a set of rules the group follows in order to address issues and take clear decisions. Even the absence the rules (anarchy) is a form of governance! At the opposite end of the spectrum is dictatorship, where all decisions are made by one person. Open source projects are groups of humans, and they are no exception to this. They can opt for various governance models, which I detailed in a previous article four years ago (how time flies!).

That article compared various overall models in terms of which one would best ensure the long-term survival of the community, avoiding revolutions (or forks). It advocated for a representative democracy model, and since then I've been asked several times for the best recipe to implement it. However there are numerous trade-offs in the exercise of building governance, and the "best" depends a lot on the specifics of each project situation. So, rather than detail a perfect one-size-fits-all governance recipe, in this article I'll propose a framework of three basic rules to keep in mind when implementing it.

This simple 3-rule model can be used to create just enough governance, a lightweight model that should be sustainable over the long run, while avoiding extra layers of useless bureaucracy.

Rule #1: Contributor-driven bodies

Governance bodies for an open source project should be selected by the contributors to the project. I'm not talking of governance bodies for open source Foundations (which generally benefit from having some representation of their corporate sponsors chiming in on how their money shall be spent). I'm talking about the upstream open source project itself, and how the technical choices end up being made in a community of contributors.

This rule is critical: it ensures that the people contributing code, documentation, usage experience, mentoring time or any other form of contribution to the project are aligned with the leadership of the project. When this rule is not met, generally the leadership and the contributors gradually drift apart, to the point where the contributors no longer feel like their leadership represents them. This situation generally ends with contributors making the disruptive decision to fork the project under a new, contributor-aligned governance, generally leaving the old governance body with a trademark and an empty shell to govern.

One corollary of that first rule is that the governance system must regularly allow replacement of current leaders. Nobody should be appointed for life, and the contributors should regularly be consulted, especially in fast-moving communities.

Rule #2: Aligned with their constituencies

This is another corollary of the first rule. In larger projects, you need enough governance bodies to ensure that each is aligned with its own constituency. In particular, if your community is made of disjoint groups with little to no overlap in membership, and those groups each need decisions to be made, they probably need to each have their own governance body at that level.

The risk we are trying to avoid here is dominance of the larger group over smaller groups. If you use a single governance body for two (or more) disjoint groups, chances are that the larger group will dominate the representative governance body, and therefore will end up making decisions for the smaller group. This is generally OK for global decisions that affect every contributor equally, but matters that are solely relevant to the smaller group should be decided at the smaller group level, otherwise that group might be tempted to fork to regain final call authority over their own things.

Rule #3: Only where decisions are needed

Strict application of rule #2 tends to result in the creation of a large number of governance bodies, that's why you need to balance it with rule #3: only create governance bodies where decisions are actually needed. The art of lightweight governance is, of course, to find the best balance between rule #2 and rule #3.

This rule has two practical consequences. The first one is obvious: you should not create vanity governance bodies, just to give people or organizations a cool title or badge. Numerous communities fall in the trap of creating "advisory" boards with appointed seats, to thank long-standing community members, or give organizations the illusion of control. Those bodies create extra bureaucracy while not being able to make a single call, or worse, trying desperately to assert authority to justify their existence.

The second consequence is, before creating a governance body at a certain level in the project organization, you should question whether decisions are really needed at that level. If the group needs no final call, or can trust an upper decision body to make the call if need be, maybe that governance body is not needed. If two governance bodies need to cooperate to ensure things work well between them, do you really need to create a governance body above them, or just encourage discussion and collaboration ? This trade-off is more subtle, but generally boils down to how badly you need final decisions to be made, vs. letting independently-made decisions live alongside.

That is all there is to it! As I said in the introduction, those three rules are not really a magic recipe, but more of a basic framework to help you, in the specific situation of your community, build healthy communities with just enough governance. Let me know if you find it useful!

by Thierry Carrez at June 25, 2018 11:45 AM

Red Hat Stack

Red Hat Certified Cloud Architect – An OpenStack Perspective – Part Two

Previously we learned about what the Red Hat Certified Architect certification is and what exams are included in the “OpenStack-focused” version of the certification. This week we want to focus on personal experience and benefits from achieving this milestone.

Let’s be honest, even for the most skilled engineers the path to becoming an RHCA can be quite challenging and even a little bit intimidating!  Not only do the exams test your ability to perform specific tasks based on the certification requirements, but they also test your ability to repurpose that knowledge and combine it with the knowledge of other technologies while solving extremely complex scenarios.  This can make achieving the RHCA even more difficult; however, it also makes achieving the RHCA extremely validating and rewarding.

samuel-clara-69657-unsplash
Photo by Samuel Clara on Unsplash

Many busy professionals decide to prepare for the exams with Red Hat Online Learning (ROLE), which allows students to access the same robust course content and hands-on lab experience delivered in classroom training from the comfort of their own computer and at their own pace. This is made even easier through the Red Hat Learning Subscription (RHLS).

RHLS provides access to the entire Red Hat courseware catalog, including video classrooms, for a single, convenient price per year. This kind of access can help you prepare for all the certifications. We found that before sitting an exam, it was important to be able to perform 100 percent of the respective ROLE lab without referring back to any documentation for help; with RHLS this is much easier to do!  

While documentation and man pages are available during an exam, they should be used as a resource and not a replacement for deep knowledge. Indeed, it’s much better to make sure you know it by heart without needing to look! We also found that applying the comprehensive reviews found at the end of each ROLE course to real world scenarios helped us better understand how what we learned in the course applied to what we would do on a day-to-day basis.  

For example, when taking the Ansible ROLE course DO407, which uses a comprehensive virtual environment and a video classroom version, we were easily able to spawn instances in our own physical OpenStack environment and apply what we had learned in the course to the real world.  By putting the courseware into action in the real world it better allowed us to align the objectives of the course to real-life scenarios, making the knowledge more practical and easier to retain.

What about formal training?

nathan-dumlao-572047-unsplash
Photo by Nathan Dumlao on Unsplash

We wouldn’t recommend for anyone to just show up at the examination room without taking any formal training. Even if you feel that your level of proficiency in any of these technologies is advanced, keep in mind that Red Hat exams go very deep, covering large portions of the technology. For example, you might be an ‘Ansible Ninja’ writing playbooks for a living. But how often do you work with dynamic inventories or take advantage of delegation, vaults or parallelism? The same applies for any other technology you want to test yourself in, there is a good chance it will cover aspects you are not familiar with.

The value comes from having the combination of skills.  Take the example of an auto mechanic who is great at rebuilding a transmission, but may not know how to operate a manual transmission!  You can’t be an expert at one without knowing a lot about the other.

For us, this is where Red Hat training has been invaluable. With every exam there is a corresponding class provided. These classes not only cover each aspect of the technology (and beyond) that you will be tested on, but also provide self-paced lab modules and access to lab environments. They are usually offered with either a live instructor or via an online option so you can juggle the education activities with your ‘day job’ requirements!

More information about the classes for these exams can be found on the Red Hat Training site. 

How long does it take?

It doesn’t have to take long at all. If you already have an RHCE in Red Hat Enterprise Linux and OpenStack is not a new subject to you, the training will serve as an excellent reminder rather than something that you have to learn from scratch. Some people may even be able to complete all 5 exams in less then a month.

But does everyone want to go that fast? Probably not.

estee-janssens-396876-unsplash
Photo by Estée Janssens on Unsplash

When our customers ask us about what we recommend to achieve these certifications in a realistic timeframe we suggest the Red Hat Learning Subscription to them. As mentioned, it gives you amazing access to Red Hat courseware.

But it is more than that.

The Red Hat Learning Subscription is a program for individuals and organizations that not only provides the educational content to prepare you for the exams (including videos and lab access), but also, in some cases, may includes actual exams (and some retakes) at many Red Hat certified facilities. It is is valid for one year, which is plenty of time to work through all the courses and exams.

This kind of flexibility can help to shape an individual learning path.

For instance, imagine doing it like this:

With the Red Hat Learning subscription you could schedule all the exams in advance in two month intervals. These exams then become your milestones and give you a good predictable path for studying. You can always reschedule them if something urgent comes up. This lets you sign up for classes, but don’t take them too far apart before your exam. Then re-take all the self paced labs a week before your exam, without reading guided instructions. After that you should be in a position to assess your readiness for the exams and reach the ultimate goal of an RHCA.

Don’t get discouraged if you don’t pass on the first try, it’s not unusual even for subject experts to fail at first try! Simply close the knowledge gaps and retake the exam again. And with RHLS, you’ve got the access and time to do so!

The benefits of becoming RHCA can be substantial. Outside of gaining open source “street cred”, the most important aspect is, of course, for your career – it’s simple: you can get better at your job.

clark-tibbs-367075-unsplash
Photo by Clark Tibbs on Unsplash

And of course, being better at your job can translate to being more competitive in the job market, which can lead to being more efficient in your current role and potentially even bring additional financial compensation!

But becoming an RHCA is so much more. It helps to broaden your horizons. You can learn more ways to tackle real life business problems, including how to become more capable of taking leadership roles through translating problems into technology solutions.

As a proud Red Hat Certified Architect you will have the tools to help make the IT world a better place!

So what are you waiting for … go get it!


Icon_RH_Transportation_Space-Rocket_RGB_Flat (1)Ready to start your certification journey? Get in touch with the friendly Red Hatters at Red Hat Training in your local area today to find all the ways you can master the skills you need to accelerate your career and run your enterprise cloud!


About the authors:

Screen Shot 2018-06-20 at 12.23.14 pmChris Janiszewski is an Red Hat OpenStack Solutions Architect. He is proud to help his clients validate their business and technical use cases on OpenStack and supporting components like storage, networking or cloud automation and management. He is the father of two little kids and enjoys the majority of his free time playing with them.  When the kids are asleep he gets to put the “geek hat” on and build OpenStack labs to hack crazy use cases!


Screen Shot 2018-06-20 at 12.23.23 pmKen Holden is a Senior Solution Architect with Red Hat.  He has spent the past 3 years on the OpenStack Tiger Team with the primary responsibility of deploying Red Hat OpenStack Platform Proof-Of-Concept IaaS Clouds for Strategic Enterprise Customers across North America.  Throughout his 20 year career in Enterprise IT, Ken has focussed on Linux, Unix, Storage, Networking, and Security with the past 5 years being primarily focused on Cloud Solutions. Ken has achieved Red Hat Certified Architect status (RHCA 110-009-776) and holds Certified OpenStack Administrator status (COA-1700-0387-0100) with the OpenStack Foundation. Outside of work, Ken spends the majority of his time with his wife and two daughters, but also aspires to be the world’s most OK Guitar Player when time permits!

by Chris Janiszewski - Senior OpenStack Solutions Architect - Red Hat Tiger Team at June 25, 2018 02:54 AM

June 24, 2018

Aija Jauntēva

Outreachy: Adding support for Redfish BIOS resource in sushy

Time flies and this is my 6th week in Outreachy project. My first 'big' task is to implement support for BIOS resource in sushy library.

BIOS resource is one of the Redfish resources beside many others, like Ethernet Interface, Processor which are already implemented in sushy. To shortly describe BIOS resource - it returns a list of BIOS attributes and allows to update them, it also has 2 "actions": Reset the attributes to default value and change BIOS password.

My starting point was an API scheme PDF[1] referenced in the ticket and merged patch of Ethernet Interface [2] which gave me idea which parts of the project I need to touch: implement classes for BIOS resource, add a release note, add a BIOS property to parent resource called System and add unit tests for the new BIOS resource and for new parts of System.

Sounds straightforward, could take a day or two? Of course it did not.

One of the first things I encountered was that for action "Change Password" there are no parameters specified. How to pass new password then? Looked around and discovered that the PDF referenced in the ticket is not the latest version. Found the latest version[3] released 14 May 2018, after the ticket was created and this spec has parameters for "Change Password": NewPassword, OldPassword and PasswordName. At least this makes sense now, but what to do with those who are still on previous version? At some point in freenode IRC #openstack-ironic channel I asked this question and people there knew that this was issue with generating JSON schemas from CSDL. Turns out there are CSDL files which are schemas in XML and they appear to be source of truth, not JSON. Good to know, but somehow I'm still sticking to JSON and if in doubt look at CSDL. But I started to look more at JSON schemas themselves, rather then generated PDF. For example, BIOS has it schema at [4]. There is also Schema index [5], but it is easy to guess the URL if know the resource name. With this resolved I was preparing to submit this patch for code review and while pre-reviewing myself I took another look at sample JSON file for BIOS resource to check if everything is covered. Here is the sample:

{
  "@odata.type": "#Bios.v1_0_3.Bios",
  "Id": "BIOS",
  "Name": "BIOS Configuration Current Settings",
  "AttributeRegistry": "BiosAttributeRegistryP89.v1_0_0",
  "Attributes": {
    "AdminPhone": "",
    "BootMode": "Uefi",
    "EmbeddedSata": "Raid",
    "NicBoot1": "NetworkBoot",
    "NicBoot2": "Disabled",
    "PowerProfile": "MaxPerf",
    "ProcCoreDisable": 0,
    "ProcHyperthreading": "Enabled",
    "ProcTurboMode": "Enabled",
    "UsbControl": "UsbEnabled"
  },
  "@Redfish.Settings": {
    "@odata.type": "#Settings.v1_0_0.Settings",
    "ETag": "9234ac83b9700123cc32",
    "Messages": [
      {
        "MessageId": "Base.1.0.SettingsFailed",
        "RelatedProperties": [
          "#/Attributes/ProcTurboMode"
        ]
      }
    ],
    "SettingsObject": {
      "@odata.id": "/redfish/v1/Systems/437XR1138R2/BIOS/Settings"
    },
    "Time": "2016-03-07T14:44.30-05:00"
  },
  "Actions": {
    "#Bios.ResetBios": {
      "target": "/redfish/v1/Systems/437XR1138R2/BIOS/Actions/Bios.ResetBios"
    },
    "#Bios.ChangePassword": {
      "target": "/redfish/v1/Systems/437XR1138R2/BIOS/Actions/Bios.ChangePassword"
    }
  },
  "@odata.context": "/redfish/v1/$metadata#Bios.Bios",
  "@odata.id": "/redfish/v1/Systems/437XR1138R2/BIOS"
}

Id, Name, AttributeRegistry, Attributes covered, Actions covered, but what is this @Redfish.Settings? It's not in the schema. I start searching and find this described in Redfish Scalable Platforms Management API Specification[6], this has also HTML version [7], it's handier for me. The field is described in section 7.2.2[8]. Turns out this field is used where resource cannot be updated directly, as is the case for BIOS attributes - updates would require system reboot. Ok, looks like a reusable field. I talk with mentors and we decide this should be implemented as a separate patch to keep the patches smaller and make the BIOS patch dependent on this, a chained patch. Also speaking of this specification[6][7], this is better starting point for all Redfish things than Schema PDF, schema PDF is just a human-readable presentation of JSON/CSDL schemas, but then again if reading this first it might be hard to relate to anything in it. With only few days in I still was not comfortable with sushy project and was confusing myself how I should implement this re-usable field. In hindsight there is nothing to be confused about, but until I got there I peeked at base modules of sushy library to see what's happening there, otherwise I was just copying whatever was done in Ethernet Interface without really knowing and understanding what's happening under the hood. One thing to mention, which I did not see at the beginning, is that sushy is not a serialization library that serializes JSON to Python classes and back - it does more. sushy users wouldn't have to know Redfish to use sushy with bare metals. It adds some processing where necessary to make its use handy and hides all the web service details. At least that's how I see it now.

With the Settings field done and submitted for code review I returned to the original BIOS patch to make it dependent on Settings patch and update to use Settings field. Then I noticed that there might be a thing that should have been implemented differently in Settings patch. The Settings object holds a URI (see SettingsObject/@odata.id in the sample above) where clients should PATCH and where committed attributes are visible. Committed attributes are those that user updated, but haven't been applied yet as they are waiting for the system reboot. Having this realization that sushy is not just a dumb serialization library, an idea came that sushy should load this resource automatically when user accesses the property in Python. And I made the update to do it. As the Settings field is re-usable field I had to support that it can create a new instance of its parent type. The code introduced some structures and approaches that were not seen anywhere in the project, so I was not sure that this was the way to go, but the only way to find this out is to submit it for code review. So in the next patch update for Settings field I have a re-usable field that could load a settings resource automatically and dynamically based on parent resource type.

With both patches updated and submitted I started to wait for code reviewer feedback. I have been warned that code reviews can take a while, so during that I had some other smaller tickets to work on, and later I started to look at the next bigger ticket.

Looong story short, there were many suggestions how to improve this (still keeping in mind that this is not just serialization library), how to change some things, at one point there was a dict inherited with __setitem__ override that instead of updating the dict values, did PATCHing so that sushy user can write attributes['ProcTurboMode'] = 'Disabled' seamlessly without knowing what magic happens behind. This again introduced a lot of structures not seen anywhere else in the project and in the end it was deemed to be too confusing for user instead of being helpful, for example, with code above why does not attribute value in the dictionary change in the end?

Somewhere in between, one question remained unanswered regarding @Redfish.Settings - why isn't this field in the schema of BIOS? What are the other resources that could have this field? greping within provided mockup samples, found another case where Ethernet Interface[9] has this field, while other samples of Ethernet Interface did not have this property (e.g., [10]). Also in one of the older versions of mockup sample file Readme[11] there was a short intro in Redfish, and it mentioned that this can also be applied to Storage resources, but I haven't any mockup samples for it so far. This raised another question - what does it mean that @Redfish.Settings field is not present? Does it mean that settings are read-only or that user should patch at the resource itself? The spec was not entirely clear and the Redfish implementations being available on servers which I don't have or have access to, I was not able to check how it works in real life. Talked with mentors, should this be asked in Redfish user forum or are there any other options to find this out, an e-mail thread was started where people with more knowledge on Redfish promptly clarified this - @Redfish.Settings is payload annotation field that can appear in any resource, but practically it will appear where immediate updates are not possible but restart of a system or a service is necessary. When the field is not present, it means that user can patch against the resource itself (not that it is read-only). BIOS will always have this field, because there are no known BIOS that can apply changes without reboot and not expecting to have any in near future. Ethernet Interfaces and Storage might have it or not, and now it is not expected to appear in any other resources.

Another thing that came up this week is, how to determine and inform sushy user of attribute update outcome. In the @Redfish.Settings field there are list of messages returned, time when changes were applied and ETag of the version the changes were applied. How to know when to check for the messages? Could there be some notifications received when updates are done? As there were more questions than answers, it was decided that in this patch this will not be exposed to sushy user to find a good solution in the next patch.

In the end with all the magic removed and even both patches merged back together (it was hard to review as it was changing back and forth), current patch versions are very close to the first versions I proposed. They still are in review, but hopefully will be done soon. I feel like I walked around the world to return at the same point, but during the trip I saw places and had some fun with Python.

by ajya at June 24, 2018 03:00 PM

June 22, 2018

Red Hat Stack

Red Hat OpenStack Platform fast forward upgrades: A step-by-step overview

New in Red Hat®️ OpenStack®️ Platform 13, the fast forward upgrade feature lets you easily move between long-life releases, without the need to upgrade to each in-between release. Fast forward upgrades fully containerize Red Hat OpenStack Platform deployment to simplify and speed the upgrade process while reducing interruptions and eliminating the need for additional hardware. Today, we’ll take a look at what the fast forward upgrade process from Red Hat OpenStack Platform 10 to Red Hat OpenStack Platform 13 looks like in practice.

Screen Shot 2018-03-22 at 9.33.50 am

There are six main steps in the process:

  1. Cloud backup. Back up your existing cloud.
  2. Minor update. Update to the latest minor release.
  3. Undercloud upgrade. Upgrade your undercloud.
  4. Overcloud preparation. Prepare your overcloud.
  5. Overcloud upgrade. Upgrade your overcloud.
  6. Convergence. Converge your environment.

Step 1: Back up your existing cloud

First, you need to back up everything in your existing Red Hat OpenStack Platform 10 cloud, including your undercloud, overcloud, and any supporting services. It’s likely that you already have these procedures in place, but Red Hat also provides comprehensive Ansible playbooks to simply the fast forward process even more.

Manual backup procedures are likewise supported by Red Hat’s Customer Experience and Engagement (CEE) group.

A typical OpenStack backup process may involve the following steps:

  1. Notify your users.
  2. Purge your databases, including any unnecessary data stored by Heat or other OpenStack services. This will help to streamline the backup and upgrade process.
  3. Run undercloud and overcloud backups. This will preserve an initial backup of the cloud – it may take some time if you don’t have another backup to reference to this point in time.

By performing a backup before starting the upgrade, you can speed the overall upgrade process by only requiring smaller backups later on.

Step 2: Update to the latest minor release

lucas-davies-500439-unsplash
Photo by Lucas Davies on Unsplash

Next, update your Red Hat OpenStack Platform environment to the latest minor release using the standard minor update processes. This step consolidates all undercloud and overcloud node reboots required for moving to Red Hat OpenStack Platform 13. This simplifies the overall upgrade, as no reboots are needed in later steps. For example, an upgrade from Red Hat OpenStack Platform 10 to the latest, fast forward-ready minor release will update Open vSwitch (OVS) to version 2.9, Red Hat Enterprise Linux to version 7.5, and Red Hat Ceph®️ Storage to version 2.5 in your overcloud. These steps do require node reboots, so you can live-migrate workloads prior rebooting nodes to avoid downtime.

Step 3: Upgrade your undercloud

In this step, you’ll upgrade Red Hat OpenStack Platform director, known as the undercloud, to the new long-life release. This requires manual rolling updates from Red Hat OpenStack Platform 10 to 11 to 12 to 13, but does not require any reboots, as they were completed in the previous minor update. The same action pattern is repeated for each release: enable the new repository, stop main OpenStack Platform services, upgrade director’s main packages, and upgrade the undercloud. Note that Red Hat OpenStack Platform director will not be able to manage the version 10 overcloud during or after these upgrades.

Step 4: Prepare your overcloud

Red Hat OpenStack Platform 13 introduces containerized OpenStack services to the long-life release cadence. This step goes through the process to create the container registry to support the deployment of these new services during the fast forward procedure.

arnel-hasanovic-673679-unsplash
Photo by Arnel Hasanovic on Unsplash

The first part of this step is to prepare the container images to accomplish this:

  1. Upload Red Hat OpenStack Platform 13 container images to your cloud environment. These can be stored on the director node or on additional hardware. If you choose to store them on your director node, ensure that the node has enough space available for the images. Note that during this part, your undercloud will be unable to scale your overcloud.

Next, you’ll prepare your overcloud for features introduced in Red Hat OpenStack Platform 11 and 12, including composable networks and roles:

  1. Include new services in any custom roles_data files.
  2. Edit any custom roles_data files to add composable networks (new for Red Hat OpenStack Platform 13) to each role.
  3. Remove deprecated services from any custom roles_data files and update deprecated parameters in custom environment files.

If you have a Red Hat OpenStack Platform director-managed Red Hat Ceph Storage cluster or storage backends, you’ll also need to prepare your storage nodes for new, containerized configuration methods.

  1. Install the ceph-ansible package of playbooks in your undercloud and check that you are using the latest resources and configurations in your storage environment file.
  2. Update custom storage backend environment files to include new parameters and resources for composable services. This applies to NetApp, Dell EMC, and Dell EqualLogic block storage backends using cinder.

Finally, if your undercloud uses SSL/TLS for its Public API, you’ll need to allow your overcloud to access your undercloud’s OpenStack Object Storage (swift) Public API during the upgrade process.

  1. Add your undercloud’s certificate authority to each overcloud node using an Ansible playbook.
  2. Perform one last backup. This is the final opportunity for backups before starting the overcloud upgrade.

Step 5: Upgrade your overcloud

eberhard-grossgasteiger-330357-unsplash
Photo by eberhard grossgasteiger on Unsplash

This step is the core of the fast forward upgrade procedure. Remember that director is unable to manage your overcloud until this step is completed. During this step you’ll upgrade all overcloud roles and services from version 10 to version 13 using a fully managed series of commands. Let’s take a look at the process for each role.

Controller nodes

First, you’ll upgrade your control plane. This is performed on a single controller node, but does require your entire control plane to be down. Even so, it does not affect currently running workloads. Upgrade the chosen controller node sequentially through Red Hat OpenStack Platform releases to version 13. Once the database on the upgraded controller has been updated, containerized Red Hat OpenStack Platform 13 services can be deployed to all other controllers.

Compute nodes

Next, you’ll upgrade your compute nodes. As with your controller nodes, only OpenStack services are upgraded—not the underlying operating system. Node reboots are not required and workloads are unaffected by the process. The upgrade process is very fast, as it adds containerized services alongside RPM-based services and then simply switches over each service. During the process, however, compute users will not be able to create new instances. Some network services may also be affected.

To get familiar with the process and ensure compatibility with your environment, we recommend starting with a single, low-risk compute node.

Storage (Red Hat Ceph Storage) nodes

Finally, you’ll upgrade your Red Hat Ceph Storage nodes. While this upgrade is slightly different than the controller and compute nodes, it is not disruptive to services and your data plane remains available throughout the procedure. Director uses the ceph-ansible installer making upgrading your storage nodes simpler. It uses a rolling upgrade process to first upgrade your bare-metal services to Ceph 3.0, and then containerizes Ceph services.

steve-johnson-541507-unsplash
Photo by Steve Johnson on Unsplash

Step 6: Converge your environment

At this point, you’re almost done with the fast forward process. The final step is to converge all components in your new Red Hat OpenStack Platform 13 environment. As mentioned previously, until all overcloud components are upgraded to the same version as your Red Hat OpenStack Platform director, you have only limited overcloud management capabilities. While your workloads are unaffected, you’ll definitely want to regain full control over your environment.

This step finishes the fast forward upgrade process. You’ll update your overcloud stack within your undercloud. This ensures that your undercloud has the current view of your overcloud and resets your overcloud for ongoing operation. Finally, you’ll be able to operate your Red Hat OpenStack Platform environment as normal: add nodes, upgrade components, scale services, and manage everything from director.

Conclusion

Fast forward upgrades simplify the process of moving between long-life releases of Red Hat OpenStack Platform. However, upgrading from Red Hat OpenStack Platform 10 to containerized architecture of Red Hat OpenStack Platform 13 is still a significant change. As always, Red Hat is ready to help you succeed with detailed documentation, subscription support, and consulting services.

Watch the OpenStack Upgrades Strategy: The Fast Forward Upgrade video from OpenStack Summit Vancouver 2018 to learn more about the fast forward upgrade approach.

Learn more about Red Hat OpenStack Platform:

by Maria Bracho, Principal Product Manager OpenStack at June 22, 2018 09:33 PM

OpenStack Superuser

Integrating ONAP and MEF 3.0: A proof-of-concept

The Open Networking Automation Platform (ONAP) is an open-source networking project hosted by the Linux Foundation. Datavision Inc. is an IT company specializing in networking solutions for software-defined networking (SDN), network functions virtualization (NFV), edge computing, 5G, data center, network security and other next-gen technologies. With a recent demo, Datavision highlighted the interoperability between ONAP and MEF 3.0, a framework for defining, delivering, and certifying agile, assured and orchestrated communication services across global automated networks.

Superuser talks to Mark Abolafia, Datavision COO, about the importance of ONAP, open source and embracing virtualization.

About the proof-of-concept between ONAP and MEF 3.0, can you break it down for us – what’s the problem you’re solving? And who would benefit?

There are really two approaches to describe in answering that, but let me try to tie it together. ONAP serves as a services design, orchestration and analytics platform used to manage and control network services. The MEF 3.0 framework is a series of interfaces and methodologies used to provide a standardized methodology to provide agile, assured, standards-based communications both within a carrier’s systems and inter-carrier communication – a lingua franca, if you will, for operating companies to exchange service parameters and commercial-related information among themselves so that users can order a service in one operating company’s world and be able to get services provisioned in another operator’s infrastructure in a completely different geography. Global provisioning. These inter- and intra- carrier provisioning (and billing) issues are the problems being solved.

Why are open source and ONAP important to Datavision and the larger industry?

Open source, and ONAP in particular, are important to Datavision because they enable us to be uncoupled from a specific vendor’s proprietary solution. From a business perspective, where engineering and integration is “what we do,” it’s important to be able to control our destiny to some extent by being able to develop, for our clients, a platform that is extensible and open, so they are not subject to vendor lock-in and are able to move forward with the industry and the open source community at that pace, versus some product manager’s pace at a single vendor.

From an industry perspective, frankly, the same criteria apply – having the open-source community develop and advance a solution helps keep things fresh and innovative, and keeps the vendors on their toes.

You wrote that “ONAP is a platform to help service providers embrace virtualization.” What have been the obstacles/concerns that have kept them from doing it until now?

The major obstacles were (are) current legacy system complexity, severe shortage of appropriate skill levels to be networking and software engineers in the same brain, literally, and, of course, the advent of systems capable of actually fulfilling the promise being presented by vendors to the industry.

What’s been the reaction of your customers to open source in general?

Very positive, actually. They recognize the advantages of not being locked into a specific vendor, but on the other side of the coin, they are going in eyes open, realizing the TCO is similar because of the inherent levels of development and ongoing support necessary to pull an open source application/platform into your operation.

What differentiates Datavision’s services from other companies?

We have the on-the-ground experience in defining and refining the requirements for these systems from an architectural level and we have significant real-world deployment experience. We’ve also been in the space since “the beginning,” having been involved with AT&T Domain 2.0 since its inception. So we’ve got a strong history, and the skills and scars to prove it!

What’s the most common request from your customers?

Help us evaluate all the vendors’ claims, help us integrate all these disparate components together and please help us integrate all THAT into our existing systems so we can enable digital transformation of our business.

Superuser wants to hear your open infrastructure story, get in touch at editorATopenstack.org

Photo // CC BY NC

The post Integrating ONAP and MEF 3.0: A proof-of-concept appeared first on Superuser.

by Superuser at June 22, 2018 04:00 PM

About

Planet OpenStack is a collection of thoughts from the developers and other key players of the OpenStack projects. If you are working on OpenStack technology you should add your OpenStack blog.

Subscriptions

Last updated:
July 22, 2018 12:52 AM
All times are UTC.

Powered by:
Planet