November 25, 2014

Rafael Knuth

Online Meetup: Crowbar, OpenStack and Physical DevOps

Physical ops is hard because it’s so complex and heterogeneous. Join Rob Hirschfeld, Crowbar...

November 25, 2014 08:55 PM

Spilgames Engineering

OpenStack Operator tool: Novacek

About As an OpenStack operator we missed some tooling for troubleshooting and status checking. To fill this gap we created a python tool to give us this info: https://github.com/spilgames/novacek The tool is written in python and uses the OpenStack libraries to get the necessary information. Examples Typical things we use this tool for: Get the […]

The post OpenStack Operator tool: Novacek appeared first on Spil Games Engineering.

by Robert van Leeuwen at November 25, 2014 09:57 AM

Mirantis

Seeing Murano Actions in action

The Juno release of Murano introduces the new Murano actions feature, which allows you to manage application components after application deployment.  In this blog post we look at the basic concepts of using Murano actions for specific autoscaling and high availabilty (HA) use cases.

What is Murano?

As a quick review before we start digging into the technical details of actions, Murano is an OpenStack project that enables application developers and cloud developers to publish applications in a browsable catalog. With the catalog, users can create complex multi-tier application environments with the push of a button using published applications as building blocks.  

Murano provides a UI that is implemented as a plugin for the OpenStack Dashboard to simplify application management. This plugin provides:

  1. The application catalog page, where users can find applications

  2. The environments page, where users can manage multi-tier applications

  3. The package management page, where users can upload/download Murano application packages

Murano uses Heat templates to provision OpenStack resources. The application author can employ the Murano agent for  software configuration, or if the application author prefers an agentless mode, he or she can employ Heat software configuration resources.

Murano actions

Murano actions are workflows that an application author defines to expose specific operations the application can perform, such as being able to restart, backup, upgrade, and scale up/down. The Murano API server exposes the workflows as Web hooks when you deploy the application.

Now let’s take a look at auto-scaling and HA, which are two common use cases that require orchestration.

Auto-scaling

Auto-scaling is a typical use case for some applications that could be scaled horizontally. The usual example is a cluster of Web applications that can be scaled up/down depending on actual traffic.  The Murano application author can implement application autoscaling in two different ways:

  1. The first approach supported by Murano is to use Heat autoscaling resources, which works perfectly if you can scale your application with the addition of simple Heat resources.

  1. The second approach is to use Murano actions to expose specific workflows for the application scaling process. As you have full control over the scale up process, you can write complex, sophisticated scaling scenarios such as, “I need 1 additional Database host for every 10 Web Servers, and if I have more than 10 Web Servers switch from the HAProxy LoadBalancer to the F5 hardware load balancer.”

For the Auto-scaling use-case, we usually use the Ceilometer monitoring app, which enables us to create simple alarms based on CPU and memory usage. Figure 1 depicts a Java application as an example. It has three dependencies:

  1. A monitoring app

  2. The Java Servlet Container app

  3. The Database app

Figure 1 - Auto-scaling in Murano

The Java application defines the ‘scaleUP’ action workflow. When this action is triggered by the monitoring component via an HTTP call to an action web hook exposed by the API, the workflow adds a new instance to the Tomcat Cluster, deploys Tomcat and the application on it, and then adds it to the load balancer. The underlying Heat stack is then updated by adding the corresponding resources, consisting of the VM, LB Pool Member, and Ceilometer alarm associated with the new VM. This workflow can be changed to provide more logic for scaling other components, such as database scaling or changing the load balancer.

High Availability

HA for applications is not new. IT departments are very familiar with HA and have open source and proprietary methods to tackle it.  The following are the two main HA enablement approaches:

  1. Having HA inside the application so that the application itself performs all necessary actions in case of failure. Galera clusters and Tomcat clusters are good examples of HA solutions embedded into the application itself.

OR

  1. Having the underlying software and/or hardware infrastructure provide HA. VMWare HA VM is a good example of the underlying infrastructure providing HA on a virtual hardware level. Pacemaker/Corosync is another example of a software-based HA solution for applications that do not support HA directly.

You can use Murano actions to add HA to some applications, defining a workflow to be performed when an initial application host fails or the application does not respond to monitoring probes. OpenStack provides a number of options to successfully perform an HA switchover:

  1. If an application does not have any external dependencies, you can use the Nova evacuate command to simply move the VM to a new compute host or spawn a new VM instance and pause the original one.

  1. If an application keeps its state and all data on a file system, then your HA strategy will be built around volumes. When the monitoring system detects a failure, it invokes a Murano action by sending an HTTP request to an exposed web hook. Murano starts performing a workflow to pause the original application’s VM, detach the data volume, spawn a new VM with the application, and attach the data volume to the new VM.

  1. In more complex scenarios where a component requiring HA is used by other components, the triggered action performs the necessary steps to create a new instance for the failed component and then updates all of the component’s users with the new information.

Figure 2 depicts the Nagios monitoring service, which provides a fast response to the failure and has a lot of different probes that can be used by an application.

Figure 2 - An HA scenario in Murano using Nagios

Our demo Java application has two different strategies for HA. When deployed on top of a Tomcat Cluster, it just adds a new VM to the cluster if Nagios reports that one of the existing VMs is down. If the Java application is deployed on top of a single Tomcat instance, it tries to restart the Tomcat service first. If the restart is unsuccessful, the Java application spawns a new VM with Tomcat and deploys itself to the new VM.

Next steps

Murano actions also include functions such as backups, application upgrades,  and so on. In future posts, I’ll provide more detailed explanations for functions, including backups and application upgrades, with code snippets and template examples.

The post Seeing Murano Actions in action appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Georgy Okrokvertskhov at November 25, 2014 01:36 AM

Do I need OpenStack if I use Docker?

LIFE_danceDocker has broken a record in the speed in which it moved from being a disruptive technology to a commodity. The speed of adoption and popularity of Docker brings with it lots of confusion.

In this post I wanted to focus on a trend of commentary that has been gaining popularity that I’ve started to hear more often recently from users who just started using Docker: whether it makes sense to use OpenStack if they’ve already chosen to use Docker.

Before I give my take on this, I want to start with a short background for the rationale behind this question.

Background

In its simplest form, Docker provides a container for managing software workloads on shared infrastructure, all while keeping them isolated from one another. Virtual machines such as KVM do a similar job by creating a complete operating system stack of all the OS devices (through a hypervisor). However, unlike the virtual machine approach, Docker relies on a built-in feature of the Linux operating system named LXC (Linux containers).* LXC utilizes the built-in operating system features of process isolation for memory, and to a lesser degree, CPU and networking resources. Docker images do not require a complete boot of a new operating system, and as a result, provide a much lighter alternative for packaging and running applications on shared compute resources. In addition, it allows direct access to the device drivers which makes I/O operations faster than with a hypervisor approach. The latter makes it possible to use Docker directly on bare metal which, often times causes people to ask whether the use of a cloud such as OpenStack is really necessary if they’re already using Docker.

This performance difference between Docker and a hypervisor such as KVM is backed by a recent benchmark done by Boden Russell and presented during the recent DockerCon event.

The benchmark is fairly detailed, and as expected, it shows a significant difference between the time it takes to boot a KVM hypervisor to a Docker container. It also indicates a fairly big difference in the memory and CPU utilization between the two, as can be seen in the diagram below.

benchmark

This difference in performance can map to a difference in density and overall utilization difference between the two in similar proportion. This difference can easily map into a big difference in cost that is directly affected by the number of resources needed to run a given workload.

My take

  • This question has nothing specific to do with OpenStack and can be applied similarly to any other cloud infrastructure. The reason it is often brought up in the context of OpenStack, in my opinion, is due to the fact that OpenStack is fairly popular in private cloud environments which is the only environment in which we can even consider a pure Docker alternative.
  • It’s all about the hypervisor!

Many of the performance benchmarks compare Docker versus KVM, and have little to do with OpenStack. In fact, this specific benchmark of both KVM images and Docker containers was running through OpenStack which shows that the two technologies works nicely together. In that context, most of the utilization arguments become irrelevant when I choose to run OpenStack on top of a Docker based Nova stack as illustrated in the diagram below taken from the OpenStack documentation.

benchmark

  • Cloud infrastructure provides a complete data center management solution in which containers or hypervisors, for that matter, are only part of a much bigger system. Cloud infrastructure such as OpenStack includes multi-tenant security and isolation, management and monitoring, storage and networking and more. All of those services are needed for any cloud / data center management and have little dependency on whether Docker or KVM are being used.
  • Docker isn’t (yet) a full-featured VM and has some serious limitations around security, is lacking in Windows support (as indicated in the following email thread), and therefore cannot be considered a complete alternative to KVM just yet. While there’s ongoing work to bridge those gaps, it is safe to assume that adding the missing functionality may come at an additional performance cost.
  • There’s a big difference between raw hypervisor performance/containerization and application performance, as indicated in the following graphs from the benchmark results. A possible explanation for that is that applications often use the caching technique to reduce the I/O overhead.

benchmark

  • If we package Docker containers within a KVM image the difference can become negligible. This architecture often uses hypervisors for managing the cloud compute resources, and an orchestration layer such as Heat, Cloudify, or Kubernetes on top to manage containers within the hypervisor resources.

Conclusion

This brings me to the conclusion that the right way to look at OpenStack, KVM and Docker, is as a complementary stack in which OpenStack plays the role of the overall data center management. KVM as the multi-tenant compute resource management, and Docker containers as the application deployment package.

In this context a common model would be to use Docker for the following roles:

  • Docker provides deterministic software packaging and fits nicely with the immutable infrastructure model.
  • Docker for excellent great containerization of microservices PODs
  • Using Docker on top of OpenStack as well as a bare metal environment

Having said all of the above, I do see cases mostly for well-defined workloads, where the use of cloud infrastructure isn’t mandatory. For example, if I were to consider the automation of a small shop development and testing environment for DevOps purposes, I would consider using Docker directly on a bare metal environment.

Orchestration can be a great abstraction tool between the two environments.

One of the benefits of using an orchestration framework with Docker, is that it can allow us to switch between OpenStack or bare metal environments at any given point in time. In this way, we can choose either options just by pointing our orchestration engine to the target environment of choice. OpenStack Orchestration (Heat) declared support for Docker orchestration starting from the Icehouse release. Cloudify is an open source TOSCA based orchestration that works on OpenStack and other clouds such VMware, AWS and bare metal, and recently included Docker orchestrationGoogle Kubernetes is associated mostly with GCE but could be customized to work with other cloud or environments.

Creative Commons License

* As of version 0.9, the default container technology for Docker is its own libcontainer rather than LXC, but the concept is the same.

This article originally appeared at OpenSource.com.

The post Do I need OpenStack if I use Docker? appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Guest Post at November 25, 2014 12:49 AM

November 24, 2014

Rafael Knuth

Online Meetup: Automating OpenStack clouds and beyond w/ StackStorm

"Automate all the things!" is a great motto for OpenStack and automation is fundamental to DevOps....

November 24, 2014 05:52 PM

Alessandro Pilotti

reBot – using Lego for bare metal deployments on Intel NUC

Here at Cloudbase, we use a lot of Intel Next Unit of Computing (NUC) for our internal testing and development work. The NUC is a small form factor computer, designed and manufactured by Intel, very compact and powerful, sporting a Haswell i5 processor, up to 16 GB of RAM and a mSATA SSD.

You can see our NUC tempest testing rig in action here.

What those NUCs are lacking (except for a single older model, as of today) is the ability to power them on and off remotely, like higher grade servers do with technologies like IPMI, AMT, iLO. Unfortunately, the NUCs don’t have any of those, so we invented our own to have some fun. :-)

 

Lego to the rescue

Since we had a lot of Lego just lying around, we though, why not building a Lego Mindstorms robot to remotely push the NUC power button?

After a bunch of prototypes, we came out with this:

reBot prototyping

 

We call it reBot

Our first implementation of reBot was for Ubuntu Metal as a Service (MAAS). MAAS is the bare-metal deployment service for Ubuntu, which spins physical machines up just like virtual machines in OpenStack. I won’t go into the details of how MAAS & Juju work, but you can read more about our Windows implementation here.

Right now we also have a working proof-of-concept power adapter for OpenStack Ironic.

This is how a 4-node reBot MAAS setup is deploying a full OpenStack bare-metal cloud, all unattended:

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="http://www.youtube.com/embed/J3lFTYvSrcQ" width="630"></iframe>

The Empire deployed Close up of a node

 

Software

The Mindstorms EV3 brick runs a custom firmware, called leJos which provides some remote control abilities via Java RMI (ok, it ain’t Python, but we can live with it :-)). We connected it to the MAAS controller (the sitting on top of the switches in the picture) via an ethernet-over-USB connection.
Here are some of the motor actions that we wrote for pushing the lever up/down and reset the NUC. Here is an example:

import lejos.remote.ev3.*;
import lejos.utility.Delay;

class BarePlasticAction {
    public static void main(String[] args) {

       if(args.length != 4) {
            System.out.println("usage: BarePlasticAction <EV3 address> <A|B|C|D> <degrees> <pause>");
            System.exit(1);
        }

        String host = args[0];
        String port = args[1];
        int degrees = Integer.parseInt(args[2]);
        int pause = Integer.parseInt(args[3]); 

        RMIRegulatedMotor m = null;

        try {
            RemoteEV3 ev3 = new RemoteEV3(host);

            m = ev3.createRegulatedMotor(port, 'L');

            m.setAcceleration(6000);

            float speed = m.getMaxSpeed();
            m.setSpeed((int)speed);

            m.rotateTo(degrees);

            if (pause >= 0) {
                Delay.msDelay(pause);
                m.rotateTo(0);
            }
        }
        catch(Exception ex) {
            ex.printStackTrace();
        }
        finally {
            if (m != null) {
                try {
                    m.close();
                }
                catch(Exception ex) {
                    // ignore
                }

            }
        }
    }
}

All we needed to do at this point was to call it from MAAS, specifying the EV3 port, degrees of rotation and time to keep the power button pushed down (de facto simulating what you’d do with your finger when resetting a PC).

Power on:

java  -cp ../ev3classes.jar:.. BarePlasticAction 10.0.1.1 B 1440 1500

Power off:

java  -cp ../ev3classes.jar:.. BarePlasticAction 10.0.1.1 B 1440 5000

And this is how the MAAS configuration page looks like:

MAAS configuration page

 

The code and Lego model are available as open source on GitHub, so go ahead and build your own, or even improve on the design:

https://github.com/cloudbase/reBot

here’s also a handy bill of materials with all the required Lego parts.

The post reBot – using Lego for bare metal deployments on Intel NUC appeared first on Cloudbase Solutions.

by Dorin Paslaru at November 24, 2014 05:00 PM

Percona

Schema changes in MySQL for OpenStack Trove users

People using OpenStack Trove instances can hit a common issue in the MySQL world: how to perform schema change operations while minimizing the impact on the database server? Let’s explore the options that can allow online schema changes.

Summary

Schema changes in MySQL for OpenStack Trove usersWith MySQL 5.5, pt-online-schema-change from Percona Toolkit is your best option for large tables while regular ALTER TABLE statements are only acceptable for small tables. Also beware of metadata locks.

With MySQL 5.6, almost all types of schema changes can be done online. Metadata locks can also be an issue. pt-online-schema-change can still be worth using as it is also online on read replicas.

Regular ALTER TABLE with MySQL 5.5

If you are still using MySQL 5.5, almost all schema changes will require a table rebuild and MySQL will set a write lock. Therefore all writes to the table that gets modified will be blocked. As soon as the table gets large or if you cannot afford to have a maintenance window, using ALTER TABLE becomes tricky.

The only exception is that secondary indexes can be added or removed without rebuilding the whole table. The table is still write locked during the operation but it is much faster.

You can spot this ‘fast index creation’ process by looking at SHOW PROCESSLIST (see manage keys in the State field):

+----+------+-----------+--------+---------+------+-------------+-----------------------------------------+-----------+---------------+-----------+
| Id | User | Host      | db     | Command | Time | State       | Info                                    | Rows_sent | Rows_examined | Rows_read |
+----+------+-----------+--------+---------+------+-------------+-----------------------------------------+-----------+---------------+-----------+
|  1 | root | localhost | sbtest | Query   |    4 | manage keys | alter table sbtest2 add index idx_k (k) |         0 |             0 |         0 |
+----+------+-----------+--------+---------+------+-------------+-----------------------------------------+-----------+---------------+-----------+

It is helpful for small or even medium tables, but it is not fast enough for large tables.

A common workaround when you have a master-slave configuration is to perform the schema change on the slave first, promote the slave and then perform the schema change on the old master. However replication is not only available if you are using Icehouse and even with Juno, replicas are read-only. So this workaround is not usable with Trove.

pt-online-schema-change

A good alternative is to use pt-online-schema-change. In a nutshell the tool creates a new table with the desired structure, creates triggers on the original table, and copies rows in chunks from the original table to the new table.

Nice features of the tool include being able to write to the original table while the schema change is being performed (hence the name of the tool), monitoring replication lag on slaves and automatic throttling to keep lag under a reasonable threshold.

Let’s assume you want to add a new field called ‘city’ in the ‘user’ table. Instead of running:

mysql> ALTER TABLE mydb.user ADD city VARCHAR(50) NOT NULL DEFAULT '';

you will run:

# pt-online-schema-change --progress=percentage,10 --alter="ADD city VARCHAR(50) NOT NULL DEFAULT ''" h=localhost,u=root,D=mydb,t=user --execute

All ALTER TABLE statements can be converted to an equivalent pt-online-schema-change command.

Is there any tradeoff? The first one is that pt-online-schema-change will be slower than a plain ALTER TABLE. This is expected as the tool has extra code to throttle itself if it adds too much load.

How much slower is it? As always it depends. On a lightly loaded server the difference will be minimal, for instance I measured 3mn24s for ALTER TABLE and 3mn30s for pt-online-schema-change on a test table. However on a heavily loaded server, it can be like 5x slower.

The second tradeoff is that pt-online-schema-change adds triggers on the original table to capture data change. So if you already have triggers, you can’t use the tool (this will be lifted in MySQL 5.7).

Metadata Locks

Metadata locks were introduced in MySQL 5.5 for better transaction isolation.

But one side effect is: if you start a schema change operation on a table while another transaction is reading or writing on the same table, ALTER TABLE will be stuck in the Waiting for metadata lock state.

Another negative side effect of metadata locks is that in some situations ALTER TABLE will also block all queries to the table that is being altered, even reads. In the worst cases, this may create a query pileup situation that will freeze the whole server. For more on this topic, have a look at this post.

What about pt-online-schema-change and metadata locks? It is of course hit by this issue/feature. However the good thing is that the timeout for metadata locks is set by the tool to 60s instead of the default 1 year. So after 60s, pt-online-schema-change will simply retry the operation that failed because of metadata locks.

MySQL 5.6: Online Schema Changes?

Metadata locks also exist with MySQL 5.6, the same issues as described above can then happen.

However the good news with MySQL 5.6 is that most schema changes can be done online. Now the question is: should you use pt-online-schema-change or a regular online ALTER TABLE statement?

Both have pros and cons:

  • ALTER TABLE is easy to use while being confident with pt-online-schema-change requires some time.
  • There is no way for ALTER TABLE to know if it’s overloading a replica while pt-online-schema-change monitors replication lag and throttles itself if needed.
  • ALTER TABLE is only online for the master, not for replicas while pt-online-schema-change is online for all servers.

The last point can be a bit confusing: why an online ALTER TABLE on the master wouldn’t be online on the slave as well? The truth is that unless you are using multi-threaded replication, writes on slaves are serialized. So while the ALTER TABLE is running, the slave will not process any other write, which is not much different from an offline ALTER TABLE.

So if you are running a single server (all Icehouse users for instance), ALTER TABLE is probably the right choice. But if you have read replicas and specifically if the application cannot tolerate replication lag, pt-online-schema-change is definitely a better choice.

Conclusion

Performing schema changes is becoming easier with newer versions of MySQL. However potential issues can be found with any version, so always be careful when you need to run ALTER TABLE.

The post Schema changes in MySQL for OpenStack Trove users appeared first on MySQL Performance Blog.

by Stephane Combaudon at November 24, 2014 08:00 AM

Opensource.com

A look back and a look ahead for OpenStack

Interested in keeping track of what's happening in the open source cloud? Opensource.com is your source for what's happening right now in OpenStack, the open source cloud infrastructure project.

by Jason Baker at November 24, 2014 08:00 AM

Lars Kellogg-Stedman

Fedora Atomic, OpenStack, and Kubernetes (oh my)

While experimenting with Fedora Atomic, I was looking for an elegant way to automatically deploy Atomic into an OpenStack environment and then automatically schedule some Docker containers on the Atomic host. This post describes my solution.

Like many other cloud-targeted distributions, Fedora Atomic runs cloud-init when the system boots. We can take advantage of this to configure the system at first boot by providing a user-data blob to Nova when we boot the instance. A user-data blob can be as simple as a shell script, and while we could arguably mash everything into a single script it wouldn't be particularly maintainable or flexible in the face of different pod/service/etc descriptions.

In order to build a more flexible solution, we're going to take advantage of the following features:

  • Support for multipart MIME archives.

    Cloud-init allows you to pass in multiple files via user-data by encoding them as a multipart MIME archive.

  • Support for a custom part handler.

    Cloud-init recognizes a number of specific MIME types (such as text/cloud-config or text/x-shellscript). We can provide a custom part handler that will be used to handle MIME types not intrinsincally supported by cloud-init.

A custom part handler for Kubernetes configurations

I have written a custom part handler that knows about the following MIME types:

  • text/x-kube-pod
  • text/x-kube-service
  • text/x-kube-replica

When the part handler is first initialized it will ensure the Kubernetes is started. If it is provided with a document matching one of the above MIME types, it will pass it to the appropriate kubecfg command to create the objects in Kubernetes.

Creating multipart MIME archives

I have also created a modified version of the standard write-multipart-mime.py Python script. This script will inspect the first lines of files to determine their content type; in addition to the standard cloud-init types (like #cloud-config for a text/cloud-config type file), this script recognizes:

  • #kube-pod for text/x-kube-pod
  • #kube-service for text/x-kube-service
  • #kube-replica for text/x-kube-replca

That is, a simple pod description might look something like:

#kube-pod
id: dbserver
desiredState:
  manifest:
    version: v1beta1
    id: dbserver
    containers:
    - image: mysql
      name: dbserver
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: secret

Putting it all together

Assuming that the pod description presented in the previous section is stored in a file named dbserver.yaml, we can bundle that file up with our custom part handler like this:

$ write-mime-multipart.py \
  kube-part-handler.py dbserver.yaml > userdata

We would then launch a Nova instance using the nova boot command, providing the generated userdata file as an argument to the user-data command:

$ nova boot --image fedora-atomic --key-name mykey \
  --flavor m1.small --user-data userdata my-atomic-server

You would obviously need to substitute values for --image and --key-name that are appropriate for your environment.

Details, details

If you are experimenting with Fedora Atomic 21, you may find out that the above example doesn't work -- the official mysql image generates an selinux error. We can switch selinux to permissive mode by putting the following into a file called disable-selinux.sh:

#!/bin/sh

setenforce 0
sed -i '/^SELINUX=/ s/=.*/=permissive/' /etc/selinux/config

And then including that in our MIME archive:

$ write-mime-multipart.py \
  kube-part-handler.py disable-selinux.sh dbserver.yaml > userdata

A brief demonstration

If we launch an instance as described in the previous section and then log in, we should find that the pod has already been scheduled:

# kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
dbserver            mysql               /                                       Waiting

At this point, docker needs to pull the mysql image locally, so this step can take a bit depending on the state of your local internet connection.

Running docker ps at this point will yield:

# docker ps
CONTAINER ID        IMAGE                     COMMAND             CREATED             STATUS              PORTS               NAMES
3561e39f198c        kubernetes/pause:latest   "/pause"            46 seconds ago      Up 43 seconds                           k8s--net.d96a64a9--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--d872be51

The pause image here is a Kubernetes detail that is used to configure the networking for a pod (in the Kubernetes world, a pod is a group of linked containers that share a common network namespace).

After a few minutes, you should eventually see:

# docker ps
CONTAINER ID        IMAGE                     COMMAND                CREATED             STATUS              PORTS               NAMES
644c8fc5a79c        mysql:latest              "/entrypoint.sh mysq   3 minutes ago       Up 3 minutes                            k8s--dbserver.fd48803d--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--58794467   
3561e39f198c        kubernetes/pause:latest   "/pause"               5 minutes ago       Up 5 minutes                            k8s--net.d96a64a9--dbserver.etcd--3d30eac0_-_745c_-_11e4_-_b32a_-_fa163e6e92ce--d872be51

And kubecfg should show the pod as running:

# kubecfg list pods
ID                  Image(s)            Host                Labels              Status
----------          ----------          ----------          ----------          ----------
dbserver            mysql               127.0.0.1/                              Running

Problems, problems

This works and is I think a relatively elegant solution. However, there are some drawbacks. In particular, the custom part handler runs fairly early in the cloud-init process, which means that it cannot depend on changes implemented by user-data scripts (because these run much later).

A better solution might be to have the custom part handler simply write the Kubernetes configs into a directory somewhere, and then install a service that launches after Kubernetes and (a) watches that directory for files, then (b) passes the configuration to Kubernetes and deletes (or relocates) the file.

by Lars Kellogg-Stedman at November 24, 2014 05:00 AM

November 21, 2014

Ben Nemec

Paris OpenStack Summit - The Big Tent Casts a Long Shadow

So it's been a couple of weeks since I got back from the Paris Summit, but it's been a busy time and I haven't had time to write up all of my thoughts about it. Here's my belated attempt to do that.

From a purely personal perspective, I felt like this summit was much more useful than Atlanta. That's not a commentary on the summit itself, but more on my involvement in it. I don't know if I was just more comfortable jumping into discussions or if an extra six months working on everything provided me with stronger opinions, but regardless I felt like I contributed a lot more this time around. Some highlights of the discussions I was in:

Logging

Lots of good discussion around improving logging this cycle, which, as a TripleO contributor who is constantly frustrated by our useless logging at anything but DEBUG level, makes me very happy. We have a spec that is very close to ready IMHO, and once that merges we can get to work really improving things in this area.

Upgrades

Also lots of talk about upgrades. Since this and logging were the top two operator issues with OpenStack (by a long shot, IIRC), it's good that the focus is appropriately strong on them. I got the impression from some of the discussions that there's still a lot that isn't fully understood about how Nova is moving forward with their live upgrade mechanisms (and thus how the other projects can follow suit), but once all the mechanisms are in place it becomes more of an education issue than an implementation one, which should be an easier problem.

There's a long way to go, but I'm seeing a glimmer of light at the end of the live upgrades tunnel.

TripleO Review Policy

This was a long discussion, but I think it was valuable and the end result was a streamlining of the review process in TripleO. If this works out I could see it being adopted across many more projects (some have already expressed interest). Since core reviewer bandwidth is one of the biggest bottlenecks in OpenStack, optimizing it is important.

See the new review policy spec for a detailed discussion of the motivations and conclusions from this discussion.

Puppet/Chef/Ansible/etc.

While I will admit to having been poorly prepared for these discussions, it felt like there was a good dialog between some experts in the various tools and the TripleO team who are trying to implement their usage. I learned a lot and I think a general path forward was determined. It's early days for that work though, so we'll see where it goes from here.

Lunch

Not a technical point, but I thought some kudos were deserved here. Lunches at Le Meridien were terrific. The lines were kind of excessive the last two days, but I suspect that was a reflection of the quality and the fact that fewer people felt the need to go elsewhere to find food they wanted to eat.

There were a few lowlights too though:

Big Tent

The Big Tent sessions didn't feel like they made any significant progress toward an answer, at least from where I was sitting. Based on that, my suspicion is that a massive, all-at-once change to a different governance model just isn't going to happen. Hopefully we can find some incremental improvements to make along the way, and maybe some day we'll end up in a place everyone is happier with. This topic's probably above my pay grade though.

Containers

Based solely on this session, containers as a service may never be a thing in OpenStack. :-(

"The sky is falling" pessimism aside though, my reaction after the session concluded was less negative than while I was in the session. Honestly, I think the disconnect there (which was never explicitly called out in the session, to my knowledge) is that the folks working on containers want to collaborate with the OpenStack Compute team to make their new project fit in as well as possible, but the OpenStack Compute team is overloaded as it is and can't give them the time and effort they're looking for. This has led to a lot of wheel spinning in the effort to get the project off the ground.

I suspect the outcome of this topic is going to be that the containers team goes off and does their thing, and eventually they live as a largely separate entity under the Compute umbrella. But my crystal ball has been wrong before. :-)

In the end, I think it was a productive summit and we seem to be working on the right things to make OpenStack better for the people using it. I could probably write for another $TOO_MANY hours about everything that went on, but I have an RDO release for TripleO to work on so I'll stop here.

by bnemec at November 21, 2014 06:44 PM

Rafael Knuth

Online Meetup: How does OpenStack Swift play with Open vStorage?

Join us for a conversation between Swift PTL John Dickinson and Wim Provoost of Open vStorage as...

November 21, 2014 05:21 PM

OpenStack Blog

OpenStack Community Weekly Newsletter (Nov 14 – 21)

OpenStack User Survey Insights: November 2014

This is the fourth consecutive survey conducted by the User Committee prior to each Summit starting in April 2013 and the previous survey in Atlanta, May 2014. The goals of the survey are to generate insights based on a representative sample of OpenStack users, in order to better understand their organizational profiles, use cases and technology choices across different deployment stages and sizes. These insights are intended to provide feedback to the broader community, and to arm technical leaders and contributors with better data to make decisions.

Wrapping up the Travel Support Program – Kilo

The OpenStack Foundation brought 20 people to Paris for the Summit earlier this month, thanks to the grants offered by the Travel Support Program. We had 22 people accepted in the program from 11 different countries, spanning five continents. Four people traveled from Brazil, four from India, three from Europe and the rest were from South America, North America and South-east Asia. Of the selected recipients, two were unable to attend due to VISA timing issues, but we were excited to welcome the 20 attendees that were able to make the trip. Stay tuned for when we announce the applications for the Travel Support Applications for the May Summit in Vancouver.

A mascot for Ironic

The idea about what the mascot would be was easy because the RAX guys put “bear metal” in their presentation and that totally rocks! So Lucas Alvares Gomes drew a bear.

Relevant Conversations

Deadlines

Tips ‘n Tricks

Security Advisories and Notices

Reports from Previous Events

Upcoming Events

Other News

Got Answers?

Ask OpenStack is the go-to destination for OpenStack users. Interesting questions waiting for answers:

Welcome New Reviewers, Developers and Core Reviewers

Welcome Michael McCune and Sergey Reshetniak to sahara-core, Steve Heyman, Juan Antonio Osorio Robles and Steve Heyman for barbican-core.

Xiao Xi LIU Ai Jie Niu
Naohiro Tamura jiangfei
Danny Wilson Dave Chen
yatin rajiv
Richard H juigil kishore
Pawel Palucki Richard Hedlind
Park Lucas Dutra Nunes
jmurax Nicolas T
Swati Shukla Michael Hagedorn
Guillaume Giamarchi keshava
Seb Hughes
sailajay

OpenStack Reactions

Helping new contributor

Finding a great bug to solve and giving it to new contributors (and helping them)

The weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

by Stefano Maffulli at November 21, 2014 04:47 PM

Flavio Percoco

What's coming in Kilo for Glance, Zaqar and Oslo?

As usual, here's a write up of what happened last week during the OpenStack Summit. More than a summary, this post contains the plans we discussed for the next 6 months.

Glance

Lots of things happened in Juno for Glance. Work related to artifacts was done, async workers were implemented and glance_store was created. If none of these things excite you, I'm sorry to tell you that you're missing the big picture.

The 3 features mentioned above are the bases of many things that will happen in Kilo. For long time, we've been waiting for async workers to land and now that we have them we can't but use them. One of the first things that will consume this feature is image introspection, which will allow glance to read image's metadata and extract useful information from them. In addition to this, we'll messing with images a bit more by implementing basic support for image conversion to allow for automatic conversion of images during uploads and also as a manual operation. There are many things to take care of here and tons of subtle corner cases so please, keep an eye on these things and help us out.

The work on artifacts is not complete, there are still many things to do there and lots of patches and code are being written. This still seems to be the path the project is going down to for Kilo to allow more generic catalogs and support for storing data assets.

One more thing on Glance, all the work that happened in glancestore during Juno, will finally pay off in Kilo. We'll start refactoring the library and it'll likely be adopted by Nova in K-2. Noticed I said likely? That's because before we get there, we need to clean up the messy glance wrapper nova has. In that same session we discussed what to do with that code and agreed on getting rid of it and let nova consume glanceclient directly, which will happen in kilo-1 before the glancestore adoption. Here's the spec.

Zaqar

When thinking about Zaqar and Kilo, you need to keep 3 things in mind:

  1. Notifications
  2. Persistent Transport
  3. Integration with other services

Notifications is something we've wanted to work on since Icehouse. We talked about them back in Hong Kong, then in Atlanta and we finally have a good plan for them now. The team will put lots of efforts on this feature and we'd love to get as much feedback as possible on the implementation, use cases and targets. In order to implement notifications and mark a fresh start, the team has also decided to bump the API version number to 2 and use this chance to clean up the technical debt from previous versions. Some of the things that will go away from the API are:

  • Get messages by id
  • FIFO will become optional
  • Queue's will be removed from the API, instead we'll start talking about topics. Some notes on this here.

One of the projects goal is to be easily consumed regardless of the device you're using. Moreover, the project wants to allow users to integrate with it. Therefore, the team is planning to start working on a persistent Transport in order to define a message-based protocol that is both stateless and persistent as far as the communication between the peers goes. The first target is websocket, which will allow users to consume Zaqar's API from a browser and even using a library without having to go down to raw TCP connections, which was highly discouraged at the summit. This falls perfectly in the projects goals to be easily consumable and to reuse existing technologies and solutions as much as possible.

Although the above two features sound exciting, the ultimate goal is to integrate with other projects in the community. The team has long waited for this opportunity and now that it has a stable API, it is the perfect time for this integration to happen. At our integration session folks from Barbican, Trove, Heat, Horizon showed up - THANKS - and they all shared use-cases, ideas and interesting opinions about what they need and about what they'd like to see happening for Kilo with regards to this integration. Based on the results of this session Heat and Horizon are likely to be the first targets. The team is thrilled about this and we're all looking forward for this collaboration to happen.

Oslo

No matter what I work on, I'll always have time for Oslo. Just like for the other projects I mentioned, there will be exciting things happening in Oslo as well.

Let me start by saying that new libraries will be released but not many of them. This will give the team the time needed to focus on the existing ones and also to work on the other, perhaps equally important, items in the list. For example, we'll be moving away from using namespaces - YAY!, which means we'll be updating all the already released libraries. Something that's worth mentioning is that the already released libraries won't be renamed and the ones to be released will follow the same standard for names. The difference is that they won't be using namespaces internally at all.

Also related to the libraries maintenance, the team has decided to stop using alpha versions for the libraries. One of the points against this is that we currently don't put caps on stable branches, however this will change in Kilo. We will pin to MAJOR.MINOR+1 in stable, allowing bug fixes in MAJOR.MINOR.PATCH+1.

I unfortunately couldn't attend all the Oslo sessions and I missed one that I really wanted to attend about oslo.messaging. By reading the etherpad, it looks like great things will happen in the library during kilo that will help with growing its community. Drivers will be kept in tree, zmq won't be deprecated, yet. Some code de-duplication will happen and both the rabbit and qpid driver will be merged into a single one now that kombu has support for qpid. Just like other projects throughout OpenStack, we'll be targeting full Py3K support like CRAZY!

Hopefully I didn't forget anything or even worse said something stupid. Now, if you may excuse me, I gotta go offline for the next 6 month. Someone has to work on these things.

by FlaPer87 at November 21, 2014 03:33 PM

Rob Hirschfeld

To thrive, OpenStack must better balance dev, ops and business needs.

OpenStack has grown dramatically in many ways but we have failed to integrate development, operations and business communities in a balanced way.

My most urgent observation from Paris is that these three critical parts of the community are having vastly different dialogs about OpenStack.

Clouds DownAt the Conference, business people were talking were about core, stability and utility while the developers were talking about features, reorganizing and expanding projects. The operators, unfortunately segregated in a different location, were trying to figure out how to share best practices and tools.

Much of this structural divergence was intentional and should be (re)evaluated as we grow.

OpenStack events are split into distinct focus areas: the conference for business people, the summit for developers and specialized days for operators. While this design serves a purpose, the community needs to be taking extra steps to ensure communication. Without that communication, corporate sponsors and users may find it easier to solve problems inside their walls than outside in the community.

The risk is clear: vendors may find it easier to work on a fork where they have business and operational control than work within the community.

Inside the community, we are working to help resolve this challenge with several parallel efforts. As a community member, I challenge you to get involved in these efforts to ensure the project balances dev, biz and ops priorities.  As a board member, I feel it’s a leadership challenge to make sure these efforts converge and that’s one of the reasons I’ve been working on several of these efforts:

  • OpenStack Project Managers (was Hidden Influencers) across companies in the ecosystem are getting organized into their own team. Since these managers effectively direct the majority of OpenStack developers, this group will allow
  • DefCore Committee works to define a smaller subset of the overall OpenStack Project that will be required for vendors using the OpenStack trademark and logo. This helps the business community focus on interoperability and stability.
  • Technical leadership (TC) lead “Big Tent” concept aligns with DefCore work and attempts to create a stable base platform while making it easier for new projects to enter the ecosystem. I’ve got a lot to say about this, but frankly, without safeguards, this scares people in the ops and business communities.
  • An operations “ready state” baseline keeps the community from being able to share best practices – this has become a pressing need.  I’d like to suggest as OpenCrowbar an outside of OpenStack a way to help provide an ops neutral common starting point. Having the OpenStack developer community attempting to create an installer using OpenStack has proven a significant distraction and only further distances operators from the community.

We need to get past seeing the project primarily as a technology platform.  Infrastructure software has to deliver value as an operational tool for enterprises.  For OpenStack to thrive, we must make sure the needs of all constituents (Dev, Biz, Ops) are being addressed.


by Rob H at November 21, 2014 03:30 PM

Self-Exposure: Hidden Influencers become OpenStack Product Working Group

Warning to OpenStack PMs: If you are not actively involved in this effort then you (and your teams) will be left behind!

ManagersThe Hidden Influencers (now called “OpenStack Product Working Group”) had a GREAT and PRODUCTIVE session at the OpenStack (full notes):

  1. Named the group!  OpenStack Product Working Group (now, that’s clarity in marketing) [note: I was incorrect saying “Product Managers” earlier].
  2. Agreed to use the mailing list for communication.
  3. Committed to a face-to-face mid-cycle meetup (likely in South Bay)
  4. Output from the meetup will be STRATEGIC DIRECTION doc to board (similar but broader than “Win the Enterprise”)
  5. Regular meeting schedule – like developers but likely voice interactive instead of IRC.  Stefano Maffulli is leading.

PMs starting this group already direct the work for a super majority (>66%) of active contributors.

The primary mission for the group is to collaborate and communicate around development priorities so that we can ensure that project commitments get met.

It was recognized that the project technical leads are already strapped coordinating release and technical objectives.  Further, the product managers are already but independently engaged in setting strategic direction, we cannot rely on existing OpenStack technical leadership to have the bandwidth.

This effort will succeed to the extent that we can help the broader community tied in and focus development effort back to dollars for the people paying for those developers.  In my book, that’s what product managers are supposed to do.  Hopefully, getting this group organized will help surface that discussion.

This is a big challenge considering that these product managers have to balance corporate, shared project and individual developers’ requirements.  Overall, I think Allison Randall summarized our objectives best: “we’re herding cats in the same direction.”


by Rob H at November 21, 2014 03:11 PM

Leveling OpenStack’s Big Tent: is OpenStack a product, platform or suite?

Question of the day: What should OpenStack do with all those eager contributors?  Does that mean expanding features or focusing on a core?

IMG_20141108_101906In the last few months, the OpenStack technical leadership (Sean Dague, Monty Taylor) has been introducing two interconnected concepts: big tent and levels.

  • Big tent means expanding the number of projects to accommodate more diversity (both in breath and depth) in the official OpenStack universe.  This change accommodates the growth of the community.
  • Levels is a structured approach to limiting integration dependencies between projects.  Some OpenStack components are highly interdependent and foundational (Keystone, Nova, Glance, Cinder) while others are primarily consumers (Horizon, Saraha) of lower level projects.

These two concepts are connected because we must address integration challenges that make it increasingly difficult to make changes within the code base.  If we substantially expand the code base with big tent then we need to make matching changes to streamline integration efforts.  The levels proposal reflects a narrower scope at the base than we currently use.

By combining big tent and levels, we are simultaneously growing and shrinking: we grow the community and scope while we shrink the integration points.  This balance may be essential to accommodate OpenStack’s growth.

UNIVERSALLY, the business OpenStack community who wants OpenStack to be a product.  Yet, what’s included in that product is unclear.

Expanding OpenStack projects tends to turn us into a suite of loosely connected functions rather than a single integrated platform with an ecosystem.  Either approach is viable, but it’s not possible to be both simultaneously.

On a cautionary note, there’s an anti-Big Tent position I heard expressed at the Paris Conference.  It’s goes like this: until vendors start generating revenue from the foundation components to pay for developer salaries; expanding the scope of OpenStack is uninteresting.

Recent DefCore changes also reflect the Big Tent thinking by adding component and platform levels.  This change was an important and critical compromise to match real-world use patterns by companies like Swiftstack (Object), DreamHost (Compute+Ceph), Piston (Compute+Ceph) and others; however, it creates the need to explain “which OpenStack” these companies are using.

I believe we have addressed interoperability in this change.  It remains to be seen if OpenStack vendors will choose to offer the broader platform or limit to themselves to individual components.  If vendors chase the components over platform then OpenStack becomes a suite of loosely connect products.  It’s ultimately a customer and market decision.

It’s not too late to influence these discussions!  I’m very interested in hearing from people in the community which direction they think the project should go.


by Rob H at November 21, 2014 03:09 PM

IBM OpenStack Team

Code and tapas: Best when enjoyed with others

I’m off to Spain to hang out with an innovative group of developers at the Codemotion conference at the Universidad San Pablo CEU in Madrid. On Saturday, I’ll be delivering the keynote, but throughout the conference we’ll explore our craft: software development.

Here’s how Codemotion describes its conference:

“Computer Science is a motor of the economy: we control airports and hospitals, keep transactions and invoices flowing, and store passwords, documents and kitten pictures. Hell, we even did what we could for the banks.

“We live in a sector that reinvents itself every six months, where it would be easy to just relax and stop chasing after the next new and shiny thing. Codemotion is this day in which we sit down to talk about all that we have in common, stepping aside everything that makes us different. … [so]…

“Codemotion gets all the IT communities in Spain together for two days. We take an entire university building to present the best of each technology and get you far away from your comfort zone.”

This is going to be fun—and not just because my pre-Thanksgiving dinner is going to be authentic tapas (there are some patatas bravas and jamón iberico with my name on it, I’m sure). It’ll be fun because we get to talk about how developers are making a difference in the lives of everyday citizens. And I get to hang out with developers committed to getting better at doing what they do best.

codemotionBut one line in the Codemotion call-to-action stands out above the rest to me: “We live in a sector that reinvents itself every six months, where it would be easy to just relax and stop chasing after the next new and shiny thing.”

It couldn’t be more true. We are seeing an unprecedented rate of change in our industry. New languages. New technologies and devices. New skills. New use cases. New data management problems that require even newer solutions. Plus, everything needs to be secured, tracked, stored, accessed and, of course, analyzed at a moment’s notice.

So how do we keep up with this constant reinvention? If you know me, my answer won’t surprise you: Open technology.

The power of the community

At the heart of a robust open technology effort is a vibrant community of developers looking to constantly improve the code we are working on. From different countries, industries, and bringing our unique skills and use cases together, we innovate as a team. The skills here go beyond languages and coding methods. Project management and collaboration are key. Basic listening skills are as critical as software engineering training. Together, we do more than any one team or company could do on its own.

The power of the code

The outcome of a well-managed, highly engaged and talented open technology development effort is quality code. Code talks! Written for the leading use cases, tested, debugged and iterated, implementers know quality when they can use it, and when they see many other vendors and partners doing the same at increasing rates. This is how we begin to achieve more vendor implementation choices and begin to ensure cross-implementation interoperability.

Tapas is always better when you share it with friends. Coding and open technology are the same. Meticulously crafted and stacked from the appetizing ingredients, the group quickly identifies innovative new ideas. We share our standby recipes. We make improvements. Innovation demands we try new things, but it doesn’t demand we ignore the favorites—those standard recipes that work every time.

This is open technology. There is no one perfect recipe for success. There are guidelines. Open governance. Technical innovation. Iteration. The OpenStack community stands out among the crowd. This open source software community is building the most innovative technology for creating private and public clouds today. The success of the growing community is helping individual community members succeed as well. Just ask an OpenStack developer about their job prospects. Ask an implementer if they are better prepared to handle new cloud innovations and integrations because they employ OpenStack as the foundation of their cloud infrastructure.

For today’s developers, open source is a way of life. Like tapas, there is no easy answer. You have to get in there, roll up your sleeves and create something fresh. At the end of the day, a developer knows he or she was successful because something works better. Like when you have a great tapa, success tastes great.

Os veréen Madrid.

The post Code and tapas: Best when enjoyed with others appeared first on Thoughts on Cloud.

by Angel Luis Diaz at November 21, 2014 01:00 PM

Julien Danjou

Distributed group management and locking in Python with tooz

With OpenStack embracing the Tooz library more and more over the past year, I think it's a good start to write a bit about it.

A bit of history

A little more than year ago, with my colleague Yassine Lamgarchal and others at eNovance, we investigated on how to solve a problem often encountered inside OpenStack: synchronization of multiple distributed workers. And while many people in our ecosystem continue to drive development by adding new bells and whistles, we made a point of solving new problems with a generic solution able to address the technical debt at the same time.

Yassine wrote the first ideas of what should be the group membership service that was needed for OpenStack, identifying several projects that could make use of this. I've presented this concept during the OpenStack Summit in Hong-Kong during an Oslo session. It turned out that the idea was well-received, and the week following the summit we started the tooz project on StackForge.

Goals

Tooz is a Python library that provides a coordination API. Its primary goal is to handle groups and membership of these groups in distributed systems.

Tooz also provides another useful feature which is distributed locking. This allows distributed nodes to acquire and release locks in order to synchronize themselves (for example to access a shared resource).

The architecture

If you are familiar with distributed systems, you might be thinking that there are a lot of solutions already available to solve these issues: ZooKeeper, the Raft consensus algorithm or even Redis for example.

You'll be thrilled to learn that Tooz is not the result of the NIH syndrome, but is an abstraction layer on top of all these solutions. It uses drivers to provide the real functionalities behind, and does not try to do anything fancy.

All the drivers do not have the same amount of functionality of robustness, but depending on your environment, any available driver might be suffice. Like most of OpenStack, we let the deployers/operators/developers chose whichever backend they want to use, informing them of the potential trade-offs they will make.

So far, Tooz provides drivers based on:

All drivers are distributed across processes. Some can be distributed across the network (ZooKeeper, memcached, redis…) and some are only available on the same host (IPC).

Also note that the Tooz API is completely asynchronous, allowing it to be more efficient, and potentially included in an event loop.

Features

Group membership

Tooz provides an API to manage group membership. The basic operations provided are: the creation of a group, the ability to join it, leave it and list its members. It's also possible to be notified as soon as a member joins or leaves a group.

Leader election

Each group can have a leader elected. Each member can decide if it wants to run for the election. If the leader disappears, another one is elected from the list of current candidates. It's possible to be notified of the election result and to retrieve the leader of a group at any moment.

Distributed locking

When trying to synchronize several workers in a distributed environment, you may need a way to lock access to some resources. That's what a distributed lock can help you with.

Adoption in OpenStack

Ceilometer is the first project in OpenStack to use Tooz. It has replaced part of the old alarm distribution system, where RPC was used to detect active alarm evaluator workers. The group membership feature of Tooz was leveraged by Ceilometer to coordinate between alarm evaluator workers.

Another new feature part of the Juno release of Ceilometer is the distribution of polling tasks of the central agent among multiple workers. There's again a group membership issue to know which nodes are online and available to receive polling tasks, so Tooz is also being used here.

The Oslo team has accepted the adoption of Tooz during this release cycle. That means that it will be maintained by more developers, and will be part of the OpenStack release process.

This opens the door to push Tooz further in OpenStack. Our next candidate would be write a service group driver for Nova.

The complete documentation for Tooz is available online and has examples for the various features described here, go read it if you're curious and adventurous!

by Julien Danjou at November 21, 2014 12:10 PM

eNovance Engineering Teams

Distributed group management and locking in Python with tooz

With OpenStack embracing the Tooz library more and more over the past year, I think it’s a good start to write a bit about it.

A bit of history

A little more than year ago, with my colleague Yassine Lamgarchal and others at eNovance, we investigated on how to solve a problem often encountered inside OpenStack: synchronization of multiple distributed workers. And while many people in our ecosystem continue to drive development by adding new bells and whistles, we made a point of solving new problems with a generic solution able to address the technical debt at the same time.

Yassine wrote the first ideas of what should be the group membership service that was needed for OpenStack, identifying several projects that could make use of this. I’ve presented this concept during the OpenStack Summit in Hong-Kong during an Oslo session. It turned out that the idea was well-received, and the week following the summit we started the tooz project on StackForge.

Goals

Tooz is a Python library that provides a coordination API. Its primary goal is to handle groups and membership of these groups in distributed systems.

Tooz also provides another useful feature which is distributed locking. This allows distributed nodes to acquire and release locks in order to synchronize themselves (for example to access a shared resource).

The architecture

If you are familiar with distributed systems, you might be thinking that there are a lot of solutions already available to solve these issues: ZooKeeper, the Raft consensus algorithm or even Redis for example.
You’ll be thrilled to learn that Tooz is not the result of the NIH syndrome, but is an abstraction layer on top of all these solutions. It uses drivers to provide the real functionalities behind, and does not try to do anything fancy.
All the drivers do not have the same amount of functionality of robustness, but depending on your environment, any available driver might be suffice. Like most of OpenStack, we let the deployers/operators/developers chose whichever backend they want to use, informing them of the potential trade-offs they will make.

So far, Tooz provides drivers based on:

Kazoo (ZooKeeper)
Zake
memcached
redis
SysV IPC (only for distributed locks for now)
PostgreSQL (only for distributed locks for now)
MySQL (only for distributed locks for now)

All drivers are distributed across processes. Some can be distributed across the network (ZooKeeper, memcached, redis…) and some are only available on the same host (IPC). Also note that the Tooz API is completely asynchronous, allowing it to be more efficient, and potentially included in an event loop.

Features

Group membership

Tooz provides an API to manage group membership. The basic operations provided are: the creation of a group, the ability to join it, leave it and list its members. It’s also possible to be notified as soon as a member joins or leaves a group.

Leader election

Each group can have a leader elected. Each member can decide if it wants to run for the election. If the leader disappears, another one is elected from the list of current candidates. It’s possible to be notified of the election result and to retrieve the leader of a group at any moment.

Distributed locking

When trying to synchronize several workers in a distributed environment, you may need a way to lock access to some resources. That’s what a distributed lock can help you with.

Adoption in OpenStack

Ceilometer is the first project in OpenStack to use Tooz. It has replaced part of the old alarm distribution system, where RPC was used to detect active alarm evaluator workers. The group membership feature of Tooz was leveraged by Ceilometer to coordinate between alarm evaluator workers.

Another new feature part of the Juno release of Ceilometer is the distribution of polling tasks of the central agent among multiple workers. There’s again a group membership issue to know which nodes are online and available to receive polling tasks, so Tooz is also being used here.

The Oslo team has accepted the adoption of Tooz during this release cycle. That means that it will be maintained by more developers, and will be part of the OpenStack release process.

This opens the door to push Tooz further in OpenStack. Our next candidate would be write a service group driver for Nova.

The complete documentation for Tooz is available online and has examples for the various features described here, go read it if you’re curious and adventurous!

by Julien Danjou at November 21, 2014 11:29 AM

Opensource.com

Lessons from the OpenStack user survey

Every six months, the OpenStack Foundation reports on the results from its user survey. The results for the most recent iteration were released earlier this month on the OpenStack Superuser blog. Let's take a look and see what's new.

by Jason Baker at November 21, 2014 10:00 AM

Craige McWhirter

Deleting Root Volumes Attached to Non-Existent Instances

Let's say you've got an OpenStack build you're getting ready to go live with. Assume also that you're performing some, ahem, robustness testing to see what breaks and prevent as many surprises as possible prior to going into production. OpenStack controller servers are being rebooted all over the shop and during this background chaos, punters are still trying to launch instances with vary degrees of success.

Once everything has settled down, you may find that some lucky punters have deleted the unsuccessful instances but the volumes have been left behind. This isn't initially obvious from the cinder CLI without cross checking with nova:

$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | B
ootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
| 3e56985c-541c-4bdd-b437-16b3d96e9932 | in-use    |              |  3   |    block    |
 true   | 6e06aa0f-efa7-4730-86df-b32b47e53316 |
+--------------------------------------+-----------+--------------+------+-------------+--
--------+--------------------------------------+
$ nova show 6e06aa0f-efa7-4730-86df-b32b47e53316
ERROR (CommandError): No server with a name or ID of '6e06aa0f-efa7-4730-86df-b32b47e53316' exists.

It will manifest itself in Horizon like this:

Attached to None

Now trying to delete this volume is going to fail:

$ cinder delete 52aa706df17d-4599-948c-87ae46d945b2
Delete for volume 52aa706d-f17d-4599-948c-87ae46d945b2 failed: Invalid volume:
Volume status must be available or error, but current status is: creating (HTTP 400)
(Request-ID: req-f45671de-ed43-401c-b818-68e2a9e7d6cb)
ERROR: Unable to delete any of the specified volumes.

As will an attempt to detach it from the non-existent instance:

$ nova volume-detach 6e06aa0f-efa7-4730-86df-b32b47e53316 093f32f6-66ea-451b-bba6-7ea8604e02c6
ERROR (CommandError): No server with a name or ID of '6e06aa0f-efa7-4730-86df-b32b47e53316' exists.

and no, force-delete does not work either.

Here's my approach for resolving this problem:

SSH onto your MariaDB server for OpenStack and open MariaDB to the cinder database:

$ mysql cinder

Unset the attachment in the volumes table by repeating the below command for each volume that requires detaching from a non-existent instance:

MariaDB [cinder]> UPDATE volumes SET attach_status='detached', instance_uuid=NULL, \
attach_time=NULL, status="available" WHERE id='3e56985c-541c-4bdd-b437-16b3d96e9932';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Back on your OpenStack client workstations you should now be able to delete the offending volumes:

$ cinder delete 3e56985c-541c-4bdd-b437-16b3d96e9932

Happy housekeeping :-)

by Craige McWhirter at November 21, 2014 01:35 AM

November 20, 2014

Kenneth Hui

Some New OpenStack Content

OSDC_What_is_Openstack_blue

One of the things I enjoy most about being part of the OpenStack community is the amount of content available (sometime too much content) about the technology and the ecosystem.  I’ve had the privilege to contribute some of that content myself, particularly in the past month.  So, I thought it might be helpful to at least a few people if I aggregated the content in a single blog post.

OpenStack Summit

I had a busy first two days in Paris where I moderated two panels, participated as a panelist in five others, and had the opportunity to give a mini-talk.  The panels and mini-talk covered a wide range of topics from technology to community to career advice.  The OpenStack Foundation has helpfully made all the Summit panels and talks available on their YouTube channel.  Below are the panels and talk I participated in, organized in alphabetical order.

Ask The Experts: OpenStack As A Service Or A Distribution

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/jHt3Fj5XbbM?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

Bridging The Gap: OpenStack For VMware Admins

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/ePIwHZJxVNg?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

 Building A Cloud Career In OpenStack

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/0ba2W2TPrbE?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

 Meet The OpenStack Ambassadors

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/JZraN92lgZE?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

OpenStack Design Guide Panel

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/PtomtKeJ0tc?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

The Role Of Software Defined Storage In OpenStack Cinder

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/DZbrf8TgCTU?version=3&amp;rel=1&amp;fs=1&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" type="text/html" width="560"></iframe>

OpenStack Podcast

I had a blast this week being interviewed by Niki Acosta and Jeff Dickey (well sort of, since Jeff decided to dodge me by going on vacation in Mexico).  I recorded the podcast/hangout from my home office which has bright red walls (not my choice) so it looks like I am in some kind of 80’s new wave dance hall.  That aside, it was a great opportunity to talk all things OpenStack.

<iframe allowfullscreen="true" class="youtube-player" frameborder="0" height="315" src="http://www.youtube.com/embed/videoseries?list=UU2iPkY9iMBaWVCxoklBeGWQ&amp;hl=en_US" type="text/html" width="560"></iframe>

OpenStack CrowdChat

Last but not least, I had the privilege of moderating an EMC sponsored CrowdChat on the topic of “OpenStack Operations.”  The CrowdChat was headlined by a star-studded panel of OpenStack experts, including Tim Bell from CERN, Scott Carlson from PayPal, Craig Tracey from Blue Box Group, and Sean Winn from EMC/Cloudscaling.  There were great discussions on a plethora of topics and I encourage everyone to read the transcript here rather you were able to join us live or not.  We will be sponsoring more OpenStack CrowdChats in the future.

That’s all for now.  Hope folks find the content interesting and useful.


Filed under: Cloud, Cloud Computing, IT Operations, OpenStack, Private Cloud, Public Cloud Tagged: Amazon Web Services, AWS, Cloud, Cloud computing, IaaS, OpenStack, Private Cloud, Public Cloud

by kenhui at November 20, 2014 04:00 PM

The Official Rackspace Blog » OpenStack

Build Rich Network Services With OpenStack’s Neutron API

Providing users with programmatic control of their infrastructure has always been one of the primary value propositions of the Rackspace Cloud. The ability to deploy and manage a wide array of cloud resources with a few lines of code has brought new levels of automated efficiency to the IT industry and revolutionized how we think about managing our applications and workloads.

Today, we make it easier for Rackspace Cloud users to build rich network services with the availability of the OpenStack Neutron Networking API. This milestone greatly increases your ability to create and manage networking services and capabilities and makes it possible to add a number of immediate and future improvements. The API will be in Limited Availability for our Cloud Networks users only. If you are already a Rackspace Cloud Networks user, you may immediately start taking advantage of this API.

Integrating OpenStack’s Neutron API into the Rackspace Cloud introduces three new top-level resources: /networks, /ports and /subnets – all of which are available today.

The Neutron API also introduces the following new features:

  • Create and manage Cloud Networks via Neutron API
  • Assign routes to Cloud Servers at boot-time (Host Routes)
  • Configure allocation pools for subnets (CIDRs) on Cloud Networks to control the dynamic IP address assignments on your Cloud Servers
  • Provision an IP address of your choice on isolated networks ports
  • Dual stack your isolated networks so that you can have IPv4 and IPv6 addresses on the same port

The integration of the OpenStack Networking API provides an alternative to the /os-networksv2 Cloud Servers extension that was the only option when provisioning networks in the public cloud.

While the existing networking API based on the /os-networksv2 extension will continue to function, we encourage you to start using the new API in order to take advantage of the new features and services that are only available within Neutron.

If you are a heavy consumer of Rackspace’s networking API, here are a few points to take note of:

  • You will still need to use the Cloud Networks virtual interface extension (/os-virtual-interfacesv2) to attach and detach networks at this time
  • The Neutron client will not be available to use along with the new API at this time, although we plan to make this available very soon.
  • The API is not currently available to RackConnect v3 users
  • The Neutron API and all of the related improvements are only available via API at this time. We expect to start integrating some new functions into the Cloud Control Panel soon.

The documentation for the new API can be found here and the getting started guide can be found here.

As always, we greatly value your feedback and look forward to more announcements about new features for cloud users.

by Sameer Satyam at November 20, 2014 04:00 PM

Red Hat Stack

Co-Existence of Containers and Virtualization Technologies

By, Federico Simoncelli, Principal Software Engineer, Red Hat

As a software engineer working on the Red Hat Enterprise Virtualization (RHEV), my team and I are driven by innovation; we are always looking for cutting edge technologies to integrate into our product.

Lately there has been a growing interest in Linux containers solutions such as Docker. Docker provides an open and standardized platform for developers and sysadmins to build, ship, and run distributed applications. The application images can be safely held in your organization registry or they can be shared publicly in the docker hub portal (http://registry.hub.docker.com) for everyone to use and to contribute to.

Linux containers are a well-known technology that runs isolated Linux systems on the same host sharing the same kernel and resources as cpu time and memory. Containers are more lightweight, perform better and allow more density of instances compared to full virtualization where virtual machines run dedicated full kernels and operating systems on top of virtualized hardware. On the other hand virtual machines are still the preferred solution when it comes to running highly isolated workloads or different operating systems than the host.

As the Docker vendor ecosystem has grown richer, Red Hat announced Red Hat Enterprise Linux Atomic: a lightweight operating system based on Red Hat Enterprise Linux and designed to run applications in Docker containers. Other vendors are focused on providing Docker orchestration tools across different hosts. One example is Kubernetes, an open source Docker manager, recently released by Google.

So how does Red Hat Enterprise Virtualization work with Docker today? oVirt (the upstream project for Red Hat Enterprise Virtualization) supports running Docker containers inside virtual machines and simplifies the process by enabling the Project Atomic image to be imported into your datacenter from the public Glance repository (glance.ovirt.org). Additionally, we are working on providing a platform for the orchestration solutions to integrate with RHEV. Kubernetes, in fact, already includes an oVirt Cloud Provider that can be connected to your data centers to discover the virtual machines dedicated to run containers (minions in the Kubernetes jargon).

Red Hat Enterprise Virtualization therefore is capable of providing you with an optimized stack to run containers starting from the operating system on the bare-metal up to the one inside the virtual machines and in the images, preserving at the same time the freedom and possibilities of the Docker hub. It is possible to imagine a future addition to the Kubernetes oVirt Cloud Provider to register regular RHEV hosts as minions as well, giving you the option to run containers on bare-metal with a minimum effort.

Today, deploying a private or hybrid cloud that runs virtual machines and containers, as just described, is cost prohibitive. We know that you are very much interested in maximizing the efficiency and optimizing your data centers by deploying the right tools for the right workloads. To help you in this quest, we are working on enabling Red Hat Enterprise Linux (RHEL) Atomic hosts to dynamically run different types of workloads. For example RHEV virtual machines, Docker containers and Hadoop jobs. In fact under the orchestration of Mesos (a powerful scheduling framework) it is possible to maximize and balance the hosts computational power for the most important and demanding tasks at any given time.

Integrating emerging technologies such as Kubernetes, Docker and Mesos enables us to help you to meet your requirements and run efficient and reliable datacenters. Stay tuned for more blog posts that will highlight the integration of these new technologies features into Red Hat Enterprise Virtualization.

by fsimonce at November 20, 2014 02:00 PM

Opensource.com

Do I need OpenStack if I use Docker?

Docker has broken a record in the speed in which it moved from being a disruptive technology to a commodity. The speed of adoption and popularity of Docker brings with it lots of confusion.

In this post I wanted to focus on a trend of commentary that has been gaining popularity that I’ve started to hear more often recently from users who just started using Docker: whether it makes sense to use OpenStack if they’ve already chosen to use Docker.

by Nati Shalom at November 20, 2014 10:00 AM

November 19, 2014

Aptira

Product Management lies dreaming ...

This was originally posted to the product working group mailling list, which consists of people in influencing positions within their employer's OpenStack organisations. The group originally formed around the idea of Hidden Influencers, which made me think of us lying dormant somewhere, like Cthulhu lies dreaming at R'lyeh. Although with fewer tentacles. Anyway, on with the post, which I've modified slightly to add appropriate context:
 
There's a been a bit of traffic on the operators list about a proposal for an Operations project that would aim to curate a collection of assets to support the operator community. To generate a set of "Frequently Required Solutions" if you will.
 
Whilst there is some disagreement on the thread about how to proceed, it is clear there's definitely a need amongst operators for quite a bit of supporting code. As a group of product managers it behooves us to consider some of the larger questions like this and drive/enable/influence the community toward answering the questions that community considers a priority.
 
In just saying that, some more big questions are raised:
  • What are the goals of this group?
  • What strategy are we going to employ to achieve those goals?
  • How do we gain acceptance from the wider community for this effort?
  • How do we find an effective mechanism for product management to occur?
and no doubt there are more.
 
Answering these questions it's not as "easy" as starting a project, but it's necessary if this group wants to contribute in any collective or concerted fashion.
 
If we're going to have a mid-cycle meetup to bootstrap a product management group or forum, lets set some targets for what we want to achieve there, and lets set them well in advance. For me, the priority is establishing the basic existential motivations for this group:
  • Why are we here? (Purpose, Goals)
  • What are we going to do about it? (Strategy)
Anything more than that is a bonus, and if we can agree these prior to the meetup, so much the better. However without these basics, any work in the detail doesn't validate the existence of this group as anything more than just a mailing list.
 
For those who aren't on, or aren't aware of the product group mailling list: if you're in a position of influence or control of your employer's OpenStack direction think about joining up. We're going to be looking at some pretty fundamental questions about how OpenStack will move forward as it progresses further along the hype cycle.

by Roland Chan (roland@aptira.com) at November 19, 2014 10:41 PM

Marton Kiss

OpenStack Budapest User group - Summit overview

Yesterday we had the first Hungary user group meeting since Paris Summit, and of course we cannot miss the summit review from the Agenda. The event start was a bit exciting, because the projector’s cable failed to work and we realized this little technical failure in the last minute. The Bonnie Restro staff did a wonderful job, and they somehow managed to get a replacement device from the closest pub.

The first topic was an introduction of a very interesting open-source project, two guys Peter Voros and Daniel Csubak from Balabit made a Zorp GPL based load balancer service for Neutron as a part of last Goggle Summer of Code. Zorp GPL is an application level firewall, and this LbaaS implementation is a really entry-level introduction of technology feature-set. For this project the actual publication of source-code and packages was the “easy” introduction, the hard work starts now if they want to contribute back those results into upstream OpenStack code. Basically the source code consist around 500 lines of code, the most time was used for learning OpenStack, Neutron and DevStack. As they told during the project they run into several issues caused mostly by the difference of simple virtualization and cloud terminology. My personal opinion that we could bold those differences for newcomers, and somehow reflect this in documentation. (for example you cannot ssh into an instance if you don’t have proper security group configuration, and you don’t need to forget about assigning elastic ips for public access). I think this project is a nice example of a good start in OpenStack ecosystem, and I guess those guys have endless possibilities in the open-source world of cloud if they continue the hard work and continuous learning.

If you want to learn more about this project, I suggest to read this OpenSuse blog post: https://news.opensuse.org/2014/09/01/things-i-learnt-with-the-zorp-and-opensuse-team/comment-page-1/#comment-105958

The second session was a Summit overview. Gergely Szalay from Fremont, Gabriel Borkowski from Avaya and Ildiko Tuzko-Keger from HP came and share our story with the user group. I can definitely say Paris Summit was one of my favourite. I’ve a large amount of data to make this comparsion, because it was my 7th Summit, and not missed a one since Boston. I’m very happy that Hungarians joined in a relatively large number and can participate in this large-scale event, from Avaya, IBM Zürich Research lab, Ericsson, Percona, HP and Fremont. Most of them not simply visit the conference, but was an active participant in the Design Summit as ATCs. Paris was the first Summit in Europe, our home-continent, so usually it means 2 hrs of flight from Budapest, but we made a policy with Gergely and Gabriel that a trip to an OpenStack Summit cannot take less time than 10hours, so instead of flying we jump in a car in Saturday morning, and crossed half of the Europe through Austria and Germany and drove directly to Paris. Budapest-Paris distance is 1500km, and we accomplished this task during 15hrs including the mandatory stops for car-refueling and eating. (And I want to notice here, don’t try to drive in Paris if you not born there!)

With Saturday evening arrival we missed the first part of upstream training, but joined in on Sunday. The goal of this training was to teach newcomers of OpenStack contribution process and give a little feeling about day-to-day work we are doing on OpenStack projects. Our favorite part of the training was the Lego-town building collaboration. We formed teams and started to work on specified tasks, based on several sprints. It was amazing to see how an early chaos turned into a real collaboration of people and teams, and I want to remark here that most of them never meet each-other personally. As a real simulation of OpenStack project we needed to take care that the buildings (Projects) we are working on must fit together (API definition) and it is not enough if people agree inside the teams, we cannot miss a team-level collaboration to build a whole town from little fragments of different projects.

The keynotes were fantastic as usual, but Hungary was a little-bit over-represented due some political intention trying to introduce a traffic based additional tax on internet, not forgetting that we are already paying a simple VAT tax on our internet subscriptions. I hope next time our country will appear in a different spotlight with some nicely finished OpenStack projects maybe in the gov. sector. The big deal here was the BMW story. To tell the truth selling OpenStack in the EU with names like Wells Fargo, or Comcast not easy, because those brands are not so well known here. Linking OpenStack to a premium-category car vendor is a real win-win situation. Most of business leaders and decision makers know the brand, even using those cars, and BMW’s decision broadcasts a strong signal to existing OpenStack users and prepares the involvement of new users of technology.

Ildiko was a first-timer in Paris and was working hard at HP booth that was constantly filled up with people and was easier to bypass the crowd than crossing through. She told us that it was a huge surprise for her to see the diversity of the entire community, from a geeky faces to well-suited vendor vp’s, and their openness to talk with each other. I think this is a very basic core value of the OpenStack ecosystem and presents a never-seen collaboration of the IT industry.

Additionally to daily presentations and sessions, the most remarkable part of the Summit was the social activity in form of parties organized by different vendors, some of them was publicly announced, and a few was a closed secret one. All of us agree on that the biggest boom was the Machine-Machine party by HP, they rented Les Pavillons de Bercy and it is hard to describe the whole event if you were not there. There was some open-air grill, raw-meat (tartare), good quality of wines, French cheese, entertainment and music. And of course a lot of interesting people from OpenStack community we can talk personally.

I’m sure we could share some passion about the Summit, and it helps to trigger of involvement of user group members. Of course I not forget to mention that contribution and community work can help to reach the Summits through the Travel Summit program. For Paris Summit with the support of the Foundation helped 20 people to visit the event from every part of the world.

November 19, 2014 03:22 PM

Red Hat Stack

Empowering OpenStack Cloud Storage: OpenStack Juno Release Storage Overview

Wind Energy

 License: CC0 Public Domain

The OpenStack 10th release added ten new storage backends and improved testing on third-party storage systems. The Cinder block storage project continues to mature each cycle exposing more and more Enterprise cloud storage infrastructure functionalities.

Here is a quick overview of some of these key features.

Simplifying OpenStack Disaster Recovery with Volume Replication

After introducing a new Cinder Backup API to allow export and import backup service metadata in the Icehouse release, which allowed “electronic tape shipping” style backup-export & backup-import capabilities to recover OpenStack cloud deployments, the next step for Disaster Recovery enablement in OpenStack is the foundation of volume replication support at block level.

Starting with the OpenStack Juno release, Cinder has now initial support for Volume Replication which makes Cinder aware of replicas, and allows the cloud admin to define storage policies to enable replication.
With this new feature Cinder Storage backend drivers can expose different replication capabilities via volume type convention policies that enable various replication operations, such as failover, failback as well as reverse direction capabilities.
Using the new API, a volume is created with replication extra-spec that will be allocated on supported backends for replication.

Data Protection Enablement

Consistency Groups support was added to group volumes together for the purpose of application data protection (with the focus of snapshots of consistency groups for disaster recovery), where the grouping of volumes is based on the volume-type, however there is still future work to support this functionality together with Cinder backups & volume replication.

Another important aspect is also to maintain consistency at the application & filesystem level. Similar to the AWS ec2-consistent-snapshot feature that provides consistent data in the snapshot, by performing flush/freeze to the filesystem, as well as flushing and locking the database, if applicable.  It is possible to achieve similar functionality in OpenStack with QEMU guest agent during image snapshotting for KVM instances, where nova-compute libvirt driver could request QEMU Guest Agent to freeze the filesystems (and applications if fsfreeze-hook is installed) during the image snapshot.  The QEMU guest agent support is currently planned for the next release to help automate daily/weekly backup of instances with consistency.

Storage Management enhancements at the Dashboard level

The following Cinder API features were also added to the Horizon dashboard in the Juno cycle:

  • Utilize Swift to store volume backups as well as restore volumes from these backups.
  • Enabling resetting the state of a snapshot.
  • Enabling resetting the state of a volume.
  • Supporting upload-to-image.
  • Volume retype.
  • QoS (quality of service) support.

Support for Volume Pools via a new Pool-aware Scheduler

Until the Juno release, Cinder used to see each volume backend as a whole, even if the backend consisted of several smaller pools with totally different capabilities and capacities. This gap could have caused issues where a backend may appear to have enough capacity to create a copy of a volume but in fact failed to do so. Extending Cinder to support storage pools within volume backend has also improved Cinder scheduler decision making, that is now aware of storage pools within backend and also use them as finest granularity for resource placement.

Another Cinder Scheduling aspect was addressed with a new Volume Num Weighter feature that enables the user to choose a volume backend according to free_capacity and allocated_capacity. The volume number weighter feature lets the scheduler choose a volume backend based on its volume number in the volume backend, to improve volumes IO balancing performance.

Glance Generic Catalog

The Glance Image Service introduced artifacts as a broader definition for images during Juno. The scope is expanding the image repository to a generic catalog of various data assets. It is possible now to manage a catalog of metadata definitions where users can register the metadata definitions to be used on various resource types including images, aggregates, and flavors. Support for viewing and editing the assignment of these metadata tags is included in Horizon.  Other key new features included asynchronous processing and Image download improvements such as:

  • Restart of partial download  (solved a problem associated with downloads of  very large images that may be interrupted prior to completion, due to dropped connections)
  • A new download image restriction policy, to restrict users from downloading an image based on policy.

Introducing Swift Storage Policies

This large feature has been finally released in the Juno cycle of the OpenStack Object Storage project, allowing users more control over cost and performance in terms of how they want to replicate and access data across different backends and geographical regions.

Storage Policies allow for some level of segmenting the cluster for various purposes through the creation of a multiple object ring. Once configured, users can create a container with a particular policy.

Storage Policies can be set for:

  • Different Storage implementations:
  • Different Diskfile (e.g. GlusterFS, Kinetic) for a group of nodes
  • Different levels of replication
  • Different Performance profiles  (e.g. SSD-only)

Other new Swift new notable feature include the Multi-ring awareness with support for:

  • Object replication – added to be aware of the different locations on-disk that storage policies introduced
  • Large objects – refactoring work for storage policies
  • Object auditing – added to be aware of the different on-disk locations for objects in different storage policies
  • Improved partition placement – allows for better capacity adjustments, especially when adding a new region to existing clusters.

The swift-ring-builder has been updated as well to first prefer device weight, then use failure domains to break ties.

The progress on multi-ring and storage policies is the foundation to the Swift Erasure coding development that is on-going in the Kilo release cycle. Erasure coding is a storage policy with its own ring and configurable set of parameters designed to reduce  storage costs associated with massive amounts of data (both operating costs and capital costs) by providing an option that maintains the same, or better, level of durability using much less disk space, especially for “Warm” storage use cases, such as when performing  volume backup to a Swift object storage system, as backups are typically large compressed objects and are infrequently read once they have been written to the storage system.

To learn more about the new OpenStack storage features see OpenStack 2014.2 (Juno) Release notes.

by Sean Cohen at November 19, 2014 02:00 PM

November 18, 2014

The Official Rackspace Blog » OpenStack

Forrester Names Rackspace A ‘Strong Performer’ In Hosted Private Cloud; And We’ve Made It Even Stronger

Forrester Research, Inc. has named Rackspace Private Cloud (RPC) a “strong performer” in The Forrester Wave™: Hosted Private Cloud Solutions, Q4 2014. We believe that this rating is great recognition of the momentum we’re seeing with our OpenStack-powered private cloud solution.

The report states that Rackspace has “successfully been able to deliver an intuitive portal, diverse control options, granular user permissions, numerous certifications, and large global DC footprint.” And because Fanatical Support is core to our business, we are especially pleased Forrester noted that RPC offers “strong support capabilities.”

What we find even more exciting, however, are some areas of improvement that RPC can capitalize on to offer a stronger private cloud solution: reporting and metering, automation, multi-data-center portability and strength of our SLAs. Why are we so excited about those? Because they are some of the exact enhancements included in the latest release of Rackspace Private Cloud.

Forrester’s study was conducted in July 2014 and the report is based on the previous version of Rackspace Private Cloud, which was powered by OpenStack Havana. In the time between when the study was conducted and when it was published, we released an enhanced version of Rackspace Private Cloud. This new version of RPC is built on OpenStack Icehouse and includes features that address many of the opportunities Forrester highlights as areas of improvement. Specifically, with the current version of RPC we address the SLA opportunity by delivering an industry-leading 99.99 percent OpenStack API uptime guarantee. We also address the automation opportunity by enabling DevOps Automation Services with RPC, supporting the OpenStack Orchestration (Heat) project and launching several RPC solution templates that empower customers to deploy production-ready application stacks in just minutes.

We believe that the opportunities laid out by Forrester help validate the recent enhancements we’ve made to Rackspace Private Cloud. We’ll continue to enhance our solution based on our customers’ feedback and provide you with all the power of the cloud without the pain of running it, so you can focus on your core business.

by Christian Foster at November 18, 2014 03:00 PM

OpenStack @ NetApp

NetApp in Paris - OpenStack Summit Recap

NetApp in Paris: OpenStack Summit Recap

About two weeks ago, NetApp sent a large contingent of folks to the biannual OpenStack Summit in Paris – where developers, operators, and users converge - to talk about their experiences around OpenStack, learn more about what’s new in the ecosystem, and help design the next release of OpenStack!

Probably most notable was the amount of excitement around Manila. (In case you're not familiar, Manila allows users of OpenStack clouds to provision and securely manage shared file systems through a simple REST API.)

There was a standing-room-only general session on Manila, which gave an overview of Manila, its API structure and key concepts, an architectural overview of the service, and then updates on what was new in the Juno release and where our focus will be during the Kilo cycle. Here’s a link to the recording in case you weren’t able to join us - https://www.youtube.com/watch?v=PLYr5Xo3LJk

Manila Overview Diagram

Immediately following the session on Manila, NetApp hosted a "Use Case Showcase" session where we had brief presentations from customers & partners in various stages of deployment: a customer in production, one currently deploying OpenStack on NetApp, and a prototype integration scenario of Manila with SAP on SUSE Cloud. There were some great takeaways from this session - so check out the YouTube recording at https://www.youtube.com/watch?v=6XW5U2XFwEg".

As always, if you want to learn more about Manila, get started by checking our wiki page @ http://wiki.openstack.org/wiki/Manila or jump on IRC - we’re always hanging out in #openstack-manila and #openstack-netapp on Freenode! We’ve got weekly meetings at 1500 UTC in #openstack-meeting-alt on Freenode as well.

November 18, 2014 02:42 PM

Red Hat Stack

Simplifying and Accelerating the Deployment of OpenStack Network Infrastructure

plumgrid logo

RHOSCIPN_logo_small

The energy from the latest OpenStack Summit in Paris is still in the air. Its record attendance and vibrant interactions are a testimony of the maturity and adoption of OpenStack across continents, verticals and use cases.

It’s especially exciting to see its applications growing outside of core datacenter use cases with Network Function Virtualization being top of mind for many customers present at the Summit.

If we look back at the last few years, a fundamental role fueling OpenStack adoption has been played by the distributions which have taken the project OpenStack and helped turn it into an easy to consume, supported, enterprise-grade product.

At PLUMgrid we have witnessed this transformation summit after summit, customer deployment after customer deployment. Working closely with our customers and our OpenStack partners we can attest how much easier, smoother, simpler an OpenStack deployment is today.

Similarly, PLUMgrid wants to simplify and accelerate the deployment of OpenStack network infrastructure, especially for those customers that are going into production today and building large-scale environments.

If you had the pleasure to be at the summit you have learnt about all the new features that were introduced in Juno for the OpenStack networking component (and if not check out this blog which provides a good summary of all Juno’s networking feature).

Despite the tremendous progress, standing up the network infrastructure for an OpenStack environment can still be a daunting process made of opening tickets, creating VLANs and opening firewall ports, tcpdump and lengthy troubleshooting that can slow down the deployment of many weeks.
Also, once the OpenStack cloud is in place, some network component might still fail to be as on-demand, rich, programmable and flexible as the other components of the stack are.

Last, especially for NFV type of clouds, there is a strong need to leverage and combine Network Functions from a broad portfolio of partners – a task that can’t always easily be achieved today out of the box.

PLUMgrid’s comprehensive software suite, PLUMgrid ONS 2.0 for OpenStack, enables scalable and secure virtual network infrastructure for OpenStack clouds.
As a software only solution fully integrated within Red Hat’s Foreman installer, PLUMgrid enables the creation of all the necessary infrastructure in a matter of hours completely independent of the underlying physical infrastructure.

Once the OpenStack deployment is in place, PLUMgrid’s comprehensive set of L2-4 distributed Virtual Network Functions as well as its broad ecosystem powered by its Service Insertion Architecture can be used to satisfy a wide array of use cases.

Its integration and certification with Red Hat OpenStack Platform 5.0 enable PLUMgrid to effectively speed up OpenStack deployments and enable their joint customers to create scalable, secure multi-tenant solutions.

To learn more about how PLUMgrid deployed with Red Hat OpenStack Platform 5.0 can speed up NFV OpenStack deployments join us for a live webinar on November 19th at 8AM PST.

by Valentina at November 18, 2014 02:00 PM

Rafael Knuth

Recording from Online Meetup: OpenStack Juno the Complete Lowdown & Tales from the Summit

Everything new in the Juno Release, Syed Armani - Hastexo!Tales from the Summit w/ Nati Shalom,...

November 18, 2014 02:00 PM

Mirantis

It’s not a summit without a party: OpenStack Underground

One of the things we enjoy the most about the semi-annual OpenStack summit is the chance for people throughout the ecosystem to get together and really kick back together like the extended family that we are.  For those of you who couldn’t make it to OpenStack Underground (or were there but would like a refresher) here are some of the highlights.  You can see the entire photostream here.


mirantis-004
mirantis-007mirantis-115mirantis-011mirantis-013mirantis-027mirantis-039mirantis-047mirantis-078mirantis-087mirantis-098mirantis-128mirantis-134mirantis-137mirantis-151mirantis-157mirantis-162mirantis-173mirantis-175mirantis-170mirantis-174mirantis-196mirantis-201mirantis-236mirantis-220mirantis-234mirantis-205mirantis-249mirantis-250mirantis-254 mirantis-247mirantis-243

The post It’s not a summit without a party: OpenStack Underground appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Admin at November 18, 2014 03:53 AM

November 17, 2014

The Official Rackspace Blog » OpenStack

OpenStack Summit Paris: Federated Clouds Becoming A Reality [Video]

At OpenStack Summit Paris, cloud federation – the ability to easily and seamlessly leverage a multi-cloud environment – was a buzz worthy topic. The story of cloud federation was told though a number of sessions: from Tim Bell from CERN’s keynote presentation to several smaller breakouts. Rackspace and CERN openlab are working together to federate OpenStack clouds. The project has already had a number of successes.

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="360" src="http://www.youtube.com/embed/ijYeHxI4DIw" width="640"></iframe>

In the video above, Bell and Rackspace DevOps Practice CTO Chris Jackson talk about the work Rackspace and CERN openlab are doing and the importance of federated clouds.

For a deeper drilldown into cloud federation, tune in to this recorded Google Hangout featuring Bell and Jackson.

Check out more coverage from OpenStack Summit Paris.

 

by Andrew Hickey at November 17, 2014 08:00 PM

Rafael Knuth

Online Meetup: OpenStack Trove Update - Juno, Kilo and Beyond

Join us as Amrith Kumar from Tesora shares what is new in Trove in Juno release, community plans for...

November 17, 2014 05:51 PM

OpenStack Blog

Wrapping up the Travel Support Program – Kilo

The OpenStack Foundation brought 20 people to Paris for the Summit earlier this month, thanks to the grants offered by the Travel Support Program. The Travel Support Program is based on the promise of Open Design and its aim is to facilitate participation of key contributors to the OpenStack Design Summit. The program aims at covering costs for travel and accommodation for key contributors to the OpenStack project to join the community at the Summits.

Travel Support Program

We had 22 people accepted in the program from 11 different countries, spanning five continents. Four people traveled from Brazil, four
 from India, three from Europe and the rest were from South America, North America and South-east Asia. Of the selected recipients, two were unable to attend due to VISA timing issues, but we were excited to welcome the 20 attendees that were able to make the trip. 

The Foundation spent $28,400 on flights and $24,000 on hotels for a total cost for the Foundation of more than $54,000 USD, including the cost of four full access passes granted to non-ATCs. 

Stay tuned for when we announce the applications for the Travel Support Applications for the May Summit in Vancouver. The Travel Support Program will also be a sponsorship opportunity for the upcoming Summit in Vancouver. Details will be shared in the sponsorship prospectus that will be published soon.  

by Allison Price at November 17, 2014 03:37 PM

Opensource.com

OpenStack's developers make plans for the next release

Interested in keeping track of what's happening in the open source cloud? Opensource.com is your source for what's happening right now in OpenStack, the open source cloud infrastructure project.

by Jason Baker at November 17, 2014 08:00 AM

Adam Young

Dynamic Policy in Keystone

Ever get that feeling that an epiphany is right around the corner? I spent a good portion of the OpenStack summit with that feeling. I knew that it would not be earth shattering, or lead me to want to rewrite Keystone, but rather a clarification of how a bunch of things should fall together. The “click” happened on the second to last day, and it can be summarized in a few key points.

When discussing the OAUTH1.0 extension to Keystone, several people commented on how it was similar to trusts, and that we should have a unified mechanism between them for delegation. During a discussion with David Chadwick, he mentioned that the role assignments themselves were a form of delegation, and lamented that we were losing the chain of delagtion by how we delegate roles. So the first point was this:

Keystone should have a single, unified mechanism for delegation.

One key feature that feeds into that is the ability to break a big role into a small one. I had posted a spec for hierarchical roles prior to the summit, but wasn’t clear for how to implement it; I could see how it coule be implemented on the token side, but all people I talked to insisted it made more sense on the enforcement side. That is the second big point.

Role inheritance should be expanded by policy enforcement.

Policy is almost all static. Each OpenStack project had it’s own policy file in its own it repo. Extending it to cover is user requests for things like project specific policy or more granular roles has not been possible.

UPDATE: I’ve been asked to make clearer what problems this addresses.

  1. Determine what roles a user can assign to another user
  2. Allow a user to determine what roles they need to perform some action
  3. Allow some user interface to determine what a user is capable of doing based on their roles
  4. Establish an iterative process solve the long-standing bug that a user with admin on any scope has admin on all scoped.
  5. Allow a user to delegate a subset of their capabilites to a remote service.

What we have now is a simple set of specs that build on each other that will, in the end, provide a much more powerful, flexible, and consistant delegation mechanism for Keystone. Here are the General steps:

  1. Graduate oslo policy to a library
  2. Add to the policy library the essential code to enforce policy based on a keystone token.  I’ve looked at both the Keystone and Nova pieces that do this, and they are similar enough that we should not have too much problem making this happen.
  3. Add in the ability to fetch the policy.json file from Keystone.
  4. Add a rule to the Keystone policy API to return the default policy file if no policy file is specified for an endpoint.
  5. Merge the current default policy files from all of the projects into a single policy file, with namespaces that keep the rules from conflicting across services.  Reduce the duplication of rules like “admin_or_owner”  so that we have a consistent catalog of capabilities across OpenStack.  Make this merged file the default that is served out of Keystone when an endpoint asks for a policy file and Keystone does not have an endpoint specific file to give it.
  6. Make a database schema to hold the rules from the policy file.  Use this to generate the policy files served by Keystone.  There should be no functional difference between the global file and the one produced in the above merge.
  7. Use the hierarchical role definitions to generate the rules for the file above.  For example, rules that essentially say “grant access to a user with any role on this project”  will now say  “grant access to any user with the member role, or with any role that inherits the member role.  The member role will be the lowest form of access.  Admin will inherit member, as will all other defined roles.
  8. Break member up into smaller roles.  For example,  we could distinguish between actions that can only read state from those that can change it:  “Observer”  and “Editor”  Member would inherit editor, and editor would inherit observer.
  9. Change the rules for specific API policy enforcement points to know about the new roles.  For example, the API to create a new image in glance might now require the editor role instead of the member role.  But, since member inherits editor, all current users will be able to perform the same set of operations.
  10. Change the role assignment mechanism so that a user can only assign a role that they themselves have on the designated scope.  In order to assign Member, the user must have the member role, or a role that inherits Member,such as admin.  Role assignment, trusts, oauth, and any other mechanism out there will follow this limitation.  We will have to perform additional limitations, such as determining what happens to a delegated role when the person that does the delegation has that role removed;  perhaps one will need a specific role in order to perform “sticky” role assignments that last past your employment, or perhaps we will allow a user to pass some/all their delegations on to another user.

 

This is still in the planning stage.  One nice thing about a plan like this is that each stage shows value on its own, so that if we only get as far as, say stage 3, we still have a better system than we do today.  Many of the details are still hiding in the weeds, and will require more design.  But I think the above approach makes sense, and will make Keystone do what a lot of people need it to do.

by Adam Young at November 17, 2014 03:45 AM

Minimal Token Size

OpenStack Keystone tokens can become too big to fit in the headers between mod_wsgi and the WSGI applications. Compression mitigates the problem somewhat, but if token sizes continue to grow, eventually they outpace the benefits of compression. How can we keep them to a minimal size?

There are two variables to the size of the tokens: the packaging, and the data inside. The packaging for a PKIZ token has a lower bound based on the the signing algorithm. An empty CMS document of compressed data is going to be no less than 650 bytes. An unscoped token with proper compression comes in at 930 bytes. This are for headers, but it means that we have to keep additional data inside the token body as small as possible.

Encoding

Lets shift gears back to the encoding. A recent proposal suggested using symmetric encryption instead of asymmetric. The idea is that a subset of data would be encrypted by Keystone, and the data would have to be sent back to Keystone to validate. What would this save us?

Lets assume for a moment that we don’t want to pay any of the overhead of the CMS message format. Instead, keystone will encrypt just the JSON and base64 the data. How much does that save us? Depends on the encryption algorithm. An empty token will be tiny: 33 bytes when encrypted like this:

openssl bf -salt -a -in cms/empty.json -out cms/empty.bf

Which, according to the openssl man page, is blowfish encrypted and base64 encoded. What about a non-trivial token? Turns out, our unscoped token is quite a bit bigger: 780 bytes for the comparable call:

openssl bf -d -k key.data -in cms/auth_token_unscoped.json -out cms/auth_token_unscoped.bf

Compared with the PKIZ format at 929 bytes, the benefit does not seem all that great.

What about for a scoped token with role data embedded in it, but no service catalog? It turns out the compression actually makes the PKIZ format more effecient: PKIZ is 917 bytes versus 1008 for the bf.

Content

What data is in the token?

Identification. This is what you would see in an unsigned token: user id and name, domain id and possibly name.

Scope: domain and project info Roles: specific to the scope. service catalog. The sets of services and endpoints that implement those services.

It is the service catalog that is so problematic. While we have stated that you can make tokens without a service catalog, doing so is rally not going to allow the endpoints to make any sort of decisions about where to get resources.

There is a lot of redundant data in the catalog. We’ve discussed doing ID only service catalogs. That implies that each endpoint is expandable on the endpoint size: the endpoints need to be able to fetch the service catalog and then look up the endpoints by ID.

But let us think in terms of scale. If there is a service catalog with, say, 512 endpoints, we are still going to be sending tokens that are 512 * length(endpoint_id)

Can we do better? According to Jon Bently in Programming Pearls, yes we can. We can use a bitmap. No, not the image format. Here a bitmap is an array of bits, each of which, when set, indicates inclusion of the member in the set.

We need two things. One, a cached version of the service catalog on the endpoints. But now we need to put a slightly stricter constraint on it: the token must match up exactly to a version of the service catalog, and the service catalog must contain that version number. I’d take the git approach, do a sha256 hash of the service catalog document, and include that version in the token.

Second, we need to enforce ordering on the service catalog. Each endpoint must be in a repeatable location in the list. I need to be able to refer to the endpoints, not by ID, but by sequence number.

Now, what the token would contain? Two things:

The hash of the service catalog. A bitmap of the included services.

Here’s a minimal service catalog

Index | Service name | endpoint ID
 0 | Nova | N1
 1 | Glance | G1
 2 | Neutron | T1
 3 | Cinder | C1

A service catlog that had all of the endpoints would be (b for binary) b1111 or, in Hex, 0xF

A service catalog with only Nova would be b0001 or 0×1.

Just cinder would be b1000 or 0×8

A service catalog with 512 endpoints would be 512 bits in length. That would be 64 characters long, the length of a string comparable to a sha256. A comparable list of uuids would take 16384 characters, not including the JSON overhead of commas and quotes.

I’ve done a couple tests with token data in both the minimized and the endpoing_id only formats. With 30 endpoint ids, the compressed token size is 1969 bytes. Adding one ID to that increases the size to 1989. The minimized format is 1117 when built with the following data:

"minimizedServiceCatalog": { 
    "catalog_sha256": "7c7b67a0b88c271384c94ed7d93423b79584da24a712c2ece0f57c9dd2060924",
    "entrymap": "Ox2a9d590bdb724e6d888db96f846c9fd8" },

The ID only one would scale up at rougly 20 bytes per entry point, the minimized one would stay fairly fixed in length.

Are there other options? If a token without a catalog assumed that all endpoints were valid, and auth_token middleware set the environment for the request appropriately, then there is no reason to even send a catalog on over.

Project filtering of endpoints could allow for definitions of the service catalog that is a subset of the overall catalog. These subordinate service catalogs could have their own ids, and be sent over in the token. This would minimize the size of data in the token at the expense of the server; a huge number of projects, each with their own service catalog would lead to a large synchronization effort between the endpoints and the keystone server.

If a token is only allowed to work with a limited subset of the endpoints assigned to the project, then maintaining strictly small service catalogs in their current format would be acceptable. However, this would require a significant number of changes on how users and service request tokens from Keystone.

by Adam Young at November 17, 2014 02:42 AM

November 16, 2014

Shaifali Agrawal

Switch on to OpenStack :)

For the coming few months I will be working as an OPW intern for OpenStack’s Zaqar component under Flavio Percoco, (flaper87 on IRC). My task is to split control and data planes of Zaqar storage layer completely, brief explanation here. Exciting..Thrilling..  Exhilarating!. :D Starting Results are out and internship period is about to start(from 9 Dec), […]

by exploreshaifali at November 16, 2014 08:39 AM

November 15, 2014

Opensource.com

Top 5 of the week: Free book on GitHub and open cloudy weather

Every week, I tally the numbers and listen to the buzz to bring you the best of last week's open source news and stories on Opensource.com, this week: November 10 - 14, 2014.

by Jen Wike Huger at November 15, 2014 02:00 PM

Lars Kellogg-Stedman

Creating a Windows image for OpenStack

If you want to build a Windows image for use in your OpenStack environment, you can follow the example in the official documentation, or you can grab a Windows 2012r2 evaluation pre-built image from the nice folks at CloudBase.

The CloudBase-provided image is built using a set of scripts and configuration files that CloudBase has made available on GitHub.

The CloudBase repository is an excellent source of information, but I wanted to understand the process myself. This post describes the process I went through to establish an automated process for generating a Windows image suitable for use with OpenStack.

Unattended windows installs

The Windows installer supports fully automated installations through the use of an answer file, or "unattend" file, that provides information to the installer that would otherwise be provided manually. The installer will look in a number of places to find this file. For our purposes, the important fact is that the installer will look for a file named autounattend.xml in the root of all available read/write or read-only media. We'll take advantage of this by creating a file config/autounattend.xml, and then generating an ISO image like this:

mkisofs -J -r -o config.iso config

And we'll attach this ISO to a vm later on in order to provide the answer file to the installer.

So, what goes into this answer file?

The answer file is an XML document enclosed in an <unattend>..</unattend> element. In order to provide all the expected XML namespaces that may be used in the document, you would typically start with something like this:

<?xml version="1.0" ?>
<unattend
  xmlns="urn:schemas-microsoft-com:unattend"
  xmlns:ms="urn:schemas-microsoft-com:asm.v3"
  xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State">

  <!-- your content goes here -->

</unattend>

Inside this <unattend> element you will put one or more <settings> elements, corresponding to the different configuration passes of the installer:

<settings pass="specialize">
</settings>

The available configuration passes are:

Of these, the most interesting for our use will be:

  • windowsPE -- used to install device drivers for use within the installer environment. We will use this to install the VirtIO drivers necessary to make VirtIO devices visible to the Windows installer.

  • specialize -- In this pass, the installer applies machine-specific configuration. This is typically used to configure networking, locale settings, and most other things.

  • oobeSystem -- In this pass, the installer configures things that happen at first boot. We use this to step to install some additional software and run sysprep in order to prepare the image for use in OpenStack.

Inside each <settings> element we will place one or more <component> elements that will apply specific pieces of configuration. For example, the following <component> configures language and keyboard settings in the installer:

<settings pass="windowsPE">
  <component name="Microsoft-Windows-International-Core-WinPE"
    processorArchitecture="amd64"
    publicKeyToken="31bf3856ad364e35"
    language="neutral"
    versionScope="nonSxS">

    <SetupUILanguage>
      <UILanguage>en-US</UILanguage>
    </SetupUILanguage>
    <InputLocale>en-US</InputLocale>
    <UILanguage>en-US</UILanguage>
    <SystemLocale>en-US</SystemLocale>
    <UserLocale>en-US</UserLocale>
  </component>
</settings>

Technet provides documentation on the available components.

Cloud-init for Windows

Cloud-init is a tool that will configure a virtual instance when it first boots, using metadata provided by the cloud service provider. For example, when booting a Linux instance under OpenStack, cloud-init will contact the OpenStack metadata service at http://169.254.169.254/ in order to retrieve things like the system hostname, SSH keys, and so forth.

While cloud-init has support for Linux and BSD, it does not support Windows. The folks at Cloudbase have produced cloudbase-init in order to fill this gap. Once installed, the cloudbase-init tool will, upon first booting a system:

  • Configure the network using information provided in the cloud metadata
  • Set the system hostname
  • Create an initial user account (by default "Admin") with a randomly generated password (see below for details)
  • Install your public key, if provided
  • Execute a script provided via cloud user-data

Passwords and ssh keys

While cloudbase-init will install your SSH public key (by default into /Users/admin/.ssh/authorized_keys), Windows does not ship with an SSH server and cloudbase-init does not install one. So what is it doing with the public key?

While you could arrange to install an ssh server that would make use of the key, cloudbase-init uses it for a completely unrelated purpose: encrypting the randomly generated password. This encrypted password is then passed back to OpenStack, where you can retrieve it using the nova get-password command, and decrypt it using the corresponding SSH private key.

Running nova get-password myinstance will return something like:

w+In/P6+FeE8nv45oCjc5/Bohq4adqzoycwb9hOy9dlmuYbz0hiV923WW0fL
7hvQcZnWqGY7xLNnbJAeRFiSwv/MWvF3Sq8T0/IWhi6wBhAiVOxM95yjwIit
/L1Fm0TBARjoBuo+xq44YHpep1qzh4frsOo7TxvMHCOtibKTaLyCsioHjRaQ
dHk+uVFM1E0VIXyiqCdj421JoJzg32DqqeQTJJMqT9JiOL3FT26Y4XkVyJvI
vtUCQteIbd4jFtv3wEErJZKHgxHTLEYK+h67nTA4rXpvYVyKw9F8Qwj7JBTj
UJqp1syEqTR5/DUHYS+NoSdONUa+K7hhtSSs0bS1ghQuAdx2ifIA7XQ5eMRS
sXC4JH3d+wwtq4OmYYSOQkjmpKD8s5d4TgtG2dK8/l9B/1HTXa6qqcOw9va7
oUGGws3XuFEVq9DYmQ5NF54N7FU7NVl9UuRW3WTf4Q3q8VwJ4tDrmFSct6oG
2liJ8s7ybbW5PQU/lJe0gGBGGFzo8c+Rur17nsZ01+309JPEUKqUQT/uEg55
ziOo8uAwPvInvPkbxjH5doH79t47Erb3cK44kuqZy7J0RdDPtPr2Jel4NaSt
oCs+P26QF2NVOugsY9O/ugYfZWoEMUZuiwNWCWBqrIohB8JHcItIBQKBdCeY
7ORjotJU+4qAhADgfbkTqwo=

Providing your secret key as an additional parameter will decrypt the password:

$ nova get-password myinstance ~/.ssh/id_rsa
fjgJmUB7fXF6wo

With an appropriately configured image, you could connect using an RDP client and log in as the "Admin" user using that password.

Passwords without ssh keys

If you do not provide your instance with an SSH key you will not be able to retrieve the randomly generated password. However, if you can get console access to your instance (e.g., via the Horizon dashboard), you can log in as the "Administrator" user, at which point you will be prompted to set an initial password for that account.

Logging

You can find logs for cloudbase-init in c:\program files (x86)\cloudbase solutions\cloudbase-init\log\cloudbase-init.log.

If appropriately configured, cloudbase-init will also log to the virtual serial port. This log is available in OpenStack by running nova console-log <instance>. For example:

$ nova console-log my-windows-server
2014-11-19 04:10:45.887 1272 INFO cloudbaseinit.init [-] Metadata service loaded: 'HttpService'
2014-11-19 04:10:46.339 1272 INFO cloudbaseinit.init [-] Executing plugin 'MTUPlugin'
2014-11-19 04:10:46.371 1272 INFO cloudbaseinit.init [-] Executing plugin 'NTPClientPlugin'
2014-11-19 04:10:46.387 1272 INFO cloudbaseinit.init [-] Executing plugin 'SetHostNamePlugin'
.
.
.

Putting it all together

I have an install script that drives the process, but it's ultimately just a wrapper for virt-install and results in the following invocation:

exec virt-install -n ws2012 -r 2048 \
  -w network=default,model=virtio \
  --disk path=$TARGET_IMAGE,bus=virtio \
  --cdrom $WINDOWS_IMAGE \
  --disk path=$VIRTIO_IMAGE,device=cdrom \
  --disk path=$CONFIG_IMAGE,device=cdrom \
  --os-type windows \
  --os-variant win2k8 \
  --vnc \
  --console pty

Where TARGET_IMAGE is the name of a pre-existing qcow2 image onto which we will install Windows, WINDOWS_IMAGE is the path to an ISO containing Windows Server 2012r2, VIRTIO_IMAGE is the path to an ISO containing VirtIO drivers for Windows (available from the Fedora project), and CONFIG_IMAGE is a path to the ISO containing our autounattend.xml file.

The fully commented autounattend.xml file, along with the script mentioned above, are available in my windows-openstack-image repository on GitHub.

The answer file in detail

windowsPE

In the windowsPE phase, we start by configuring the installer locale settings:

<component name="Microsoft-Windows-International-Core-WinPE"
  processorArchitecture="amd64"
  publicKeyToken="31bf3856ad364e35"
  language="neutral"
  versionScope="nonSxS">

  <SetupUILanguage>
    <UILanguage>en-US</UILanguage>
  </SetupUILanguage>
  <InputLocale>en-US</InputLocale>
  <UILanguage>en-US</UILanguage>
  <SystemLocale>en-US</SystemLocale>
  <UserLocale>en-US</UserLocale>

</component>

And installing the VirtIO drviers using the Microsoft-Windows-PnpCustomizationsWinPE component:

<component name="Microsoft-Windows-PnpCustomizationsWinPE"
  publicKeyToken="31bf3856ad364e35" language="neutral"
  versionScope="nonSxS" processorArchitecture="amd64">

  <DriverPaths>
    <PathAndCredentials wcm:action="add" wcm:keyValue="1">
      <Path>d:\win8\amd64</Path>
    </PathAndCredentials>
  </DriverPaths>

</component>

This assumes that the VirtIO image is mounted as drive d:.

With the drivers installed, we can then call the Microsoft-Windows-Setup component to configure the disks and install Windows. We start by configuring the product key:

<component name="Microsoft-Windows-Setup"
  publicKeyToken="31bf3856ad364e35"
  language="neutral"
  versionScope="nonSxS"
  processorArchitecture="amd64">

  <UserData>
    <AcceptEula>true</AcceptEula>
    <ProductKey>
      <WillShowUI>OnError</WillShowUI>
      <Key>INSERT-PRODUCT-KEY-HERE</Key>
    </ProductKey>
  </UserData>

And then configure the disk with a single partition (that will grow to fill all the available space) which we then format with NTFS:

  <DiskConfiguration>
    <WillShowUI>OnError</WillShowUI>
    <Disk wcm:action="add">
      <DiskID>0</DiskID>
      <WillWipeDisk>true</WillWipeDisk>

      <CreatePartitions>
        <CreatePartition wcm:action="add">
          <Order>1</Order>
          <Extend>true</Extend>
          <Type>Primary</Type>
        </CreatePartition>
      </CreatePartitions>

      <ModifyPartitions>
        <ModifyPartition wcm:action="add">
          <Format>NTFS</Format>
          <Order>1</Order>
          <PartitionID>1</PartitionID>
          <Label>System</Label>
        </ModifyPartition>
      </ModifyPartitions>
    </Disk>
  </DiskConfiguration>

We provide information about what to install:

  <ImageInstall>
    <OSImage>
      <WillShowUI>Never</WillShowUI>

      <InstallFrom>
        <MetaData>
          <Key>/IMAGE/Name</Key>
          <Value>Windows Server 2012 R2 SERVERSTANDARDCORE</Value>
        </MetaData>
      </InstallFrom>

And where we would like it installed:

      <InstallTo>
        <DiskID>0</DiskID>
        <PartitionID>1</PartitionID>
      </InstallTo>
    </OSImage>
  </ImageInstall>

specialize

In the specialize phase, we start by setting the system name to a randomly generated value using the Microsoft-Windows-Shell-Setup component:

<component name="Microsoft-Windows-Shell-Setup"
  publicKeyToken="31bf3856ad364e35" language="neutral"
  versionScope="nonSxS" processorArchitecture="amd64">
  <ComputerName>*</ComputerName>
</component>

We enable remote desktop because in an OpenStack environment this will probably be the preferred mechanism with which to connect to the host (but see this document for an alternative mechanism).

First, we need to permit terminal server connections:

<component name="Microsoft-Windows-TerminalServices-LocalSessionManager"
  processorArchitecture="amd64"
  publicKeyToken="31bf3856ad364e35"
  language="neutral"
  versionScope="nonSxS">
  <fDenyTSConnections>false</fDenyTSConnections>
</component>

And we do not want to require network-level authentication prior to connecting:

<component name="Microsoft-Windows-TerminalServices-RDP-WinStationExtensions"
  processorArchitecture="amd64"
  publicKeyToken="31bf3856ad364e35"
  language="neutral"
  versionScope="nonSxS">
  <UserAuthentication>0</UserAuthentication>
</component>

We will also need to open the necessary firewall group:

<component name="Networking-MPSSVC-Svc"
  processorArchitecture="amd64"
  publicKeyToken="31bf3856ad364e35"
  language="neutral"
  versionScope="nonSxS">
  <FirewallGroups>
    <FirewallGroup wcm:action="add" wcm:keyValue="RemoteDesktop">
      <Active>true</Active>
      <Profile>all</Profile>
      <Group>@FirewallAPI.dll,-28752</Group>
    </FirewallGroup>
  </FirewallGroups>
</component>

Finally, we use the Microsoft-Windows-Deployment component to configure the Windows firewall to permit ICMP traffic:

<component name="Microsoft-Windows-Deployment"
  processorArchitecture="amd64"
  publicKeyToken="31bf3856ad364e35"
  language="neutral" versionScope="nonSxS">

  <RunSynchronous>

    <RunSynchronousCommand wcm:action="add">
      <Order>3</Order>
      <Path>netsh advfirewall firewall add rule name=ICMP protocol=icmpv4 dir=in action=allow</Path>
    </RunSynchronousCommand>

And to download the cloudbase-init installer and make it available for later steps:

    <RunSynchronousCommand wcm:action="add">
      <Order>5</Order>
      <Path>powershell -NoLogo -Command "(new-object System.Net.WebClient).DownloadFile('https://www.cloudbase.it/downloads/CloudbaseInitSetup_Beta_x64.msi', 'c:\Windows\Temp\cloudbase.msi')"</Path>
    </RunSynchronousCommand>
  </RunSynchronous>
</component>

We're using Powershell here because it has convenient methods available for downloading URLs to local files. This is roughly equivalent to using curl on a Linux system.

oobeSystem

In the oobeSystem phase, we configure an automatic login for the Administrator user:

  <UserAccounts>
    <AdministratorPassword>
      <Value>Passw0rd</Value>
      <PlainText>true</PlainText>
    </AdministratorPassword>
  </UserAccounts>
  <AutoLogon>
    <Password>
      <Value>Passw0rd</Value>
      <PlainText>true</PlainText>
    </Password>
    <Enabled>true</Enabled>
    <LogonCount>50</LogonCount>
    <Username>Administrator</Username>
  </AutoLogon>

This automatic login only happens once, because we configure FirstLogonCommands that will first install cloudbase-init:

  <FirstLogonCommands>
    <SynchronousCommand wcm:action="add">
      <CommandLine>msiexec /i c:\windows\temp\cloudbase.msi /qb /l*v c:\windows\temp\cloudbase.log LOGGINGSERIALPORTNAME=COM1</CommandLine>
      <Order>1</Order>
    </SynchronousCommand>

And will then run sysprep to generalize the system (which will, among other things, lose the administrator password):

    <SynchronousCommand wcm:action="add">
      <CommandLine>c:\windows\system32\sysprep\sysprep /generalize /oobe /shutdown</CommandLine>
      <Order>2</Order>
    </SynchronousCommand>
  </FirstLogonCommands>

The system will shut down when sysprep is complete, leaving you with a Windows image suitable for uploading into OpenStack:

glance image-create --name ws2012 \
  --disk-format qcow2 \
  --container-format bare  \
  --file ws2012.qcow2

Troubleshooting

If you run into problems with an unattended Windows installation:

During the first stage of the installer, you can look in the x:\windows\panther directory for setupact.log and setuperr.log, which will have information about the early install process. The x: drive is temporary, and files here will be discarded when the system reboots.

Subsequent installer stages will log to c:\windows\panther\.

If you are unfamiliar with Windows, the type command can be used very much like the cat command on Linux, and the more command provides paging as you would expect. The notepad command will open a GUI text editor/viewer.

You can emulate the tail command using powershell; to see the last 10 lines of a file:

C:\> powershell -command "Get-Content setupact.log -Tail 10"

Technet has a Deployment Troubleshooting and Log Files document that discusses in more detail what is logged and where to find it.

by Lars Kellogg-Stedman at November 15, 2014 05:00 AM

November 14, 2014

Ed Leafe

The OpenStack Big Tent and Magnum

One of the most heavily-attended design summit events at last week’s OpenStack Summit in Paris was on Magnum, a proposed service for containers that would integrate into the Nova compute service. It seems that any session at any conference these days that involves Docker attracts a lot of interest, as Docker is an amazing new way of approaching how we think about virtualization and achieving efficiencies of scale.

Disclaimer: I know Adrian Otto, the leader of the Magnum project, from my days at Rackspace, and genuinely like him. I have no doubt that he would be able to put together a team that can accomplish all that he is setting out to do with this project. My thoughts and concerns about Magnum would be the same no matter who was leading the project.

The goal of the Magnum session was to present its concept and proposed architecture to the Nova ganttteam, with the hope of being designated as the official Docker project in OpenStack. However, there was a lot of push back from many members of the Nova team. Some of it had to do with procedural issues; I learned later that Magnum had been introduced at the Nova mid-cycle meetup, and the expectations set then had not been met. I wasn’t at that meetup, so I can’t personally attest to that. But the overall sentiment was that it was just too premature to settle on one specific approach to something as important and fast-moving as Docker. While I support the idea of Magnum and hope that it is a wild success, I also think that world of Docker/containers is moving so fast that what looks good today may look totally different 6 months from now. Moving such a project into OpenStack proper would only slow it down, and right now it needs to remain as nimble as possible.

I wrote a little while ago about my thoughts on the current discussions on the Big Tent vs. Layers vs. Small Core (Simplifying OpenStack), and I think that the Magnum effort is an excellent example of why we need to modify the approach to how we handle projects like this that add to OpenStack. The danger of the current Big Tent system of designating a single effort as the official OpenStack solution to a given problem is that by doing so we might be discouraging some group with a different and potentially better solution from pursuing development, and that would short-change the OpenStack ecosystem long-term. Besides, a little competition usually improves overall software quality, right?

by ed at November 14, 2014 06:06 PM

OpenStack Blog

OpenStack Community Weekly Newsletter (Post Paris)

How Operators Can Get Involved in Kilo #OpenStackSummit

Maish Saidel-Keesing participated in the Ops Summit: How to get involved in Kilo, and shared his notes from those sessions.

Development Reports from Summit

Relevant Conversations

Tips ‘n Tricks

Security Advisories and Notices

Upcoming Events

Other News

Got Answers?

Ask OpenStack is the go-to destination for OpenStack users. Interesting questions waiting for answers:

Welcome New Reviewers, Developers and Core Reviewers

jiangfei Cserey Szilard
ZongKai LI Ran Ziv
Swati Shukla Accela Zhao
Guillaume Giamarchi yatin
Seb Hughes François Bureau
Vadim Rutkovsky habuka036
Mark McDonagh Zengfa Gao
Craige McWhirter Jorge Munoz
juigil kishore Jobin Raju
Trung Trinh John Belamaric
Scott Lowe Seb Hughes
Roozbeh Shafiee David Caro
Mike Mason Craige McWhirter
Tan Lin Jan-Erik Mångs
David Caro Adolfo Duarte
Konstantinos Papadopoulos Tan Lin
Matteo Panella hossein zabolzadeh
Lena Vinod Pandarinathan
Michael Hagedorn Pieter
Major Hayden Lan Qi song
Magnus Lundin Vidyut
Arun S A G Inessa Vasilevskaya
Pratik Mallya Gil Meir
Brian Saville Dimitri Korsch
Chris Grivas Ian Adams
Marcin Karkocha Pratik Mallya
Yash Bathia
Wei Xiaoli
Mike Mason
Anton Arefiev
Yury Konovalov
Shang Yong

OpenStack Reactions

show-the-light

When a large company join the OpenStack foundation after ignoring it for a while

The weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

 

by Stefano Maffulli at November 14, 2014 06:06 PM

Daniel P. Berrangé

Faster rebuilds for python virtualenv trees

When developing on OpenStack, it is standard practice for the unit test suites to be run in a python virtualenv. With the Nova project at least, and I suspect most others too, setting up the virtualenv takes a significant amount of time as there are many packages to pull down from PyPI, quite a few of which need compilation too. Any time the local requirements.txt or test-requirements.txt files change it is necessary to the rebuild the virtualenv. This rebuild is an all-or-nothing kind of task, so can be a significant time sink, particularly if you frequently jump between different code branches.

At the OpenStack design summit in Paris, Joe Gordon showed Matt Booth how to setup devpi and wheel to provide a cache of the packages that make up the virtualenv. Not only does it avoid the need to repeatedly download the same packages from pypi each time, but it also avoids the compilation step, since the cache is storing the final installed pieces for each python module. The end result is that it takes 20-30 seconds or less to rebuild a virtualenv instead of many minutes.

After a few painful waits for virtualenvs today, I decided to set it up too I don’t like installing non-packaged software as root on my machines, so what follows is all done as a regular user account. The first step is to pull down the devpi and wheel packages from pypi, telling pip to install them in under $HOME/.local

# pip install --user devpi
# pip install --user wheel

Since we’re using a custom install location, it necessary to update your $HOME/.bashrc file with new $PATH and $PYTHONPATH environment variables and then source the .bashrc file

# cat >> $HOME/.bashrc <<EOF
export PATH=\$PATH:$HOME/.local/bin
export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages
EOF
# . $HOME/.bashrc

The devpi package provides a local server that will be used for downloads instead of directly accessing pypi.python.org, so this must be started

# devpi-server --start

Both devpi and wheel integrate with pip, so the next setup task is to modify the pip configuration file

# cat >> $HOME/.pip/pip.conf <<EOF
[global]
index-url = http://localhost:3141/root/pypi/+simple/
wheel-dir = /home/berrange/.pip/wheelhouse
find-links = /home/berrange/.pip/wheelhouse
EOF

We’re pretty much done at this point – all that is left is to prime the cache with the all the packages that Nova wants to use

# cd $HOME/src/cloud/nova
# pip wheel -r requirements.txt
# pip wheel -r test-requirements.txt

Now if you run any command that would build a Nova virtualenv, you should notice it is massively faster

# tox -e py27
# ./run_tests.sh -V

That is basically all there is to it. Blow away the virtualenv directories at any time and they’ll be repopulated from the cache. If the requirements.txt is updated with new deps re-running the ‘pip wheel’ command above will update the cache to hold the new bits.

That was all incredibly easy and so I’d highly recommend devs on any non-trivially sized python project make use of it.  Thanks to Joe Gordon for the pointing this out at the summit !

by Daniel Berrange at November 14, 2014 05:28 PM

Red Hat Stack

Delivering Public Cloud Functionality in OpenStack

Talligent-logo

RHOSCIPN_logo_small

When it comes to delivering cloud services, enterprise architects have a common request to create a public cloud-type rate plan for showback, chargeback, or billing. Public cloud packaging is fairly standardized across the big vendors as innovations are quickly copied by others and basic virtual machines are assessed mainly on price. (I touched on the concept of the ongoing price changes and commoditization of public clouds in an earlier post.) Because of this standardization and relative pervasiveness, public cloud rate plans are well understood by cloud consumers. This makes them a good model for introducing enterprise users to new cloud services built on OpenStack.Enterprise architects are also highly interested in on-demand, self-service functionality from their Openstack clouds in order to imitate the immediate response of public clouds. We will cover how to deliver on-demand cloud services in a future post.

Pricing and Packaging Cloud Services
Public cloud rate plans are very popular, seeing adoption within enterprises, private hosted clouds, and newer public cloud providers alike. Most public cloud providers use the typical public cloud rate plan as a foundation for layering on services, software, security, and intangibles like reputation to build up differentiated offerings.Enterprise cloud architects use similar rate plans to demonstrate to internal customers that they can provide on-demand, self-service cloud services at a competitive price. To manage internal expectations and encourage good behavior, enterprises usually introduce cloud pricing via a showback model which does not directly impact budgets or require exchange of money. Users learn cloud cost structures and the impact of their resource usage. Later, full chargeback can be applied where internal users are expected to pay for services provided.

As evidenced by easily accessible published rate plans, on-demand compute instances can represent a wide range of sizes, locations, operating systems, and optimizations (memory, storage, compute). Reserved instances provide the user the opportunity to make a one-time upfront payment in exchange for a discount on the hourly charge for the instance over the course of one or three years. Spot instances provide a discount in exchange for flexibility of running the instance at a time when the cloud has unused compute capacity.

Red Hat Enterprise Linux OpenStack Platform and Talligent OpenBook
Red Hat Enterprise Linux OpenStack Platform provides an integrated set of OpenStack modules and APIs that Talligent OpenBook connects to. OpenBook leverages the Keystone and Horizon modules for customer authentication and self-service. OpenBook also directly connects to other OpenStack components such as Nova, Swift, and Cinder to create an initial list of cloud tenants and assigned resources.

openstack stack
The key attributes of the rate plan are available from OpenStack via the Ceilometer metering component. Ceilometer data is non-repudiable and therefore auditable for billing of cloud services. OpenBook captures a full set of configuration details and usage metrics from OpenStack via the Ceilometer module. This includes all the meters associated with Nova (Compute), Neutron (Network), Glance (Image), Cinder (Volume), Swift (Object Storage), and Kwapi (Energy). A detailed list of the meters is available here.

As new metrics are added to Ceilometer, those metrics are picked up by the resource manager in OpenBook and made available as billable elements included in rate plans. For example, the Juno release of OpenStack is expected to add key SDN metrics such as Load Balancer as a Service, Firewall as a Service, and VPN as a Service. We plan to make those SDN metrics available in OpenBook within a few weeks of the GA release of the Juno version of OpenStack.

Creating an Public Cloud Rate Plan in OpenStack
Using a solution like OpenBook from Talligent, cloud architects can create a rate plan for Infrastructure-as-a-Service similar to any of the large public cloud providers. In order to approximate typical public cloud pricing in OpenStack, here are the key elements of the rate plan:

  • Server images in Openstack correspond to on-demand instances. Generally, they are categorized according to size (small, medium, large, extra large, etc.) and have vCPU, memory, and instance storage configured accordingly.
  • While OpenBook can prorate charges for fractional billing periods, most public cloud providers rounds up all fractions to the next hourly interval when computing charges.
  • OpenStack includes the concept of geographical regions. The region attribute enables the cloud architect to charge different rates for the different regions. This might be done because of variations in cost structures, data center capabilities, currencies, taxes, or other location specific variables.
  • Once the image, billing period, and regions have been established, users can designate a unit charge for each server image. Based on a recent survey of public cloud prices, prices range from $0.013 per hour for a tiny instance to $0.28 per hour for an extra large instance.
  • Block storage can be billed by volume type (SSD, SATA,..) and by GB hour for volumes and GB month for images and snapshots. GB hour and GB month calculations can be based on either peak or average hourly size.
  • Object storage can be billed by storage object size, container size, outgoing bytes transferred, and number of API requests.

That covers the basics. Advanced users can develop services that take into account dedicated connections between the customer and cloud, GPUs, DNS, VPN Connection, high i/o instances, high storage instances, high performance computing, etc. Public cloud rates change regularly as the market grows and competition increases. Instance types might not correspond directly to instances configured within OpenStack so please note that these rates will be an approximation only. Talligent.com for more information

About Talligent
A Business-Ready CloudTM requires business processes and controls. Once your OpenStack cloud is deployed, you need the visibility and control to maintain high operational efficiency, as well as the billing flexibility to keep up with ever-changing customer requirements and market conditions. Self-service capabilities from Talligent support the high response on-demand cloud functionality that customers expect. Public and private cloud offerings and prices are evolving rapidly – Talligent supplies the tools to evolve your cloud and compete.

by johnmeadowsjr at November 14, 2014 02:00 PM

Percona

Q&A: Percona XtraDB Cluster as a MySQL HA solution for OpenStack

Thanks to all who attended my Nov. 12 webinar titled, “Percona XtraDB Cluster as a MySQL HA Solution for OpenStack.” I had several questions which I covered at the end and a few that I didn’t. You can view the entire webinar and download the slides here.

Q&A: Percona XtraDB Cluster as a MySQL HA solution for OpenstackQ: Is the read,write speed reduced in Galera compared to normal MySQL?

For reads, it’s the same (unless you use the sync_wait feature, used to be called causal reads).

For writes, the cost of replication (~1 RTT to the worst node), plus the cost of certification will be added to each transaction commit.  This will have a non-trivial impact on your write performance, for sure.  However, I believe most OpenStack meta store use cases should not suffer overly by this performance penalty.

Q: Does state transfers affect a continuing transaction within the nodes?

Mostly, no.  The joining node will queue ongoing cluster replication while receiving its state transfer and use that to do its final ‘catchup’.  The node donating the state transfer may get locked briefly during a full SST, which could temporarily block writes on that node.  If you use the built-in clustercheck (or check the same things it checks), you can avoid this by diverting traffic away from a donor.

Q: Perhaps not the correct webinar for this question, but I was also expecting to hear something about using PXC in combination with OpenStack Trove. If you’ve got time, could you tell something about that?

Trove has not supported the concept of more than a single SQL target.  My understanding is a recent improvement here for MongoDB setups may pave the way for more robust Trove instances backed by Percona XtraDB Cluster.

Q: For Loadbalancing using the Java Mysql driver, would you suggest HA proxy or the loadbalancing connection in the java driver. Also how does things work in case of persistent connections and connection pools like dbcp?

Each node in a PXC cluster is just a mysql server that can handle normal mysql connections.  Obviously if a node fails, it’s easy to detect that you should probably not keep using that node, but in Percona XtraDB Cluster you need to watch out for things like cluster partitions where nodes lose quorum and stop taking queries (but still allow connections)  I believe it’s possible to configure advanced connection pooling setups to properly health check the nodes (unless the necessary features are not in the pool implementation), but I don’t have a reference architecture to point you to.

Q: Are there any manual solutions to avoid deadlocks within a high write context to force a query to execute on all nodes?

Yes, write to a single node, at least for the dataset that has high write volume.

Remember what I said in the webinar:  you can only update a given row once per RTT, so there’s an upper cap of throughput on the smallest locking granularity in InnoDB (i.e., a single row).

This manifests itself in two possible ways:

1) In a multi-node writing cluster by triggering deadlocks on conflicts.  Only approximately 1 transaction modifying a given row per RTT would NOT receive a deadlock

2) In a single-node writing cluster by experiencing longer lock waits.  Transaction times (and subsequently lock times) are extended by the replication and certification time, so other competing transactions will lock wait until the blocking transaction commit. There are no replication conflict deadlocks in this case, but the net effect is exactly the same:  only 1 transaction per row per RTT.

Galera offers us high data redundancy at the cost of some throughput.  If that doesn’t work for your workload, then asynchronous replication (or perhaps semi-sync in 5.7) will work better for you.

Note that there is wsrep_retry_autocommit.  However, this only works for autocommited transactions.  If your write volume was so high that you need to increase this a lot to get the conflict rate down, you are likely sacrificing a lot of CPU power (rollbacks are expensive in Innodb!) if a single transaction needs multiple retries to commit.  This still doesn’t get around the law:  1 trx per row per RTT at best.

That was all of the questions. Be sure to check out our next OpenStack webinar on December 10 by my colleague Peter Boros. It’s titled “MySQL and OpenStack Deep Dive” and you can register here (and like all of our webinars, it’s free). I also invite you to visit our OpenStack Live 2015 page and learn more about that conference in Santa Clara, California this coming April 13-14. The Call for Papers ends Nov. 16 and speakers get a full-access conference pass. I hope to see you there!

The post Q&A: Percona XtraDB Cluster as a MySQL HA solution for OpenStack appeared first on MySQL Performance Blog.

by Jay Janssen at November 14, 2014 02:00 PM

Sylvain Bauza

How to compare 2 patchsets in Gerrit ?

Apples & Oranges - They Don't Compare (Flickr, CC2.0)

Apples & Oranges – They Don’t Compare (Flickr, CC2.0)

Reviewing is one of my duties I’m doing daily. I try to dedicate around 2 hours each day in reading code, understanding the principles, checking if everything is OK from a Python perspective, verifying the test coverage and eventually trying to understand if it’s good for the project I’m supporting and not breaking anything even if CI is happy.

All that stuff can take time. And as I’m lazy, I really dislike the idea of reviewing again the same change that I previously approved if I’m sure that the new patchset is only a rebase. So, the question became in my mind very quickly : how can I compare that 2 patchsets are different ?

Actually, there are many ways to do so with Gerrit and Git. The obvious one is to make use of the Gerrit UI and ask for comparing 2 patchsets.

The main problem is that it shows all the differences, including the changes coming from the rebase so that’s not really handy, unless the change is very small.

Another is maybe to make use of the magical option “-m” of git review which allows to rebase each patchset on master and compare them.

git review -m <CHANGE_NUMBER>,<OLD_PS>[-<NEW_PS>]

That’s a pretty nice tool because it will allow you to make sure that the diffs are made against the same thing (here, master) so that it works fine provided you have your local master up-to-date. That said, if when rebasing your old patchset, you get a conflict, you should be stuck and sometimes the output would be confusing.

On my own, I finally ended up by doing something different : I’m fetching each patchset on a local branch and comparing each branch with the previous commit it has in the branch. Something like that :

vimdiff <(git diff ${MY_PS1_BRANCH}^ ${MY_PS1_BRANCH}) <(git diff ${MY_PS2_BRANCH}^ ${MY_PS2_BRANCH})

Here, MY_PS1_BRANCH would just be the local branch for the previous patchset, and MY_PS2_BRANCH would be the local branch for the next patchset.

By comparing the previous commit and the actual commit (ie. the change) on one branch and the another, then I’m quite sure that I won’t have to carry all the rebasing problems with my local master.


by Sylvain Bauza at November 14, 2014 10:26 AM

November 13, 2014

Mirantis

How small is too small? A minimal OpenStack testbed

When I started working on OpenStack I wanted to test all kinds of things and had no budget to build even a scaled-down test environment. And I know many engineers often test changes on mission critical systems in production because of their organization’s financial constraints, even at the real risk of downtime, lost revenue, and unhappy customers. Not to mention that testing in production is not as thorough as it could be.  It doesn’t have to be this way.

An inexpensive testing solution?

An OpenStack testbed does not have to be a full-size representation of the cluster. The architecture should be the same, but the number of compute and storage nodes, the specs of the hardware, and the networking infrastructure can be scaled down. A cloud with a hundred compute nodes and 25 Swift or Ceph storage nodes of 20TB each can be represented by a mini cloud with the same number of controllers and Swift proxies, but with only 5-10 compute nodes and five Swift or Ceph nodes at 10% of the total cost of the cloud. This is a great solution when you consider that a single large outage in a production environment can eat up the cost of a relatively small testbed.

Of course some things cannot be adequately tested with a scaled-down testbed, such as the behavior of a large number of nodes under load or bottleneck testing and similar issues that are only seen at scale. If testing such issues you may need a full-scale replica of the existing environment; however, the vast majority of configuration changes and failure scenarios can be successfully tested in a miniature environment.

The scaled-down environment is still too expensive?

If a scaled-down version of a real environment is still too expensive, you can test specific issues in a smaller environment. For example, you can use a single-controller configuration  instead of an HA configuration.

So what is the minimum size of a physical environment? For testing Nova, a single controller and one compute node can be sufficient. If a high load on the compute node is expected, you can use SSDs on the storage side, LACP on the network side, and more memory on the system side to alleviate the worst bottlenecks.

A Swift cluster can consist of a single storage node and a single proxy, which can reside on the same hardware. You can install Ceph alongside proxies or controllers, a technique Mirantis OpenStack employs to provide Ceph for internal storage without additional nodes.

Unlike an architecturally similar environment, a stripped down environment does not lend itself to running the same configuration you would run in production, so you will have to hand-prep changes tested in the small environment and plan for the impact of the scaled down version.

Still not cheap enough?

Looking at virtualization options I wondered how far I could get with a single node. I bought what I would normally consider a gaming machine – a tower case with a beefy power supply, a mainboard with a single AMD 8-core CPU, 32GB of memory, two 500GB SSDs, and an additional GbE network card. Total cost was roughly $1000. VMWare ESXi provided a number of VMs that I bound together into clouds.

Initially designed as a learning tool, this machine has seen a lot of test cases, and other developers have borrowed it on occasion. Mirantis OpenStack is preinstalled on one of the VMs, ready to deploy whatever cluster I may need for research. The machine is still sitting under my desk, waiting for the next test case.

And don’t forget one of the biggest advantages of virtualization. Before doing something potentially destructive you can make a snapshot of a VM and reinstate it if something does go awry.

I would never have believed that such a small and seemingly restricted machine would be able to do so much good in a world where scale is of paramount importance, but with a little imagination and ingenuity this mini testbed lets me try out a lot of things that would have cost time and money if I had wanted to set them up in a test environment.

Think outside the box

This old management adage still has merit, but occasionally it pays to think about a box — a box you can use to educate yourself, and develop software, troubleshoot, and test new features on.

The post How small is too small? A minimal OpenStack testbed appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Christian Huebner at November 13, 2014 04:03 PM

Percona

Percona Live London 2014 Wrap Up

Percona Live London 2014 SummaryThe 2014 edition of Percona Live London brought together attendees from 30 countries to hear insightful talks from leaders in the MySQL community. The conference kicked off on Monday with a full day of tutorials followed by the very popular Community Dinner featuring a double decker bus shuttle from the conference to the event.

Tuesday started with keynote talks by representatives from MySQL, VMware, HGST, Codership, and Percona. I particularly enjoyed the talks by Tomas Ulin of MySQL (which highlighted the upcoming MySQL 5.7 release) and Robert Hodges of VMware (which focused on the evolution of MySQL). The remainder of the day was filled by six time slots of breakout sessions (30 sessions in all) broken into 10 tracks. The day wrapped up with the always popular Community Networking Reception. Attesting to the quality of the conference, 4 out of 5 respondents to our post conference survey indicate they are likely to attend the conference again in 2015.

The session slides are available by visiting the Percona Live London 2014 conference website (look for the “Session Slides” button in the right hand column). Slides are added as they come in from the speakers so please check back if the slides are not yet available for a specific talk that interests you.

Special thanks goes out to the Percona Live London 2014 Conference Committee which put together such a great program:

  • Cedric Peintre of Dailymotion
  • David Busby of Percona
  • Colin Charles of MariaDB
  • Luis Motta Campos of the ebay Classified Group
  • Nicolai Plum of Booking.com
  • Morgan Tocker of Oracle
  • Art van Scheppingen of Spil Games

Percona Live London 2014 Attendee Survey

Percona Live London 2014 Attendee SurveyThis year we collaborated with ComputerworldUK to run a short survey at the conference which should appear in that publication in the near future. We had 64 responses, all of them being form MySQL professionals who attended the conference. The results were interesting:

Do you agree with the statement that “Oracle has been a good steward of MySQL over the past twelve months”?
YES = 81%
NO = 19%

Are you currently running a production OpenStack environment in your organization?
YES = 17%
NO = 83%

Have you evaluated OpenStack within the past twelve months?
YES = 25%
NO = 75%

Do you plan to evaluate OpenStack in the next twelve months?
YES = 48%
NO = 52%

Are you currently using an AWS product to run MySQL in the Cloud?
YES = 28%
NO = 72%

Are you more likely to switch to a non-MySQL open source database now than you were twelve months ago?
YES = 35%
NO = 65%

The sentiment about Oracle’s stewardship of MySQL compares favorably with the comments by our own Peter Zaitsev in a recent ZDNet article titled “MySQL: Why the open source database is better off under Oracle“.

Percona Live MySQL Conference and OpenStack Live Conference

Percona Live London 2014The ratings related to OpenStack mirror our experience with the strong growth in interest in that technology. In response, we are launching the inaugural OpenStack Live 2015 conference in Silicon Valley which will focus on making attendees more successful with OpenStack with a particular emphasis on the role of MySQL and Trove. The event will be April 13-14, 2015 at the Santa Clara Convention Center. The call for proposals closes on November 16, 2014.

Our next Percona Live MySQL Conference and Expo is April 13-16, 2015 in Silicon Valley. Join us for the largest MySQL conference in the world – last year’s event had over 1,100 registered attendees from 40 countries who enjoyed a full four days of tutorials, keynotes, breakout sessions, networking events, BOFs, and more. The call for speaking proposals closes November 16, 2014 and Early Bird registration rates are still available so check out the conference website now for full details.

Thanks to everyone who helped make Percona Live London 2014 a great success. I look forward to the Percona Live MySQL Conference and the OpenStack Live Conference next April in Silicon Valley!

The post Percona Live London 2014 Wrap Up appeared first on MySQL Performance Blog.

by Terry Erisman at November 13, 2014 02:16 PM

Stefano Maffulli

Tips to improve organizations contributing to OpenStack

The slides of my presentation at OpenStack Summit in Paris 2014 (download ODP source, 9M).

<iframe frameborder="0" height="400" marginheight="0" marginwidth="0" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/41502830" width="476"></iframe>

And the sessions’ recording:

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="http://www.youtube.com/embed/4sXuadpEhk8" width="560"></iframe>


© stefano for ][ stefano maffulli, 2014. | Permalink | No comment | Add to del.icio.us
Post tags: , , , , , ,

Feed enhanced by Better Feed from Ozh

by stefano at November 13, 2014 11:19 AM

Opensource.com

8 new tips for getting things done with OpenStack

Want to get more done with OpenStack? We've got you covered.

We've put together some of the best how-tos, guides, tutorials, and tips published over the past month into this handy collection. And if you need more help, the official documentation for OpenStack is always a great place to turn.

by Jason Baker at November 13, 2014 10:00 AM

November 12, 2014

Mirantis

See the Mirantis talks from Paris OpenStack Summit on video

The OpenStack Kilo 2014 Summit in Paris is now behind us. With thousands of attendees, a couple hundred presentations and design sessions, there was a lot to take in. And of course, the city of lights provided plenty of distractions.  

For those of you who didn’t have a chance to travel to Paris or attend the 20+ presentations that we took part in, we’ve curated a quick list of videos of those presentations (produced by the OpenStack Foundation) below. 

A full list of all conference videos is here

The post See the Mirantis talks from Paris OpenStack Summit on video appeared first on Mirantis | The #1 Pure Play OpenStack Company.

by Admin at November 12, 2014 05:26 PM

Tesora Corp

Short Stack: Execs discuss Kilo, OpenStack and Software-defined Economy and the Mr. Spock view of OpenStack

short stack_b small_0_0.jpgWelcome to the Short Stack, our weekly feature where we search for the most intriguing OpenStack links to share with you. These links may come from traditional publications or company blogs, but if it's about OpenStack, we'll find the best links we can to share with you every week.

If you like what you see, please consider subscribing.

Here we go with this week's links:

OpenStack welcomes you to the software-defined economy | ZDNet

As we transform to a software-defined economy (software really is eating the world, as Marc Andreessen once said), we need to be faster to market and quicker to react to competitive pressure and proprietary software inhibits that.  OpenStack started as a small project and now it's a huge project because people are recognizing the value of an open source project in today's rapidly changing world.

As OpenStack Stabilises Big Questions Remain for the Foundation | Forrester/Computerworld

There is little dispute that OpenStack is a maturing project now, but Forrester believes that as we head into 2015, we will move from experimentation to full blown implementations and Forrester would like to hear more from the enterprise user base and less from large vendors about the future of the project

OpenStack: Do The Needs of the Many Outweigh the Needs of the Few? [VIDEO] | Datamation

How can you resist an article that uses a Star Trek reference? And when it comes to the OpenStack community, you may need to look at the path forward the Vulcan way: the needs of the many outweigh the needs of the few. Sounds good.

OpenStack Execs Discuss Kilo Release, Ironic Project and More | eWeek

Leaders of the OpenStack community had a discussion and one of the things that came out of it was the support for Bare Metal servers in the next release of OpenStack, which could provide performance improvements to drive additional use cases for OpenStack moving forward.

5 things we learnt from OpenStack Summit 2014 | Computer Business Review

Now that we've wrapped up the OpenStack Paris Summit, there are few things that are clear. The community has grown enormously (although the exact size is unclear) and the big companies are very interested in getting involved. This article provides a summary of themes and take-aways.

by 693 at November 12, 2014 01:15 PM

Aptira

What is Ceilometer's Minimum Viable Product?

I believe the basic goals of Ceilometer are laudable. It attempts to gather data about the state of an OpenStack instance, which is a useful goal. It then attempts to solve a bunch of problems related to that data, which is also useful.

However, many of the problems it tries to solve work at cross purposes to each other, and so choices that are made to accommodate solutions to these problems prevent the project from solving any of them satisfactorily. Solutions to some require high resolution data, some do not. Solutions to other problems require large amounts of historical data, others do not.

Ceilometer attempts to address these problems in a single database (OK a few databases, but one for each class of data: meters, events, alarms). This can never meet the requirements of every solution. Instead Ceilometer should focus on gathering the information, getting it to consuming systems in a fast, reliable and easy to consume manner, and then it should stop. There's plenty of work to do to reach this smaller set of goals, and there's an enormous amount of work to be done creating consumer systems that deliver real customer value.

Let other projects start to solve the problems we need to solve.

- Lets build a usage corellation system that takes the data stream and emits simple rateable usage records. Lets build another system that rates the records according to rules defined in a product catalogue.

- Lets build a policy engine that can pull the levers on various APIs based on a stream of information coming from Ceilometer.

- Lets build a repository for long term storage of data and accompanying analysis tools

and so on. But let's make them separate projects, not attempt to solve all the problems from a single DB.

The projects can focus on their own specific needs and because they are loosely coupled to metric collection and completely decoupled from one another, they can ignore needs of others that might be destructive to their own value. They can adopt an attitude of excellence rather than one of compromise.

More generally, lets examine our approach to adding features to OpenStack and see whether continually adding features to existing projects is necessarily the right way to go.

by Roland Chan (roland@aptira.com) at November 12, 2014 05:41 AM