Planet OpenStack

Open development, releases and quality

2012-02-21T14:07:11Z

Every 6 months, as a cycle ends and we prepare the next, we look back at our release model and try to see how we can improve it. My opinion is that we need (once more) to evolve it, and here is why.

Objectives and past evolutions

Our main objective is to produce stable and usable software. Our secondary objective is let stable new features reach the hands of our users in a timely manner. Our tertiary objective is to empower developers to work efficiently on the features and improvements of tomorrow. In simple solutions, those three objectives are not really compatible, and in the past we tried to make adjustments without acknowledging the fundamental incompatibility between those objectives.

The Austin-Cactus system was a 3-month cycle with various freezes. The issues with this system were that it took too long to get features in the hands of testers and users, developers had trouble working during the frozen periods, and not so many people actually helped during the QA period.

For Diablo, we decided to switch to a 6-month cycle with monthly milestones. Those milestones were supposed to address the “get features into the hands of users early” issue. To empower developers, we introduced an almost-always-open trunk: we only had 2 weeks of feature freeze before the Essex branch opened (and coexisted with the Diablo release branch). The problem was the resulting quality was not good, since we accumulated 6 months’ worth of features and only had 2 weeks’ worth of QA on them.

So for Essex, there was a decision that each project would decide where to place its feature freeze and its first release candidate (which would be when the Folsom branch opens). Most of them decided to use the essex-3 milestone as a soft feature freeze and essex-4 as a hard one, which reintroduced the “closed trunk” issue. And the work needed to stabilize 6 months’ worth of features is still daunting. And we saw lots of feature freeze exceptions for last-minute “almost there” features that can’t afford to wait another 6 months.

So what’s the solution ? Is there a good way to (at the same time) reduce the pain of “missing a release”, have always-open development branches, and get higher quality ?

The kernel model

The only way to reduce the pain of missing a release is to have shorter time-based releases. The only way to have always-open development branches without sacrificing release quality is to separate them completely from release branches. Where do those truisms lead us ?

Another famous open innovation project has already been there, and that’s the Linux kernel. Development happens in various always-open topic branches, and regularly a merge window opens to propose stable features for inclusion in the mainline kernel.

If we manage to efficiently separate OpenStack in topics, we could adopt the same model. We could have several topic branches where core developers on a specific area can collaborate and review code affecting that area. We could have frequent releases (every 6-8 weeks ?), and for each release we could have a “merge window” where stuff from topic branches can be proposed for release, if deemed stable enough. Between the moment the merge window closes and the moment the final release is cut, various release candidates can be produced, on which serious QA can be unleashed without blocking other developers.

This solves all objectives. If a feature is not ready yet (according to the team maintaining the topic branch or according to release management), it can bake until the next merge window, which is not far away. Regular releases ensure that improvements reach the users in a timely manner. Development branches are always open. And with a reasonable amount of new code in every release, QA work is facilitated, theoretically resulting in higher release quality.

Additional benefits

Splitting development into topics has an additional benefit: smaller groups developing an expertise on a specific area make better reviewers on that specific area than a random nova-core developer that can’t be an expert in all things Nova. I’d say that each topic should be small enough to be manageable with a team of 6 core reviewers at the maximum. Of course, nothing prevents anyone from being active on multiple topics if they wish.

More frequent releases also allows you to set themes for each release. It’s difficult to refuse some feature in a 6-month cycle, but it’s easy to delay a feature to the next 6-week cycle. In the same way some kernel releases introduce large architectural changes and some others are more geared towards performance improvements or stability, we could also define themes for every release — after all the next one is not so far away.

Challenges

There are a few issues with this model obviously. Code needs to be componentized enough so that merge pains can be limited. Changes from every release need to be merged back into topic branches. Code needs to be clearly separated into a set of relevant topics, each with its own set of core reviewers maintaining the topic branch. The release team must be staffed enough to be able to review proposed code for stability. Bug fixes need to easily end up in the release branch. And how do those “releases” match the 6-month period between summits ? What means “Folsom” in that respect ?

I hope we can use the following weeks to discuss the devil in the details of this possible evolution, and be ready to take a decision when the time comes for us to gather at the Folsom Design summit.

Community Weekly Review (Feb 10-17)

2012-02-17T23:03:41Z

OpenStack Community Newsletter –February 17, 2012

HIGHLIGHTS

TryStack.org – A Sandbox for OpenStack is live http://www.openstack.org/blog/2012/02/trystack-org-a-sandbox-for-openstack/
The brand new API reference website http://api.openstack.org/
SoftLayer announced its Object Storage product built on OpenStack Swift http://blog.softlayer.com/2012/softlayer-openstack-swift-softlayer-object-storage/
Action needed: OpenStack Governance Elections http://www.openstack.org/blog/2012/02/openstack-governance-elections-spring-2012/
Setting up the OpenStack Foundation: video of the meetup in Santa Clara http://www.collierclan.net/mark/?p=425

A draft for the Foundation’s structure is up for discussion http://lists.openstack.org/pipermail/foundation/2012-February/000134.html

Last days to answer the Call for speaker for OpenStack Conference http://www.openstack.org/conference/san-francisco-2012/speakers/
Book your room for the week https://resweb.passkey.com/go/DRAC2012

EVENTS

FLOSSTalk Kosovo Feb 22, 2012 – UNICEF Innovations Lab Kosovo in Prishtina, Kosovo http://www.flossk.org/en/blog/flosstalk-arturo-suarez-openstack
OpenStack Spring 2012 Design Summit Apr 16 – 18 and Conference Apr 19-20 – San Francisco, California http://openstack.org/conference/

OTHER NEWS

Under the hood of Swift: the Ring (by Mirantis) http://www.openstack.org/blog/2012/02/1997/
DNS management system for OpenStack (by Grid Dynamics) http://openstackgd.wordpress.com/2012/02/17/dns-for-openstack/
OpenStack Wiki Recent Changes – http://wiki.openstack.org/RecentChanges

New report to track changes only in main branch http://wiki.openstack.org/CommunityMetrics/Code
OpenStack sprint at PyCon2012 http://wiki.openstack.org/Sprints/PyCon2012
How to update the Corporate Copyright License Agreement http://wiki.openstack.org/HowToUpdateCorporateCLA

Documentation Day, March 6th http://wiki.openstack.org/Documentation/DocDay
Documentation HowTo http://wiki.openstack.org/Documentation/HowTo
Getting Started with OpenStack in Japanese http://wiki.openstack.org/GettingStarted/ja

Project meeting summary http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-02-14-21.00.html

COMMUNITY STATISTICS

Activity on the main branch of OpenStack repositories, lines of code added and removed per developer during week 6 of 2012 (from Mon Feb 06 00:00:00 UTC 2012 to Mon Feb 13 00:00:00 UTC 2012)

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

DNS management system for OpenStack

2012-02-17T15:50:02Z

`nova-dns` package was developed to solve two tasks:

map instance’s hostnames to ip addresses in DNS
provide REST API to manage DNS

To solve first task service monitor message bus (RabbitMQ). For every started instance service add DNS record, for terminated – remove. DNS name is choose in form hostname.tenant_name.root_zone. If zone for tenant_name doesn’t exists yet, it is created and populated automatically.

To solve second task service start REST API server on specified ip/port. REST API supports authentication against keystone and utilize keystone’s RBAC.

As a DNS backend PowerDNS is used, but other backends (for example for Bind) can be added later.

Next we are going to add support for PTR zones and add ability to create personal zone for every started instance.

Links:

TryStack.org – A Sandbox for OpenStack!

2012-02-16T15:34:44Z

Today, a project that has been a long time in the making is finally coming to fruition. Back last summer, when I was working at Rackspace, Nati Ueno from NTT PF Labs came up with the idea of establishing a “Free Cloud” — a site running OpenStack that developers using the OpenStack APIs could use to test their software.

The involvement of a number of companies — Dell, NTT PF Labs, Rackspace, Cisco, Equinix and HP Cloud Services — eventually drove the idea of “The Free Cloud” from concept to reality. Though there were many delays, I’m happy to announce that this new OpenStack testing sandbox has now been launched.

The new name of the artist formerly known as The Free Cloud is now called TryStack. The front page at http://trystack.org now houses some information about the effort and a few FAQs. If you’re interested in trying out OpenStack, TryStack.org is right up your alley, and we encourage you to sign up. Instructions for signup are on the front page.

Using TryStack

Once you’ve gone through the TryStack registration process, you will receive a username and password for logging in to the TryStack Dashboard and Nova API. After logging in, you’ll see the user dashboard for TryStack. This dashboard is essentially the stock upstream OpenStack Dashboard — only lightly modified with a couple extensions that Nati wrote to show consumption of Stack Dollars and the TryStack logo.

Stack Dollars are the “unit of currency” for TryStack. When you perform certain actions in TryStack — launching instances, creating volumes, etc — you consume Stack Dollars. Likewise, instances consume Stack Dollars as long as they are running. When you run out of Stack Dollars, you won’t be able to use TryStack services until your Stack Dollars are replenished. Stack Dollars will be replenished on a periodic basis (haven’t quite decided on the interval yet…)

To prevent people from gaming the system or using TryStack as a tool for evil, instances will remain alive for up to 24 hours or until your Stack Dollar balance is depleted. Also, always keep in mind that TryStack should only be used for testing. Under no circumstances should you use it for any production uses. There is no service level agreement with TryStack.

An Automation and Administration Sandbox Too!

In addition to being a useful sandbox for developers using the OpenStack APIs and others interested in seeing what OpenStack is all about, TryStack.org is also a very useful testbed for work that the OpenStack community is doing to automate the deployment and administration OpenStack environments.

TryStack is deployed using the Chef Cookbooks from the upstream repository, and changes that are needed will be pushed back upstream immediately for consumption by the broad community. We have a limited HA setup for the initial TryStack availability zone and lessons learned from the deployment of these HA setups are being incorporated into an online TryStack Administrator’s Guide that will serve as input for the upstream documentation teams as well.

Roadmap for TryStack

In the next three to six months, we’re planning to bring on-line at least one more availability zone. The next availability zone will be running HP hardware and will be housed in a datacenter in Las Vegas. It is likely that this new zone will be deployed with the Essex release of OpenStack components, enabling users to test against both a Diablo-based OpenStack installation and an Essex-based installation.

This first availability zone does not contain an installation of Swift. Of course, we want to change that, so an installation of Swift is definitely on the roadmap for either the next availability zone or as a separate service itself. Note that, just like the instances launched in TryStack, objects stored in a TryStack Swift cluster would be temporary. After all, TryStack is for trying out OpenStack services, not for providing a free CDN or storage system!

We will also eventually move towards a different registration process to accomodate non-Facebook users. If you are interested in helping with this effort, please do let us know.

Finally, we’ll be adding things like an automated Twitter status feed for each zone, lots of documentation gathered from running TryStack, and hopefully a number of videos showing usage of TryStack as well as common administrative tasks — all with the goal of providing more and better information to the broad and growing OpenStack community. I fully expect numerous hiccups and growing pains in these first couple months of operation, but we promise to turn any pain points into lessons learned and document them for the benefit of the OpenStack community.

Please do check out the trystack.org service. We look forward to your feedback. You can find us on Freenode.net #trystack. Nati and I will be hosting a webinar February 23, and we’ll be speaking at a San Francisco meetup March 6 if you’re interested in learning more or getting involved.

Update: I totally goofed and left Cisco off the list of donor organizations. My apologies to Mark Voelker and the excellent folks at Cisco who provided two 4948-10GE switches that are in use in the TryStack cloud. I also got the link wrong to HP Cloud… which is pretty lame, considering I work for HP. That’s been corrected.

TryStack.org – A Sandbox for OpenStack!

2012-02-16T14:23:35Z

Using TryStack

An Automation and Administration Sandbox Too!

Roadmap for TryStack

We will also eventually move towards a different registration process to accomodate non-Facebook users. If you are interested in helping with this effort, please do let us know.

Under the hood of Swift: the Ring

2012-02-15T09:45:57Z

This is the first post in series that summarizes our analysis of Swift architecture. We’ve tried to highlight some points that are not clear enough in the official documentation. Our primary base was an in-depth look into the source code.

The following material applies to version 1.4.6 of Swift.

The Ring is the vital part of Swift architecture. This half database, half configuration file keeps track of where all data resides in the cluster. For each possible path to any stored entity in the cluster, the Ring points to the particular device on the particular physical node.

There are three types of entities that Swift recognizes: accounts, containers and objects. Each type has the ring of its own, but all three rings are put up the same way. Swift services use the same source code to create and query all three rings. Two Swift classes are responsible for this tasks: RingBuilder and Ring respectively.

Ring data structure

Every Ring of three in Swift is the structure that consists of 3 elements:

a list of devices in the cluster, also known as devs in the Ring class;
a list of lists of devices ids indicating partition to data assignments, stored in variable named _replica2part2dev_id;
an integer number of bits to shift an MD5-hashed path to the account/container/object to calculate the partition index for the hash (partition shift value, part_shift).

List of devices

A list of devices includes all storage devices (disks) known to the ring. Each element of this list is a dictionary of the following structure:

Key	Type	Value
id	integer	Index of the devices list
zone	integer	Zone the device resides in
weight	float	The relative weight of the device to the other devices in the ring
ip	string	IP address of server containing the device
port	integer	TCP port the server uses to serve requests for the device
device	string	Disk name of the device in the host system, e.g. `sda1`. It is used to identify disk mount point under `/srv/node` on the host system
meta	string	General-use field for storing arbitrary information about the device. Not used by servers directly

Some device management can be performed using values in the list. First, for the removed devices, the 'id' value is set to 'None'. Device IDs are generally not reused. Second, setting 'weight' to 0.0 disables the device temporarily, as no partitions will be assigned to that device.

Partitions assignment list

This data structure is a list of N elements, where N is the replica count for the cluster. The default replica count is 3. Each element of partitions assignment list is an array('H'), or Python compact efficient array of short unsigned integer values. These values are actually index into a list of devices (see previous section). So, each array('H') in the partitions assignment list represents mapping partitions to devices ID.

The ring takes a configurable number of bits from a path’s MD5 hash and converts it to the long integer number. This number is used as an index into the array('H'). This index points to the array element that designates an ID of the device to which the partition is mapped. Number of bits kept from the hash is known as the partition power, and 2 to the partition power indicates the partition count.

For a given partition number, each replica’s device will not be in the same zone as any other replica’s device. Zones can be used to group devices based on physical locations, power separations, network separations, or any other attribute that could make multiple replicas unavailable at the same time.

Partition Shift Value

This is the number of bits taken from MD5 hash of '/account/[container/[object]]' path to calculate partition index for the path. Partition index is calculated by translating binary portion of hash into integer number.

Ring operation

The structure described above is stored as pickled (see Python pickle) and gzipped (see Python gzip.GzipFile) file. There are three files, one per ring, and usually their names are:

account.ring.gzcontainer.ring.gzobject.ring.gz

These files must exist in /etc/swift directory on every Swift cluster node, both Proxy and Storage, as services on all these nodes use it to locate entities in cluster. Moreover, ring files on all nodes in the cluster must have the same contents, so cluster can function properly.

There are no internal Swift mechanisms that can guarantee that the ring is consistent, i.e. gzip file is not corrupt and can be read. Swift services have no way to tell if all nodes have the same version of rings. Maintenance of ring files is administrator’s responsibility. These tasks can be automated by means external to the Swift itself, of course.

The Ring allows any Swift service to identify which Storage node to query for the particular storage entity. Method Ring.get_nodes(account, container=None, obj=None) is used for identification of target Storage node for the given path (/account[/container[/object]]). It returns the tuple of partition and dictionary of nodes. The partition is used for constructing the local path to object file or account/container database. Nodes dictionary elements have the same structure as the devices in list of devices (see above).

Ring management

Swift services can not change the Ring. Ring is managed by swift-ring-builder script. When new Ring is created, first administrator should specify builder file and main parameter of the Ring: partition power (or partition shift value), number of replicas of each partition in cluster, and the time in hours before a specific partition can be moved in succession:

swift-ring-builder  create

When the temporary builder file structure is created, administrator should add devices to the Ring. For each device, required values are zone number, IP address of the Storage node, port on which server is listening, device name (e.g. sdb1), optional device meta-data (e.g., model name, installation date or anything else) and device weight:

swift-ring-builder  add z-:/_

Device weight is used to distribute partitions between the devices. More the device weight, more partitions are going to be assigned to that device. Recommended initial approach is to use the same size devices across the cluster and set weight 100.0 to each device. For devices added later, weight should be proportional to the capacity. At this point, all devices that will initially be in the cluster, should be added to the Ring. Consistency of the builder file can be verified before creating actual Ring file:

swift-ring-builder

In case of successful verification, the next step is to distribute partitions between devices and create actual Ring file. It is called ‘rebalance’ the Ring. This process is designed to move as few partitions as possible to minimize the data exchange between nodes, so it is important that all necessary changes to the Ring are made before rebalancing it:

swift-ring-builder  rebalance

The whole procedure must be repeated for all three rings: account, container and object. The resulting .ring.gz files should be pushed to all nodes in cluster. Builder files are also needed for the future changes to rings, so they should be backed up and kept in safe place. One of approaches is to put them to the Swift storage as ordinary objects.

Physical disk usage

Partition is essentially the block of data stored in the cluster. This does not mean, however, that disk usage is constant for all partitions. Distribution of objects between the partitions is based on the object path hash, not the object size or other parameters. Objects are not partitioned, which means that an object is kept as a single file in storage node file system (except very large objects, greater than 5Gb, which can be uploaded in segments – see the Swift documentation).

The partition mapped to the storage device is actually a directory in structure under /srv/node/. The disk space used by this directory may vary from partition to partition, depending on size of objects that have been placed to this partition by mapping hash of object path to the Ring.

In conclusion it should be said that the Swift Ring is a beautiful structure, though it lacks a degree of automation and synchronization between nodes. I’m going to write about how to solve these problems in one of the following posts.

More information

More information about Swift Ring can be found in following sources:
Official Swift documentation – base source for description of data structure
Swift Ring source code on Github – code base of Ring and RingBuilder Swift classes.
Blog of Chmouel Boudjnah – contains useful Swift hints

OpenStack Governance Elections Spring 2012

2012-02-13T07:00:01Z

The time is once again upon us for our OpenStack Governance Elections. The OpenStack community is called to elect the Project Technical Leads and two seats of the Project Policy Board. The election committee is made of Stefano Maffulli, Lloyd Dewolf and Dave Nielsen.

February 16 – 26 11:59 PST: Nominations open.
February 28 – March 3 11:59 PST: Online voting open.
March 3 11:59 PST: Voting closed.

Final results will be posted immediately upon election close.

What seats are up for election

NOVA Project Team Lead (1 Position)
SWIFT Project Team Lead (1 Position)
GLANCE Project Team Lead (1 Position)
HORIZON Project Team Lead (1 Position)
KEYSTONE Project Team Lead (1 Position)
Project Policy Board (2 Open Positions)

How to nominate yourself or others as Project Technical Lead

Only OpenStack community members who have code in the respective OpenStack subproject are eligible to be elected as that subproject’s Project Team Lead. Please nominate someone from the developer community or yourself at http://etherpad.openstack.org/Spring2012-Nominees under the Nominees heading. Please provide the name and email address of the nominee. The election committee will then confirm with the nominee that they are willing to run for the position.The list of Approved Candidates will be announced with a new blog post on openstack.org/blog when online voting opens (Feb 28).

How to nominate yourself or others as member of the Project Policy Board

Any registered member of the OpenStack Launchpad group is eligible to run or be nominated for a position on the Project Policy Board. If you want to vote and/or run for a seat you need to register on Launchpad and add yourself to the public OpenStack group on https://launchpad.net/~openstack. Please nominate someone from the community or yourself at http://etherpad.openstack.org/Spring2012-Nominees under the Nominees heading. Please give the name and email address of the nominee. The election committee will then confirm with the nominee that they are willing to run for the position. The list of Approved Candidates will be announced with a new blog post on openstack.org/blog right before the election starts.

How to register to vote for PTL

Only OpenStack community members who have code in the respective OpenStack subproject are eligible to vote for that subproject’s Project Team Lead. The authoritative list of eligible voters and nominees is the Authors file in each repository. For example, the list of Nova authors is https://github.com/openstack/nova/blob/master/Authors.
Make sure your name and correct email address is there or you won’t be able to vote.

How to register to vote for Project Policy Board

Any registered member of the OpenStack Launchpad group is eligible to vote for the Project Policy Board. If you want to vote you need to register to Launchpad and add yourself to the public OpenStack group on https://launchpad.net/~openstack before registering as a voter using the form at http://ppbelectionsregistration.openstack.org/. Company affiliation is only collected as an interesting statistic; it has no effect on the outcome of the election.

Voting process

Like previous OpenStack Governance Elections, we will use the Condorcet Internet Voting Service from Cornell University, http://www.cs.cornell.edu/andru/civs.html. This tool uses the Condorcet method of voting which invokes ranking the nominees instead of just selecting one choice. More information on this methodology is at http://www.cs.cornell.edu/w8/~andru/civs/rp.html.

All registered voters will receive an email with a unique link allowing them to privately vote.

Please note that the voting system is run using private polls with restricted access to ensure voter authenticity; however all results will be made public once the election ends. Voter anonymity is guaranteed. The result’s ranking will be evaluated using Schulze (also known as Beatpath or CSSD) completion rule.

Thanks for participating in this essential process. Please remind your friends and colleagues to get involved, register and vote!

Community Weekly Review (Feb 3-10)

2012-02-10T17:10:04Z

OpenStack Community Newsletter –February 10, 2012

HIGHLIGHTS

OpenStack Bug Squashing day was a success and we should repeat it (see graphs below)
OpenStack Foundation Update http://lists.openstack.org/pipermail/foundation/2012-February/000123.html
Ceph Lords meetup in San Francisco has links to recorded presentations http://www.openstack.org/blog/2012/02/recap-ceph-lords-openstack-sf-meetup-feb-02/
Feedback from FOSDEM 2012 meetings http://fnords.wordpress.com/2012/02/08/fosdem-2012-feedback/ and a visual https://twitter.com/#!/smaffulli/status/165746062181670912/photo/1
Canonical published guide to automatic testing on Ubuntu http://www.openstack.org/blog/2012/02/automating-openstack-testing-on-ubuntu/
How does a PUT to a swift object server look like http://blog.chmouel.com/2012/02/06/anatomy-of-a-swift-put-query-to-object-server/
The “GitHub effect” in open source project http://fnords.wordpress.com/2012/02/01/about-collaboration/
How to audit a Swift cluster http://blog.chmouel.com/2012/02/01/audit-a-swift-cluster/
Opened the Call for speaker for OpenStack Conference http://www.openstack.org/conference/san-francisco-2012/speakers/
Book your room for the week https://resweb.passkey.com/go/DRAC2012

EVENTS

OpenStack Party @ CloudConnect 2012 http://www.openstack.org/blog/2012/02/openstack-party-cloudconnect-2012/
OpenStack Foundation Meetup @CloudConnect Feb 15, 2012 – Santa Clara, CA http://www.meetup.com/openstack/events/51553992
Inaugural OpenStack DC Meetup Feb 16, 2012 – Washington, DC http://www.meetup.com/OpenStackDC/events/47106902/
FLOSSTalk Kosovo Feb 22, 2012 – UNICEF Innovations Lab Kosovo in Prishtina, Kosovo http://www.flossk.org/en/blog/flosstalk-arturo-suarez-openstack
OpenStack Spring 2012 Design Summit Apr 16 – 18 and Conference Apr 19-20 – San Francisco, California http://openstack.org/conference/

OTHER NEWS

Do you want to help OpenStack at Google Summer of Code? Chime in https://lists.launchpad.net/openstack/msg07441.html and http://wiki.openstack.org/GSoC2012
Swift 1.4.6 release proposed candidate https://lists.launchpad.net/openstack/msg07432.html
Mailing list etiquette http://wiki.openstack.org/MailingListEtiquette
Do we need a mailing list only for developers? Chime in https://lists.launchpad.net/openstack/msg07277.html
OpenStack Wiki Recent Changes – http://wiki.openstack.org/RecentChanges

Keystone Folsom summit topics http://wiki.openstack.org/KeystoneFolsomSummitTopics
Documentation HowTo http://wiki.openstack.org/Documentation/HowTo
Getting Started with OpenStack in Japanese http://wiki.openstack.org/GettingStarted/ja

Project meeting summary http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-02-07-21.03.html

COMMUNITY STATISTICS

The charts below represent the work done on bugs during Bug Squashing Day Feb 2 2012.

Open Advice Book Now Available

2012-02-10T15:54:54Z

What we wish we had known when we started working on free open source software (FOSS) – that is the premise of this essay collection, Open Advice. What’s especially interesting to me after having read all 42 essays is there isn’t really a pattern like you’d see when looking at “what I would tell my younger self” such as “believe in yourself” or “don’t worry about what others think.” Those themes do come through, but the heart of the collection centers on open source projects, software, users, coders, and the myriad roles that make an open source project great. It’s a collection of great stories and great experiences.

One theme that stuck out to me was “I wish I had been less arrogant.” So many people, women especially, can be put off by the attitudes displayed by an open source online chat room or mailing list. I know I’m happy to see more women’s names on the OpenStack mailing list, asking and answering questions. I’m also happy to note that the OpenStack community is polite, professional, and welcoming to all.

An essay that fascinated me was “27 Things I’m Happy I Didn’t Know” by Alexandra Leisse. Ignorance is bliss in many arenas, open source is no exception. Sometimes learning from mistakes is the best lesson.

I also loved the “Never on a Friday” section of Sally Khudairi‘s essay that leads with “Everyone is a marketer.” She launched a new homepage for W3C on a Friday then boarded a plane for Paris. She landed to a flood of messages about a particular tag choice. Fabulous story.

Dave Neary‘s essay about conference planning, “Getting People Together,” has a detailed section about budgets and funding plus content and parties. Valuable and practical, this essay should be required reading for both conference planners and attendees.

What did I write about? Documentation and My Former Self is the title, and in it I realize how many contradictions I’ve discovered on my journey. Often, adding more and more observations will bring a flip turn to my stance on a documentation strategy. Fascinating.

The Open Advice book in PDF format is downloadable for all and I’d encourage you to read it, enjoy it, learn from it, and share it. Then, join in an open source community with those lessons already learned.

Recap: “Ceph Lords” OpenStack SF Meetup Feb 02

2012-02-10T03:24:59Z

On February 2^nd, a league of extraordinary gentlemen gathered inside a crowded chamber for the “Ceph Lords” OpenStack SF Meetup. About 75 Stackers were in attendance making this event a smashing success!

If you missed this Meetup, then you should watch the recorded presentations in order to experience the lively discourse. Also, check out the photos that were taken by the Piston Cloud crew. They did a good job capturing the fun, relaxed ambiance of the gathering.

The Meetup was facilitated by Piston Cloud and hosted by DreamHost. Scrumptious tacos were catered by Tacolicious, and Magnolia Brewery provided a keg of ice-cold California Kolsh beer.

Ceph was in the spotlight throughout the Meetup. For those few who are still unfamiliar with Ceph, it’s a massively scalable, open source, distributed storage system. The presenters focused primarily on how Ceph works with the cloud software stack, and how it’s currently being implemented in production.

The DreamHost team kicked off the event. Ben Cherian provided a brief overview on the reasons why DreamHost chose to use Ceph as the storage foundation for their current hosting products and upcoming cloud services.

Tommi Virtanen dived into the Ceph platform from a technical perpective. He described the major components of Ceph, its storage architecture, and how it distributes data.

Carl Perry talked about how DreamHost is currently deploying Ceph. He touched on specifics such as the hardware and tools involved, the level of automation, and things learned along the way.

Christopher MacGown, Co-Founder and CTO of Piston Cloud, was the final presenter. He opened with how storage should work in a cloud environment and why Piston Cloud chose Ceph as the backend storage solution. He finished by describing how his company is using Ceph with the Piston Enterprise OS™ software.

FOSDEM 2012 feedback

2012-02-08T14:49:26Z

I’m back from Brussels, where happened the coldest FOSDEM ever. It started on Friday night with the traditional beer event. Since the Delirium was a bit small to host those thousands of frozen geeks, the FOSDEM organizers had enlisted the whole block as approved bars !

On the Saturday, I spent most of my time in the Cloud and Virtualization devroom, which I escaped only to see Simon Phipps announce the new membership-based OSI, and Paolo Bonzini talking about the KVM ecosystem (in a not technical enough way, IMO). My own OpenStack talk was made a bit difficult due to the absence of mike to cover the 550-seat Chavanne auditorium… but the next talks got one. The highlight of the day was Ryan Lane’s “infrastructure as an open source project” presentation, about how Wikimedia Labs uses Git, Gerrit, Jenkins and OpenStack to handle its infrastructure like a contributor-driven open source project. The day ended with a good and frank discussion between OpenStack developers, with upstream projects and downstream distributions.

On Sunday I tried to hop between devrooms, but in a lot of cases the room was full and I couldn’t enter, so I spent more time in the hallway track. I enjoyed Soren’s talk about using more prediction algorithms (instead of simple thresholds) in monitoring systems, introducing his Surveilr project. The highlight of the day was Dan Berrangé’s talk about using libvirt to run sandboxed applications, using virt-sandbox. There are quite a few interesting uses for this, and the performance penalty sounds more than acceptable.

Overall it was a great pleasure for me to attend FOSDEM this year. Congratulations to the organizers again. I’ll be back next year, hopefully it will be warmer

Automating Openstack Testing on Ubuntu

2012-02-08T14:44:31Z

(Original Post)

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.

The scope of this work was:

Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

Juju – used to deploy Openstack charms in the Lab
Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
Schroots for Oneiric and Precise for building packages locally
A reprepro managed local archive for Oneiric and Precise
Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch. It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases. After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle. To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation. The process is as follows:

Test run begins
Juju deploys a service (i.e. nova-compute)
A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
The next test run begins. All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root. If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel. With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets. The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot. This should minimize the test cycle even more. Credit to James Blair for the idea (see http://amo-probos.org/post/11 ).

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab. Jenkins (using the git plugin) monitors the upstream github.com repositories for commits on the master branch. When a change is detected the following process is triggered:

Build

Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev//essex
Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing//essex
New changelog entries are automatically created for the new upstream commits.
The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
The testing packaging branch is push back to lp:~openstack-ubuntu-testing//essex.
The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (tarball.sh); Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.

Deploy

Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.

Test

Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud. This testing is able to help us achieve multiple goals:

Early detection of upstream bugs that break Openstack functionality on Ubuntu
Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
- Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
- Upload an image into the Glance server for use during testing
- Creates EC2 credentials in the Keystone server for use during testing.
Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
- Basic euca-tools EC2 API for starting and stopping instances
- EC2 AMI bundle uploads
- Floating IP allocation, association and connectivity to instance
- Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.

Reporting

The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to http://jenkins.qa.ubuntu.com so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu. Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release. We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784. In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit testing of changes to stable branches; The Ubuntu Server team are working upstream on maintaining the stable branches of released versions of OpenStack – this work will validate patches proposed to stable branches in review.openstack.org against the current version of the packaging in released versions of Ubuntu. Initially this will target Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once released. Ideally the testing process will provide feedback on review.openstack.org to help the stable release team review proposed patches.

References

Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc: https://code.launchpad.net/~openstack-ubuntu-testing/

Credits

Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

Automating Openstack Testing on Ubuntu

2012-02-08T10:29:09Z

Reblogged from JavaCruft:

During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu. The scope of this work was: Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack. SRU testing for Openstack Diablo on Ubuntu 11.10. Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we …

OpenStack Party @ CloudConnect 2012

2012-02-07T23:08:44Z

For those attending CloudConnect 2012 in Santa Clara – Join stackers from all over the world at the OpenStack CloudConnect 2012 party at Fahrenheit Lounge, hosted by Mirantis, Rackspace and Cloudscaling.

Open bar, Hors D’oeuvres and music all night long. This is the place to be at CloudConnect on a Wednesday night.

We’ll have shuttle buses available every 30 minutes, traveling between Santa Clara Convention center parking lot and Fahrenheit Lounge starting at 8pm, immediately after the Cloudscaling cocktail reception.

Registration is first come, first serve and the space is limited. Visit openstackparty.eventbrite.com to register.

OpenStack Talk hosted by the Computer Society of India Pune Chapter

2012-02-07T15:49:13Z

This is a guest post from Devdatta Kulkarni. Thanks Dev for sharing!

The Computer Society of India (CSI) Pune chapter organized an OpenStack talk with me, Racker Devdatta Kulkarni, on Saturday January 21, 2012 from 5.00 pm – 6.30 pm.

Approximately 35 people attended. The audience primarily consisted of people with a technical background. Technology professionals were the most represented category followed by college students, followed by researchers.

I divided my talk into two parts. In the first part, I touched upon the need for OpenStack, the project’s history and mission, and the current projects. In the second part I delved deeper into design and architectures of Nova, Swift, Glance, and Keystone, and concluded with information about how to participate in the community.

At the end of the talk I did a quick show of hands to find out how many attendees knew about OpenStack prior to the talk. Given that I saw only three hands in response, I think the talk certainly helped in raising the awareness of OpenStack within the technical community in Pune.

Here are some of the questions that came up at the talk. Anne Gentle wrote the answers for the questions and I want to share with the attendees as well as OpenStack blog readers.
Question 1) Performance benchmarks of OpenStack deployments. They have experimented with deployment of about 200 VMs and were seeing average VM creation time of about 20 minutes. They wanted to know if this was something expected. Also, they were wondering if there are any OpenStack performance benchmark results that can be shared with the community.
Anne: A 20 minute wait sounds like a long time to me for a single VM but a short time for 200 Vms. We haven’t found a good way to share performance benchmarks yet but a post to the mailing list would probably elicit responses. I’ve also seen John Dickinson talk to folks on IRC about their Object Storage benchmarks.

Question 2) Guidelines on topology. They wanted to know if there are any published guidelines regarding the optimal topology, such as number of glance servers, number of compute, volume, and network nodes in Nova deployments?
Anne: I’d recommend they take a look at http://referencearchitecture.org for both physical and logical architecture diagrams that show the number of servers and how to scale out a deployment.

Question 3) Active Directory support in Keystone. Is this being discussed within the Keystone working group?
Anne: It’s often discussed but no one has stepped up to write an AD plugin for Keystone yet that I know of.

Question 4) Is there a QEMU-based development environment for OpenStack?
Anne: Try out http://devstack.org and if you run it in a VM, it’ll use QEMU.

Question 5) Can you give pointers to learning material?
Anne: Each of the projects has a development docs site (nova.openstack.org, glance.openstack.org, swift.openstack.org, and so on). You’ll find API and admin docs at docs.openstack.org.

How does a PUT to a swift object server look like.

2012-02-06T10:00:33Z

I have been trying lately to get a better understanding of the Swift code base, and I found the best way to know it was to read it from top to bottom and document it along the way. Here is some of my notes, hopefully more will come.

I am starting with an object PUT when the request is coming from the proxy server. The request in the log-file will look like this :

“PUT /sdb1/2/AUTH_dcbeb7f1271d4374b951954a4f1be15f/foo/file.txt” 201 – “-” “txdw08eca2842e344bb8e11b5869c81cb52″ “-” 0.0308

The WSGI controller send the request to the method swift.obj.server.ObjectController->PUT and start to do the following :

splits the request.path to :

device(sdb1), partition(2), account(AUTH_ACCOUNT_ID), container(foo), obj(file.txt)

Make sure that partition is mounted. (there is a mount_check option that can toggle this).
Ensure that there is a X-Timestamp header which should be set by the proxy server.
Start the check method check_object_creation which does the following :

Make sure the content_length is not greater than the MAX_FILE_SIZE.
Make sure there is a content_length header (except if the transfer has been chunked).
Make sure that there is no content_length (ie: zero byte body) when doing a X-Copy-From.
Make sure the object_name is not greater than MAX_OBJECT_NAME_LENGTH (1024 bytes by default).
Making sure we have a Content-Type in the headers passed (this could be set by the user or auto-guessed via mimetypes.guess_type on the proxy server).
When we have an header of x-object-manifest (for large files support) it makes sure the value is a container/object style and not contain chars like ? & / in the referenced objects names.
Checks metadata, make sure at first that the metadata name are not empty.
The metadata name length are not greater than MAX_META_NAME_LENGTH (default: 128).
The metadata value is not greater than MAX_META_VALUE_LENGTH (default: 256).
We don’t have a greater amount of metadatas than MAX_META_COUNT (default: 90).
The size of the headers combined (name+value) is not over MAX_META_OVERALL_SIZE (default: 4096).
If we have ‘X-Delete-At‘ (for the object expiration feature) we are making sure this is not happening in the past or we will exit with an HTTPBadRequest.
The class swift.obj.server.DiskFile will be the class that takes care to actually write the file locally. It gets instantiated and do the following in the constructor method:
It will hash the following value (account, container, obj) which will become hashed for our example into :

46acec4563797178df9ec79b28146fe1

It will get the path where this is going to be store which going to be :

/srv/node/sdb1/objects/2/fe1/46acec4563797178df9ec79b28146fe1

/srv/node is the devices path which is the configuration directive [proxy]->devices (default to /srv/node).
sdb1 being the mounted device name.
add the datadir type, ”objects” for us.
and the partition power (2)
last three chars of the hashed name (fe1)
the hash itself 46acec4563797178df9ec79b28146fe1
It will get the temporary directory which become in our case to: /srv/node/sdb1/tmp it is basically the devices dir, the device and /tmp
If the directory didn’t exists before then it just return.
If the directory was existing (already uploaded) then it will parse all files in there and would looks if we have :
Files ending up with .ts which will be the tombstone (a deleted file). NB: Replication process will take care to os.unlink() the file properly later.
In case of a POST and if we have fast post setting enabled (see config object_post_as_copy in proxy_server) we will detect it and only do a copy of metadata.
It calculates the expiration time which is from now + the max_upload_time setting.
It start the etag hashing to gradually calculate the md5 of the object.
Using the method mkstemp of DiskFile it will start to write to tmpdir, which does the creation of the file like that :
Make sure to create the tmpdir.
make a secure temporary file (using mkstemp(3)) and yield the file descriptor back to PUT.
If there is a content-length in the headers (assigned by the client) it will use the posix function fallocate(2) to pre-allocate that disk space to the file descriptor.
It will then iterate over chunk of data size defined by the configuration variable network_chunk_size (default: 64m) reading that chunk from the request wsgi.input :
It will update the upload_size value.
It will make sure we are not going over our upload expiration time (or get back HTTPRequestTimeout HTTP Error).
It will update the calculated md5 with that chunk.
It will write the chunk using python os.write
For large file sync which is over the configuration variable bytes_per_sync it will do a fdatasync(2) and drop the kernel buffer caches (so we are not filling up too much the kernel memory).
if we have a content-length in the client headers that doesn’t match the calculated upload_size we return a 499 Client Disconnected as it means we had a problem somewhere during the upload.
It will bail out if we have a etag in the client headers that doesn’t match the calculated etag.

And now we are starting defining our metadatas that we are going to store with the file :

metadata = {
‘X-Timestamp’: timestamp generated from the proxy_server.
‘Content-Type‘: defined by the user or ‘guessed’ by the proxy server
‘ETag‘: calculated value from the request.
‘Content-Length‘: an fstat(2) on the file to get the proper value of what is stored on the disk.
}

It will add to the metadata every headers starting by ‘x-object-meta-’.
It will add to the metadata the allowed headers to be stored which is defined in the config variable allowed_headers (default: allowed_headers = Content-Disposition, Content-Encoding, X-Delete-At, X-Object-Manifest).
It will write the file using the put method of the DiskFile class, which finalise the writing on the file on disk and renames it from the temp file to the real location:
It will write the metadata using the xattr(1) feature which is stored directly with the file.
If there is a Content-Length with the metada it will drop the kernel cache of that metadata length.
It will invalidate the hashes of the datadir directory using the function swift.obj.replicator.invalidate_hashes
It will set the hash of the dir as None, which would hint the replication process to have something to do with that dir (and that hash will be generated).
This file is stored by partition as python pickle which is in our case: /srv/node/sdb1/objects/2/hashes.pkl
Move the file from the tmp dir to go to the datadir.
It will use the method unlinkold from DiskFile to remove any older versions of the object file which is any files that has older timestamp.
It will start construct the request to make to a containers by going passing the following:
account, container, obj as request path.
the original headers.
the headers Content-Length, Content-Type, X-Timestamp, Etag, X-trans-ID.
It will get the headers X-Container-{Host,Partition,Device} from the original headers which is defined by the proxy to know on which container server it going to update. Every different PUT will have assigned a different container to each their own.
It will use the async_update method (by self since it’s part of the same class) to make an asynchronous request:
Passing the aforementioned build headers and req.path.
If the request success (between 200 to 300) it will return to the main (PUT) method.
the request didn’t succeed it will create a async_pending file locally in the tmp dir which is going to be picked-up by the replication process to update the container listing when the container is not too busy.
When finish it will respond by a HTTPCreated

Wow, qemu-img is fast

2012-02-04T01:16:00Z

I wanted to determine if its worth putting ephemeral images into the libvirt cache at all. How expensive are these images to create? They don't need to come from the image service, so it can't be too bad, right? It turns out that qemu-img is very very fast at creating these images, based on the very small data set of my laptop with an ext4 file system...

mikal@x220:/data/temp$ time qemu-img create -f raw disk 10g
Formatting 'disk', fmt=raw size=10737418240 

real	0m0.315s
user	0m0.000s
sys	0m0.004s

mikal@x220:/data/temp$ time qemu-img create -f raw disk 100g
Formatting 'disk', fmt=raw size=107374182400 

real	0m0.004s
user	0m0.000s
sys	0m0.000s

Perhaps this is because I am using ext4, which does funky extents things when allocating blocks. However, the only ext3 file system I could find at my place is my off site backup disks, which are USB3 attached instead of the SATA2 that my laptop uses. Here's the number from there:

$ time qemu-img create -f raw disk 100g
Formatting 'disk', fmt=raw size=107374182400 

real	0m0.055s
user	0m0.000s
sys	0m0.004s

So still very very fast. Perhaps its the mkfs that's slow? Here's a run of creating a ext4 file system inside that 100gb file I just made on my laptop:

$ time mkfs.ext4 disk 
mke2fs 1.41.14 (22-Dec-2010)
disk is not a block special device.
Proceed anyway? (y,n) y
warning: Unable to get device geometry for disk
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
6553600 inodes, 26214400 blocks
1310720 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
800 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

real	0m4.083s
user	0m0.096s
sys	0m0.136s

That time includes the time it took me to hit the 'y' key, as I couldn't immediately find a flag to stop prompting.

In conclusion, there is nothing slow here. I don't see why we'd want to cache ephemeral disks and use copy on write for them at all. Its very cheap to just create a new one each time, and it makes the code much simpler.

Tags for this post: openstack qemu ephemeral mkfs swap speed canonical
Related posts: Further adventures with base images in OpenStack; Openstack compute node cleanup; Taking over a launch pad project; Speed limit; Slow git review uploads?; My machine was thrashing a lot; Large inodes = faster samba; Are you in a LUG? Do you want some promotional materials for LCA 2013?; Announcement video; linux.conf.au Returns to Canberra in 2013; The next thing; Linux USB quandary

Comment

Slow git review uploads?

2012-02-03T00:53:00Z

jeblair was kind enough to help me debug my problem with slow "git review" uploads for Openstack projects just now. It turns out that part of my standard configuration for ssh is to enable ControlMaster and ControlPersist. I mostly do this because the machines I use at Canonical are a very long way away from my home in Australia, and its nice to have slightly faster connections when you ssh to a machine. However, gerrit is incompatible with these options as best as we can tell.

So, if your git reviews are taking 10 to 20 minutes to upload like mine were, check that you're not using persistent connections. Excluding review.openstack.org from that part of my configuration has made a massive difference to the speed of uploads for me.

Tags for this post: openstack git review gerrit ssh canonical
Related posts: More reviews; Book reviews; Taking over a launch pad project; Working on review comments for Chapters 2, 3 and 4 tonight; A ssh quickie; Review; Andrew's SSH filtering causes me pain; Further adventures with base images in OpenStack; Twisted conch; The Wild Palms Hotel; Wow, qemu-img is fast; Status of the book; Are you in a LUG? Do you want some promotional materials for LCA 2013?; Announcement video; linux.conf.au Returns to Canberra in 2013; The next thing; clusterssh; Openstack compute node cleanup

Comment

About collaboration

2012-02-01T17:38:59Z

In recent years, as open source becomes more ubiquitous, I’ve seen a new breed of participants appearing. They push their code to GitHub like you would wear a very visible good behavior marketing badge. They copy code from multiple open source projects, modify it, but don’t contribute back their changes to upstream. They seem to consider open source as a trendy all-you-can-eat buffet combined with a cool marketing gimmick.

In my opinion, this is not what open source is about. I see open source, and more generally open innovation (which adds open design, open development and open community), as a solution for the future. The world is facing economical and ecological limits: it needs to stop designing for obsolescence, produce smarter, reduce duplication of effort, and fix the rift between consumers and producers. Open innovation encourages synergy and collaboration. It reduces waste. It enables consumers to be producers again. That’s a noble goal, but without convergence, we can’t succeed.

The behavior of these new participants goes against that. I call this the GitHub effect: you encourage access to the code, forking and fragmentation, while you should encourage convergence and collaboration on a key repository. And like having a “packaging made from recyclable materials” sign on your product doesn’t make it environment-friendly, just publishing your own code somewhere under an open source license doesn’t really make it open.

On the extreme fringe of that movement, we also see the line with closed source blurring. Building your own closed product on top of open source technology, and/or abusing the word “Open” to imply that all you do is open source, using the uncertainty to reap easy marketing benefits. I’ve even seen a currently-closed-source project being featured as an open source project to watch in 2012. We probably need to start playing harder, denounce fake participants and celebrate good ones.

Some people tell me my view goes against making money with open source. That might be true for easy, short-term money. But I don’t think you need to abuse open source to make money out of it. The long-term benefits of open innovation are obvious, and like for green businesses, good behavior and long-term profit go well together. Let’s all make sure we encourage collaboration and promote the good behavior, and hopefully we’ll fix this.

Audit a swift cluster

2012-02-01T10:18:11Z

Swift integrity tools.

There is quite a bit of tools shipped with Swift to ensure you have the right object on your cluster.

At first there is the basic :

swift-object-info

It will take a swift object stored on the filesystem and print some infos about it, like this :

swift@storage01:0/016/0b221bab535ac1b8f0d91e394f225016$ swift-object-info 1327991417.01411.data
Path: /AUTH_root/foobar/file.txt
Account: AUTH_root
Container: foobar
Object: file.txt
Object hash: 0b221bab535ac1b8f0d91e394f225016
Ring locations:
192.168.254.12:6000 – /srv/node/sdb1/objects/0/016/0b221bab535ac1b8f0d91e394f225016/1327991417.01411.data
Content-Type: text/plain
Timestamp: 2012-01-31 06:30:17.014110 (1327991417.01411)
ETag: 053a0f8516a5023b9af76c49ca917d3e (valid)
Content-Length: 24 (valid)
User Metadata: {‘X-Object-Meta-Mtime’: ’1327968327.21′}

PS: If you don’t know where is your object on which node, you can you use swift-get-nodes

For auditing, the Etag value is important because swift-object-info will compare the object recorded etag in the metadata with what we have on the disks. Let’s try to see if that works :

swift@storage01:0/016/0b221bab535ac1b8f0d91e394f225016$ cp 1327991417.01411.data /tmp
swift@storage01:0/016/0b221bab535ac1b8f0d91e394f225016$ echo “foo” >> 1327991417.01411.data
swift@storage01:0/016/0b221bab535ac1b8f0d91e394f225016$ swift-object-info 1327991417.01411.data|grep ‘^Etag’
Etag: 053a0f8516a5023b9af76c49ca917d3e doesn’t match file hash of 9ff871e5ce5dcb5d3f2680a80a88ff38!

swift-object-info has detected that this file is not the one we have uploaded.

There is an other tool called swift-drive-audit which as explained in the admin guide will parse the /var/log/kern.log and have predefined regexp to detect disk failure notified by the kernel. It is usually run periodically by cron and there is a config file for it called /etc/swift/drive-audit.conf. If the script find any errors for a certain drive it will unmount it and comment it in /etc/fstab(5). Afterwards the replication process will pick it up from other replicas and put the object on that drive in handover.

Swift provide as well different type of auditor daemons for account/container/object :

swift-account-auditor
swift-container-auditor
swift-object-auditor

swift-account-auditor will open all sqlite db of an account server and launch a SQL query to make sure all the dbs are valid.
swift-container-auditor will do the same but for containers.
swift-object-auditor will open all object of an object server and make sure of :

Metadata are correct.
We have the proper size.
We have the proper MD5.

Those auditors needs to be set in each type-server.conf, for example for account server you will add something like this to /etc/swift/account-server.conf :

[account-auditor]
# You can override the default log routing for this app here (don’t use set!):
# log_name = account-auditor
# log_facility = LOG_LOCAL0
# log_level = INFO
# Will audit, at most, 1 account per device per interval
interval = 1800
# log_facility = LOG_LOCAL0
# log_level = INFO

For container this is about the same options but for object-server does are the options :

[object-auditor]
# You can override the default log routing for this app here (don’t use set!):
# log_name = object-auditor
# log_facility = LOG_LOCAL0
# log_level = INFO
# files_per_second = 20
# bytes_per_second = 10000000
# log_time = 3600
# zero_byte_files_per_second = 50

Another tool shipped with swift is swift-account-audit which will audit a full account and report if there is missing replicas or incorrect object in that account.

OpenStack Design Summit & Conference Updates

2012-01-30T22:25:20Z

We’re making progress on the next OpenStack Design Summit (April 16-18) & Conference (April 19-20) at the Hyatt Regency San Francisco — one week, two events.

Hotel Rooms
We have a discounted hotel room block at the Hyatt under OpenStack, which is now available to book. Please make sure to designate yourself with the OpenStack Design Summit & Conference when booking.

Sponsorship Prospectus
Sponsorships are going fast, and the prospectus is available to download at the Conference website. There are a limited number of opportunities at the top levels, which are first come, first serve, with a signed agreement. If you have any questions about the prospectus, please contact sponsors@openstack.org.

Speakers & OpenStack Demo Session
We’ve also opened the OpenStack Conference call for speakers. We need your help to build out an informative and compelling agenda, including user stories, technical advancements, best practices and visions on the future of OpenStack. New to this Conference, we are also planning an OpenStack demo session, a chance for companies building products around OpenStack to present in front of the community and a panel of judges. The deadline to submit speaking sessions is February 15, and more details and deadlines for the OpenStack demo session will be announced shortly.

As a reminder, the OpenStack Design Summit is made up of working sessions for developers contributing to OpenStack. The OpenStack Conference reaches a broader audience, including users and the business ecosystem, in addition to the OpenStack technical community. Because the events are co-located, the sponsorship prospectus and hotel room block cover both events, but the call for papers is strictly for the OpenStack Conference. The Design Summit sessions and schedule will be determined by blueprint submissions, the Project Technical Leads and Release Manager.

We encourage you to make travel arrangements for the April events, and registration will open shortly. Look forward to seeing everyone in San Francisco!

Community Weekly Review (Jan 20-27)

2012-01-27T21:30:06Z

OpenStack Community Newsletter –January 27, 2012

HIGHLIGHTS

OpenStack Bug Squashing Day Is Coming On Feb 2nd http://www.openstack.org/blog/2012/01/the-first-openstack-bug-squashing-day-is-coming-on-feb-2nd/ and you can follow its evolution on http://wiki.openstack.org/bugstats/
Meet OpenStack developers at FOSDEM http://fnords.wordpress.com/2012/01/27/openstack-developers-meeting-at-fosdem/
OpenStack Jenkins dashboard available for testing Ubuntu snapshots http://www.openstack.org/blog/2012/01/openstack-jenkins-dashboard-available-for-testing-ubuntu-snapshots/
GrydDynamics team contributed a billing plugin for Horizon https://openstackgd.wordpress.com/2012/01/24/billing-plugin-for-horizon
OpenStack Australian community had another successful meetup http://www.openstack.org/blog/2012/01/openstack-melbourne-australia-meetup-jan-17/
Opened the Call for speaker for OpenStack Conference http://www.openstack.org/conference/san-francisco-2012/speakers/
Book your room for the week https://resweb.passkey.com/go/DRAC2012

EVENTS

FOSDEM 2012 Feb 04 – 05, 2012 – Bruxelles, BE http://fosdem.org/2012/schedule/track/virtualization_and_cloud_devroom
OpenStack Bug Squash Day San Francisco Feb 02, 2012 – Rackspace San Francisco, 620 Folsom, San Francisco, CA http://www.meetup.com/openstack/events/48362422/
OpenStack Bug Squash Day Austin Feb 02, 2012 – Rackspace Austin 2420 Ridgepoint Dr, Austin, TX http://www.meetup.com/OpenStack-Austin/events/48406252/
“Ceph Lords” San Francisco Stackers Free Event! Feb 02, 2012 – DreamHost San Francisco, 221 Pine St [2nd Floor] http://www.meetup.com/openstack/events/48829962/
Inaugural OpenStack DC Meetup Feb 16, 2012 – Washington, DC http://www.meetup.com/OpenStackDC/events/47106902/
FLOSSTalk Kosovo Feb 22, 2012 – UNICEF Innovations Lab Kosovo in Prishtina, Kosovo http://www.flossk.org/en/blog/flosstalk-arturo-suarez-openstack
OpenStack Spring 2012 Design Summit Apr 16 – 18 and Conference Apr 19-20 – San Francisco, California http://openstack.org/conference/

OTHER NEWS

Three (and more) things about documentation https://lists.launchpad.net/openstack/msg07082.html
Cleaning up code after Essex feature-freeze https://lists.launchpad.net/openstack/msg07065.html
Announcing a new and improved Keystone https://lists.launchpad.net/openstack/msg07063.html
Ubuntu package / ppa for the git-review tool https://lists.launchpad.net/openstack/msg07061.html
Essex-3 milestone available for Keystone, Glance, Nova and Horizon https://lists.launchpad.net/openstack/msg06989.html
Nexenta SAN volume driver for Nova https://lists.launchpad.net/openstack/msg06961.html
OpenStack Wiki Recent Changes – http://wiki.openstack.org/RecentChanges

MultiRealm Keystone http://wiki.openstack.org/MultiRealmKeystone
Smokestack http://wiki.openstack.org/smokestack
OpenStack Design Summit http://wiki.openstack.org/Summit

Project meeting summary http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-24-21.01.html

COMMUNITY STATISTICS

We’re working to improve the community stats. We hope to be back next week.

OpenStack developers meeting at FOSDEM

2012-01-27T15:54:28Z

Next week, the European free and open source software developers will converge to Brussels for FOSDEM. We took this opportunity to apply for an OpenStack developers gathering in the Virtualization and Cloud devroom.

At 6pm on Saturday (last session of the day), in the Chavanne room, we will have a one-hour town hall meeting. If you’re an existing OpenStack contributor, a developer considering to join us, an upstream project developer, a downstream distribution packager, or just curious about OpenStack, you’re welcome to join us ! I’ll be there, Stefano Maffulli (our community manager) will be there, and several OpenStack core developers will be there.

We’ll openly discuss issues and solutions about integration with upstream projects, packaging, governance, development processes, community or release cycles. In particular, we’ll have a distribution panel where every OpenStack distribution will be able to explain how they support OpenStack and discuss what we can improve to make things better for them.

And at the end of the session we can informally continue the discussion around fine Belgian beers or their famous Carbonade !

OpenStack Jenkins dashboard available for testing Ubuntu snapshots

2012-01-27T13:50:36Z

The keener eyed of you may have noticed:

https://jenkins.qa.ubuntu.com/view/Precise%20OpenStack%20Testing/

James Page has setup the jobs in the Ubuntu OpenStack QA Lab to start publishing to the public Jenkins QA instance this morning. We now have automated build testing of all core OpenStack components triggered from upstream trunk commits. This is followed by automated deployment (-deploy) of OpenStack in the lab with a serving of testing (-test) once its all up and running.

Credit to Adam Gandelman for the Juju charm work, deployment framework and test execution and to Chuck Short for the hugely misnamed tarball.sh script which completes the git/bzr/packaging fu to build and deploy OpenStack packages!

The plan is to get the upstream Tempest test suite running in the lab; at the moment we are running a more limited test script just to ensure that you can spin up and instance and see it on the network.

(Crossposted from cloud.ubuntu.com)

Billing plugin for Horizon

2012-01-24T11:38:06Z

Now nova-billing has a Django-based web interface – horizon-billing (https://github.com/griddynamics/horizon-billing).

horizon-billing is packaged to a homonymous RPM (path for RHEL: http://yum.griddynamics.net/yum/diablo/, for CentOS: http://yum.griddynamics.net/yum/diablo-centos/).

To enable it in dashboard, install the horizon_billing package and turn it
on in /etc/openstack-dashboard/local/local_settings.py:

add ‘horizon_billing’ to INSTALLED_APPS tuple;

add ‘billing’ to ‘dashboards’ key in HORIZON_CONFIG.

After installation, a new “Billing” panel is added after “Project” and “Admin”.

Grid Dynamics’ GitHub repos are renamed

2012-01-24T11:27:03Z

In order to unify our repo names with openstack GitHub origanization, “osc-robot-” prefix is removed.

So, these repos are renamed:

osc-robot-keystone;

osc-robot-nova;

osc-robot-openstackx;

osc-robot-glance;

osc-robot-swift;

osc-robot-openstack-compute;

osc-robot-noVNC;

osc-robot-python-novaclient.

Also, we add python-keystoneclient repo (forked from openstack/python-keystoneclient) that contains patches for RedHat packaging.

Looking for a Few Good Engineers

2012-01-23T20:21:51Z

Do you know Python? Do you get a thrill breaking other people’s code? Do you have experience with Chef, Puppet, Cobbler, Orchestra, or Jenkins? Have you ever deployed or worked on highly distributed systems? Do you understand virtualization technologies like KVM, Xen, ESX or Hyper-V?

If you answered “Yes!” to any of the questions above and are interested in working in a distributed, high-energy engineering team on solving complex problems with cloud infrastructure software, I want to hear from you. Experience with OpenStack is a huge plus.

I’m looking for QA software engineers, software deployment and/or automation engineers and software developers that can hit the ground running and make a big impact from Day One. Feel free to email me at REVERSE('moc.liamg@sepipyaj').

OpenStack Melbourne Australia Meetup Jan 17

2012-01-22T11:39:32Z

On Tuesday January 17 at the Exchange Hotel in Melbourne the inaugural Australian OpenStack Users Group meetup Part 2 took place. This followed up on the Sydney event last month and took the same format, being a casual informal get together for some drinks and conversation focused on OpenStack. We kicked off around 6pm and had an attendance of around 45 OzStackers. Many many thanks to everyone that came along!

Once again we had our attending vendors present a short overview of their company’s involvement in the project. The speakers were Mark Randall, Rackspace Country Manager for AU/NZ, Daniel Pendlebury, Citrix Lead Systems Engineer for Datacenter and Cloud, Gavin Coulthard, Manager – Field Systems Engineering A/NZ at F5, Peter Jung, Cloud Solutions Architect at Dell, and Andrew White, Data Centre Architect from Cisco. Following the vendors, an awesome contribution to the evening came from Dr Steven Manos, ITS Research Director from the University of Melbourne, who presented an overview of the NeCTAR project. Rounding out the talks again was Phil Rogers from Aptira.

Again as in Sydney, there was a great sense of community, lots of smiles and laughter and much conversation and enthusiasm to share information and experiences. As social events go, both this and the Sydney events have been very successful, the next round of meetups scheduled for early March will see us presenting a more structured meetup schedule with a focus on technical, with demos and the like.

Head to our Australian Meetup group to get involved, or join the AU Google group.

Community Weekly Review (Jan 13 – 20)

2012-01-20T22:41:32Z

OpenStack Community Newsletter –January 20, 2012

HIGHLIGHTS

The First OpenStack Bug Squashing Day Is Coming On Feb 2nd http://www.openstack.org/blog/2012/01/the-first-openstack-bug-squashing-day-is-coming-on-feb-2nd/

Making more solid OpenStack releases http://fnords.wordpress.com/2012/01/18/making-more-solid-openstack-releases/

Performance testing tool for Glance https://lists.launchpad.net/openstack/msg06724.html

OpenStack Nova and Glance 2011.3.1 released https://lists.launchpad.net/openstack/msg06834.html

GridDynamics released “nova-billing”, resource usage accounting system for OpenStack http://openstackgd.wordpress.com/2012/01/19/resource-usage-accounting-system-for-openstack/

Dan Prince and Russell Bryant giving an introduction to OpenStack ad FUDCon http://alt.fedoraproject.org/pub/alt/videos/2012/fudcon-blacksburg/openstack.ogg

EVENTS

Southern California Linux Expo SCALE10x, Jan 20-22 – Los Angeles, CA http://www.socallinuxexpo.org/scale10x/presentations/state-openstack

Lecture on OpenStack: Open source cloud computing Jan 21, 2012 – Pune, India (no URL: email me for details)

FOSDEM 2012 Feb 04 – 05, 2012 – Bruxelles, BE http://fosdem.org/2012/schedule/track/virtualization_and_cloud_devroom

Inaugural OpenStack DC Meetup Feb 16, 2012 – Washington, DC http://www.meetup.com/OpenStackDC/events/47106902/

OpenStack Spring 2012 Conference Apr 16 – 20, 2012 – San Francisco, California http://openstack.org/conference/

OTHER NEWS

OpenStack Community picks “Folsom” as the official name of next release https://lists.launchpad.net/openstack/msg06785.html

OpenStack Wiki Recent Changes – http://wiki.openstack.org/RecentChanges

OpenStack Academic Initiative http://wiki.openstack.org/AcademicInitiative

Puppet configuration for Nova http://wiki.openstack.org/PuppetConfigForNova

Team meeting summary http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-17-21.01.html

COMMUNITY STATISTICS

We’re taking a break from producing the graphs this week. We hope to be back next week showing more precise information from commits to the master branch of OpenStack projects.

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

Diagnose and fix PEP8 issues during code review

2012-01-20T19:13:25Z

I figured I’d write a quick post about how to deal with “pep8 issues” that come up during code reviews on OpenStack core projects. These issues come up often for new contributors, and it can be a source of frustration until the contributor understands how to diagnose and fix the issues that come up.

PEP8 is the Python PEP that deals with a recommended code style. All core (and periphery Python) OpenStack projects validate that new code pushed to the source tree is “pep8-compliant”. When a new patchset is pushed from code review to Jenkins for the set of automated pre-merge tests, the pep8 command-line tool is run against the new source tree to ensure it meets PEP8 code style standards.

If this PEP8 Jenkins job fails, the code submitter will see a notification that the job failed, and the contributor must fix up any pep8 issues and push those fixes up for review again. Typically, this notification looks something like this:

Change subject: Added Keypair extension (os-keypairs) client and tests LP#900139 ...................................................................... Patch Set 2: I would prefer that you didn't submit this Build Unstable https://jenkins.openstack.org/job/gate-tempest-pep8/38/ : UNSTABLE https://jenkins.openstack.org/job/gate-tempest-merge/78/ : SUCCESS -- To view, visit https://review.openstack.org/3179 To unsubscribe, visit https://review.openstack.org/settings Gerrit-MessageType: comment Gerrit-Change-Id: I34c7e9aa6a1796b8d4c3ac9b3b69438796752866 Gerrit-PatchSet: 2 Gerrit-Project: openstack/tempest Gerrit-Branch: master Gerrit-Owner: kavan-patil Gerrit-Reviewer: Brian Waldon Gerrit-Reviewer: Jay Pipes Gerrit-Reviewer: Jenkins Gerrit-Reviewer: kavan-patil

There are a couple ways you can diagnose what style points your code violated. Probably the easiest and fastest is to just follow the link in the notification email to the Jenkins job. Clicking the link above, I get to the Jenkins job page, which looks like this:

Clicking on the graph, I get to a details screen showing the source files and lines of code that violated pep8 rules:

I can then go to line 86 of tempest/openstack.py and investigate the code style

Alternately, I could run the pep8 CLI tool on my local branch, which will tell me the pep8 violations and what to fix, as shown here:

jpipes@uberbox:~/repos/tempest$ pep8 --repeat tempest tempest/openstack.py:86:73: W292 no newline at end of file

There we are… the tempest/openstack.py file doesn’t end with a newline. An easy fixup.

Using LVM as disk storage for OpenStack

2012-01-20T10:19:15Z

We have added ability to use LVM volumes for instances disks in OpenStack Nova according to the blueprint.

You can specify which storage type you want to use in nova.conf:

–local_images_type={raw, qcow, lvm, legacy}

legacy value (used by default) was added for backward compatibility. In case of legacy storage type will be controlled by old value:

–use_cow_images={True, False}

If you use other values, use_cow_images flag will be ignored. With lvm type you should specify LVM group to use for VM storage with lvm_volume_group flag:

–lvm_volume_group=

From point of user there is no difference at all between LVM and other storage types. He can do all usual operations on instance: reboot, migrate, resize, snapshot. One difference is that we disable snapshotting of running instance and add force_snapshot option to snapshot_instance method of OpenStack API to override this behaviour.

Code: https://github.com/griddynamics/osc-robot-nova/commit/a1cd719481ff76b871d58bc3d5bf683850dc4a04

RHEL repository: http://yum.griddynamics.net/yum/diablo

CentOS repository: http://yum.griddynamics.net/yum/diablo-centos

The Science (or Art?) of Commit Messages

2012-01-19T19:26:22Z

There are some things in the world of development that you appreciate much more when you do a lot of code reviews. One of those things is commit messages.

At first glance, commit messages seem to be a small, relatively innocuous thing for a developer. When you commit code, you type in some description about your code changes and then, typically, push your code somewhere for review by someone.

Regardless of whether the code you pushed is going to an open source project, an internal proprietary code repository, or just some code exchanged between friends working on a joint project, that simple little commit message tells the person reading your code a whole lot about you. It speaks volumes about the way you feel about the code you submit and the quality of the review you expect for your code.

As an example, suppose I was working on some code that fixed a bug. I got my code ready for initial review and I did the following:

$> git commit -a -m "Fixes some stuff"

And then I push my commit somewhere using git push …

Inevitably, what happens is that another developer will get some email or notification that I have pushed code up to some repository. It is likely that this notification will look something like this:

Change subject: Fixes some stuff ...................................................................... Fixes some stuff Change-Id: I79bbac32b5c99742b5cb283c6e55e6204bf92adc --- M path/to/some/changed/file 1 file changed, 1 insertion(+), 1 deletion(-)

And in the notification will be some link to a place to go do a code review.

Now, what do you think is the first thought that goes through the reviewer’s mind? My guess would be: Really? Fixes what stuff? By not including any context about what the patch is attempting to solve, you leave the reviewer with a bad taste in their mouth. And a bad taste in the reviewer’s mouth generally means one thing: a reluctance to review your patch.

OK, so what could we do to make the commit message better, to provide the reviewer with more initial context about your patch? Well, the first thing that comes to mind is to reference a specific bug that you are fixing with this patch.

Alright, so we amend our commit message to include a bug identifier:

$> git commit --amend -m "Fixes Bug 123456"

And subsequently push our amended commit message. The reviewer now gets a new notification that you’ve amended a previous patch. Now the notification includes the bug identifier. What do you think the next thought a typical reviewer might have? My guess is this: What, does this developer think that I’ve memorized all the bug IDs for all open bugs? How should I know what Bug 123456 is about? And here comes that bad taste in the mouth again.

OK, so this time, we will forgo the use of the time-saving -m switch to git commit and actually type a proper, multi-line commit message in our editor of choice that describes the bug that our patch fixes, including a brief description of how we fixed the bug:

git commit --amend # This will open up your editor...

Now we’d enter a good commit message … something like this would work:

Fixes Bug 123456 - ImportError raised improperly in Storage Driver Due to a circular dependency, an ImportError was improperly being thrown when the storage driver was set to XYZ. Rearranged code to remove circular dependency.

The commit message now will give the reviewer everything they need in the notification to understand what the patch is for and how you solved a bug, without needing to go to their bug tracker to figure out what the bug was about.

A detailed commit message shows you care about the time that reviewers spend on your patch and that you value the code you are submitting.

Recording of OpenStack Foundation mission and roadmap

2012-01-19T18:45:09Z

For those that couldn’t participate to past week’s webinar, here is the recording of Mark Collier and Jonathan Bryce presenting the draft mission of OpenStack Foundation and its roadmap. (The audio starts around minute 2:50). A total of 86 people participated to the events, 51 on Thursday and 35 on Friday. Since the chat was not visible to every participant, we’re disclosing the complete logs of the chat for both events, unedited . All questions were addressed by Mark and Jonathan. We’re looking for a different tool to use for future webinars.

Resource usage accounting system for OpenStack

2012-01-19T11:09:52Z

`nova-billing` service is ready for community review. It is a service which accounts resource usage per project (tenant). This information can be used for charging client later.

Currently it collects information about instances and images and snapshost in glance. Local disk usage (Gb*h), memory (Mb*h) and virtual CPU count (vcpus*h) are accounted for VM instances and disk space (Gb*h) is accounted for images in glance .

Next we are going to include other resources like network storages (volumes).

Links:

source code

issue tracker

documentation

yum repository with RHEL package

yum repository with CentOS package

Information can be received from REST interface or you can use CLI command `nova2ools-billing` from `nova2ools` set (available from Yum repositories).

FUDCon Blacksburg videos

2012-01-18T23:36:31Z

I shot videos of several of the presentations at the Fedora User and Developer Conference yesterday. For your viewing pleasure:

“State of Fedora” from the Fedora Project Leader, Jared Smith [ogg]

Mike McGrath, team lead for OpenShift, demoing OpenShift [ogg]

Jon Masters and Chris Tyler, on the ARM architecture in Fedora [ogg]. ARM is a secondary architecture today. By Fedora 18, with your help, it needs to become a primary architecture.

David Nalley presented on CloudStack, which is aiming for Fedora 17 inclusion. [ogg]

Dan Prince and Russell Bryant giving an introduction to OpenStack [ogg]

Mo Morsi presenting the Aeolus cloud management project [ogg]

[Update 1/18/2012] I was able to upload all the videos to YouTube. http://www.youtube.com/playlist?list=PL2BAA7FF83E6482C2
is a playlist with all 6.

Making more solid OpenStack releases

2012-01-18T14:16:57Z

As we pass the middle of the Essex development cycle, questions about the solidity of this release start to pop up. After all, the previous releases were far from stellar, and with more people betting their business on OpenStack we can’t really afford another half-baked release.

Common thinking (mostly coming from years of traditional software development experience) is that we shouldn’t release until it’s ready, or good enough, and calls early for pushing back the release dates. This assumes the issue is incidental: that we underestimated the time it would take our finite team of internal developers working on bugs to reach a sufficient level of quality.

OpenStack, being an open source project produced by a large community, works differently. We have a near-infinite supply of developers. The issue is, unfortunately, more structural than incidental. The lack of solidity for a release comes from:

Lack of focus on generic bugfixes. Developers should work on fixing bugs. Not just the ones they filed or the ones blocking them in their feature-adding frenzy. Fixing identified, targeted, known issues. The bugtracker is full of them, but they don’t get attention.

Not enough automated testing to efficiently catch regressions. Even if everyone was working on bug fixes, if half your fixes end up creating a set of regressions, then there is no end to it.

Lack of bug triaging resources. Only a few people work on confirming, triaging and prioritizing the flow of incoming bugs. So the bugs that need the most attention are lost in the noise.

For the Diablo cycle, we had less than a handful of people focused on generic bugfixing. The rest of our 150+ authors were busy working on something else. Pushing back the release for a week, a month or a year won’t help OpenStack solidity if the focus doesn’t switch. And if our focus switches, then there will be no need for a costly release delay.

Acting now to make Essex a success

During the Essex cycle, our Project Technical Leads have done their share of the work by using a very early milestone for their feature freeze. Keystone, Glance and Nova will freeze at Essex-3, giving us 10 weeks for bugfixing work (compared to the 4 weeks we had for Diablo). Now we need to take advantage of that long period and really switch our mindset away from feature development and towards generic bug fixing.

Next week we’ll hit feature freeze, so now is the time to switch. If we could:

have some more developers working on increasing our integration and unit test coverage

have the rest of the developers really working on generic bug fixing

have very active core reviewers that get more anal-retentive as we get closer to release, to avoid introducing regressions that would not be caught by our automated tests

…then I bet that it will lead to a stronger release than any delaying of the release could give you. Note that we’ll also have a bug squashing day on February 2 that will hopefully help us getting on top of old, deprecated and easy fixes, and give us a clear set of targets for the rest of the cycle.

That’s on our ability to switch our focus that hinges the quality of future OpenStack releases. That’s on what we’ll be judged. The world awaits, and the time is now.

A Change in Direction

2012-01-18T09:59:58Z

In 2008 my career took a sudden unexpected turn into the world of MySQL when I was offered a job at Sun. Since then MySQL and it's forks have been a big part of my life. The whole community (I mean the people, not the companies) around MySQL are part of what really drove me.

Unfortunately to me something has changed. I am not exactly sure what it is, but I am sure it is not just me because others have expressed it in conversation too. I wasn't enjoying things as much as I used to and for several reasons, some related to this, I have been quite ill.

Recently I was approached by HP's new cloud division who wanted me to work on OpenStack. It seemed the perfect opportunity to start something new inside a new vibrant community. That is not to say I have anything against my previous employer, SkySQL. They are doing a fantastic job with a great vision and I wish them well in the future.

So, I have been working on OpenStack's Core Infrastructure team for a week and a half now and so far I am loving it. The community is very welcoming and it is refreshing to see a development model similar to the one used in Drizzle so widely adopted. I look forward to diving deeper into the OpenStack world and blogging about it as I go.

The First OpenStack Bug Squashing Day Is Coming On Feb 2nd

2012-01-16T22:37:53Z

Shine your keyboards, hackers of OpenStack: on February 2nd 2012 you will be called to fight the ever growing number of bugs that keep creeping in our beloved code. On Bug Squashing Day all the OpenStack developer community will focus mainly on Nova to:

Close old fixed bugs. Old bugs are nasty. Even when they are long dead, they clog bug views and render the lists unusable. Just look at old bugs and check if they still apply ! If they don’t, close them as FixReleased (if you can pinpoint when they were fixed) or Invalid (if you can’t).

Fix bugs. The best thing you can do on a bug squashing day is to kill a live one. Just look at the list of Confirmed or Triaged and pick your target. Submit a change that fixes it. Ask for review help on the channel.

Triage incoming bugs. It’s sometimes hard to distinguish fresh bugs from false alarms. You can help by using your expertise or reproduction skills on New bugs. If you can confirm the issue, set the bug to Confirmed. If you can fix it, read the previous chapter. If you need more info from the reporter, set it to Incomplete. And if it happens to not really be valid, set it to Invalid.

You don’t have to be an experienced Nova developer to participate, and we believe that February 2nd will be a great way to get started with the OpenStack community. You can get started by looking at Devstack to build your complete OpenStack development environment. The other projects are welcome to focus on quality that day but Nova is the one that will get more attention.

The event will happen mostly online, in a dedicated #openstack-bugsquash IRC channel on Freenode (that all participants are encouraged to join for the duration of the event). There will also be live meetings in Austin and San Francisco, hosted by Rackspace with food, drinks and games.

Do you want to host a Bug Squash Day on Feb 2nd? Let us know and we’ll add it to the list.

Testing framework

2012-01-16T13:01:19Z

Today we would like to share with you one of our recent subprojects. It is our testing framework. We have used to automate our tests for OpenStack with Lettuce. However we found that we need more features to build robust framework with it. So we created a Bunch tool. The main reason for Bunch was the motivation to write more flexible and powerful tests with Lettuce. Key points to improve were:

Avoid hardcoded values in test scenarios. Lettuce offers the only way for data-driven scenarios. This is done via scenario outlines. What means that all data is stored in script itself. External data sources may be only supported by your own test code. We may fill that gap introducing Jinja2 templates and YAML configuration files. Feature templates stay readable but gain flexibility. The regular Lettuce scripts are generated on every test execution thus saving BDD style. Stories remain comprehensible for an end-user.

Write test fixtures explicitly. It should be possible to write setup and teardown BDD stories. It is also important to have behavior specifications for installation procedures.

Share and re-use test fixtures, specify that test depends on specific fixtures Tests should be self-suffucient and should not rely on the state created by other tests. However it is often a huge overhead when each test performs its own setup which may always be the same. It is required to provide the concept of “dependencies” between tests and test fixtures.

Tests are environment agnostic but test fixtures are environment dependent. We may enable writing environment agnostic tests by moving all environment specific action into test fixtures. Thus test fixtures may have multiple versions aimed for different environments.

So we gathered these requirements and planned the following features for implementation:

Support for scenario templates. Scenarios become parameterizable and do not loose BDD spirit. Foreign Lettuce scenarios can be adopted by replacing hardcoded values with template variables. (YAML and Jinja2 were used for that)

Explicit separation of test fixtures from tests (setup, teardown and test scripts). Test scenario should not contain any actions that are specific to platform/configuration and should only perform actions on the product under test.

Dependency specification for setup/teardown fixtures and sharing fixtures among tests. Most of tests rely on the same state while other may have specific prerequisites.

Parallel test execution. This is often required for long running end-to-end scenarios. That feature introduces a requirement that which are planned to be executed in parallel should be independent from each other and does not use the same resources.

Points 1-3 are implemented now. That should be enough to start using it. However there is a plenty of stuff planned on the road-map.

We also shared our OpenStack test suite which runs within our CI workflow. So you may have a look how Bunch works. Links to tool, doc and tests are below:

Bunch source code: https://github.com/griddynamics/bunch

Bunch documentation: http://www.griddynamics.com/openstack/docs/bunch/

OpenStack test suite: https://github.com/griddynamics/openstack-core-test

Just install Bunch, check out tests, adjust config.yaml and execute:
bunch -e clean openstack-core-test/smoketests/basic/

Both Bunch and tests are under active development, so there are not much tests in the repo. This is just the beginning. Stay tuned.

Grid Dynamics Open Stack development activities

2012-01-16T09:57:31Z

Hello everybody.

We want to share our roadmap with a community. In the nearest future Grid Dynamics are going to focus on new component and services development. All services will be open sourced.

What are we going to present soon:

Now we are developing resource accounting system for OpenStack (nova-billing).

We are developing DNS service, so every VM in a cloud will be resolved by its name and administrators of tenants will own one subdomain and will be able to add custom DNS records or subdomains.

We have created separate set of utilities called nova2ools and going to present them soon. They will offer utilities to work with nova and utilities to work with our services like billing or DNS.

We are going to start working on hardware provisioning for OpenStack. Sometimes it is very important to be able to work on real hardware. Ability to get hardware in a OpenStack way through REST API should be very useful.

To reflect this activities we have created a site with documentation and additional information: http://www.griddynamics.com/openstack.

Presentation: OpenStack QA – Walkthrough of Processes, Tools and Code

2012-01-13T22:22:38Z

Last night I gave a short webinar to some folks about the basics of contributing to the Tempest project, which is the OpenStack integration test suite. It was the first time I’d used Google Docs to create and give a presentation and I must say I was really impressed with the ease-of-use of Google Docs Presentation. Well done, Google.

Anyway, I’ve uploaded a PDF of the presentation to this website and provided a link to the Google Docs presentation along with a brief overview of the topics covered in the slides below. As always, I love to get feedback on slides. Feel free to leave a comment here, email me or find me on IRC. Enjoy!

Google Presentation (HTML)
PDF slides

Topics included in the slides:

OpenStack Contribution Process

Running Devstack Locally

Running Tempest against an Environment

Walkthrough the Tempest Source Code

Progressively improving a test case

Common Scenarios in Code Review and Submission

Community Weekly Review (Jan 6 – 13)

2012-01-13T21:27:05Z

OpenStack Community Newsletter –January 13, 2012

HIGHLIGHTS

Announced the date and venue for next OpenStack Conference and Design Summit http://www.openstack.org/blog/2012/01/the-openstack-spring-2012-conference-in-san-francisco-california-april-16-20-2012/

Comment on OpenStack Foundation mission and roadmap http://wiki.openstack.org/Governance/Foundation

Let the community know how you would want Swift to manage object versioning https://lists.launchpad.net/openstack/msg06666.html

New section on openstack.org tells stories from OpenStack users : what’s your story? http://www.openstack.org/user-stories/

Help make history, name OpenStack ‘F’ release https://launchpad.net/~openstack/+poll/f-release-naming

AT&T joins OpenStack community http://www.openstack.org/blog/2012/01/please-welcome-att-to-the-openstack-community/

EVENTS

Australian OpenStack Users Group Inaugural Meetup Part 2 Jan 17, 2012 – Melbourne, AU http://aosug.openstack.org.au/

Southern California Linux Expo SCALE10x, Jan 20-22 – Los Angeles, CA http://www.socallinuxexpo.org/scale10x/presentations/state-openstack

FOSDEM 2012 Feb 04 – 05, 2012 – Bruxelles, BE http://fosdem.org/2012/schedule/track/virtualization_and_cloud_devroom

OTHER NEWS

OpenStack Wiki Recent Changes – http://wiki.openstack.org/RecentChanges

How to triage OpenStack bugs http://wiki.openstack.org/BugTriage

Quantum L2 Linux Bridge Plugin http://wiki.openstack.org/Quantum-Linux-Bridge-Plugin

How OpenStack picks the names for its release http://wiki.openstack.org/ReleaseNaming

Using the HybridFox plugin to manage Openstack Nova http://docs.stackops.org/display/documentation/Using+the+HybridFox+plugin+to+manage+Openstack+Nova

Team meeting summary http://eavesdrop.openstack.org/meetings/openstack-meeting/2012/openstack-meeting.2012-01-10-21.06.html

COMMUNITY STATISTICS

Top 10 contributors of 2011 for each repository, from Jan 2 00:00:00 UTC 2012 to Jan 9 00:00:00 UTC 2012

This weekly newsletter is a way for the community to learn about all the various activities occurring on a weekly basis. If you would like to add content to a weekly update or have an idea about this newsletter, please leave a comment.

The OpenStack Spring 2012 Design Summit & Conference in San Francisco, California: April 16-20, 2012

2012-01-13T19:36:49Z

We are excited to announce that the Spring 2012 OpenStack Design Summit & OpenStack Conference are planned for April 16-20, 2012 in San Francisco, California. Please start making plans to join us!

Learn more about the events on The Official Event Page.

Please note: Registration has not yet opened. More details, including accommodations, call for papers and the sponsorship prospectus will be available shortly.

Note that two events take place the same week: The Design Summit and the Conference. So, which should you attend?

Everyone: If you are an OpenStack user, provider, researcher, developer, or enthusiast, you’ll want to attend the OpenStack Conference held Thursday and Friday, April 19 & 20th. This is the main event and it includes keynotes, panel sessions, and vendor exhibitors who can help you get the most out of OpenStack.

Developers: If you are an active developer in the OpenStack community, you’ll also want to attend the Design Summit, happening right before the conference (April 16-18th). The Design Summit is kept intentionally small so that we can have productive working sessions as we discuss and plan for development on the next release of OpenStack.

We hope you’ll make the trip and join us at what’s shaping up to be a fantastic and informative event.

Virtualization & Cloud devroom at FOSDEM

2012-01-13T14:43:06Z

The Free and Open source Software Developers’ European Meeting, or FOSDEM, is an institution that happens every year in Brussels. A busy, free and open event that gets a lot of developers together for two days of presentations and cross-pollination. There are typically the FOSDEM main tracks (a set of presentations chosen by the FOSDEM organization) and a set of devrooms, which are topic-oriented or project-oriented and can organize their own schedule freely.

This year, FOSDEM will host an unusual devroom, the Virtualization and Cloud devroom. It will happen in the Chavanne room, a 550-seat auditorium that was traditionally used for main tracks. And it will last for two whole days, while other devrooms typically last for a day or a half-day.

The Virtualization and Cloud devroom is the result of the merging of three separate devroom requests: Virtualization, Xen and OpenStack devrooms. It gives us a larger space and a lot of potential for cross-pollination across projects ! We had a lot of talks proposed, and here is an overview of what you’ll be able to see there.

Saturday, February 4

Saturday will be the “cloud” day. We will start with a set of talks about OpenStack, past, present and future. I will do an introduction and retrospective of what happened last year in the project, Soren Hansen will guide new developers to Nova, and Debo Dutta will look into future work on application scheduling and Donabe. Next we’ll have a session on various cloud-related technologies: libguestfs, pacemaker-cloud and OpenNebula. The afternoon will start with a nice session on cloud interoperability, including presentations on the Aeolus, CompatibleOne and Deltacloud efforts. We’ll continue with a session on cloud deployment, with a strong OpenStack focus: Ryan Lane will talk about how Wikimedia maintains infrastructure like an open source project, Mike McClurg will look into Ubuntu+XCP+OpenStack deployments, and Dave Walker will introduce the Orchestra project. The day will end with a town hall meeting for all OpenStack developers, including a panel of distribution packagers: I will blog more about that one in the next weeks.

Sunday, February 5

Sunday is more “virtualization” day ! The day will start early with two presentations by Hans de Goede about Spice and USB redirection over the network. Then we’ll have a session on virtualization management, with Guido Trotter giving more Ganeti news and three talks about oVirt. In the afternoon we’ll have a more technical session around virtualization in development: Antti Kantee will introduce ultralightweight kernel service virtualization with rump kernels, Renzo Davoli will lead a workshop on tracing and virtualization, and Dan Berrange will show how to build application sandboxes on top of LXC and KVM with libvirt. The day will end with another developers meeting, this time the Xen developers will meet around Ian Campbell and his Xen deployment troubleshooting workshop.

All in all, that’s two days packed with very interesting presentations, in a devroom large enough to accomodate a good crowd, so we hope to see you there !

Presenting OpenStack Foundation mission and roadmap [webinar]

2012-01-10T18:39:46Z

Jonathan Bryce and Mark Collier will host two webinars to illustrate the draft mission for the future OpenStack Foundation. The draft was published on http://wiki.openstack.org/Governance/Foundation/ and it’s the fist building block of the organisation that was announced in Boston. They will be available to answer questions about the roadmap and deadlines proposed for the project. We believe a live conversation will complement the online discussion on the foundation mailing list. We picked the times of the webinars to accommodate the needs of people living across the world and we picked the technology that is most accessible. You can use your favourite operating system or dial in using toll free numbers for virtually anywhere in the world. If you encounter problems please let us know.

The first webinar will be held on Thursday Jan 12, 2012 at 06:00 PM CST (register here https://cc.readytalk.com/r/ogxx717wjy06) and the following day on Friday, Jan 13, 2012 at 9:00 AM CST (register here https://cc.readytalk.com/r/9o9bdh6hb3vn).

You may find the converter useful http://www.timeanddate.com/worldclock/converter.html to find your local time.

Fixing a very broken instance live migration manually

2012-01-06T02:13:57Z

I had a situation recently where a number of live migrations failed in a truly nasty way. The live migration failed part way through, but didn’t properly back-out the changes. This left the instance running nowhere, but in a “migrate” state in the database. I tried to reboot the instance, which then left the instance in the “running” state.

Of course, the instance wasn’t actually running anywhere and the reboot command wouldn’t start the instance, because it thought it was running. The logs complained that the instance wasn’t running whether I tried to restart the migration, or reboot. What a full of fail situation.

So, to fix this, I needed to make the instance actually start. In this situation, the database thought the instance was running on host virt2, but the instance’s libvirt files were on virt4. I copied the nwfilter file across to /etc/libvirt/nwfilter, then the domain file across to /etc/libvirt/qemu. I then created the nwfilter, then the domain:

virsh nwfilter-define /etc/libvirt/nwfilter/.xml virsh create /etc/libvirt/qemu/.xml

Once the instance was started, I re-migrated the instance and all was good.

As a side note, I think what caused the migration failure was that I tried to migrate too many instances at the same time from a host that was already slightly overloaded. Of course, this is no excuse for nova to fail.

TempAuth Users and Administrators

2012-01-04T18:09:00Z

A few questions have come up about TempAuth, Swift's built-in authentication system; in particular, what the various administrator levels are. This is how it works.

In our small lakeside town, Albert runs a small gas station that also sells OpenStack Swift object storage to other local businesses. He has two customers: Bob's Bait Shack (on the north side of town) and Hook, Line, and Sinker (on the south side).

At Bob's Bait Shack, there are two people who need accounts. Bob, the owner, is the account administrator, while his employee Sue has more limited access. At Hook, Line, and Sinker, there's Harold (the owner), Liam, and Sue (different Sue), in a similar situation.

Here's what Albert's TempAuth configuration would look like:

... [filter:tempauth] use = egg:swift#tempauth user_alsgasandswift_al = swordfish .reseller_admin user_bobsbaitshack_bob = catfish .admin user_bobsbaitshack_sue = dogfish user_hooklineandsinker_harold = goldfish .admin user_hooklineandsinker_liam = tunafish user_hooklineandsinker_sue = blowfish ...

Username scoping

The first thing to understand is that users are scoped under accounts. There are two users named "sue", but they are under different accounts, have different passwords, and can access different things. The user "bobsbaitshack:sue" has access to containers and objects under the "bobsbaitshack" account, while the user "hooklineandsinker:sue" is restricted to things in the "hooklineandsinker" account.

Account-level super-user status

In TempAuth, it's possible to assign super-user status at the account level by saying .admin in the config file. The user "bobsbaitshack:bob" is in the .admin group, and so he can create, delete, and modify any container or object in the bobsbaitshack account. However, he can't touch anything in the hooklineandsinker account. Similarly, the user "hooklineandsinker:harold" has full rights over every container and object in the hooklineandsinker account, but has no rights to anything in the bobsbaitshack account.

Cluster-level super-user status

To make someone a cluster-level super-user, you put them in the .reseller_admin group. In our example, the user "alsgasandswift:al" is a member of the .reseller_admin group, and so he can do anything to any object or container in the entire system. He can upload files to one of Bob's containers, he can delete files from one of Harold's containers, and he can revoke and grant permissions for any existing non-admin user. He's got full access to anything in the Swift cluster.

Note that the privileges granted by .reseller_admin are a superset of those granted by .admin. This means that putting a user in both groups doesn't accomplish anything over putting them only in .reseller_admin.

Further adventures with base images in OpenStack

2012-01-03T07:13:00Z

I was bored over the New Years weekend, so I figured I'd have a go at implementing image cache management as discussed previously. I actually have an implementation of about 75% of that blueprint now, but its not ready for prime time yet. The point of this post is more to document some stuff I learnt about VM startup along the way so I don't forget it later.

So, you want to start a VM on a compute node. Once the scheduler has selected a node to run the VM on, the next step is the compute instance on that machine starting the VM up. First the specified disk image is fetched from your image service (in my case glance), and placed in a temporary location on disk. If the image is already a raw image, it is then renamed to the correct name in the instances/_base directory. If it isn't a raw image then it is converted to raw format, and that converted file is put in the right place. Optionally, the image can be extended to a specified size as part of this process.

Then, depending on if you have copy on write (COW) images turned on or not, either a COW version of the file is created inside the instances/$instance/ directory, or the file from _base is copied to instances/$instance.

This has a side effect that had me confused for a bunch of time yesterday -- the checksums, and even file sizes, stored in glance are not reliable indicators of base image corruption. Most of my confusion was because image files in glance are immutable, so how come they differed from what's on disk? The other problem was that the images I was using on my development machine were raw images, and checksums did work. It was only when I moved to a slightly more complicated environment that I had enough data to work out what was happening.

We therefore have a problem for that blueprint. We can't use the checksums from glance as a reliable indicator of if something has gone wrong with the base image. I need to come up with something nicer. What this probably means for the first cut of the code is that checksums will only be verified for raw images which weren't extended, but I haven't written that code yet.

So, there we go.

Tags for this post: openstack cloud computing nova glance qemu image management canonical
Related posts: Openstack compute node cleanup; Wow, qemu-img is fast; Playing with StumbleUpon; Why document management is good; Being Geek; Over-analysed Friday comments on Half Nekkid Thursday; Taking over a launch pad project; Perl sample source code; Old ImageMagick packages in Debian and Ubuntu; Slow git review uploads?; Color ebook!; Giving the ACS the benefit of the doubt; Cataloguing meta data against multi media formats; Looking for Women studying computing in Australia; MacOS' RSS screen saver really unreliable?; Supporting more than one browser for your website, or does Firefox really make my life harder?; Life hacking; Scott Adam's blog: the boner theory of management; The architecture of PerlMagick; Gartner recommends blogging over electronic content management; On delegation and event running

Comment

An Evening of Swift

2011-12-29T23:11:00Z

Earlier this month, we had great turnout for an OpenStack meetup in the San Francisco bay area focusing on Swift. Many thanks to Dave Neilson who helped organize the event and to Nebula who sponsored food & drinks. In one night we had a 2-hour Swift install training workshop, and had 3 speakers giving their perspectives on the state of OpenStack Swift. It wasn't just a meetup, it was a Swift conference! There was about 4 hours of OpenStack Swift workshops and speakers.

We started with a Swift Install Training Workshop. The packed room was working with Darrell Bishop and Joe Arnold on getting Swift up and running. For those of you who missed the class, we put the workshop online for those who want to take it. http://swiftstack.com/swiftworkshop

We feel that the best way to learn about something is to actually do it. So the workshop philosophy went through each step of configuration so that everyone could understand Swift cluster architecture and its configuration. For example, we walked the class through how to format and mount drives in the system, what all the configuration settings mean, what processes are running on the server and what those processes are responsible for.

In the end, most everyone walked away with a virtual machine running Swift on their laptop.

After the Swift install training session, we had a great lineup of speakers.

The first speaker was Daniel Ford of Nebula who kicked us off with a lightning talk - "Swift within the Storage Ecosystem". It was a great overview of the storage landscape and where object storage fits / falls into the storage ecosystem.

Then Reinhardt Quelle of Cisco talked about the cloud initiatives within Cisco. He addresed process around selecting which platform services they are offering internally and why OpenStack is interesting to Cisco beyond networking.

Lastly, John Dickinson, the Technical Project lead of Swift who works at Rackspace, gave a talk on the current state of Swift. It was a great overview of how Swift is architected and the future direction of Swift. His slides are below.

Looking forward to the next meetup!

OpenStack One of 10 Most Important Open Source Projects in 2011

2011-12-23T18:23:37Z

Joe “Zonker” Brockmeier recently posted his list of the top 10 most important open source projects of 2011, and OpenStack was in there.

I agree that OpenStack is one of the most compelling open source projects this year. I think back to the Cactus release earlier in the year and really am amazed at the amount of change in many of the core OpenStack projects — Nova in particular. We’ve had an explosion of contribution from a ton of individuals working at a who’s who list of technology companies, and I expect to see that passion and enthusiasm for OpenStack only continue to grow through 2012. Lots to look forward to.

Here’s to an innovative 2012 and continued collaboration in the open source community on this terrific set of cloud infrastructure projects.

OpenStackManager version 1.3 released

2011-12-22T15:30:10Z

The OpenStackManager extension is a web interface for OpenStack, and a manager for a fully integrated test and development network being written primarily for Wikimedia Foundation use.

I’ve been busy enough lately working on our OpenStack infrastructure that I haven’t made an OpenStackManager release in a while. Over the past seven months I’ve continued to make small changes to the software, and the past few weeks I’ve added features I feel deserve another release.

This is a bugfix and features release. Major changes include compatibility for cactus and diablo releases of nova, and 1.18 compatibility for MediaWiki. The changes in this release focused mainly on making workflow easier. Here’s a complete list of changes:

Added a reboot action for instances

Made compatibility changes for cactus and diablo nova releases

Made compatibility changes for MediaWiki 1.18

Added support for configurable naming attributes

Added support for adding objectclasses and attributes for users that are missing them

It’s now possible for MediaWiki to no longer have to create users, only update select user attributes and objectclasses

Made a bunch of bugfixes regarding security groups

Added support for wildcard DNS entries

Added realm and instancename variables to puppet default variables, so that they can be used in puppet runs

Added support for wiki page creation for Projects

Added support for configuring default images for instances

Added support for creating server admin logs per project

Added support for default security group rules on project creation

Added dialog to project creation for adding members to projects and roles upon creation

Added support for managing puppet classes and variables through the interface, rather than LocalSettings.php

Made a usability change to instance creation: the default security group will always be selected by default

Added support for including the ssh key fingerprint of an instance in the “your instance is ready” emails

Made a usability change to only show actions to users if they can perform them

Lots of other minor bug fixes

Bugfixes from John Du Hart, Sam Reed and Mark Hershberger

If you’d like to help develop this extension, I’ve created a development environment in a project in Wikimedia Labs. Find me on #wikimedia-labs on Freenode or email me to get a labs account and access to the project.

Related posts:

OpenStackManager version 1.2 released

OpenStackManager version 1.1 released

Announcing OpenStackManager extension for MediaWiki

Ending the year well: OpenStack Essex-2 milestone

2011-12-20T17:03:15Z

2011 is almost finished, and what a year it has been. We started it with two core projects and one release behind us. During 2011, we got three releases out of the door, grew from 60 code contributors to about 200, added three new core projects, and met for two design summits.

The Essex-2 milestone was released last week. Here is our now-regular overview of the work that made it to OpenStack core projects since the previous milestone.

Nova was the busiest project. Apart from my work on a new secure root wrapper (detailed on previous articles of this blog), we added a pair of OpenStack API extensions to support the creation of snapshots and backups of volumes, the metadata service can now run separately from the API node, network limits can now be set using a per-network base and a per-flavor multiplier, and a small usability feature lets you retrieve the last error that occurred using nova-manage. But Essex is not about new features, it’s more about consistency and stability. On the consistency front, the HA network mode was extended to support XenServer, KVM compute nodes now report capabilities to zones like Xen ones, and the Quantum network manager now supports NAT. Under the hood, VM state transitions have been strengthened, the network data model has been overhauled, internal interfaces now support UUID instance references, and unused callbacks have been removed from the virt driver.

The other projects were all busy starting larger transitions (Keystone’s RBAC, Horizon new user experience, and Glance 2.0 API), leaving less room for essex-2 features. Glance still saw the addition of a custom directory for data buffering. Keystone introduced global endpoints templates and swauth-like ACL enforcement. Horizon added UI support for downloading RC files, while migrating under the hood from jquery-ui to bootstrap, and adding a versioning scheme for environment/dependencies.

The next milestone is in a bit more than a month: January 26th, 2012. Happy new year and holidays to all !

Openstack compute node cleanup

2011-12-19T08:59:00Z

I've never used openstack before, which I imagine is similar to many other people out there. Its actually pretty cool, although I encountered a problem the other day that I think is worthy of some more documentation. Openstack runs virtual machines for users, in much the same manner as Amazon's EC2 system. These instances are started with a base image, and then copy on write is used to write differences for the instance as it changes stuff. This makes sense in a world where a given machine might be running more than one copy of the instance.

However, I encountered a compute node which was running low on disk. This is because there is currently nothing which cleans up these base images, so even if none of the instances on a machine require that image, and even if the machine is experiencing disk stress, the images still hang around. There are a few blog posts out there about this, but nothing really definitive that I could find. I've filed a bug asking for the Ubuntu package to include some sort of cleanup script, and interestingly that led me to learn that there are plans for a pretty comprehensive image management system. Unfortunately, it doesn't seem that anyone is working on this at the moment. I would offer to lend a hand, but its not clear to me as an openstack n00b where I should start. If you read this and have some pointers, feel free to contact me.

Anyways, we still need to cleanup that node experiencing disk stress. It turns out that nova uses qemu for its copy on write disk images. We can therefore ask qemu which are in use. It goes something like this:

$ cd /var/lib/nova/instances $ find -name "disk*" | xargs -n1 qemu-img info | grep backing | \ sed -e's/.*file: //' -e 's/ .*//' | sort | uniq > /tmp/inuse

/tmp/inuse will now contain a list of the images in _base that are in use at the moment. Now you can change to the base directory, which defaults to /var/lib/nova/instances/_base and do some cleanup. What I do is I look for large image files which are several days old. I then check if they appear in that temporary file I created, and if they don't I delete them.

I'm sure that this could be better automated by a simple python script, but I haven't gotten around to it yet. If I do, I will be sure to mention it here.

Tags for this post: openstack cloud computing nova glance qemu image management ubuntu canonical
Related posts: Further adventures with base images in OpenStack; Old ImageMagick packages in Debian and Ubuntu; Wow, qemu-img is fast; Playing with StumbleUpon; Why document management is good; Being Geek; Over-analysed Friday comments on Half Nekkid Thursday; Nice touch; Taking over a launch pad project; What packages are taking all my disk?; Perl sample source code; Installing Linux on a Dell Dimension e310; MySQL Workbench; Ubuntu Breezy and Dovecot hate me; Slow git review uploads?; Color ebook!; Upgrading Ubuntu Feisty now that its end of lifed; Giving the ACS the benefit of the doubt; Cataloguing meta data against multi media formats; Looking for Women studying computing in Australia; MySQL Workbench

Comment

From Cement to Spandex – Making PDF and ePub

2011-12-02T21:46:52Z

Which statement is true:

“PDFs are like cement.”

or

“Gentlemen prefer PDF.”

Turns out both are true! See my recent OpenStack blog entry, Hacking on Ebooks, for more context and attributions for those statements.

We recently held a hackathon which I blogged about earlier to discuss the prep work for creating epub from DocBook XML source for the OpenStack and Rackspace manuals. We had a very successful day of hacking on 11/11/11. A team of about seven writers, testers, and developers worked all day to try to make epub files. And sure enough, we did it!

Our list of bugs matches up with what others have noted about difficulties making ebooks, such as sizing images properly and enabling tables that scale when zooming in or out or being displayed on a small smartphone or a larger tablet screen. Turns out, many “pro” epub creators turn all the tables into images to avoid the sizing problem. We also noticed the problem with mobi output putting a new line for each list item, and I haven’t gotten the fix working yet. We’re starting with the OpenStack Starter Guide available as an epub download and automated outputting epub for that book. We’re checking out the downloads to see whether there’s interest and we’ll go from there!

Improving Nova privilege escalation model, part 3

2011-11-30T14:29:16Z

In the previous two posts of this series, we explored the deficiencies of the current model and the features of an alternative implementation. In this last post, we’ll discuss the advantages of a Python implementation and open discussion on how to secure it properly.

Python implementation

It’s quite easy to implement the features that were mentioned in the previous post in Python. The main advantage of doing so is that the code can happily live inside Nova code, in particular the filters definition files can be implemented as Python modules that are loaded if present. That solves the issue of shipping definitions within Nova and also the separation of allowed commands based on locally-deployed nodes. The code is simple and easy to review. The trick is to make sure that no malicious code can be injected in the elevated rights process. This is why I’d like to present a model and open it for comments in the community.

Proposed security model

The idea would be to have Nova code optionally use “sudo nova-rootwrap” instead of “sudo” as the root_helper. A generic sudoers file would allow the nova user to run /usr/bin/nova-rootwrap as root, while stripping environment variables like PYTHONPATH. To load its filters definitions, nova-rootwrap would try to import a set of predefined modules (like nova.rootwrap.compute), but if those aren’t present, it should ignore them. Can this model be abused ?

The obvious issue is to make sure sys.path (the set of directories from which Python imports its modules) is secure, so that nobody can insert their own modules in the process. I’ve given some thoughts to various checks, but actually there is no way around trusting the default sys.path you’re given when you start python as root from a cleaned env. If that’s compromised, you’re toasted the moment you “import sys” anyway. So using sudo to only allow /usr/bin/nova-rootwrap and cleaning the environment should be enough. Or am I missing something ?

Insecure mode ?

One thing we could do is check that sys.path all belongs to root and refuse to run in the case it’s not. That would tell the user that his setup is insecure (potentially allowing him to bypass that by running “sudo nova-rootwrap –insecure” as the root_helper). But that’s a convenience to detect insecure setups, not a security addition (the fact that it doesn’t complain doesn’t mean you’re safe, it could mean you’re already compromised).

Test mode ?

For tests, it’s convenient to allow to run code from branches. To allow this (unsafe) mode, you would tweak sudoers to allow it to run $BRANCH/bin/nova-rootwrap as root, and prepend “..” to sys.path in order to allow modules to be loaded from $BRANCH (maybe requiring –insecure mode for good measure). It sounds harmless, since if you run from /usr/bin/nova-rootwrap you can assume that /usr is safe… Or should that idea be abandoned altogether ?

Audit

Nothing beats peer review when it comes to secure design. I call all Python module-loading experts and security white-hats out there: would this work ? Are those safe assumptions ? How much do you like insecure and test modes ? Would you suggest something else ? If you’re one of those that can’t think in words but require code, you can get a glimpse of work in progress here. It will all be optional (and not used by default), so it can be added to Nova without much damage, but I’d rather do it right from the beginning Please comment !

OpenStack Dev Tip — Easily Pull a Review Branch

2011-11-28T18:06:08Z

Just a quick tip for developers working on OpenStack projects that work on multiple development machines or want to pull a colleague’s code from the Gerrit review system and test it locally.

If you have followed the instructions about setting up a development environment successfully, you will have installed the git-review tool that Jim Blair and Monty Taylor maintain. The git-review tool has a nice little feature that enables you to easily pull any branch that anyone has pushed up to code review:

$> git review -d $REVIEW_NUM

The $REVIEW_NUM variable should be replaced with the identifier of the review branch in Gerrit.

For example, I developed some code on my laptop that I now want to pull to my beefier work machine. The original branch is failing a few tests in Jenkins and I want to diagnose what’s going on. The review branch is here: https://review.openstack.org/#change,1656. The review number (ID) is 1656.

To grab that branch into my local environment and check it out, I do:

jpipes@uberbox:~/repos/glance$ git review -d 1656 Downloading refs/changes/56/1656/2 from gerrit into review/jay_pipes/bug/850377

Doing a git status, you’ll note that I am now in the local branch called review/jay_pipes/bug/850377:

jpipes@uberbox:~/repos/glance$ git status # On branch review/jay_pipes/bug/850377 # Your branch and 'gerrit/master' have diverged, # and have 1 and 2 different commit(s) each, respectively. # nothing to commit (working directory clean)

I can now run tests, diagnose the issue(s), fix code up and do a:

$> git commit -a --amend $> git review

And my changes will be pushed up to the original review in Gerrit for others to look at.

The debate about foundations for open source projects

2011-11-27T17:44:27Z

Two interesting articles debating about foundations for open source projects. Mikeal Rogers argues that Apache Software Foundation has served its purpose in its 12 years and now is only preserving the problem to justify its existence.

Ten years ago open source projects faced a long list of barriers to entry. Source hosting was a pain in the ass. Wiki hosting, Mailing List hosting, bug tracking, all of these things we can now take for granted were actually quite hard to set up and maintain as recently as 5 years ago. [...]

and then he adds that

the world has changed Apache has become a net negative for its projects.

Mainly, Mikael says, GitHub has leveled the entry field and the layer of politics added by the Apache Foundation are not necessary anymore.

Mike Milinkovich, executive director of Eclipse Foundation wrote a response, titled of course Foundations Considered Useful. He lists some of the important things that Eclipse does:

A small sampling of the core value-add that happens within the Eclipse Foundation and its community would include:

IP management. Although we often get criticized for being overly focused on the topic, nobody does IP management better than Eclipse. Which is a huge part of fulfilling our mission of delivering product-ready software platforms. Yes, it’s a lot of work, but it is absolutely a core value, and one not easily replicated.

Predictability. The Eclipse community has shipped its major platform release on time to the day for eight years. Last year’s release was 46 MLOC, so we are talking about a non-trivial amount of code. The processes that we have in place to co-ordinate the activities and release engineering for 60+ projects absolutely require some amount of centralized support.

Branding and community. The Eclipse brand means something to people. Millions of developers around the world use Eclipse or Eclipse-based products every day. They have confidence in the software and the community that delivers it. Looking inside the community, there is definitely a pride and a sense of community that comes with being part of Eclipse. Anyone who has ever been to an EclipseCon has seen this firsthand.

Industry collaboration. Obviously GitHub has been wildly successful in fostering community-led open source. However, there are lots of instances where large and conservative corporations are looking at how to get involved in open source. In many cases, their business motivation is to collaborate with other industry players to create shared industry platforms. The kind of work that goes into facilitating these ventures goes far beyond picking a license and starting to hack some code. The processes that organizations like Eclipse and Apache bring to the table for project incubation, development processes, license management and IP contribution management are critical success factors.

Both articles are interesting reading for the OpenStack community while we think about our own foundation. What do you think?
Share and Enjoy:

Tweet
Tweet

Most Commented Posts
Dove sono finiti i fondi per il Software Libero?
Notes to develop Funambol for Android in Eclipse
The end of a phase

Stef for ][ stefano maffulli, 2011. | Permalink | 3 comments
Post tags:

Improving Nova privilege escalation model, part 2

2011-11-25T11:00:45Z

In the previous post in this series we explored the current privilege escalation model used in OpenStack Compute (Nova), and discussed its limitations. Now that we are able to plug an alternative model (thanks to the root_helper option), we’ll discuss in this post what features this one should have. If you think we need more, please comment !

Command filters

The most significant issue with the current model is that sudoers filters the executable used, but not the arguments. To fix that, our alternative model should allow precise argument filtering so that only very specific commands are allowed. It should use lists of filters: if one matches, the command is executed.

The basic CommandFilter would just check that the executable name matches (which is what sudoers does). A more advanced RegexpFilter would check that the number of arguments is right and that they all match provided regular expressions.

Taking that concept a step further, you should be able to plug any type of advanced filter. You may want to check that the argument to the command is an existing directory. Or one that is owned by a specific user. The framework should allow developers to define their own CommandFilter subclasses, to be as precise as they want when filtering the most destructive commands.

Running as

In some cases, Nova runs, as root, commands that it should just run as a different user. For example, it runs kill with root rights to interact with dnsmasq processes (owned by the nobody user). It doesn’t really need to run kill with root rights at all. Filters should therefore also allow to specify a lower-privileged user a specific matching command should run under.

Shipping filters in Nova code

Filter lists should live within Nova code and be deployed by packaging, rather than live in packaging. That allows people adding a new escalated command to add the corresponding filter in the same commit.

Limiting commands based on deployed nodes

As mentioned in the previous post, nova-api nodes don’t actually need to run any command as root, but in the current model their nova user is still allowed to run plenty of them. The solution for that is to separate the command filters based on the type of node that is allowed to run them, in different files. Then deploy the nova-compute filters file only on nova-compute nodes, the nova-volume filters file only on nova-volume nodes… A pure nova-api node will end up with no filters being deployed at all, effectively not being allowed any command as root. So this can be solved by smart packaging of filter files.

Missing features ?

Those are the features that I found useful for our alternative privilege escalation model. If you see others, please comment here ! I’d like to make sure all the useful features are included. In the next post, we’ll discuss a proposed Python implementation of this framework, and the challenges around securing it.

Swift and Keystone middleware – Part1

2011-11-24T09:20:39Z

It seems that integrating Swift and Keystone together present some challenges to people and this is absolutely normal as there is a lot of changes going on. This is my attempt to document how everything is plugged together.

I am not going to explain how a middleware is supposed to work as this is nicely documented on Wikipedia :

http://en.wikipedia.org/wiki/Middleware

or how the auth middlewares works on Swift :

http://swift.openstack.org/development_auth.html

or even how this is plugged inside Keystone :

http://keystone.openstack.org/middleware_architecture.html

At first let’s get some of the wordings right :

A tenant in keystone is an account in swift.

A user in keystone is also a user in swift.

A role in keystone is a group in swift.

Now that you keep this in mind let’s walk-though how a request will
look like.

At first your user connect to keystone and says this is my username for this
tenant and here is the secret/api key, give me the endpoints for the
services and add a token to it. This will look like this in curl :

curl -s -d '{"auth": {"tenantName": "demo", "passwordCredentials": {"username": "demo", "password": "password"}}}' -H 'Content-type: application/json' http://localhost:5000/v2.0/tokens

If successfully authenticated you get back in Json those public/internal urls
for swift so you are able to connect, here is some part of the replied request :

{ "endpoints": [ { "adminURL": "http://localhost:8080/", "internalURL": "http://localhost:8080/v1/AUTH_2", "publicURL": "http://localhost:8080/v1/AUTH_2", "region": "RegionOne" } ], "name": "swift", "type": "object-store" } [...] "token": { "expires": "2011-11-24T12:35:56", "id": "ea29dae7-4c54-4e80-98e1-9f886acb389a", "tenant": { "id": "2", "name": "demo" } },

So now the clients is going to get the publicURL (or can be internal) with the token and able to give request to swift with it. Let’s take the simple request which list the container, this is a basic GET on the account :

curl -v -H 'X-Auth-Token: ea29dae7-4c54-4e80-98e1-9f886acb389a' http://localhost:8080/v1/AUTH_2

which should come back by a 20* http code if that work.

What’s happening here is that when you connect to swift it will pass it to the middleware to make sure we are able to have access with that token.

The middleware will take that token connect to keystone admin url with the admin token and pass that user token to be validated. The query looks like this in curl :

curl -H 'X-Auth-Token: 7XX' http://localhost:35357/v2.0/tokens/ea29dae7-4c54-4e80-98e1-9f886acb389a

note: localhost:35357 is the keystone admin url and 7XX is the admin token set in the configuration of the middleware.

if successful keystone will come back with a reply that look like this :

{ "access": { "token": { "expires": "2011-11-24T12:35:56", "id": "ea29dae7-4c54-4e80-98e1-9f886acb389a", "tenant": { "id": "2", "name": "demo" } }, "user": { "id": "2", "roles": [ { "id": "2", "name": "Member", "tenantId": "2" }, { "id": "5", "name": "SwiftOperator", "tenantId": "2" }, ], "username": "demo" } } }

Let’s step back before more Curl command and understand a thing about Swift, a user of an account in Swift by default don’t have any rights at all but there is one user in that account whose able to give ACL on containers for other users. In swift keystone middleware we call it an Operator.

The way the middleware knows which user is able to be admin on an account is by using the roles matching to whatever configuration we have on the middleware setting called :

keystone_swift_operator_roles = Admin, SwiftOperator

since this user is part the SwiftOperator then it has access and he’s allowed to do whatever he wants for that account like creating containers or giving ACL to other users.

So let’s say we have a user called demo2 which is part of the demo account and have only the role Member to it and not SwiftOperator by default as we say before he will not be able to do much.

But if demo user give access to the group/role Memeber to a container via acl then demo2 will be able to do stuff on it.

We can all have fun with bunch of curl commands but since swift 1.4.7 the swift CLI tool have support for the auth server version 2.0 and allow you to connect to keystone for auth so we are going to use that instead.

Let first create a testcontainer and upload a file into it with our ‘operator’ user :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer

now let’s give access to the Member group for that container on reading :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo -K password post testcontainer -r Member

and now if we try to read that file directly with demo2 it will be allowed :

swift --auth-version 2 -A http://localhost:5000/v2.0 -U demo:demo2 -K password download testcontainer etc/issue -o-

Hope this make things a bit more clears how everything works, in the next part I am going to explain how the config files and packages will look like for installing keystone and swift.

Improving Nova privilege escalation model, part 1

2011-11-23T16:31:24Z

In this series, I’ll discuss how to strengthen the privilege escalation model for OpenStack Compute (Nova). Due to the way networking, virtualization and volume management work, some Nova nodes need to be able to run some commands as root. To reduce the effects of a potential compromise (attacker being able to run arbitrary code as the Nova user), we want to limit the commands that Nova can run as root on a given node to the strict necessary. Today we’ll explain how the current model works, its limitations, and the groundwork already implemented during the Diablo cycle to improve that.

Current model: sudo and sudoers

Currently, in a typical Nova deployment, the nodes run under an account with limited rights (usually called “nova”). When Nova needs to run a command as root, it prepends “sudo” to the command. The nova packages of your distribution of choice are supposed to ship a sudoers file that contains all the commands that nova is allowed to run as root without providing a password. This is a privilege escalation security model which is pretty well-known and easy to audit.

Limitations of the current model

That said, in the context of Nova, this model is very limited. The sudoers file does not allow to efficiently filter arguments, so you can basically pass any argument to the allowed command… and some of the commands that nova wants to use are rather open-ended. As an example, the current nova_sudoers file contains commands like chown, kill, dd or tee, which are more than enough to compromise a target system completely.

There are a couple other limitations. The sudoers file belongs to the distributions packaging, so it’s difficult to keep it in sync with the rest of Nova code when someone wants to add a privileged command. Last but not least, the same nova_sudoers file is used for any type of Nova node. A Nova API server, which does not need to run any command as root, is still allowed to run all the commands that a compute node requires, for example. Those other limitations could be fixed while still using sudo and sudoers files, but the first limitation would remain. Can we do better ?

Substitute a wrapper to sudo

To be able to propose alternative privilege escalation security models, we first needed to be able to change all the “sudo” calls in the code and make them potentially use something else. That’s what I worked on late during the Diablo timeframe: creating a run_as_root option in nova.utils.execute that would use a configurable root_helper command (by default, “sudo”), and force all the existing calls to go through that (rather than blindly calling “sudo” themselves).

Thanks to the default root_helper, everything still behaves the same, but now we have the possibility to use something else, if we can be smarter than sudoers files. Like call a wrapper that will do advanced filtering of the command that nova wants to use. In part 2 of this series, we’ll look into a proposed, alternative Python-based root_helper and open discussion on its security model.