As mentioned in an older post, I’m building a test and development environment using OpenStack. The environment is intended to be fairly integrated. Part of this integration is a consistent working environment between instances in a project. Providing home directories via NFS is the easiest way of ensuring this consistent working environment.
The problem with NFS home directories, however, is that they are fairly insecure. They can be used between instances to escalate privileges. In our environment, this isn’t a problem for instances within a project. If a user is a member of a project, they have shell on all instances. If they are given sudo access in a project, they are given sudo access on all instances. Between projects, however, is a problem. user-A with root on instance-A in project-A could su to to user-B on instance-A, modify the user’s authorized_keys file, and then have access to project-B if home directories are shared across projects.
To avoid cross-project escalation, each project needs its own set of home directories. This means we can’t simply export /home to the instance’s private network and be done with it. We’ll need to create an exports file, and share different directory trees with specific instances. We’ll also need to mount home directories differently on the instances, depending on the project they belong to.
To do so, we’ll use a combination of puppet, LDAP, autofs, and Nova.
Creating the exports file
To create the exports file we need three things:
- A list of projects
- A list of instances within each project
- A list of home directory locations for each project
The first two can be found via LDAP. The query for a list of projects is: ‘(&(objectclass=groupofnames)(owner=*))’. The query for a list of instances within each project is: ‘(puppervar=instanceproject=<project>)’. Of course, this approach is only usable for people using the OpenStackManager extension for MediaWiki; I’ll mention more portable ways to get this information later in the post.
The third we can extrapolate, we just need a base directory. I chose to use /export/home/<project> for the locations.
I wrote a python script that will pull this information, and create an exports file that looks like this:
/export/home/<project1> <project1-instance1>(rw,no_subtree_check) <project-instance2>(rw,no_subtree_check) <project_instance...>(rw,no_subtree_check)
/export/home/<project2> <project2-instance1>(rw,no_subtree_check) <project2-instance2>(rw,no_subtree_check) <project2_instance...>(rw,no_subtree_check)
Mounting the shares from the instances
Each instance needs to mount the share, depending on its project. There’s a number of ways we can do this, but I like the flexibility of using autofs and LDAP to manage NFS mounts. To add slightly more flexibility we’ll involve the help of puppet as well.
In LDAP, we can create autofs entries by making maps and objects. The following objects add support for /home:
dn: nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.master
dn: nisMapName=auto.home,<basedn>
objectClass: top
objectClass: nisMap
nisMapName: auto.home
dn: nisMapName=/home,nisMapName=auto.master,<basedn>
objectClass: top
objectClass: nisObject
cn: /home
nisMapEntry: ldap:nisMapName=auto.home,<basedn>
nisMapName: auto.master
We also need to add entries for the specific home directories. Here we are going to invoke a little awesome magic that autofs has: variables. Here’s the entry we are using for all home directories in all projects:
dn: cn=*,nisMapName=auto.home,<basedn>
changetype: add
nisMapEntry: ${SERVNAME}:${HOMEDIRLOC}/&
objectClass: nisObject
objectClass: top
nisMapName: auto.home
cn: *
We only need the one entry, which saves us from having to create and delete entries on creation and deletion of projects. Using this, however, means we need to set the variables. This is where puppet comes in. First, though, let’s look at the node in LDAP:
dn: dc=i-0000005c,dc=pmtpa,ou=hosts,dc=wikimedia,dc=org
objectClass: domainrelatedobject
objectClass: dnsdomain
objectClass: domain
objectClass: puppetclient
objectClass: dcobject
objectClass: top
puppetVar: realm=labs
puppetVar: writable=false
puppetVar: db_cluster=s1
puppetVar: instancecreator_email=rlane@wikimedia.org
puppetVar: instancecreator_username=Ryan Lane
puppetVar: instancecreator_lang=en
puppetVar: instanceproject=testlabs
puppetClass: base
puppetClass: ldap::client::wmf-test-cluster
puppetClass: exim::simple-mail-sender
puppetClass: db::core
puppetClass: mysql::mysqluser
puppetClass: mysql::datadirs
puppetClass: mysql::conf
l: pmtpa
associatedDomain: i-0000005c.pmtpa.wmflabs
associatedDomain: labs-db2.pmtpa.wmflabs
dc: i-0000005c
aRecord: 10.4.0.12
All the above objectclasses and attributes are available for use in puppet. The really important one here is instanceproject=testlabs.
We can set the autofs variables via the OPTIONS variable in the /etc/default/autofs file:
OPTIONS=”-DSERVNAME=<%= nfs_server_name %> -DHOMEDIRLOC=<%= homedir_location %>”
Here SERVNAME and HOMEDIRLOC autofs variables are being set. nfs_server_name and homedir_location are being set via a puppet template. Both are being determined via a manifest:
$homedir_location = "/export/home/${instanceproject}"
nfs_server_name is a hash, based on the project:
$nfs_server_name = $instanceproject ? {
default => "labs-nfs1",
}
I chose to use a hash based on project so that I can choose to separate the server based on project as well, if needed for performance, or extra security.
Managing user home directories
Everything up to this point is just creating the shares. However, we must maintain the users’ home directories as well. For this, we need to know which users are in which projects, and we need to manage their home directories per project.
I wrote a script to search for users, based on a group (the project), that selectively creates/deletes/renames home directories and authorized_keys files. I should note here that I don’t use nova’s mechanisms for SSH key management, as it isn’t portable between applications. I instead store the keys in the user’s LDAP entry.
There’s a security issue with management of home directories. If a user is added to a project and we create a home directory, with a populated authorized_keys file, then remove the user from the project, but don’t remove their home directory, the user will still have access to the project’s instances. There’s two ways I go about solving this issue:
- Ensure the user only has access to the instance if they are in the project, using access.conf. In my architecture, when projects are added, they are also made a posixgroup, with a gid. Thanks to this, we can treat the project as a system group in all instances. In access.conf we limit access to the project group.
- User’s home directories are moved from /export/home/<project>/<user> to /export/home/<project>/SAVE/<user> when they are removed from the project.
Problems with this solution, and future improvements to make
The major shortcoming of this solution is that it isn’t terribly portable. It is dependent on using LDAP, and storing specific information in the LDAP directory. Using the nova tools, or having nova manage the exports on instance creation/deletion would make this a much more portable solution.
Another shortcoming is that it isn’t terribly scalable. The exports file is being created from scratch every single script run (which needs to happen fairly frequently). Ideally, nova would write to a queue, and the NFS instance would add/remove instances from the exports as instances are created/deleted.
Thankfully, I didn’t have a shortage of ideas about how to accomplish this, as shown in my proposal. I decided upon the quick and dirty approach, opting to do one of the more reusable approaches later. I’ll likely add support to nova for this at some time in the future.
Related posts:
- Why I chose MediaWiki for my OpenStack Manager project
- A process for puppetization of a service using Nova
- Announcing OpenStackManager extension for MediaWiki
We had an informal 
Please comment !
