Using puppet in UEC/EC2: puppet support in Ubuntu images

March 24, 2010

One of the focus for the Lucid release cycle in the Ubuntu Server team is to improve the integration between puppet and UEC/EC2. I’ll discuss in a series of articles how to setup a puppet infrastructure to manage Ubuntu Lucid instances running on UEC/EC2. I’ll focus on the bootstrapping process rather than writing puppet recipes.

Today we’ll look at configuring a puppetmaster into an instance and how to start instances that will register automatically with the puppetmaster.

We’ll work with the Lucid Beta1 image on EC2. All the instances started through out this article will be based on this AMI.

Puppetmaster setup

Let’s start by creating a puppetmaster running on EC2. We’ll setup all the puppet configuration via ssh using a bzr branch on Launchpad: lp:~mathiaz/+junk/uec-ec2-puppet-config-tut1.

Start an instance of the Lucid Beta1 AMI using an ssh key. Once it’s running write down its public and private DNS addresses. The public DNS address will be used to setup the puppetmaster via ssh. The private DNS address will be used as the puppetmaster hostname given out to puppet clients.

We’ll actually install the puppetmaster using puppet itself.

Log on the started instance via ssh to install and setup the puppet master:

  1. Update apt files:

    sudo apt-get update

  2. Install the puppet and bzr packages:

    sudo apt-get install puppet bzr

  3. Change the ownership of the puppet directory so that the ubuntu user can directly edit the puppet configuration files:

    sudo chown ubuntu:ubuntu /etc/puppet/

  4. Get the puppet configuration branch:

    bzr branch –use-existing-directory lp:~mathiaz/+junk/uec-ec2-puppet-config-tut1 /etc/puppet/

    Before doing the actual configuration let’s have a look at the content of the /etc/puppet/ directory created from the bzr branch.

    The layout follows the recommended puppet practices. The puppet module available in the modules directory defines a puppet::master class. The class makes sure that the puppetmaster package is installed and that the puppetmaster service is running. The manifests/puppetmaster.pp file defines the default node to be configured as a puppetmaster.

  5. We’ll now run the puppet client to setup the instance as a puppetmaster:

    sudo puppet /etc/puppet/manifests/puppetmaster.pp

Starting a new instance

Now that we have puppetmaster available in our cloud we’ll have look at how a new instances of the Lucid Beta1 AMI can be started and automatically setup to register with the puppetmaster.

We’re going to use the cloud-config puppet syntax to boot an instance and have it configure itself to connect to the puppetmaster using its user data information:

  1. On the puppetmaster instance create a user-data.yaml file to include the relevant puppetmaster configuration:

    cp /usr/share/doc/cloud-init/examples/cloud-config-puppet.txt user-data.yaml

  2. Update the server setting to point to the puppetmaster private dns hostname. I also strongly recommend to include the puppmaster ca certificate as the ca_cert setting.

    The example certname setting uses a string extrapolation to make each puppet client certificate unique: for now %i is replace by the instance Id while %f is replaced by the FQDN of the instance.

    The sample file has extensive comments about the format of the file. One of the key point is that you can set any of the puppet configuration options via the user data passed to the instance.

    Note that you can remove all the comments to make the user-data.yaml file easier to copy and paste. However don’t remove the first line (#cloud-config) as this is used by the instance boot process to start the puppet installation.

  3. Launch a new instance using the content of the user-data.yaml file you’ve just created as the user-data option passed to the new instance.

  4. You can watch the puppetmaster log on the puppetmaster instance to see when the new instance will request a new certificate:

    tail -f /var/log/syslog

  5. After some time you should see a request coming in:

    puppetmasterd[2637]: i-fdb31b96.ip-10-195-18-227.ec2.internal has a waiting certificate request

    During the boot process of the new instance the puppet cloud-config plugin used the user-data information to automatically install the puppet package, generate the /etc/puppet/puppet.conf file and start the puppetd daemon.

  6. You can then approve the new instance:

    sudo puppetca -s i-fdb31b96.ip-10-195-18-227.ec2.internal

  7. Watching the puppetmaster log you’ll see that after some time the new instance will connect and get its new manifest compiled and sent:

    puppetmasterd[2637]: Compiled catalog for i-fdb31b96.ip-10-195-18-227.ec2.internal in 0.03 seconds

In conclusion we now have an instance acting as a puppetmaster and have a single user-data configuration for the whole puppet infrastructure. That user data can be passed to new instances which will automatically register with our puppetmaster.

Even though we’re able to make all our instances automatically register with our puppetmaster we still need to manually sign each request as outlined in step 6 above. We’ll have a look at automating this step in the next article.


FOSDEM 2010

February 17, 2010

I had the opportunity to attend FOSDEM this year. The most amazing (and frustrating) part of the event was the huge number of talks that were given. Making choices was sometimes hard. However the FOSDEM team recorded most of the sessions and the videos are available now. Perfect for such a busy event!

Here a few highlights of the presentations I attended:

Linux distribution for the cloud

I started the day by attending Peter Eisentraut (a Debian and PostgreSQL core developer) session about Linux distributions for the cloud. He focused on the provisioning aspect of clouds by giving a history of how operating systems were installed: from floppy drives to Cloud images. He dedicated one slide to Ubuntu’s cloud offering including Canonical Landscape commenting that Ubuntu is clearly the leader of distributions in the cloud space. He also outlined what were the current problems such as lack of standards and integration of existing software stacks. He pointed out that linux distributions could drive this.

The second part of his talk was focused on the Linux desktop and the impact of cloud services on it. Giving ChromeOS as an example he outlined how applications themselves were moved to the cloud. He then listed the problems with Cloud services with regards to the Free Software principles: non-free server side code, non-free hosting, little or not control over data, lack of open-source community.

He concluded by outlining the challenge in the domain: how could free software principles be transposed to the Cloud and its services? One great reference is Eben Moglen talk named “Freedom in the Cloud”.

Beyond MySQL GA

Kristian Nielsen, a MariaDB developer, gave an overview of the Developer ecosystem around MySQL. He listed a few patches that were available to add new functionalities and fix some bugs: the Google patch, Percona patches, eBay patches, Galera multi-master replication for InnoDB as well as a growing list of storage engines. Few options are available to use them:

  • packages from third party repositories (such as ourdelta and percona)
  • MariaDB maintains and integrates most of the patches
  • a more do-it-yourself approach where you maintain a patch serie.

I talked with Kristian after the presentation about leveraging bzr and LP to make the maintenance easier. It could look like this:

  1. Each patch would be available and maintained in a bzr branch – in LP or else where.
  2. The Ubuntu MySQL package branch available in LP would be used as the base for creating a debian package (or the Debian packaging branch since Debian packages are also available in LP via bzr)
  3. bzr-looms would glue the package bzr branch with the patches bzr branches. The loom could be available from LP or elsewhere.
  4. bzr-builder would be used to create a recipe to build binary packages out of the loom branch.
  5. Packages would be published in PPAs ready to be installed on Ubuntu systems.

The Cassandra distributed database

I finally managed to get in the NoSQL room to attend Eric Evans overview of the Cassandra project. He is a full time developer and employee of Rackspace. The project was started by Facebook to power their inbox search. Even though the project had been available for some years the developer community really started to grow in March 2009. It is now part of the Apache project and about to graduate to a Top Level Project there.

It is inspired by Dynamo from Amazon and provide a O(1) DHT with eventual consistency and a consistent hashing. Multiple client APIs are available:

  • Thrift
  • Ruby
  • Python
  • Scala

I left before the end of the talk as I wanted to catch the complete presentation about using git for packaging.

Cross distro packaging with (top)git

Thomas Koch gave an overview of using git and related tools to help in maintaining Debian packaging. He works in web shop where every web application is deployed as a Debian package.

The upstream release tarball is imported in a upstream git branch using the pristine-tar tool. Packaging code (ie the debian/ directory) is kept in a different branch.

Patches to the upstream code are managed by topgit as seperate git branches. He also noted that topgit was able to export the whole stack of patches in the quilt Debian source format using the tg export command.

Here is the list of tools associated with his workflow:

  • pristine-tar
  • git-buildpackage
  • git-import-orig
  • git-dch
  • topgit

The workflow he outlined looked very similar to the one based around bzr and looms.

Scaling Facebook with OpenSource tools

David Recordon from Facebook gave a good presentation on the challenges that Facebook runs into when it comes to scale effectively.

Here are a few numbers I caught during the presentation to give an idea about the scale of the Facebook infrastructure (Warning: they may be wrong – watch the video to double check):

  • 8 billion minutes spent on Facebook every day
  • 2.5 billions of pictures uploaded every month
  • 400 billion page/view per month
  • 25 TB of log per day
  • 40 billions pictures stored in 4 resolutions which bring a grand total of 160 billions photos
  • 4 millions of line codes in php

Their overall architecture can be broken into the following components:

  1. Load balancers
  2. Web server (php): Most of the code is written in PHP: the language is simple, it fits well for fast development environments and there are a lot of developers available. A few of the problems are CPU, Memory, how to reuse the PHP logic in other systems and the difficulty to write extensions to speed up critical parts of the code. An overview of the HipHop compiler was given: a majority of their PHP code can be converted to C++ code which is then compiled and deployed on their webserver. An apache module is coming up soon probably as a fastcgi extension.
  3. memcached (fast, simple): A core component of their infrastructure. It’s robust and scales well: 120 millions queries/second. They wrote up some patches which are now making their way to upstream.
  4. Services (fast, complicated): David gave an overview of some of the services that Facebook opensourced:
    • Thrift: an rpc server, now part of the Apache incubator.
    • Hive: build on top of hadoop it is now part of the Apache project. It’s an SQL-like frontend to hadoop aiming at simplifying access to the hadoop infrastructure so that more people (ie non-engineers) can write and run data analysis jobs.
    • Scribe: a performant and scalable logging system. Logs are stored in a hadoop/hive cluster to help in data analysis.
  5. Databases (slow, simple): 1000’s of MySQL servers are used as a persistence layer. InnoDB is used for the storage engine and multiple independent clusters are used for reliability. Joins are done at the webserver layer. The database layer is actually the persistence storage layer with memcached acting as a distributed index.

Other talks that seemed interesting

I had planned to a attend a few other talks as well. Unfortunately either their schedule was conflicting with another interesting presentation or the room was completely full (which seemed to happen all day long with the NoSQL room). Here is a list of them:

  1. NoSQL for Fun & Profit
  2. Introduction to MongoDB
  3. Cloudlets: universal server images for the cloud
  4. Continuous Packaging with Project-Builder.org

RFC: Boot-time configuration syntax for UEC/EC2 images

December 21, 2009

As part of the Boot-time configuration for UEC/EC2 images specification a configuration file can be passed to instances as user-data to customize some part of the instance without writing and maintaining custom scripts.

The goal is to support most common operations done on instance boot as well as help to bootstrap the instance to be part of an existing configuration management infrastructure.

It currently supports:

  • apt configuration
  • package installation

Other requested features looked into include:

  • runurl support
  • ssh host keys setup

Should these be included as well?

Here is an example of a configuration file (using YAML as the syntax):

# Update apt database on first boot
# (ie run apt-get update)
#
# Default: true
#
apt_update: false

# Upgrade the instance on first boot
# (ie run apt-get upgrade)
#
# Default: false
#
apt_upgrade: true

# Add apt repositories
#
# Default: none
#
apt_sources:

 # PPA shortcut:
 #  * Setup correct apt sources.list line
 #  * Import the signing key from LP
 #
 #  See https://help.launchpad.net/Packaging/PPA for more information
 #
 - source: "ppa:user/ppa"    # Quote the string

 # Custom apt repository:
 #  * Creates a file in /etc/apt/sources.list.d/ for the sources list entry
 #  * [optional] Import the apt signing key from the keyserver
 #  * Defaults:
 #    + keyserver: keyserver.ubuntu.com
 #    + filename: 00-boot-sources.list
 #
 #    See sources.list man page for more information about the format
 #
 - source: "deb http://archive.example.org lucid main restricted" # Quote the string
 keyid: 12345678 # GPG key ID published on a key server
 keyserver: keyserver.example.org
 filename: 01-mirror-example.org.list

 # Custom apt repository:
 #  * The apt signing key can also be specified
 #    by providing a pgp public key block
 #  
 #  The apt repository will be added to the default sources.list file:
 #  /etc/apt/sources.list.d/00-boot-sources.list
 #
 - source: "deb http://mirror.example.net/karmic/ ./" # Quote the string
 key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
 -----BEGIN PGP PUBLIC KEY BLOCK-----
 Version: SKS 1.0.10

 mI0ESXTsSQEEALuhrVwNsLIzCoaVRnrBIYraSUYCJatFcuvnhi7Q++kBBxx32JE487QgzmZc
 ElIiiPxz/nRZO8rkbHjzu05Yx61AoZVByiztP0MFH15ijGocqlR9/R6BMm26bdKK22F7lTRi
 lRxXxOsL2GPk5gQ1QtDXwPkHvAhjxGydV/Pcf81lABEBAAG0HUxhdW5jaHBhZCBQUEEgZm9y
 IE1hdGhpYXMgR3VniLYEEwECACAFAkl07EkCGwMGCwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAK
 CRANXKLHCU0EIIJHBAC1NCwdLwchCPIQU2bd562/YWcB7QSgYD3j+Llqm8v6ghFQ0Bdygbn1
 M6tzpwDiPxXQfZRqGhJsluCVHGLCQYNm0HDNisP4+YrZF3UkmAXDwZuh8K3LmvUPM+lLY8YJ
 1qnFHp3eN9M8/SYEFN0wlaVAurZD13NaU34UePd46vPtzA==
 =eVIj
 -----END PGP PUBLIC KEY BLOCK-----

# Add apt configuration files
#  Add an apt.conf.d/ file with the relevant content
#
#  See apt.conf man page for more information.
#
#  Defaults:
#   + filename: 00-boot-conf
#
apt_conf:

 # Creates an apt proxy configuration in /etc/apt/apt.conf.d/01-proxy
 - filename: "01-proxy"
 content: |
 Acquire::http::Proxy "http://proxy.example.org:3142/ubuntu";

 # Add the following line to /etc/apt/apt.conf.d/00-boot-conf
 #  (run debconf at a critical priority)
 - content: |
 DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt -p critical|| true";

# Provide debconf answers
#
# See debconf-set-selections man page.
#
# Default: none
#
debconf_selections: |     # Need to perserve newlines
 # Force debconf priority to critical.
 debconf debconf/priority select critical

 # Override default frontend to readline, but allow user to select.
 debconf debconf/frontend select readline
 debconf debconf/frontend seen false

# Install additional packages on first boot
#
# Default: none
#
packages:
 - openssh-server
 - postfix

I would like to get feedback on the format as well as ideas for other features, either on the wiki page or in the comments section.


RFP: packages to promote to main and demote to universe for Lucid Lynx LTS

November 30, 2009

The Ubuntu Server team is requesting feedback on the list of packages to be promoted to main and demoted to universe during this release cycle.

Lucid being an LTS release we wanna make sure that packages in main are maintainable for 5 years.  Useful packages should be promoted to main while packages that provide duplicated functionalities or are not maintained anymore should be demoted to universe.

The LucidServerSeeds wiki page is used to track packages under discussion. If you want to add a package to this discussion you should edit the relevant section (either Proposed universe demotions or Proposed main promotions) of the wiki page.

For example the current list of proposed packages to be moved to universe includes

  • nis
  • elinks
  • lm-sensors
  • sensord
  • cricket
  • radvd
  • logwatch
  • vlock
  • lilo
  • libxp6

The current packages being discussed for main promotion include acl, ctdb and tdb-tools  (to support Samba cluster). A switch from autofs 4 to autofs 5 is also under discussion.

Any feedback is welcome and should be added to the wiki page.


Oct 13 – Oct 16 Wrap-up

October 16, 2009

UEC

  • loads of testing. Uncovered new bugs and help Dusting fix most of them.
    • multiple installs on two sets of hardware in Montreal.
    • stress testing.
  • help scott and other to debug their UEC install.
  • review and upload image-store-proxy (working now).

Upgrades testing

  • help out mvo to add logic to handle mysql 5.0 upgrade from jaunty to karmic.
  • Support MySQL cluster setup.

Sponsoring

  • review and sponsor checkbox for marc.
  • review and sponsor landscape-client new upstream release.

Sep 28 – Oct 02 Wrap-up

October 5, 2009

Loads of Karmic Beta -server isos testing.

One day of UEC Beta testing: chased down with Dustin and Matt the failure of the auto-registration upstart scripts. Turns out to be a bug in Upstart – known by Scott who has a simple fix (dbus related).

Investigate failed RAID installation: this is a known boot loader issue. Added a section about it to the Karmic Release note.

Install UNR Karmic beta on my mini 10v. Write up blog post about it. Looks slick.

Put shorewall back into main. Fell off to universe due to a package rename in Debian.

More work on directory/krb5 infrastructure using puppet: add support for slapd modules and schemas to the puppet configuration. Slow progress towards a fully automated deployment of a directory+krb5 infrastructure for testing purposes in EC2.

Update server team knowledge with the lists of daily New,Undecided Bugs so that daily triaging can be kicked off. The lists are automatically generated on qa.ubuntu.com.


Test run: Ubuntu Netbook Remix 9.10 Beta on my Dell Mini 10v

October 1, 2009

Impressive for a beta release. Of course there are few glitches but overall it feels great: I’m writing this article from my Mini 10v running an Ubuntu Netbook Remix 9.10 Beta live system.

At the begining of the week I received a Dell Mini 10v I had ordered a few of weeks ago. I had chosen to upgrade some of the default components: my Mini 10v comes with 2Gb of RAM and a 16 GB SSD drive. And of course Ubuntu Hardy 8.04 LTS is installed by default at the factory. Now that the Beta of Karmic has been released I decided to take the opportunity to download the Ubuntu Netbook Remix iso and boot from a usb stick to see how this variant of Ubuntu looked like.

Load Ubuntu Netbook Remix on a USB key

But first things first. In order to be able to boot the UNR Beta iso, I had to put it on a usb stick. The USB Startup Disk Creator application located under System -> Administration proved to be best option:

  1. Download the UNR Beta iso image.
  2. Connect your usb key to the computer. I was actually using a 1GB SD card from my camera with an USB adapter.
  3. Open USB Startup Disk Creator.
  4. Select the UNR beta iso image and the usb drive (which may need to be formatted).
  5. And make the startup disk.

The boot experience

I plugged the usb stick in one of the Mini 10v usb port, powered on my netbook and hit F12 early in the boot sequence to bring the boot menu. And there – as the second choice – was my USB stick.

Loading the whole system took some time in which I had the time to admire the new boot experience – well I wasn’t that surprised as my main laptop had been running Karmic for a while now. But still it looked slick as the new black and white theme matched very well with my Mini 10v colours – black for most of the parts with a light grey stripe below the keyboard.

After being auto-logged in I was greeted with the new launcher and started to poke around. Turns out that tapping on the touch pad doesn’t work. I had to use the buttons at the bottom to actually click (which is a bit annoying since the pad is sensitive around the click area – it can lead to some mouse movement while trying to click).

No wireless available

Restricted driver popped up to tell me that I could install some non-free drivers. I had two choices all related to the wireless card:

  1. The B43xxx wireless driver. I tried to activate it: packages seemed to get installed – however the driver was still disabled after that.
  2. The STA wireless driver. Tried to activate it as well. This time the driver seemed to have correctly installed. However a reboot of the system was required – which is a bit annoying when you run from a live USB key.

Selecting each driver popped a prompt for entering a password in order to be able to install packages. Turns out the password is empty and just pressing the Enter key make things go away. I wonder if this dialogue could be completely deactivated during a live session – that would improve the experience of complete new user.

So no wireless available on my Mini 10v running from the live USB key. Time to plug a wired network cable. And a few seconds later I was connected to the Internet.

Application Names …

In the Favorites sub menu – which is the first thing you see when your session starts – there are a couple of applications: Mozilla Web Browser, Evolution Mail and Calendar, Cheese, Empathy, Help, Ubuntu One and Install Ubuntu Netbook Remix 9.10. All of these choices have recognizable names except for Cheese and Empathy. Of course I know about these being a long time Ubuntu user – however it may be more difficult for a first time user. Even though there is a small webcam as part of the Cheese icon and the Empathy icon kind of relates to communication having a descriptive name would probably be helpful.

… and Ubuntu One …

As for the Ubuntu One option, it doesn’t give a clue of what this is about. So my curious nature lead me to start the application (well… I knew what Ubuntu One was as I had been an early beta-tester). The Ubuntu One icon appeared in the top menu bar. I could go the web and log into my account by right-clicking on the icon. However I didn’t find an obvious way to associate my local instance with my remote account.

… and sound

Further poking around lead me to the Sound and Video sub menu where I tried to record a sound. First attempt failed. Opening the Volume Control from the File Menu and going to the Input tab showed me that the input was actually muted. Unmute it and voila – a few moments later I could hear my voice being played back!

So all in all I was pleasantly surprised by the beta version of UNR. A few glitches here and there (to be reported in LP of course) but overall the experience was positive!

Next step:

Actually install the system on the local SSD drive and experience the fast boot of Ubuntu on my Mini 10v. With an SSD drive I expect it to be below (9.)10 seconds.


Sep 20 – Sep 25 Wrap-up

September 27, 2009

Spent most of my week in Portland to attend conferences.

Conferences

  • Attended LDAPCon 2009 and published report.
  • Attended LinuxCon 2009.

Image Store Proxy

  • Updated image-store-proxy to 1.0. This version brings support for gpg signed images. Still need testing against the real-world Canonical Image Store infrastructure.

A summary of LDAPCon 2009

September 25, 2009

On Sunday, September 20th and Monday, September 21st I attended LDAPCon 2009 in Portland, OR. Most of the open source projects were there – with the notable absence of Port 389 (Redhat) – as well as some vendors (Apple and UnboundID). Most of the slides are available online.

Apache Directory project

The Apache Directory folks gave several presentations:

Apache Directory Server provides an integrated product with most of the standard network services: in addition to ldap, dns, dhcp, ntp and kerberos services can be enabled as part of a deployment. Kerberos support seems to be in early stage as it almost works. Another interesting aspect of the project is its integration with the Eclipse environment. Apache Directory Server is embedded in Apache Directory Studio. The latter provides a management tool for Directory administrator. If the Eclipse integration in Ubuntu is improved Apache Directory Studio would be a very good addition to the archive.

An overview of implementing replication in the Apache Directory Server project was given. RFC 4533 is used as the basis for LDAP replication in OpenLDAP. The goal here was to be able to replicate between Apache Directory Server and OpenLDAP. This may be the start to a standard replication protocol between different directory products.

Three components needed to be implemented:

  • the consumer part is the easiest and can be a standalone component. It receives LDAP entries updates and can do whatever it wants with them. It reminds me of similar requests I heard at the MySQL User Conference last April where people were interested in having an easier access to the MySQL replication log.
  • the producer is more complex to implement as it requires to keep a log of the modifications done on the server.
  • conflict resolution is the hardest part and mandatory if multi-master is to be supported. The Apache Directory Server decided to implement a strategy of last writer wins as they’re trying to not require any user intervention for conflict resolution. I’m not convinced this is the best strategy though.

While implementing replication support they’ve also added support for store procedures and triggers.

LSC Project: LDAP Synchronization Connector

Corporate environments usually have multiple identity repositories and keeping all of them in sync can be quite a challenge. The LSC project aims at automating the task of keeping all identity stores up-to-date. Written in java it can read and write to any database or LDAP directory. On-the-fly transformation of data sources are possible and the framework tries to make it easy to implement new synchronisation policies.

Another great tool that could be added to the directory administrator toolbox to help integrate Ubuntu in existing infrastructures.

Storing LDAP Data in MySQL Cluster (OpenLDAP and OpenDS)

This was a joined presentation between the OpenLDAP and OpenDS projects. A new backend has been added to store entries using the MySQL Cluster NDB API. The main advantage is to be able to access the same data over SQL and LDAP as well as providing a highly-available infrastructure with data distributed on multiple nodes. Both OpenDS and OpenLDAP have worked together to create a common data model highlighting that cooperation does happen in the LDAP space.

A Panel discussion among the representatives of the various LDAP Projects on roadmaps

Sunday ended up with a panel where representatives of different directory vendors answered questions from the audience. Each open source project briefly outlined a few points they were trying to improve: documentation for OpenLDAP, data migration for Apache Directory and multiple schema support for OpenDS. The issue of virtual directories was also discussed with the need of more GUIs to cover administration tools as well as workflows. Apache Directory Studio was given as a potential good starting point to build these higher level tools. The subject of standard ACL’s was also covered. It seems that this is still a sensitive issue in the community and projects are still arguing about a common solution. One option put forward was to look at the X500 ACL model and start from there.

The last item of discussion covered how to expand the user base of directories. The world of directories is rather small and its use cases are usually associated with Identity Management (User and Group, Authentication). Having good client APIs was mentioned as an option. However the whole group ran out of ideas quickly and got kind of stuck in front of this problem.

Directory Standardization Status

Directory standardization happens within two bodies: X500 in ISO/IEC and LDAP in IETF. The most important topic currently discussed in both bodies is password policies. A new draft of an IETF document is being worked on by Howard Chu and Ludovic Poitou.

Other topics being worked on cover:
  • Internationalization (with Unicode support in LDAPprep and SASLprep)
  • simple LDAP Transactions (to cover adding entries to different containers)
  • replacing DIGEST-MD5 with SCRAM
  • vCard support

On the front of Directory Application schemas support for NFSv4 Federated Filesystem and an Information Model for Kerberos are currently being worked on with drafts available for review.

The question of starting a new LDAP working group within the IETF was raised. Topics that could be covered include:
  • LDAP Chaining Operation
  • Access controls: based on the X.500 model with extensibility added.
  • LDIF update
  • LDAP Sync/ LDAP Sync-based Replication
  • Complex Transactions
  • Password Policies
  • Directory views
  • Schema versioning

LDAP in the java world

LDAP support in java is being actively worked on especially on the SDK front. OpenDS, Apache Directory Server and UnboundID have released new open-sourced SDKs to improve the aging JNDI and Netscape java SDKs. All of them are rather low-level implementations. The three projects are also working together to find a common ground.

There is also some progress made at the persistence level. The DataNucleus project gave an overview of adding LDAP support to the standard JDO interface. The goal is to provide a reference implementation of JDO for an LDAP data store.

Unified Authentication Service in OpenLDAP

Howard Chu gave an overview of the new modules developed in OpenLDAP related user authentication. Based on the work from nss-ldapd the nssov overlay provides integration with the pam stack as well as the nss stack. Disconnected mode in the pcache overlay has been added in the latest version of openldap as discussed during the Ubuntu Developer Summit last May. Most of this work is already available in Ubuntu Karmic and improvements should be made during the Lucid release cycle.

Another interesting module is the integrated certification authority. If a search request for the userCertificate and userKey attributes for an entry is made and these attributes don’t exist they’re generated on the fly. This should help out in creating an X.509 base PKI.

LDAP Innovations in the OpenDS project

The last session of the conference was given by Ludovic Poitou of the OpenDS project. New features available in OpenDS include tasks as well as extended syntax rules. Time matching rules have also been added so that queries like “give me entries that have a last login time older than 3 weeks” can be expressed directly in ldap and processed by the server. That brought some interesting issues when clients and servers don’t share the same timezone.

A few gems from beer conversations

After the official sessions ended most of the attendees congregated to have diner followed by beers. Howard showcased his G1 phone running slapd while Ludovic was showing off an LDAP client application on his iPhone. And of course by then end of the conference both systems were connected: the iPhone was able to look up contact information on the G1 running slapd.

On an unrelated note OpenLDAP is faster than OpenDS, even in beer drinking. However the OpenLDAP project was compared to a Beetle car with a Porsche engine whereas OpenDS was actually building a Porsche.

Even though not all the players in the directory space were represented at the conference, most of the key players from the open source world were there presenting their work. Friendly competition exists amongst the different projects which turns into cooperation on topics that matters such as interoperability and data formats.

It seems that the directory world is rather small and its use cases are restricted to specific situations compared to RDBMS. This is rather unfortunate as directories offer a compelling alternatives to databases as a data store infrastructure. The community seems to be aware of this issue and is looking into breaking out of its traditional fields of applications.


Sep 11 – Sep 18 Wrap-up

September 18, 2009

Image-store-proxy

Package image-store-proxy to enable the Image Store tab in Eucalyptus. The package (python-image-store-proxy) has made its way to main and on the -server isos in time for alpha6 with the help of Thierry and Kees.

Server-karmic-directory-enabled-user-login

Kept on investigating the use of puppet to build an ldap/krb5 infrastructure on EC2. Integrated dnsmasq and puppetmaster configuration. Discovered a few bugs along the way and reported them upstream. My current work is available from an LP branch. And puppet is awesome!

Alpha6 ISO testing

Loads of alpha6 testing.

Landscape-client Stable Release Update

Reviewed the landscape-client and smart SRU requests from the Landscape team.

Bug scripts

With the help of Brian my bug scripts are now regularly run on qa.ubuntu.com. All bug lists used in the SRU review and the triaging process can be found on qa.ubuntu.com.

Misc

Updated my status report script to publish a draft of my activity report on my blog as the weekly “wrap-up”.