MySQL 5.1 Bug Zap: Bug day result

March 30, 2010

Today was targeted at looking through the mysql-dfsg-5.1 bugs to triage them. We ended up with all bugs having their importance set and their status set to Triaged or Incomplete.

Tomorrow will be dedicated to fixing most of them as well as some upgrade testing. I’ll also have a look at the new mysql upstart job that has replaced the mysld_safe script.

Looking at the bugs today I’ve found a couple of bugs that look easy to fix:

  • Bug 552053:  mysqld_safe should be available in mysql-server
  • Bug 498939: mysql- packages section on synaptic

To get started grab a copy of the package branch:

bzr init-repo mysql-dfsg-5.1/
cd mysql-dfsg-5.1/
bzr branch lp:ubuntu/mysql-dfsg-5.1

Fix a bug and push the branch to launchpad:

bzr push lp:~YOUR-LOGIN/ubuntu/mysql-dfsg-5.1/zap-bug-XXXXXX

And finish up by creating a merge proposal for the Lucid package branch. I’ll take a look at the list of merge proposal throughout the day and include them in the upload schedule for tomorrow.

Ubuntu Server Bug Zap: MySQL 5.1

March 29, 2010

Following up on the kvm and samba bug zap days I’m organizing a two day bug
zap around MySQL.

First phase: bug triaging

First in line is triaging all the bugs related to mysql-dfsg-5.1 package. As
of Tue Mar 30 00:23:04 UTC 2010 there are 27 bugs waiting to be looked at.

The goal is to have the importance set for all bugs and have as many bugs
status moved to either Triaged or Invalid/Won’t Fix.

A few resources are available to help out:

Objective: get the list of bugs to zero.

Using puppet in UEC/EC2: Automating the signing process

March 25, 2010

I outlined in the previous article how to setup a puppetmaster instance on UEC/EC2 and how to start instances that will automatically register with the puppetmaster. We’re going to look at automating the process of signing puppet client certificate requests.


Our puppet infrastructure on the cloud can be broken down into three components:

  • The Cloud conductor responsible for starting new instances in our cloud.
  • A Puppetmaster responsible for configuring all the instances running in our cloud.
  • Instances acting as puppet clients asking to be setup correctly.

The idea is to have the Cloud conductor start instances and notify the puppetmaster that these new instances are coming up. The puppetmaster can then automatically sign their certificate requests.

We’ll use S3 as the way to communicate between the Cloud conductor and the puppetmaster. The Cloud conductor will also assign a random certificate to each instance it starts.

The Cloud conductor will be located on a sysadmin workstation while the puppetmaster and instances will be running in the cloud. The bzr branch contains all the scripts necessary to setup such a solution.

The Cloud conductor:

  1. Get the tutorial2 bzr on the Cloud conductor (an admin workstation):

    bzr branch lp:~mathiaz/+junk/uec-ec2-puppet-config-tut2

    In the scripts/ directory plays the role of the Cloud conductor. It creates new instances and stores their certname in S3. The start_instance.yaml configuration file provides almost the same information as the user-data.yaml file we used in the previous article.

  2. Edit the start_instance.yaml file and update each setting:

    • Choose a unique S3 bucket name.
    • Use the private DNS hostname of the instance running the puppetmaster.
    • Add the puppetmaster ca certificate found on the puppetmaster.
  3. Make sure your AWS/UEC credentials are available in the environment. The uses these to access EC2 to start new instances and S3 to store the instance certificate names.

  4. Start a new instance of the Lucid Beta1 AMI:

    ./ -c ./start_instance.yaml ami-ad09e6c4 starts a new instance using the AMI specified on the command line. The instance user data holds a random UUID for the puppet client certificate name. also creates a new file in its S3 bucket named after the puppet client certificate name.

  5. On the puppetmater looking at the puppetmaster log you should see a certificate request show up after some time:

    Mar 19 19:09:33 ip-10-245-197-226 puppetmasterd[20273]: a83b0057-ab8d-426e-b2ab-175729742adb has a waiting certificate request

Automating the signing process on the puppetmaster

It’s time to setup the puppetmaster to check if there are any certificate requests waiting and signs only the ones started by the Cloud conductor. We’ll use the cron job that will get the list of waiting certificate requests via puppetca --list and checks whether there is a corresponding file in the S3 bucket.

  1. On the puppetmaster get the tutorial2 bzr branch:

    bzr pull –remember lp:~mathiaz/+junk/uec-ec2-config/tut2 /etc/puppet/

  2. The puppetmaster.pp manifest has been updated to setup the cron job to run every 2 minutes. You need to update the cron job command line in /etc/puppet/manifests/puppetmaster.pp with your own S3 bucket name.

  3. Update the puppetmaster configuration:

    sudo puppet /etc/puppet/manifests/puppetmaster.pp

  4. Watching /var/log/syslog you should see check_csr being run by cron every other minute:

    Mar 19 19:10:01 ip-10-245-197-226 CRON[21858]: (root) CMD (/usr/local/bin/check_csr –log-level=debug

    check_csr gets the list of waiting certificate requests and checks if there is a corresponding file in its S3 bucket:

    Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: List of waiting csr: a83b0057-ab8d-426e-b2ab-175729742adb
    Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: Checking a83b0057-ab8d-426e-b2ab-175729742adb
    Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: DEBUG: Checking url

    If so it will sign the certificate request:

    Mar 19 19:10:03 ip-10-245-197-226 check_csr[21859]: INFO: Signing request: a83b0057-ab8d-426e-b2ab-175729742adb

S3 bucket ACL

For now the S3 bucket ACL is set so that anyone can get the list files available in the bucket. However only authenticated requests can create new files in the bucket. Given that the filename are just random UUID this is not a big issue.

Using SQS instead of S3

Another implementation of the same idea is to use SQS to handle the notification of the puppetmaster by the Cloud conductor about new instances. While SQS would seem to be the best tool to provide that functionality it is not available in UEC in Lucid.


We end up with a puppet infrastructure where legitimate instances are automatically accepted. Now that instances can easily show up and be automatically enrolled what should these be configured as? We’ll dive into this issue in the next article.

Using puppet in UEC/EC2: puppet support in Ubuntu images

March 24, 2010

One of the focus for the Lucid release cycle in the Ubuntu Server team is to improve the integration between puppet and UEC/EC2. I’ll discuss in a series of articles how to setup a puppet infrastructure to manage Ubuntu Lucid instances running on UEC/EC2. I’ll focus on the bootstrapping process rather than writing puppet recipes.

Today we’ll look at configuring a puppetmaster into an instance and how to start instances that will register automatically with the puppetmaster.

We’ll work with the Lucid Beta1 image on EC2. All the instances started through out this article will be based on this AMI.

Puppetmaster setup

Let’s start by creating a puppetmaster running on EC2. We’ll setup all the puppet configuration via ssh using a bzr branch on Launchpad: lp:~mathiaz/+junk/uec-ec2-puppet-config-tut1.

Start an instance of the Lucid Beta1 AMI using an ssh key. Once it’s running write down its public and private DNS addresses. The public DNS address will be used to setup the puppetmaster via ssh. The private DNS address will be used as the puppetmaster hostname given out to puppet clients.

We’ll actually install the puppetmaster using puppet itself.

Log on the started instance via ssh to install and setup the puppet master:

  1. Update apt files:

    sudo apt-get update

  2. Install the puppet and bzr packages:

    sudo apt-get install puppet bzr

  3. Change the ownership of the puppet directory so that the ubuntu user can directly edit the puppet configuration files:

    sudo chown ubuntu:ubuntu /etc/puppet/

  4. Get the puppet configuration branch:

    bzr branch –use-existing-directory lp:~mathiaz/+junk/uec-ec2-puppet-config-tut1 /etc/puppet/

    Before doing the actual configuration let’s have a look at the content of the /etc/puppet/ directory created from the bzr branch.

    The layout follows the recommended puppet practices. The puppet module available in the modules directory defines a puppet::master class. The class makes sure that the puppetmaster package is installed and that the puppetmaster service is running. The manifests/puppetmaster.pp file defines the default node to be configured as a puppetmaster.

  5. We’ll now run the puppet client to setup the instance as a puppetmaster:

    sudo puppet /etc/puppet/manifests/puppetmaster.pp

Starting a new instance

Now that we have puppetmaster available in our cloud we’ll have look at how a new instances of the Lucid Beta1 AMI can be started and automatically setup to register with the puppetmaster.

We’re going to use the cloud-config puppet syntax to boot an instance and have it configure itself to connect to the puppetmaster using its user data information:

  1. On the puppetmaster instance create a user-data.yaml file to include the relevant puppetmaster configuration:

    cp /usr/share/doc/cloud-init/examples/cloud-config-puppet.txt user-data.yaml

  2. Update the server setting to point to the puppetmaster private dns hostname. I also strongly recommend to include the puppmaster ca certificate as the ca_cert setting.

    The example certname setting uses a string extrapolation to make each puppet client certificate unique: for now %i is replace by the instance Id while %f is replaced by the FQDN of the instance.

    The sample file has extensive comments about the format of the file. One of the key point is that you can set any of the puppet configuration options via the user data passed to the instance.

    Note that you can remove all the comments to make the user-data.yaml file easier to copy and paste. However don’t remove the first line (#cloud-config) as this is used by the instance boot process to start the puppet installation.

  3. Launch a new instance using the content of the user-data.yaml file you’ve just created as the user-data option passed to the new instance.

  4. You can watch the puppetmaster log on the puppetmaster instance to see when the new instance will request a new certificate:

    tail -f /var/log/syslog

  5. After some time you should see a request coming in:

    puppetmasterd[2637]: i-fdb31b96.ip-10-195-18-227.ec2.internal has a waiting certificate request

    During the boot process of the new instance the puppet cloud-config plugin used the user-data information to automatically install the puppet package, generate the /etc/puppet/puppet.conf file and start the puppetd daemon.

  6. You can then approve the new instance:

    sudo puppetca -s i-fdb31b96.ip-10-195-18-227.ec2.internal

  7. Watching the puppetmaster log you’ll see that after some time the new instance will connect and get its new manifest compiled and sent:

    puppetmasterd[2637]: Compiled catalog for i-fdb31b96.ip-10-195-18-227.ec2.internal in 0.03 seconds

In conclusion we now have an instance acting as a puppetmaster and have a single user-data configuration for the whole puppet infrastructure. That user data can be passed to new instances which will automatically register with our puppetmaster.

Even though we’re able to make all our instances automatically register with our puppetmaster we still need to manually sign each request as outlined in step 6 above. We’ll have a look at automating this step in the next article.


February 17, 2010

I had the opportunity to attend FOSDEM this year. The most amazing (and frustrating) part of the event was the huge number of talks that were given. Making choices was sometimes hard. However the FOSDEM team recorded most of the sessions and the videos are available now. Perfect for such a busy event!

Here a few highlights of the presentations I attended:

Linux distribution for the cloud

I started the day by attending Peter Eisentraut (a Debian and PostgreSQL core developer) session about Linux distributions for the cloud. He focused on the provisioning aspect of clouds by giving a history of how operating systems were installed: from floppy drives to Cloud images. He dedicated one slide to Ubuntu’s cloud offering including Canonical Landscape commenting that Ubuntu is clearly the leader of distributions in the cloud space. He also outlined what were the current problems such as lack of standards and integration of existing software stacks. He pointed out that linux distributions could drive this.

The second part of his talk was focused on the Linux desktop and the impact of cloud services on it. Giving ChromeOS as an example he outlined how applications themselves were moved to the cloud. He then listed the problems with Cloud services with regards to the Free Software principles: non-free server side code, non-free hosting, little or not control over data, lack of open-source community.

He concluded by outlining the challenge in the domain: how could free software principles be transposed to the Cloud and its services? One great reference is Eben Moglen talk named “Freedom in the Cloud”.

Beyond MySQL GA

Kristian Nielsen, a MariaDB developer, gave an overview of the Developer ecosystem around MySQL. He listed a few patches that were available to add new functionalities and fix some bugs: the Google patch, Percona patches, eBay patches, Galera multi-master replication for InnoDB as well as a growing list of storage engines. Few options are available to use them:

  • packages from third party repositories (such as ourdelta and percona)
  • MariaDB maintains and integrates most of the patches
  • a more do-it-yourself approach where you maintain a patch serie.

I talked with Kristian after the presentation about leveraging bzr and LP to make the maintenance easier. It could look like this:

  1. Each patch would be available and maintained in a bzr branch – in LP or else where.
  2. The Ubuntu MySQL package branch available in LP would be used as the base for creating a debian package (or the Debian packaging branch since Debian packages are also available in LP via bzr)
  3. bzr-looms would glue the package bzr branch with the patches bzr branches. The loom could be available from LP or elsewhere.
  4. bzr-builder would be used to create a recipe to build binary packages out of the loom branch.
  5. Packages would be published in PPAs ready to be installed on Ubuntu systems.

The Cassandra distributed database

I finally managed to get in the NoSQL room to attend Eric Evans overview of the Cassandra project. He is a full time developer and employee of Rackspace. The project was started by Facebook to power their inbox search. Even though the project had been available for some years the developer community really started to grow in March 2009. It is now part of the Apache project and about to graduate to a Top Level Project there.

It is inspired by Dynamo from Amazon and provide a O(1) DHT with eventual consistency and a consistent hashing. Multiple client APIs are available:

  • Thrift
  • Ruby
  • Python
  • Scala

I left before the end of the talk as I wanted to catch the complete presentation about using git for packaging.

Cross distro packaging with (top)git

Thomas Koch gave an overview of using git and related tools to help in maintaining Debian packaging. He works in web shop where every web application is deployed as a Debian package.

The upstream release tarball is imported in a upstream git branch using the pristine-tar tool. Packaging code (ie the debian/ directory) is kept in a different branch.

Patches to the upstream code are managed by topgit as seperate git branches. He also noted that topgit was able to export the whole stack of patches in the quilt Debian source format using the tg export command.

Here is the list of tools associated with his workflow:

  • pristine-tar
  • git-buildpackage
  • git-import-orig
  • git-dch
  • topgit

The workflow he outlined looked very similar to the one based around bzr and looms.

Scaling Facebook with OpenSource tools

David Recordon from Facebook gave a good presentation on the challenges that Facebook runs into when it comes to scale effectively.

Here are a few numbers I caught during the presentation to give an idea about the scale of the Facebook infrastructure (Warning: they may be wrong – watch the video to double check):

  • 8 billion minutes spent on Facebook every day
  • 2.5 billions of pictures uploaded every month
  • 400 billion page/view per month
  • 25 TB of log per day
  • 40 billions pictures stored in 4 resolutions which bring a grand total of 160 billions photos
  • 4 millions of line codes in php

Their overall architecture can be broken into the following components:

  1. Load balancers
  2. Web server (php): Most of the code is written in PHP: the language is simple, it fits well for fast development environments and there are a lot of developers available. A few of the problems are CPU, Memory, how to reuse the PHP logic in other systems and the difficulty to write extensions to speed up critical parts of the code. An overview of the HipHop compiler was given: a majority of their PHP code can be converted to C++ code which is then compiled and deployed on their webserver. An apache module is coming up soon probably as a fastcgi extension.
  3. memcached (fast, simple): A core component of their infrastructure. It’s robust and scales well: 120 millions queries/second. They wrote up some patches which are now making their way to upstream.
  4. Services (fast, complicated): David gave an overview of some of the services that Facebook opensourced:
    • Thrift: an rpc server, now part of the Apache incubator.
    • Hive: build on top of hadoop it is now part of the Apache project. It’s an SQL-like frontend to hadoop aiming at simplifying access to the hadoop infrastructure so that more people (ie non-engineers) can write and run data analysis jobs.
    • Scribe: a performant and scalable logging system. Logs are stored in a hadoop/hive cluster to help in data analysis.
  5. Databases (slow, simple): 1000’s of MySQL servers are used as a persistence layer. InnoDB is used for the storage engine and multiple independent clusters are used for reliability. Joins are done at the webserver layer. The database layer is actually the persistence storage layer with memcached acting as a distributed index.

Other talks that seemed interesting

I had planned to a attend a few other talks as well. Unfortunately either their schedule was conflicting with another interesting presentation or the room was completely full (which seemed to happen all day long with the NoSQL room). Here is a list of them:

  1. NoSQL for Fun & Profit
  2. Introduction to MongoDB
  3. Cloudlets: universal server images for the cloud
  4. Continuous Packaging with

RFC: Boot-time configuration syntax for UEC/EC2 images

December 21, 2009

As part of the Boot-time configuration for UEC/EC2 images specification a configuration file can be passed to instances as user-data to customize some part of the instance without writing and maintaining custom scripts.

The goal is to support most common operations done on instance boot as well as help to bootstrap the instance to be part of an existing configuration management infrastructure.

It currently supports:

  • apt configuration
  • package installation

Other requested features looked into include:

  • runurl support
  • ssh host keys setup

Should these be included as well?

Here is an example of a configuration file (using YAML as the syntax):

# Update apt database on first boot
# (ie run apt-get update)
# Default: true
apt_update: false

# Upgrade the instance on first boot
# (ie run apt-get upgrade)
# Default: false
apt_upgrade: true

# Add apt repositories
# Default: none

 # PPA shortcut:
 #  * Setup correct apt sources.list line
 #  * Import the signing key from LP
 #  See for more information
 - source: "ppa:user/ppa"    # Quote the string

 # Custom apt repository:
 #  * Creates a file in /etc/apt/sources.list.d/ for the sources list entry
 #  * [optional] Import the apt signing key from the keyserver
 #  * Defaults:
 #    + keyserver:
 #    + filename: 00-boot-sources.list
 #    See sources.list man page for more information about the format
 - source: "deb lucid main restricted" # Quote the string
 keyid: 12345678 # GPG key ID published on a key server

 # Custom apt repository:
 #  * The apt signing key can also be specified
 #    by providing a pgp public key block
 #  The apt repository will be added to the default sources.list file:
 #  /etc/apt/sources.list.d/00-boot-sources.list
 - source: "deb ./" # Quote the string
 key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
 Version: SKS 1.0.10


# Add apt configuration files
#  Add an apt.conf.d/ file with the relevant content
#  See apt.conf man page for more information.
#  Defaults:
#   + filename: 00-boot-conf

 # Creates an apt proxy configuration in /etc/apt/apt.conf.d/01-proxy
 - filename: "01-proxy"
 content: |
 Acquire::http::Proxy "";

 # Add the following line to /etc/apt/apt.conf.d/00-boot-conf
 #  (run debconf at a critical priority)
 - content: |
 DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt -p critical|| true";

# Provide debconf answers
# See debconf-set-selections man page.
# Default: none
debconf_selections: |     # Need to perserve newlines
 # Force debconf priority to critical.
 debconf debconf/priority select critical

 # Override default frontend to readline, but allow user to select.
 debconf debconf/frontend select readline
 debconf debconf/frontend seen false

# Install additional packages on first boot
# Default: none
 - openssh-server
 - postfix

I would like to get feedback on the format as well as ideas for other features, either on the wiki page or in the comments section.

RFP: packages to promote to main and demote to universe for Lucid Lynx LTS

November 30, 2009

The Ubuntu Server team is requesting feedback on the list of packages to be promoted to main and demoted to universe during this release cycle.

Lucid being an LTS release we wanna make sure that packages in main are maintainable for 5 years.  Useful packages should be promoted to main while packages that provide duplicated functionalities or are not maintained anymore should be demoted to universe.

The LucidServerSeeds wiki page is used to track packages under discussion. If you want to add a package to this discussion you should edit the relevant section (either Proposed universe demotions or Proposed main promotions) of the wiki page.

For example the current list of proposed packages to be moved to universe includes

  • nis
  • elinks
  • lm-sensors
  • sensord
  • cricket
  • radvd
  • logwatch
  • vlock
  • lilo
  • libxp6

The current packages being discussed for main promotion include acl, ctdb and tdb-tools  (to support Samba cluster). A switch from autofs 4 to autofs 5 is also under discussion.

Any feedback is welcome and should be added to the wiki page.