Sam Pointer

I read an interesting paper that was presented at USENIX recently. It is worth going away to read it - it’s only five pages long.

The rub, if you are lazy, is that not all instances of a given instance type on Amazon EC2 are created equally. There is a range of gear you can land on when you stand up an m1.xlarge or a c1.medium, and as expected, performance therefore varies (sometimes quite significantly) within an instance type. 

This set me thinking. We have Chef and Ohai (although this applies equally to Puppet and Facter) storing a bunch of attributes about our nodes in our index. This could be used to see what the distribution of ‘good’ servers was over our estate and allow us to see where we were not getting the most performant kit for our outlay. This is important, as perhaps if we landed on the most performant kit we could remove a few nodes in a tier and save ourself some money. Or perhaps we might want to avoid a given instance type altogether.

I’m a big fan of using Sinatra to build quick dashes. I’m also a big fan of using Rails to front-up these self-contained dashes, but that is another post for another day. So, I sat down and wrote what amounted to 44 lines of ruby/sinatra using the chart_topper gem, which provides a nice way to work with gruff under Sinatra.

The Sinatra app queries the Chef API and therefore index and pulls out a bunch of CPU profile information. It then uses this to plot a distribution of hardware for each given instance type. The output looks like this.

The greater the surface area of the graph, the more performance we’re getting for our outlay over the whole estate. The further up a given axis, the better the range of kit we have for that instance type over the estate. To me this kind of thing demonstrates the 50% of configuration management tools that most people miss: estate intelligence. We can collect inumerable metrics about individual nodes and whole estates through Ohai and Facter. Exposing this data in meaningful ways is a return on the investment in configuration management that the business as a whole can see, use and understand. 

Moreover, these kinds of views are compelling insofar as they are “real-time”, where real-time is your client execution interval and splay period.

I’m speaking Tueday 15th of January at the LDN Devops conference in London on this and a number of other topics regarding Chef implementations at scale. If you can’t attend, then follow #ldndevops on Twitter and look out for the live stream.

Introduction

The Ruby world moves at an astounding pace. Pat Shaughnessy wrote an excellent series of articles in December 2011 documenting the options avaliable for using Twitter’s Bootstrap framework version 1.3 with Rails 3.1. At the time of writing Bootstrap has moved onto version 2.0, Rails is on 3.2.1 and Pat’s example application no longer builds as described.

A Slight Digression

In the rest of this post I’ll explain what little needs to be done if you’d like to follow those articles but use Rails 3.2 and Bootstrap 2.0, but first a quick digression on Bootstrap.

For the visually inept and graphically challenged amongst us, a set of professional and consistent design elements is a God-send. I’ve been using Perl (CGI.pm through to the later frameworks) and Rails since version 1.x to generate front-ends and dashboards and the like for all sorts of Infrastructure and traditional sysadmin tasks.

It just so happens that the Devops world follows, in part, the same route: assuming that system administrators can develop something other than shell script splattered with global variables, adopting Ruby the language from which the most prominent tools are built, and absorbing a huge amount from the Rails world: be it RESTful web services, rapid development or DRY. So, it is nice to finally be able to produce tools that look nice if for no other reason than quite often some fantastic operational work is trivialized and missed as it’s fronted by a bunch of crap in cgi-bin spitting out table elements.

Digression over, here’s what you need to do.

formtastic-bootstrap

cgunther has kindly done the hard work in getting mjbellantoni’s formtastic-bootstrap working with Bootstrap 2.0 in his bootstrap2-rails3-2-formtastic-2-1 branch. However, it requires the 2.1 version of formtastic, which is still in rc at the time of writing. However, the following in your Gemfile should do it:

gem 'formtastic', :git => 'git://github.com/justinfrench/formtastic.git', :branch => '2.1-stable'
gem 'formtastic-bootstrap', :git => 'https://github.com/cgunther/formtastic-bootstrap.git', :branch => 'bootstrap2-rails3-2-formtastic-2-1'

Also, when editing your ./app/views/layouts/application.html.erb you should use the new Bootstrap 2.0 classes:

<body>

<div class="navbar navbar-fixed-top">
 <div class="navbar-inner">
  <div class="container">
   <a class="brand" href="#">OrigamiHub</a>
   <div class="nav-collapse">
    <%= tabs %>
   </div>
  </div>
 </div>
</div>

<div class="container">
  <%= yield %>
</div>

</body>

Finally, you should heed the formtastic deprecation options, and construct your semantic forms thus:

<%= semantic_form_for @widget do |f| %>
  <%= f.semantic_errors %>
  <%= f.inputs do %>
    <%= f.input :name, :hint => "The wangdoodle is a best-seller" %>
    <%= f.input :type, :hint => "We only do three sizes!" %>
  <% end %>
  <%= f.actions do %>
    <%= f.action(:submit) %>
  <% end %>
<% end %>

tabulous

You’ll need to use the:

config.tabs_ul_class = "nav nav-pills"

option within tabulous to get the navigation bar to behave properly, along with the options recommended in the original article.

Conclusion

I’m sure the work above will be merged back into the mainline for each of the respective gems, and it’s a tribute to the community of github that such a lot of good work is given and fixed freely.

Audience

The follow steps will allow you to start a Rails instance automatically on boot under an Ubuntu instance in the simplest possible way. If you know about alternative Rails servers, deployment architectures beyond “webserver + mod_proxy + Mongrel” and the like then you are not the intended audience for this post.

Below is the simplest possible thing you can do to start a Rails instance automatically when you reboot your Ubuntu server, in keeping with the startup scripts that are already there. I hope that this is useful for someone starting out.

Steps

1. save the following script in a file called ‘rails’ within the /tmp directory
2. edit the USER, PORT, RAILS_ROOT variables to suit your application
3. run the following commands:

sudo cp /tmp/rails /etc/init.d
sudo update-rc.d rails defaults

Reboot and test.

Script

#! /bin/sh
### BEGIN INIT INFO
# Provides:          rails
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start a Rails instance
# Description:       Do the simplest thing possible in keeping with
#             upstart to spin up a single Rails instance.
### END INIT INFO

# Author: Sam Pointer 
#
# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
USER="myappuser"
PORT=3000
RAILS_ROOT="/home/myapp/current"
COMMAND="ruby script/server -e production -p $PORT -d"
DESCRIPTION="Rails instance"

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    su -c "cd $RAILS_ROOT && $COMMAND" $USER
}

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESCRIPTION"
        do_start
        case "$?" in
                0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
                2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
        esac
        ;;
esac

I’ve recently had cause to try to parse the output of the `mailq’ command, by dint of it being the common queue output format between sendmail and postfix, and a desire to interoperate with them both. It is a complete bastard!

Here is some to save you running the command:

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
637F5CFF9C*    1497 Sat Dec 18 21:40:34  sender@domain.com 
                                          recepient@domain.com

4C4D5A32E6*    1481 Sat Dec 18 21:39:45  sender@domain.com 
                                         recepient@domain.com

EBBE4D5C93*    1481 Sat Dec 18 21:51:52  sender@domain.com 
                                         recepient@domain.com

 ...


36BAAA3C9E     1471 Sat Dec 18 18:19:16  sender@domain.com 
(delivery temporarily suspended: host mailxx-xx.xxxx.aol.com[xxxxxx] refused to talk to me: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html)
                                         recepient@domain.com
  
350FE10A66C     1438 Wed Dec 15 01:33:38  sender@domain.com 
(lost connection with mx-c1.xxxx.net[61.20.xx.xx] while receiving the initial server greeting)
                                          recepient@domain.com

...

-- 359752 Kbytes in 174586 Requests.

This is awkward to parse because:

• The queue to which a mail identified by a given queue-id belongs to is denoted by a single character appended to the queue-id. Except if you’re in the deferred queue, in which case there is no special character - an atribute by absence, if you like.
• The ‘arrival time’ doesn’t include a year. We either have to assume that the year is the current year and open ourselves up to year-end bugs, or we have to compensate for edge cases with time and boundary checks.
• Records span multiple lines, and how many depends on the queue that the mail is in. This leads to nasty “if the last character of the first field of the following line was not an asterix of exclamation mark read two more line, otherwise read three more lines. And that field that was on the second line before is now on the third” logic in parsers.

  Whilst parsable with somewhat icky code, it just seems designed not to be parsed or piped: “records” spanning multiple lines is a definite no-no when you’re ‘xargs’-ing or piping through awk or anything else remotely UNIXy. I wonder what was going through the minds of the original authors of the `sendmail -bp` option, that `mailq` ultimately emulates?

  I’m currently working on some Ruby code to parse the output of `mailq` and to populate an ActiveRecord model with the appropriate attributes. Wrapped up as a gem, it would be a convenient way to provide Rails applications access to the mail queue.