a thin god

Nothing spirtual here, but the cool monitoring framework god together with thin. You’ve probably read a lot of benchmarks about thin, but whats the fastest webserver when you can’t ensure they are up and running ? Right here is where god comes into play. God is (beside the silly name) a nice process monitoring framework with a ruby configuration script. You can easily start/stop/monitor daemon processes and put advanced flags on them, like CPU or memory usage.

The default god startup file looks quite neat if you only run one site, and is easily converted to work together with thin, as thin and mongrel share the same parameters. Now i don’t have the luxury of only supporting one site but rather quite a buch of (mostly small and inactive) sites which share quite a lot of critera. Creating this kind of config file for each of them seems quite exessive and against the rails DRY principle.

Now, with a recent thin version (I did try 0.6.3) you can have the configuration for each site in a seperate YAML file, there’s even a small god startup script included in the examples directory.

Now first, lets take a look at the thin yaml config of a test ruby installation, in this case test.yml in /etc/thin :

servers: 3
user: www-data
group: www-data
chdir: /var/www/rails
pid: tmp/pids/test
port: 8000
address: 0.0.0.0
log: log/thin.log

Most is pretty self explainig, the pid and logfile is relative to the startdir, and the uid/gid is set to www-data (the default on ubuntu machines)
Unfortunately the thin.god startup file in thin 0.6.3 seems to have a small bug when it comes to allocating ports (or i’m doing something wrong, but the config file works flawlessly with just thin), so here’s a fixed version of the thin.god file:

# == God config file
# http://god.rubyforge.org/
# Author: Gump
#
# fixed thin ports <michael@glauche.de>
#
# Config file for god that configures watches for each instance of a thin server for
# each thin configuration file found in /etc/thin.

require 'yaml'

config_path = "/etc/thin"

Dir[config_path + "/*.yml"].each do |file|
  config = YAML.load_file(file)
  num_servers = config["servers"] ||= 1
  for i in 0...num_servers
    God.watch do |w|
      w.group = "thin-" + File.basename(file, ".yml")
      port = config["port"] + i

      w.name = w.group + "-#{port}"

      w.interval = 30.seconds

      w.uid = config["user"]
      w.gid = config["group"]

      w.start = "thin start -C #{file} -o #{port}"
      w.start_grace = 10.seconds

      w.stop = "thin stop -C #{file} -o #{port}"
      w.stop_grace = 10.seconds

      w.restart = "thin restart -C #{file} -o #{port}"

      pid_path = config["chdir"] + "/" + config["pid"]
      ext = File.extname(pid_path)

      w.pid_file = pid_path.gsub(/#{ext}$/, ".#{port}#{ext}")

      w.behavior(:clean_pid_file)

      w.start_if do |start|
        start.condition(:process_running) do |c|
          c.interval = 5.seconds
          c.running = false
        end
      end
      w.restart_if do |restart|
        restart.condition(:memory_usage) do |c|
          c.above = 150.megabytes
          c.times = [3,5] # 3 out of 5 intervals
        end

        restart.condition(:cpu_usage) do |c|
          c.above = 50.percent
          c.times = 5
        end
      end

      w.lifecycle do |on|
        on.condition(:flapping) do |c|
          c.to_state = [:start, :restart]
          c.times = 5
          c.within = 5.minutes
          c.transition = :unmonitored
          c.retry_in = 10.minutes
          c.retry_times = 5
          c.retry_within = 2.hours
        end
      end
    end
  end
end

Note, there was a bug with older god installations and ubuntu installations, but they worked flawlessly for me with the 0.7.0 god release. (there were some problems with the 0.5.0 release and ubuntu, so check if you have an up to date god version)

It will look in /etc/thin for all .yml files and will start and supervise them accordlingly. To start up the monitoring use

 # god -c thin.god

After its up and running you can check the status of your servers with:

 # god status

That command should give the following output:

thin-test-8000: up
thin-test-8001: up
thin-test-8002: up