Pete Keen

Articles tagged with 'Programming'

Using Stripe Checkout for Subscriptions

30 March 2014

Stripe provides a gorgeous pre-built credit card form called Stripe Checkout. Checkout is mainly intended for one-off purchses like Dribbble or my book. Many people want to use it for their Stripe-powered subscription sites so in this article I'm going to present a good way of doing that.

Tagged: Programming  Stripe 

Read More

Self-hosted Git Server

16 March 2014

I've had a GitHub account since 2008. June 16th, to be exact. For almost six years I've been hosting my code on someone else's servers. It was sure convenient, and free, and I don't regret it one bit, but the time has come to move that vital service in-house.

I've run my own private git server on the Mac mini in my living room since 2012. For the last few years, then, my GitHub account has become more of a public portfolio and mirror of a selection of my private repos. As of today, my GitHub account is deprecated. If you want to see what I'm working on now you can go to my Projects page. I'll be gradually moving old projects over to this page, and new projects will show up there first.


The projects page has three moving pieces. The git repos themselves, read-only public clone access, and finally displaying the projects on the page.

For the git repos, I was able to just re-use the puppet recipe I put together to install Gitolite on my Mac mini. It has a much simpler config because it just needs dynamic repos and no other crazy hooks.

To add read-only clone access I turned to a project named Grack. Grack implements git's smart HTTP protocol as a Rack handler which makes it super simple to add to bugsplat.rb, the software that runs this site. Here's what config.rb looks like:

require 'dotenv'

require 'app'
require 'grack'

grack_config = {
  project_root: ENV['PROJECTS_REPOS_ROOT'],
  adapter: Grack::GitAdapter,
  git_path: ENV['GIT_BINARY'],
  upload_pack: true

puts grack_config.to_json

use Rack::ShowExceptions
run \
  '/'       =>,
  '/source' =>

Every git repo in the directory named by PROJECTS_REPOS_ROOT is available for read-only public cloning.

To display these repos, I added a new Project class. Here it is, in it's entirety:

require 'grit'
require 'yaml'

class Project
  def initialize(path)
    @path = path

  def load_config
    data = repo_data(".repo.yml")
    @config = data.nil? ? {} : YAML.load(data)

  def name
    @config['name'] || base_path.gsub('.git', '')

  def description
    @config['description'] || "No description"

  def clone_url

  def information_page
    "/projects/#{base_path.gsub('.git', '')}"

  def base_path

  def repo_data(path)
    repo =
    obj = repo.tree / path
    if obj'UTF-8')

  def readme_contents
    repo_data("") || ""

  def self.all
    Dir[File.join(ENV['PROJECTS_REPOS_ROOT'], "*.git")] do |dir|

  def self.find(name)
    path = File.join(ENV['PROJECTS_REPOS_ROOT'], "#{name}.git")

It uses Grit to pull out the file from the repo, as well as a small YAML config file that contains a few pieces of metadata.

The app then uses the existing Redcarpet-based Markdown renderer that renders the rest of the pages and throws the content up on the page.

Future Additions

There are a lot of little things that this system lacks. Easy code and commit browsing are both huge features that I'd like to add at some point. I thought about using Gitlab which has all kinds of nice features, but hacking things together is kind of my thing.

I realize that it's a little ironic that most of the links in this post point at GitHub. My reasons for moving to this system are pretty simple: I want to control my own destiny, free of even the possibility that someone else will be able to decide how or what I choose to share with the world.

For a lot of people, GitHub or Bitbucket or another 3rd party service presents a reasonable compromise for them, and that's fine. For myself, today is the last day that I'm pushing new repos to GitHub as well as the last day I'm paying them for my organization account.

Tagged: Programming 

Read More

Using the Mailchimp API for Sales

6 March 2014

One of the very first things I did when I started working on the idea that eventually became Mastering Modern Payments was set up a Mailchimp mailing list. People would land on the teaser page and add themselves to the list so that when the book came out they would get a little note. After the book launch (with 30% of that initial list eventually buying) I started putting actual purchasers on the list.

For three whole months my process was:

  1. Use a rake task to export the entire list as a CSV
  2. Navigate to Mailchimp
  3. Remind myself where the import button is and click it
  4. Paste in the CSV
  5. Assign field labels
  6. Hit the "import" button
  7. Wait for an email saying it was done

I did this every single time I sent an email to the list, which was quite often when I was actively fixing bugs in the book. Clearly this couldn't continue.

Mailchimp turns out to have a very nice API these days and there's a plethora of good libraries out there to help you take advantage. Because the Mastering Modern Payments application is written in Ruby I chose to go with Gibbon.

The code is remarkably straight forward:

class MailchimpWorker
  include Sidekiq::Worker

  def perform(guid)
    ActiveRecord::Base.connection_pool.with_connection do
      sale = Sale.find_by(guid: guid)

      gb =

        id: Rails.configuration.mailchimp[:list_id],
        update_existing: true,
        email: {email:},
        merge_vars: {
          PRODUCT: sale.product.permalink,
          PURCH: 't',
          GUID: sale.guid,
          AMOUNT: sale.amount,
          PURCHAT: sale.created_at.strftime('%Y-%m-%d %H:%M:%S')

To start out, it's a Sidekiq worker. Every customer-initiated interaction with Stripe and Mailchimp in the MMP sales app goes through Sidekiq. Feel free to substitute Sidekiq for whatever other background worker you use, or if you're feeling cheeky just don't use one at all.

Then it goes on to look up the sale, build a new instance of the Gibbon API, and call the subscribe method. Three things to note here. First, the update_existing flag. I don't want to stomp on the user's record if it exists, I just want to update it with their sales info.

Second, the merge_vars keys are in all caps. Mailchimp merge vars are case sensitive, in so far as they are always upper case. If you specify the wrong merge vars nothing will happen, the update will just silently fail (I spent probably two hours debugging that one).

Third, that PURCH merge var. This list is a mixture of people who are possibly interested in buying the book and also people who have definitely purchased the book. Frequently I'll want to send to either of those groups, but not both, and Mailchimp's search interface makes it surprisingly difficult (impossible, actually) to check for null values. Instead, I set this little true/false flag and then I can do a search for purchased == 't' or purchased != 't'.

The Sidekiq job that processes the sale with Stripe kicks this job off as the very last step in the process. I can instead sit back with a bottle of beer and write emails to the list, confident in the knowledge that everyone who should get an email will.

Tagged: Programming  Marketing 

Read More

The Life of a Stripe Charge

20 January 2014

One of the most common issues that shows up in the #stripe IRC channel is people setting up their front-end Stripe Checkout integration and then expecting a charge to show up, which isn't really how Stripe works. In this post I'm going to walk through a one-off Stripe charge and hopefully illustrate how the whole process comes together.

Tagged: Programming  Stripe 

Read More

A Practical Exercise in Web Scraping

15 December 2013

Yesterday a friend of mine linked me to a fictional web serial that he was reading and enjoying, but could be enjoying more if it was available as a Kindle book. The author, as of yet, hasn't made one available and has asked that fan-made versions not be linked publicly. That said, it's a very long story and would be much easier to read using a dedicated reading app, so I built my own Kindle version to enjoy. This post is the story of how I built it.

Tagged: Programming 

Read More

Simple Git-backed Microsites

5 December 2013

A few days ago I built a new tool I'm calling Sites. It builds on top of git-backed wikis powered by GitHub's Gollum system and lets me build and deploy microsites in the amount of time it takes me to create a CNAME.

Something that I've wanted for a very long time is a way to stand up new websites with little more than a CNAME and a few clicks. I've gone through a few rounds of trying to make that happen but nothing ever stuck. Furthest progressed was a Rails app exclusively hosting Comfortable Mexican Sofa, a simple CMS engine. I never ended up putting any sites on it, though.

GitHub's Pages are of course one of the best answers, but I'm sticking to my self-hosting, built-at-home guns.

A Short Code Tour

The code is split up into four distinct parts:

  • viewer is a Sinatra app that presents wiki content as web pages. It also can serve static assets right from the wiki repo and caches everything in an in-memory LRU cache. If you have a file layout.erb it will wrap the pages in that layout, otherwise it'll pass the content straight to the browser.

  • manager is another Sinatra app that allows me to create new sites on the fly. Because a site is just a Gollum wiki and a Gollum wiki is just a git repo, it just has to create a git repo at the right place and do a redirect. It bakes in HTTP Basic Auth so other people can't create sites all willy nilly.

  • some small extensions to Gollum add basic auth and override the method Gollum uses to find the correct wiki repo. By default Gollumn wants to be told exactly where the repo is in a class-level Sinatra setting, but that doesn't work when things are dynamic.

  • a Rack middleware ties it all together. The middleware has three jobs. If the incoming hostname maps to a CNAME that one of the sites has declared in a special wiki page named cnames, pass the request to the viewer app. Otherwise, either pass the request to Gollum for existing sites or to the manager to create a new site. The hard work of building the CNAME to site mapping is cached for a short period of time to minimize disk hits.

The neatest part about this setup is that, since sites are just git repos, I can clone the repo to my laptop and work with it in Emacs instead of directly editing in Gollum if I don't want to. This also lets me easily add asset files and layouts.

Demo Time

Here's what the manager app presents when you just go to

I'm the only one that's ever going to be looking at this, so it doesn't really need to be anything fancy.

If you click on one of those links, you'll get the familiar Gollum interface:

To create a new site, just append its name to

Notice the new little button. Clicking that creates the repo and sends you back to Gollum to populate the home page:

If you want to see how the demo site is put together, it's running here and the source is on GitHub here.


I built Sites to fit my infrastructure which is a delightful bastardization of 12 Factor so I haven't tried installing it elsewhere. I know it won't run properly on Heroku because it needs to be able to put the git repos in a persistent place, but it might work well as a Docker image. If you get it running somewhere please let me know and I'll link it here.

Tagged: Programming 

Read More

Simulating a Market in Ruby

2 December 2013

Trading markets of all kinds are in the news pretty much continuously. The flavor of the week is of course the Bitcoin markets but equity and bond markets are always in the background. Just today there is an article on Hacker News about why you shouldn't invest in the stock market. I've participated in markets in one way or another for about a decade now but I haven't really understood how they work at a base level. Yesterday I built a tiny market simulator to fix that.

Tagged: Programming 

Read More

Little Data: How do we query personal data?

12 November 2013

My wife and I recently moved from Portland, OR to Ann Arbor, MI. Among the cacophony of change that is involved with a move like that, we of course changed to the local utility company. Browsing around in their billing application one day I came across a page that showed a daily graph of our energy usage, supposedly valid through yesterday for both gas and electric. And it has a button that spits out a CSV file of the date, which means if I actually wanted to I could build my own tool to analyze our usage.

Energy Usage Graphs

This got me thinking about all of the little databases out there that have data about me that I might actually consider useful. Useful, in this case, means that by looking some combination of the data I could draw some useful conclusions and take some sort of action. Here's a list of the ones that immediately came to mind:

Some of the things in that list have easily accessed APIs, like RunKeeper and Withings. Some of them you can export the data manually to CSVs. And then some of them really should have an API but don't, like LoseIt. And then what if we could add more data? Things like how often we run the clothes dryer, how often the furnace or AC kicks in and for how long, whether we're in the house or not, etc.

None of this is Big Data. This is all Little Data. It's reminiscent of the Quantified Self movement, but it's more about combining existing data sets into a unified whole than synthesizing new data.

I've gone so far as to build isolated silos for some of these data sets. For example, I track my finances using ledger and a variety of home-grown tools. My email gets backed up hourly and that's provided some useful data occasionally. Even so, all of it is still in individual silos. Ideally I would be able to dump all of this into some kind of personal "data soup" that I could query with a unified interface and build analytics tools on top of.

At this point I have a bunch of questions and no answers. What shape does the data soup take? How does data get in, how would it get queried? What questions could I realistically answer? I'm going to keep thinking about this and hopefully you will too.

Tagged: Programming  Data 

Read More

Post-mortem of a Dead-on-Arrival SaaS Product

19 October 2013

A little over a year ago I announced the launch of my latest (at the time) product named Marginalia. The idea was to be a sort of online journal. A cheaper, more programmer friendly alternative to Evernote. It never took off, despite my best intentions, and so a few months ago I told the only active user that I was going to shut it down, and today I finally took that sad action. This post is a short history of the project and a few lessons learned.

Tagged: Marginalia  Programming 

Read More

DRY your Rails CRUD with Simple Form and Inherited Resources

22 September 2013

When you're writing a Rails application you usually end up with a lot of CRUD-only controllers and views just for managing models as an admin. Your user-facing views and controllers should of course have a lot of thought and care put into their design, but for admin stuff you just want to put data in the database as simply as possible. Rails of course gives you scaffolds, but that's quite a bit of duplicated code. Instead, you could use the one-two-three combination of Simple Form, Inherited Resources, and Rails' built-in template inheritance to DRY up most of the scaffolding while still preserving your ability to customize where appropriate. This lets you build your admin interface without having to resort to something heavy like Rails Admin or ActiveAdmin while also not having to build from scratch every time.

Tagged: Programming  Rails 

Read More

Essential Tools for Starting a Rails App in 2013

15 September 2013

Over the past few years I've written a number of Rails applications. It's become my default "scratch an itch" tool for when I need to build an app quickly to do a task. Even though Rails is mostly batteries-included, there are a few tools that make writing new applications so much easier. This is my list of tools that I use for pretty much every new Rails project.

Edit: The discussion on Hacker News has some great gems that you should consider using as well.

Tagged: Programming  Rails 

Read More

Mastering Modern Payments Is Out Today!

15 August 2013

I'm so proud to announce that Mastering Modern Payments: Using Stripe with Rails is officially launching this morning. Mastering Modern Payments is your guide to integrating Stripe with your Rails application and is packed with sample code and best practices that will make sure your integration works now and in the future.

I've collected for you the best resources from around the web and added my own experiences and stories to help you make your Stripe integration as robust as it can be. Click below to check it out!

Tagged: Programming  Books 

Read More

Announcing: Mastering Modern Payments: Using Stripe with Rails

15 July 2013

Over the past few years I've put together a number of projects that use Stripe and their Ruby API to collect payments and manage subscriptions. I've learned quite a bit about how to effectively use the things that Stripe provides to my best advantage. Two months ago I decided that I would like to share that knowledge and so I started working on a guide to integrating Stripe with Rails and today I'd like to announce that Mastering Modern Payments: Using Stripe with Rails will be available on August 15th, 2013.

Tagged: Programming  Books  Stripe 

Read More

Shipping with Stripe and EasyPost

2 July 2013

Let's say that instead of running a Software as a Service, you're actually building and shipping physical products. Let's say quadcopter kits. People come to your website, buy a quadcopter kit, and then you build it and ship it to them. It takes you a few days to build the kit, though, and you would rather not charge the customer until you ship. Traditionally Stripe has been focused on paying for online services but recently they added the ability to authorize and capture payments in two steps. In this post we're going to explore billing with Stripe and shipping with EasyPost with separate charge and capture.

Tagged: Programming  Stripe 

Read More

Page Viewer, a Simple Markdown Viewer

15 June 2013

For various projects including Mastering Modern Payments I've found it really useful to be able to view the Markdown source rendered as HTML but I don't really care about editing it online. I put together a little gem named page_viewer which renders Markdown files like this:

Tagged: Programming 

Read More

Design for Failure: Processing Payments with a Background Worker

30 May 2013

Processing payments correctly is hard. This is one of the biggest lessons I've learned while writing my various SaaS projects. Stripe does everything they can to make it easy, with quick start guides and great documentation. One thing they really don't cover in the docs is what to do if your connection with their API fails for some reason. Processing payments inside a web request is asking for trouble, and the solution is to run them using a background job.

Tagged: Programming  Stripe 

Read More

Distributed Personal Wiki

10 May 2013

For as long as I can remember I've been trying to find a good way to keep personal text notes. Recipes, notes, ideas, that kind of thing. Things that aren't really suited to blogging. Along the way I've used (and stuck with) PmWiki, DocuWiki, TiddlyWiki, and most recently I built my own sort-of-pseudo-wiki Marginalia.

Lately, though, it's been kind of a drag to use a web-based application just to write down some work notes. Having sort of an obsession with Markdown I decided to just start keeping notes in Markdown-formatted files in a directory. Of course, files that aren't backed up are likely to disappear at any moment, so I naturally stuck them in a git repository and pushed to my personal git server. But then, how do I deal with synching my work and home machines? I guess I'll manually merge changes...

Tagged: Programming  Git 

Read More

Increasing the Encryption Noise Floor

27 January 2013

Inspired by Tim Bray's recent post about encrypting his website, I decided to enable and force HTTPS for The process was straightforward and, turns out, completely free. Read on to find out how and why.

Tagged: Programming  Meta 

Read More

Full Text Search with Whistlepig

9 January 2013

Yesterday I suddenly developed the intense need to add search to this site. Among the problems with this is that the site is kind of a weird hybrid between static and dynamic, and it has no database backend. If posts were stored in Postgres this would be a trivial matter, but they're just markdown files on disk. After flailing around for awhile I came across a library named Whistlepig which purported to do in-memory full text indexing with a full query language.

November 5, 2013: I've removed search because nobody used it and this way the site can be 100% static.

Tagged: Programming  Meta 

Read More

Deploy 12-Factor Apps with Capistrano::Buildpack

30 December 2012

Last month I wrote a short article describing a method of deploying a 12-factor application application to your own hardware or VPS, outside of Heroku. Today I'm happy to announce a gem named capistrano-buildpack which packages up and formalizes this deployment method.

Tagged: Programming  Heroku 

Read More

Docverter is now Open Source

23 November 2012

A few months ago I created a hosted document conversion service named Docverter. The idea was to collect together the best document conversion tools I could find into one comprehensive service and sell access. Many of these tools are difficult to install if you're used to a service like Heroku, so it only made sense to wrap it all up.

After two months of trying to make a go of it, I've made the decision to open source the server component of Docverter. I've put the source on Github, along with installation instructions. I'm also available on a consulting basis for installation and integration work. Email me if you're interested.

Tagged: Docverter  Programming 

Read More

Deploying a 12-Factor App with Capistrano

11 November 2012

Deploying Heroku-style 12 factor applications outside of Heroku has been an issue for lots of people. I've written several different systems that scratch this particular itch, and in this post I'll be describing a version that deploys one particular app using a Heroku-style buildpack, Foreman, and launchd on Mac OS X via Capistrano.

Tagged: Programming  Heroku 

Read More

Run Anything on Heroku with Custom Buildpacks

5 November 2012

Heroku is a Platform as a Service running on top of Amazon Web Services where you can run web applications written using various frameworks and languages. One of the most distinguishing features of Heroku is the concept of Buildpacks, which are little bits of logic that let you influence Heroku as it builds your application. Buildpacks give you almost unlimited flexibility as to what you can do with Heroku's building blocks.

Hanging out in the #heroku irc channel, I sometimes see some confusion about what buildpacks are and how they work, and this article is my attempt to explain how they work and why they're cool.

Tagged: Programming  Heroku 

Read More

Private Git Repositories with Gitolite and S3

27 October 2012

Earlier this year I bought a new Mac mini for various reasons. One of the big ones was so I would have a place to stash private git repositories that I didn't want to host on 3rd party services like Github or Bitbucket. This post describes how I set up Gitolite and my own hook scripts, including how I mirror my git repos on S3 using JGit.

Tagged: Git  Programming 

Read More

On-the-fly Markdown Conversion to PDF and Docx

20 October 2012

Today I added PDF, Docx, and Markdown download links to the bottom of every post here on Bugsplat. Scroll down to the bottom to see them, the scroll back up here to read how it works.

Tagged: Docverter  Programming  Meta 

Read More

Keeping a Programming Journal with Marginalia

8 September 2012

In addition to writing on this blog, I've been keeping notes for various things on Marginalia, my web-based note taking and journaling app. In my previous post I talked about the why and how of Marginalia itself. In this post I'd like to talk more about what I actually use it for day to day, in particular to keep programming journals.

Update 2013-10-19: Marginalia is shut down and open source on GitHub

Tagged: Marginalia  Programming 

Read More

Marginalia: A web-based journaling and note taking tool

3 September 2012

I'd like to present my new webapp, Marginalia, a web based journaling and note taking tool. Notes are written in Markdown, and there are some simple shortcuts for appending timestamped entries at the end of a note, as well as a few email-based tools for creating and appending to notes. You should check it out. Look below the fold for technical details and the origin story.

Update 2013-10-19: Marginalia is shut down and open source on GitHub

Tagged: Heroku  Programming  Marginalia 

Read More

Task-oriented Dotfiles

11 August 2012

Recently I sat down and reorganized my dotfiles around the tasks that I do day-to-day. For example, I have bits of configuration related to ledger and some other bits related to Ruby development. In my previous dotfile setup, this stuff was all mixed together in the same files. I had started to use site-specific profiles (i.e. home vs work), but that led to a lot of copied config splattered all over. I wanted my dotfiles more organized and modifiable than that.

Tagged: Programming 

Read More

ProcLaunch v1.2

15 April 2011

Just a few bug fixes this time:

  • When you send proclaunch SIGHUP, it will send all of the profiles their respective stop signals and then wait for them to shut down. You can tell proclaunch to stop without waiting by sending SIGHUP again.
  • You can pass the --log-path command line option to change where proclaunch writes it's log. By default this is $profile_dir/error.log

Get it from github! These changes were generously sponsored by Zipline Games, who are using proclaunch to launch lua mongrel2 handlers as part of their Moai Cloud platform.

Tagged: Programming  Proclaunch 

Read More

ProcLaunch Improvements and v1.1

4 March 2011

ProcLaunch has learned a bunch of new things lately. I've fixed a few bugs and implemented a few new features, including:

  • A --log-level option, so you can set a level other than DEBUG
  • Kill profiles that don't exist
  • Instead of killing the process and restarting, proclaunch can send it a signal using the reload file
  • Instead of always sending SIGTERM, the stop_signal file can contain the name of a signal to send when proclaunch wants to stop a profile
  • Pid files are properly cleaned up after processes that don't do it themselves
  • You won't get two copies of proclaunch if one is already running as root

Get version 1.1 from github! Thanks a bunch to Matt, who hunted down the bugs and helped me figure out the features.

(Also, I added highlight.js syntax highlighting. Hope you like it!)

Tagged: Programming  Proclaunch 

Read More

ProcLaunch v1.0

23 September 2010

I kind of started [ProcLaunch][] as a lark. Can I actually do better than the existing user space process managers? It turns out that at least a few people think so. I've gotten a ton of great feedback from thijsterlouw, who actually filed bug reports and helped me work through a bunch of issues. ProcLaunch even has some tests now!

Tagged: Programming  Proclaunch 

Read More

Perl with a Lisp

22 August 2010

Browsing around on hacker news one day, I came across a link to a paper entitled "A micro-manual for Lisp - Not the whole truth" by John McCarthy, the self-styled discoverer of Lisp. One commentor stated that they have been using this paper for awhile as a code kata, implementing it several times, each in a different language, in order to better learn that language. The other day I was pretty bored and decided that maybe doing that too would be a good way to learn something and aleviate said boredom. My first implementation is in perl, mostly because I don't want to have to learn a new language and lisp at the same time. The basic start is after the jump.

Tagged: Perl  Programming 

Read More

Managing Your Processes with ProcLaunch.

8 August 2010

Edit 2010-08-08: ProcLaunch now has a CPAN-compatible install process. See below for details.

I finally got the chance to work some more on proclaunch, my implementation of a user space process manager, like runit or mongrel or god. I wrote up a big overview of the currently available options [previously][12], but in summary: all of the existing options suck. They're either hard to setup, have memory leaks, have a weird configuration language, or are just plain strange. The only viable option was procer, and even that was just sort of a tech demo put together for the Mongrel2 manual.

That's why I started putting together proclaunch. I need some of the features of runit, namely automatic restart, with none of the wackyness, and I wanted it to be easy to automatically configure. I also wanted it to be standalone so I wouldn't have to install a pre-alpha version of Mongrel2 just to manage my own processes.

Tagged: Programming  Proclaunch 

Read More

Blog Generator Updates

6 August 2010

I've made some small changes to the way is generated. First, I refactored quite extensively. Instead of being a huge mess of spaghetti-perl, it's nicely factored out into functions, each one doing as little as possible. It got a little longer, but I think it's worth the tradeoff in readability.

Second, I added self-generated shortlinks. Each post on the site has an internal id, which is actually a monotonically increasing sequence number. The short link for a post is<id>. For this post, it's These are implemented as mod_rewrite rules in .htaccess which are generated using a template, just like every other piece of content on the site.

Third, I wrote a new convenience script named, the idea for which I shamelessly stole from technosorcery. Basically, it'll prompt me for a post title using bash's read function, then generate a URL and some date strings, as well as comb through the entries/ directory to find the highest id, then increase it by one. It writes all this to a file and then opens emacsclient right at the correct spot to start typing an entry.

Fourth, I worked on the CSS a little bit. Hopefully it looks a little snazzier than it did before.

Tagged: Programming  Meta 

Read More

Daemons are Our Picky, Temperamental Friends

1 August 2010

Modern web applications are complicated beasts. They've got database processes, web serving processes, and various tiers of actual application services. The first two generally take care of themselves. PostgreSQL, MySQL, Apache, Nginx, lighttpd, they all have well-understood ways of starting and keeping themselves up and running.

But what do you do if you have a bunch of processes that you need to keep running that aren't well understood? What if they're well-understood to crash once in a while and you don't want to have to babysit them? You need a user space process manager. Zed Shaw seems to have coined this term specifically for the Mongrel2 manual, and it describes pretty accurately what you'd want: some user-space program running above init that can launch your processes and start them again if they stop. Dropping privilages would be nice. Oh, and it'd be cool if it were sysadmin-friendly. Oh, and if it could automatically detect code changes and restart that'd be nifty too.

Tagged: Programming  Perl  Proclaunch 

Read More

Data Mining "Lost" Tweets Part 1

2 June 2010

Note: this article uses the Twitter V1 API which has been shut down. The concepts still apply but you'll need to map them to the new V2 API.

As some of you might know, Twitter provides a streaming API that pumps all of the tweets for a given search to you as they happen. There are other stream variants, including a sample feed (a small percentage of all tweets), "Gardenhose", which is a stastically sound sample, and "Firehose", which is every single tweet. All of them. Not actually all that useful, since you have to have some pretty beefy hardware and a really nice connection to keep up. The filtered stream is much more interesting if you have a target in mind. Since there was such a hubbub about "Lost" a few weeks ago I figured I would gather relevant tweets and see what there was to see. In this first part I'll cover capturing tweets and doing a little basic analysis, and in the second part I'll go over some deeper analysis, including some pretty graphs!

Tagged: Programming  Perl 

Read More

Iterating Elements in boost::tuple, template style

30 May 2010

In my day job I use a mix of perl and C++, along with awk, sed, and various little languages. In our C++ we use a lot of boost, especially simple things like the date_time libraries and tuple. Tuple is a neat little thing, sort of like std::pair except it lets you have up to 10 elements of arbitrary type instead of just the two. One of the major things that it gives you is a correct operator<, which gives you the ability to use it as a key in std::map. Very handy. One tricky thing, though, is generically iterating over every element in the tuple. What then?

Tagged: Programming  C++ 

Read More

Everyone Needs Goals

27 May 2010

Creating actionable information out of raw data is sometimes pretty simple, requiring only small changes. Of the few feature requests that I've received for Calorific, most (all) of them have been for goals. Always listen to the audience, that's my motto!

Tagged: Programming 

Read More

Building Battle Bots with Clojure

16 May 2010

Once in a while at [Rentrak][] we have programming competitions, where anyone who wants to, including sysadmins and DBAs, can submit an entry for whatever the problem is. The previous contest involved writing a poker bot which had to play two-card hold'em, while others have involved problems similar in spirit to the Netflix Prize. This time we chose to build virtual robots that shoot each other with virtual cannons and go virtual boom! We'll be using [RealTimeBattle][], which is a piece of software designed specifically to facilitate contests of this sort. It's kind of like those other robot-battle systems, except instead of requiring you to write your robot in their own arbitrary, broken, horrible language, this lets you write your bot in any language that can talk on stdin and stdout.

Tagged: Programming 

Read More

Actionable Information

12 May 2010

Let's pretend, just for a second, that you want to make some money on the stock market. Sounds easy, right? Buy low, sell high, yadda yadda blah blah blah. Except, how do you know when to buy and when to sell? Not so easy. Being a nerd, you want to teach your computer how to do this for you. But where to start? I discovered a few months ago that there are services out there that will sell you a data feed that literally blasts every single anonymous transaction that happens on any market in the US in real time. They'll also sell you access to a historical feed that provides the same tick-level information going back for several years.

Tagged: Programming 

Read More

Moose vs Mouse and OOP in Perl

9 May 2010

After using Calorific for a month two things have become very clear. First, I need to eat less. Holy crap do I need to eat less. I went on to SparkPeople just to get an idea of what I should be eating, and it told me between 2300 and 2680 kcal. I haven't implemented averaging yet, but a little grep/awk magic tells me I'm averaging 2793 kcal per day. This is too much. So. One thing to work on.

Tagged: Perl  Programming 

Read More

Calorific, a Simple Calorie Tracker

8 April 2010

I'm a nerd. I write software for a living. I spend a lot of my day either sitting in a chair in front of a computer, or laying on my couch using my laptop. I'm not what you'd call... athletic. I did start lifting weights about six months ago but that's really just led to gaining more weight, not losing it. A few years back I started counting calories and I lost some weight, and then stopped counting calories and gained it all back. Time to change that.

Tagged: Perl  Programming  Ledger 

Read More