Dead Man’s Snitch

I stopped blogging every day a while back, but I’ve kept up a steady stream of links on Saturday mornings since. Hopefully this new instalment will join that roster. I might call it Friday Dev Love, or maybe Things A Geek Just Can’t Live Without. First up, Dead Man’s Snitch.

The world runs on cron jobs. As sites grow, ingesting more data, doing more things and managing more tasks, you end up with a wealth of background jobs. Multiply this by a dozen separate projects, some that might not be touched for weeks or months and you have a dangerous recipe for things failing and no one noticing.

There’s a simple solution. You get each job to email you after it completes. Now you wake up to an inbox of a dozen reports, every day. Did you spot that “purge activity index” didn’t run though?

Now you upgrade to a watchdog. Each task checks in, says “hey, I’m alive”, and only if one fails does your watchdog send you an email. If you’re smart, you’ll put this watchdog on another server. Ideally a completely separate data centre. If you’ve got a nice big ELK set-up, or New Relic, you can start pinging off metrics, set up alerts that way. It’s a lot of set-up for something that should be really simple.

And that’s where Dead Man’s Snitch comes in.

You set-up a snitch, with a name like “checked feeds” or “updated database”, and it’s reporting schedule; should this run every day, every hour? And then you just POST to it’s unique URL. Simple. If DMS doesn’t get a message, it emails you. You can pause, add messages, various extras. But the basic concept is stupidly simple.

It’s become part and parcel of every new project for any background task. Drop it in a trait or a decorator, pass in the ID and done. That’s it.

It’s saved my ass. Like that time Google changed scopes and broke active directory. Or when a stock ticker decided they needed moar XML. It’s also a steal for any reasonably sized project, once you factor in spinning up (and forever maintaining) your own watchdog.

So anyway, that’s my first Dev Love post. Hopefully not the last.