Hutils - Explore your structured log data

September 5, 2014 by

Many of Heroku’s internal components make heavy use of logfmt to log information about what’s going on in production. The format is hugely valuable in that it allows us to retroactively analyze what happened during any arbitrary request to our components, query our log traces in very flexible ways, and combined with Splunk, easily generate arbitrary metrics on historical data. It’s unquestionably been an invaluable tool for fixing countless bugs, tracking down the root cause of many production incidents, and assessing usage in ways that would have been difficult otherwise.

That said, when viewed in the wrong light, logfmt is capable of producing hard-to-read walls of text like nothing else, which can be slow for a human to parse and even more difficult to visualize. The wall of text below for example, represents a standard fulfillment from the build API and is a whopping 51 pages long:

To make exploring this data a slightly less onerous task, we’ve written a tiny suite of tools called “hutils” that are installable via Rubygems:

gem install hutils

See the README for full details on the usage information, but a few of the more important tools in hutils are:

As dictated by the philosophy of small, sharp tools described in The Art of Unix Programming, tools from the suite are designed to be chained together to augment their usefulness. Combining ltap and lviz looks something like this:

ltap "95a89bef-ff65-49c8-95a6-fbc0fbeca8cd earliest=-1d" | head -n 50 | lviz

And the ugly Splunk trace above is transformed into something a little more digestible:

Note that lviz also ships with a --compact option if the output above is too verbose for your taste.

Hutils also ship with a few other commands to help work with logfmt:

Give it a try today and if you have any requests or want to contribute then check out the project on github