Heroku Engineering Blog

At Heroku, we’re always striving to provide the best operational experience with the services we offer. As we’ve recently launched Heroku Kafka, we were excited to help out with testing of the latest release of Apache Kafka, version 0.10, which landed earlier this week. While testing Kafka 0.10, we uncovered what seemed like a 33% throughput drop relative to the prior release. As others have noted, “it’s slow” is the hardest problem you’ll ever debug, and debugging this turned out to be very tricky indeed. We had to dig deep into Kafka’s configuration and operation to uncover what was going on.

Read more →

Heroku Metrics: There and Back Again

For almost two years now, the Heroku Dashboard has provided a metrics page to display information about memory usage and CPU load for all of the dynos running an application. Additionally, we’ve been providing aggregate error metrics, as well as metrics from the Heroku router about incoming requests: average and P95 response time, counts by status, etc.

Read more →

Speeding up Sprockets

The asset pipeline is the slowest part of deploying a Rails app. How slow? On average, it’s over 20x slower than installing dependencies via $ bundle install. Why so slow? In this article, we’re going to take a look at some of the reasons the asset pipeline is slow and how we were able to get a 12x performance improvement on some apps with Sprockets version 3.3+.

Read more →

Introducing React Refetch

Heroku has years of experience operating our world-class platform, and we have developed many internal tools to operate it along the way; however, with the introduction of Heroku Private Spaces, much of the infrastructure was built from the ground up and we needed new tools to operate this new platform. At the center of this, we built a new operations console to give ourselves a bird’s eye view of the entire system, be able to drill down into issues in a particular space, and everything in between.

Read more →

How We Migrated to Active Record 4

If your application is successful, there may come a time where you’re on an unsupported version of a dependency. In the case of the Heroku Platform API, this dependency was a very old version of Active Record from many years ago. Due to the complexity involved in the upgrade, this core piece of infrastructure had been pegged at version 2.3.18, which was released in March 2013. We’re happy to announce that we’ve overcome the challenge and are now running Active Record 4.2.4, the latest version, in production. In this post, we’ll describe the technical challenges we faced in the upgrade process and take a look at how your organization could benefit from upgrading legacy software dependencies.

Read more →

Patching Rails Performance

In a recent patch we improved Rails response time by >10%, our largest improvement to date. I’m going to show you how I did it, and introduce you to the tools I used, because.. who doesn’t want fast apps?

Read more →

Incremental Garbage Collection in Ruby 2.2

This article introduces incremental garbage collection (GC) which has been introduced in Ruby 2.2. We call this algorithm RincGC. RincGC achieves short GC pause times compared to Ruby 2.1.

Read more →

Time Out Quickly

Working with our support team, I often see customers having timeout problems. Typically, their applications will start throwing H12 errors.

Read more →

Benchmarking Rack Middleware

Performance is important, and if we can’t measure something, we can’t make it fast. Recently, I’ve had my eye on the ActionDispatch::Static middleware in Rails. This middleware gets put at the front of your stack when you set config.serve_static_assets = true in your Rails app. This middleware has to compare every request that comes in to see if it should render a file from the disk or return the request further up the stack. This post is how I was able to benchmark the middleware and give it a crazy speed boost.

Read more →

Django and Node together on Heroku

Heroku Connect is written primarily in Python using Django. It’s an add-on and a platform app, meaning it’s built on the Heroku platform. Part of our interface provides users with a realtime dashboard, so we decided to take advantage of socket.io and node.js for websocket communication. But like all Heroku apps, only one type of dyno can serve traffic. This left us with two choices: manage 2 apps, each with its own repo, and carefully consider when and how we deployed them, or find a way to serve both node and Django traffic from the same app.

Read more →

The Heroku Mobile App Template

One of the challenges when starting a mobile app project is deciding what technology stack to use. Should the client app use iOS or Android native, mobile web, or a hybrid? Do the backend in Node, Ruby, or Java? Or skip the backend and use an Mobile Backend-as-a-Service?

Read more →

Securing Celery on Heroku

Celery is by far the most popular library in Python for distributing
asynchronous work using a task queue. If you’re building a Python web app,
chances are you already use it to send email, perform API integrations, etc.
Many people choose Redis as their message broker of choice because
it’s dead simple to set up: provision a Redis addon, use its envrionment
variable as your BROKER_URL, and you’re done. But the simplicity of Redis
comes at a cost. Redis does not currently support SSL, and
it doesn’t seem like that’s going to change any time soon.
Because Heroku add-ons communicate over the public web, that means the contents
of Celery jobs are traveling unencrypted between dynos and Redis.

Read more →

Hutils - Explore your structured log data

Many of Heroku’s internal components make heavy use of logfmt to log information about what’s going on in production. The format is hugely valuable in that it allows us to retroactively analyze what happened during any arbitrary request to our components, query our log traces in very flexible ways, and combined with Splunk, easily generate arbitrary metrics on historical data. It’s unquestionably been an invaluable tool for fixing countless bugs, tracking down the root cause of many production incidents, and assessing usage in ways that would have been difficult otherwise.

Read more →

Retrospectives

Retrospectives are a valuable tool for software engineering teams. Heroku consistently uses retrospectives to review operational incidents, root cause problems, and generate remediation tasks to improve our systems. Increasingly we use retrospectives for another purpose: to improve teamwork and interactions on projects. Here we intentionally avoid technical discussions and focus on the emotional and human aspects of work, with the goal of creating positive insights into how to improve as a team.

Read more →

Subscribe to the full-text feed.