Using HAProxy to Build a More Featureful Elastic Load Balancer

13916729435

Though Shareaholic is hosted in the AWS cloud, we avoid depending on Amazon’s virtualized cloud services whenever possible. If we ever hit a bottleneck in AWS, I want to be able to switch providers without needing to rebuild a core piece of our architecture. I also don’t want our tech team to have to make product and infrastructure sacrifices just so that we conform to AWS standard practices.
Load balancing with HAProxy was the first example of a service that Amazon provides, that we felt was better to manage ourselves.
Here’s how we did it.

Why HAProxy?

I’ve used HAProxy nonstop since 2009, and to date it has never been the culprit of a single problem I’ve experienced. It is extremely resource efficient and it is impeccably maintained by Willy Tarreau.
It provides both HTTP and TCP load balancing, and a powerful feature set not found in Amazon’s ELB.

Monitoring

HAProxy provides a dashboard with global stats, server stats, and aggregate stats for each frontend and backend. On a single page you can see which servers are busiest, which ones are up and which ones are down. You can also view this data at a service level, so if you have a three node cluster and only two of the servers were down at any point in time, HAProxy correctly summarizes that you did not experience any downtime.
While writing this post, one of our rails boxes had degraded performance due to too many background jobs running. Using the HAProxy dashboard, I quickly spotted it. You probably can too:

(Mouseover the image for the spoiler).

Configurability

HAProxy is highly configurable. You have over a half dozen load balancing algorithms to choose from. You can provide relative weighting for different servers in case you have boxes of differing resources. You can specify small health check timeouts on servers that should resond quickly and longer ones for servers that are slower. You can set these preferences globally, on a per-backend basis or on a per-server basis.

Requests as Mutable Objects

We often think of a web requests as being static, but with HAProxy we can bend them to our will to do important things.
Let’s start with the prototypical load balancer example: adding an X-Forwarded-Proto header so that our app server knows it’s receiving an SSL request even though it’s coming through on port 80:

reqadd X-Forwarded-Proto:\ https

Now let’s try something we can’t do with ELB: perform HTTP Basic Authentication before a request hits our app server.

userlist admins
user myusername insecure-password mypassword
frontend restricted_cluster
acl auth_tintoretto http_auth(admins)
http-request auth realm ShareaholicRestricted

Any web request being routed to restricted_cluster will need to contain valid HTTP auth credentials in order for that request to be sent to an application server. At Shareaholic we use this for our internal products and dashboards. This lets us manage authentication in a single place, and prevents us from needing to worry about it at the application level.

Frontends

HAProxy supports an infinite number of independently configurable load balancing frontends. In addition to HTTP load balancing, it can be used in TCP mode for general purpose load balancing. This means that if you have a redis database with three read slaves, you can use the same HAProxy instance that balances your web traffic to balance your redis traffic.

Backends and ACLs

Backends and ACLs enable you to host large-scale web sites and web services in ways not possible with simple request forwarding.
For example, did you know that the shareaholic.com domain is hosted partly in a PHP cluster, partly in a Rails cluster, partly in a Python cluster and partly as static content on github.com? We accomplish this with backends and ACLs.
First, we define an ACL based on the path of the URL request. An abbreviated example:

# Tech blog traffic goes to tech blog
acl techblog path_beg /tech
use_backend sharingthetech if techblog
# Other traffic goes to rails app
default_backend rails_app

Then we define backends composed of one or more servers to handle the different kinds of traffic:

backend rails_app :80
option httpchk /haproxy_health_check
server rails-1 10.1.1.1:8080
server rails-2 10.1.1.2:8080
server rails-3 10.1.1.3:8080
backend sharingthetech :80
reqirep ^Host: Host:\ shareaholic.com
server github-pages shareaholic.github.com:80

There are two things to note in this example:
First, our rails servers are configured to receive their health checks at /haproxy_health_check. These requests are handled by a proc in our routes.rb. We chose this rather than a static file on disk because we want the rails stack to be invoked during the health check; if apache is up but rails isn’t working, we want the health check to fail.
Second, Github pages only works for a single host (CNAME), so in our sharingthetech backend, we rewrite the Host header to ensure it is shareaholic.com, which is what’s contained in our Github Pages CNAME file.
Note: Since writing this post we have moved our tech blog from http://shareaholic.com/tech to http://tech.shareaholic.com. This example illustrates how we had hosted it at http://shareaholic.com/tech.

Shareaholic’s HAProxy Production Setup

Deployment

We run HAProxy on EC2 small boxes backed by instance storage. We do not back up these instances because Chef generates the config files on the fly. We use a custom fork of the haproxy cookbook that provides support for compiling HAProxy from source. We use the latest 1.5-dev12 build.
The shareaholic.com A record points to an Elastic IP address, which we assign to our load balancer. A second load balancer is always running; if the primary load balancer goes down, we simply reassign the Elastic IP address to our backup load balancer.
Because HAProxy consumes negligible CPU cycles and memory when not in use, we save money by avoiding a single tenant backup balancer. Instead we run our backup balancer on a utility box that is doing other work, and that utility box doubles as a hot standby that can handle traffic in a pinch.

HAProxy Configuration

Here is our production haproxy.cfg. The only changes are 1) redactions to protect authentication credentials and 2) Changes to IP addresses and paths to protect potential attack vectors:

haproxy.cfg

[color=#586e75 !important]1 [color=#586e75 !important]2 [color=#586e75 !important]3 [color=#586e75 !important]4 [color=#586e75 !important]5 [color=#586e75 !important]6 [color=#586e75 !important]7 [color=#586e75 !important]8 [color=#586e75 !important]9 [color=#586e75 !important]10 [color=#586e75 !important]11 [color=#586e75 !important]12 [color=#586e75 !important]13 [color=#586e75 !important]14 [color=#586e75 !important]15 [color=#586e75 !important]16 [color=#586e75 !important]17 [color=#586e75 !important]18 [color=#586e75 !important]19 [color=#586e75 !important]20 [color=#586e75 !important]21 [color=#586e75 !important]22 [color=#586e75 !important]23 [color=#586e75 !important]24 [color=#586e75 !important]25 [color=#586e75 !important]26 [color=#586e75 !important]27 [color=#586e75 !important]28 [color=#586e75 !important]29 [color=#586e75 !important]30 [color=#586e75 !important]31 [color=#586e75 !important]32 [color=#586e75 !important]33 [color=#586e75 !important]34 [color=#586e75 !important]35 [color=#586e75 !important]36 [color=#586e75 !important]37 [color=#586e75 !important]38 [color=#586e75 !important]39 [color=#586e75 !important]40 [color=#586e75 !important]41 [color=#586e75 !important]42 [color=#586e75 !important]43 [color=#586e75 !important]44 [color=#586e75 !important]45 [color=#586e75 !important]46 [color=#586e75 !important]47 [color=#586e75 !important]48 [color=#586e75 !important]49 [color=#586e75 !important]50 [color=#586e75 !important]51 [color=#586e75 !important]52 [color=#586e75 !important]53 [color=#586e75 !important]54 [color=#586e75 !important]55 [color=#586e75 !important]56 [color=#586e75 !important]57 [color=#586e75 !important]58 [color=#586e75 !important]59 [color=#586e75 !important]60 [color=#586e75 !important]61 [color=#586e75 !important]62 [color=#586e75 !important]63 [color=#586e75 !important]64 [color=#586e75 !important]65 [color=#586e75 !important]66 [color=#586e75 !important]67 [color=#586e75 !important]68 [color=#586e75 !important]69 [color=#586e75 !important]70 [color=#586e75 !important]71 [color=#586e75 !important]72 [color=#586e75 !important]73 [color=#586e75 !important]74 [color=#586e75 !important]75 [color=#586e75 !important]76 [color=#586e75 !important]77 [color=#586e75 !important]78 [color=#586e75 !important]79 [color=#586e75 !important]80 [color=#586e75 !important]81 [color=#586e75 !important]82 [color=#586e75 !important]83 [color=#586e75 !important]84 [color=#586e75 !important]85 [color=#586e75 !important]86 [color=#586e75 !important]87 [color=#586e75 !important]88

[color=#dc322f !important]global   [color=#dc322f !important]maxconn [color=#dc322f !important]15000 [color=#586e75 !important]# Max simultaneous connections from an upstream server   [color=#dc322f !important]spread-checks [color=#dc322f !important]5 [color=#586e75 !important]# Distribute health checks with some randomness   [color=#dc322f !important]log [color=#dc322f !important]127.0.0.1 [color=#dc322f !important]local0   [color=#dc322f !important]log [color=#dc322f !important]127.0.0.1 [color=#dc322f !important]local1 [color=#dc322f !important]notice   [color=#586e75 !important]#debug # Uncomment for verbose logging [color=#dc322f !important]defaults [color=#586e75 !important]# Apply to all services   [color=#dc322f !important]log [color=#dc322f !important]global   [color=#dc322f !important]mode [color=#dc322f !important]http   [color=#dc322f !important]balance [color=#dc322f !important]roundrobin   [color=#dc322f !important]option [color=#dc322f !important]abortonclose [color=#586e75 !important]# abort request if client closes output channel while waiting   [color=#dc322f !important]option [color=#dc322f !important]httpclose [color=#586e75 !important]# add "Connection:close" header if it is missing   [color=#dc322f !important]option [color=#dc322f !important]forwardfor [color=#586e75 !important]# insert x-forwarded-for header so that app servers can see both proxy and client IPs   [color=#dc322f !important]option [color=#dc322f !important]redispatch [color=#586e75 !important]# any server can handle any session   [color=#dc322f !important]timeout [color=#dc322f !important]client [color=#dc322f !important]60s   [color=#dc322f !important]timeout [color=#dc322f !important]connect [color=#dc322f !important]9s   [color=#dc322f !important]timeout [color=#dc322f !important]server [color=#dc322f !important]30s   [color=#dc322f !important]timeout [color=#dc322f !important]check [color=#dc322f !important]5s   [color=#dc322f !important]stats [color=#dc322f !important]enable   [color=#dc322f !important]stats [color=#dc322f !important]uri [color=#dc322f !important]/stats [color=#586e75 !important]# Real path redacted   [color=#dc322f !important]stats [color=#dc322f !important]realm [color=#dc322f !important]Haproxy\ [color=#dc322f !important]Statistics   [color=#dc322f !important]stats [color=#dc322f !important]auth [color=#dc322f !important]username:password [color=#586e75 !important]# Real credentials redacted   [color=#dc322f !important]monitor-uri [color=#dc322f !important]/monitor [color=#586e75 !important]# Returns 200 if we're up; real path redacted   [color=#dc322f !important]errorfile [color=#dc322f !important]503 [color=#dc322f !important]/etc/haproxy/errors/503.http   [color=#dc322f !important]userlist [color=#dc322f !important]shareaholic_admins     [color=#dc322f !important]user [color=#dc322f !important]myuser [color=#dc322f !important]insecure-password [color=#dc322f !important]mypass [color=#586e75 !important]# Real auth redacted   [color=#dc322f !important]frontend [color=#dc322f !important]incoming     [color=#dc322f !important]bind [color=#dc322f !important]*:80     [color=#dc322f !important]reqadd [color=#dc322f !important]X-Forwarded-Proto:\ [color=#dc322f !important]http     [color=#dc322f !important]acl [color=#dc322f !important]sharingthetech [color=#dc322f !important]path_beg [color=#dc322f !important]/tech     [color=#dc322f !important]acl [color=#dc322f !important]tintoretto_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]tintoretto     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]path_beg [color=#dc322f !important]/about [color=#dc322f !important]/contact [color=#dc322f !important]/help [color=#dc322f !important]/media     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]path [color=#dc322f !important]/api [color=#dc322f !important]/api/ [color=#dc322f !important]/tools/browser [color=#dc322f !important]/tools/browser/     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]blog.shareaholic.com     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]sexybookmarks.shareaholic.com     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]www.sexybookmarks.shareaholic.com     [color=#dc322f !important]use_backend [color=#dc322f !important]sharingthetech [color=#dc322f !important]if [color=#dc322f !important]sharingthetech     [color=#dc322f !important]use_backend [color=#dc322f !important]tintoretto_app [color=#dc322f !important]if [color=#dc322f !important]tintoretto_app     [color=#dc322f !important]use_backend [color=#dc322f !important]php_app [color=#dc322f !important]if [color=#dc322f !important]php_app     [color=#dc322f !important]default_backend [color=#dc322f !important]rails_app   [color=#dc322f !important]frontend [color=#dc322f !important]incoming_ssl     [color=#dc322f !important]bind [color=#dc322f !important]*:8443 [color=#dc322f !important]accept-proxy     [color=#dc322f !important]reqadd [color=#dc322f !important]X-Forwarded-Proto:\ [color=#dc322f !important]https     [color=#dc322f !important]acl [color=#dc322f !important]sharingthetech [color=#dc322f !important]path_beg [color=#dc322f !important]/tech     [color=#dc322f !important]acl [color=#dc322f !important]tintoretto_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]tintoretto     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]path_beg [color=#dc322f !important]/about [color=#dc322f !important]/contact [color=#dc322f !important]/help [color=#dc322f !important]/media     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]path [color=#dc322f !important]/api [color=#dc322f !important]/api/ [color=#dc322f !important]/tools/browser [color=#dc322f !important]/tools/browser/     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]blog.shareaholic.com     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]sexybookmarks.shareaholic.com     [color=#dc322f !important]acl [color=#dc322f !important]php_app [color=#dc322f !important]hdr_dom(host) [color=#dc322f !important]-i [color=#dc322f !important]www.sexybookmarks.shareaholic.com     [color=#dc322f !important]use_backend [color=#dc322f !important]sharingthetech [color=#dc322f !important]if [color=#dc322f !important]sharingthetech     [color=#dc322f !important]use_backend [color=#dc322f !important]tintoretto_app [color=#dc322f !important]if [color=#dc322f !important]tintoretto_app     [color=#dc322f !important]use_backend [color=#dc322f !important]php_app [color=#dc322f !important]if [color=#dc322f !important]php_app     [color=#dc322f !important]default_backend [color=#dc322f !important]rails_app   [color=#dc322f !important]backend [color=#dc322f !important]sharingthetech [color=#dc322f !important]:80     [color=#dc322f !important]reqirep [color=#dc322f !important]^Host: [color=#dc322f !important]Host:\ [color=#dc322f !important]shareaholic.com     [color=#dc322f !important]server [color=#dc322f !important]github-pages [color=#dc322f !important]shareaholic.github.com:80 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]50   [color=#dc322f !important]backend [color=#dc322f !important]php_app [color=#dc322f !important]:80     [color=#dc322f !important]option [color=#dc322f !important]httpchk [color=#dc322f !important]HEAD [color=#dc322f !important]/version.txt [color=#dc322f !important]HTTP/1.1\r\nHOST:\ [color=#dc322f !important]www.shareaholic.com     [color=#dc322f !important]server [color=#dc322f !important]master [color=#dc322f !important]10.1.1.1:80 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1     [color=#dc322f !important]server [color=#dc322f !important]slave [color=#dc322f !important]10.1.1.2:80 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1   [color=#dc322f !important]backend [color=#dc322f !important]rails_app [color=#dc322f !important]:80     [color=#dc322f !important]option [color=#dc322f !important]httpchk [color=#dc322f !important]/haproxy_health_check     [color=#dc322f !important]server [color=#dc322f !important]rails-1 [color=#dc322f !important]10.1.1.3:8080 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]rise [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1 [color=#dc322f !important]maxconn [color=#dc322f !important]100     [color=#dc322f !important]server [color=#dc322f !important]rails-2 [color=#dc322f !important]10.1.1.4:8080 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]rise [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1 [color=#dc322f !important]maxconn [color=#dc322f !important]100     [color=#dc322f !important]server [color=#dc322f !important]rails-secondary-a-1 [color=#dc322f !important]10.1.1.5:8080 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]rise [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1 [color=#dc322f !important]maxconn [color=#dc322f !important]100     [color=#dc322f !important]server [color=#dc322f !important]rails-secondary-c-1 [color=#dc322f !important]10.1.1.6:8080 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]rise [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1 [color=#dc322f !important]maxconn [color=#dc322f !important]100   [color=#dc322f !important]backend [color=#dc322f !important]tintoretto_app [color=#dc322f !important]:80     [color=#dc322f !important]server [color=#dc322f !important]tintoretto-1 [color=#dc322f !important]10.1.1.7:8080 [color=#dc322f !important]check [color=#dc322f !important]inter [color=#dc322f !important]5000 [color=#dc322f !important]fastinter [color=#dc322f !important]1000 [color=#dc322f !important]fall [color=#dc322f !important]1 [color=#dc322f !important]rise [color=#dc322f !important]1 [color=#dc322f !important]weight [color=#dc322f !important]1     [color=#dc322f !important]acl [color=#dc322f !important]auth_tintoretto [color=#dc322f !important]http_auth(shareaholic_admins)     [color=#dc322f !important]http-request [color=#dc322f !important]auth [color=#dc322f !important]realm [color=#dc322f !important]Tintoretto [color=#dc322f !important]if [color=#dc322f !important]!auth_tintoretto

SSL Termination

Terminating SSL at the Load Balancer: Why?

Many web sites defer SSL termination to the app servers that handle the individual requests. Because of the way we route traffic at Shareaholic, this is not possible for us. In order to choose a backend based on the request’s path or its Host header, we need to be able to inspect the request. This is only possible once the request has been decrypted.
An additional benefit of terminating SSL at the load balancer is the operational DRY it provides. Rather than configure SSL termination for JBoss and Apache and nginx and any other app server we run, we can configure it in one place. When it’s time to update our SSL keys, we just drop them into that one place and redeploy the box.

How We Terminate SSL

As of build 1.5-dev12 (September 10, 2012), HAProxy can be configured to perform SSL termination. We however use stud for this task because HAProxy did not have this feature when we began using it. We considered the more popular stunnel but found that stud provided substantially better performance. This is very important to us because SSL termination is the most expensive part of load balancing, and we want to squeeze as many connections as possible out of our EC2 boxes.
I’ve open sourced the stud Chef cookbook that I wrote for deploying stud to our production servers. Even if you don’t use Chef, that cookbook is instructive in how to get stud up and running quickly.

Performance and Scalability

While writing this post, HAProxy reports 161 concurrent connections with a new connection rate of 67/second. It is using about 4.1% of the single EC2 compute unit allocated to our EC2 small box.
stud is consuming an additional 9.7% of the CPU. We force SSL on all pages, so that is with near-100% of traffic flowing over SSL. (I say “near” to account for initial HTTP requests being redirected to their HTTPS equivalents). Based on these numbers, a single EC2 compute unit can handle 1167 concurrent connections, with an arrival rate of 486 connections/second. (RAM is a non-factor; combined, HAProxy and stud are consuming only 73.6 MB).
We can scale vertically on AWS until we need more than 88 EC2 compute units. That means we won’t need to tackle horizontal scalability until we are handling over 100,000 concurrent connections or accepting over 40,000 new connections per second.
If you’re fortunate enough to have that problem, the simplest way to scale horizontally is with Round-Robin DNS. This enables you to point your A record to multiple load balancers straight from your DNS.

Go Give It a Try!

Among our Chef cookbooks and the configuration posted above, we’ve open sourced everything you need to build and deploy your own load balancer, complete with SSL termination and examples for path-based and domain-based routing.
If you’ve ever wanted to host your blog on your application domain, or if you’re trying to avoid yet another provider-specific dependency in your tech stack, I encourage you to give this a try and see how easy it is to do load balancing yourself.
 
[Source URL: https://tech.shareaholic.com/2012/10/26/haproxy-a-substitute-for-amazon-elb/]