It’s always fun to spend a whole day debugging something that should
be simple. Actually I think it’s always the things that should be
simple that end up in a day of debugging. Sharing tales of woe can
sometimes help people. Or at least people can laugh at your misery.
Here’s one such tale.
For one of our production apps, we have a setup with a load balancer
and some app servers behind it. In this case the load balancer is
HAproxy and the app servers are running Rails with a Sinatra
application mounted, all on top of Phusion Passenger on Nginx.
This is a great setup for production systems.
###The Unhappy Middle Bit
The load balancer system needs to handle SSL termination which
HAproxy does not support. HAproxy is, however, a great load balancer
through which I have in other jobs run absolutely massive traffic
without issue. It has the benefit of actively monitoring your
servers so that it knows they are not responding before some request
gets hung up checking for you. It has great capability for routing
traffic based on all kinds of HTTP header information. Finally,
it has a great stats page that gives you a lot of live information
about the services it is handling. We wanted to use HAproxy.
There are a number of solutions for running HAproxy where SSL
termination is needed. The best of these is this right at hand.
Nginx supports SSL termination, is really lightweight, and is
event-based. It scales to massive proportions without much trouble.
At an unnamed previous employer we were doing 35,000 rpm in production
through a single Nginx install. I know Nginx works fine as a load
balancer, but it’s nowhere near as nice to run as HAproxy in
But… one final requirement, self-imposed for purposes of debugging,
was that the app server logs actually contain the original source
address of the client. This now means that the original IP address
needs to be relayed from Nginx to HAproxy, to Nginx, to Rails and
Sinatra. The easiest way to do that is to set HTTP headers like
on the load balancer.
is more common and lots of things muck
with it. I thought, to avoid trouble, let’s just use
in the SSL terminator’s
. HAproxy will leave it
alone and pass it along to Nginx and Rails/Sinatra on the app
servers. I can have Nginx log it on the app servers and it will
be available to put in the
This all seemed to work fine in Rails. Just as expected. Alas,
any attempt to connect to the Sinatra apps mounted on the Rails
installation resulted in
and an entire page body
consisting of the word “Forbidden”. This was from our Sinatra app
as well as from Resque-web.
connecting directly to HAproxy without going through
the SSL-terminating Nginx works as expected.
a tcpdump of the traffic sent to HAproxy
from Nginx shows that both
are set but only
is set by HAproxy.
Poking around with
I only have the problem when both headers are present. Then after
poking at this for awhile I discover that it only doesn’t work when
they are both set and NOT the same. What’s going on here? Well
Nginx is diligently setting
as expected, but
HAproxy is configured to add
from a previous config). So
is always the real
is always the IP address of the
###The Happy Ending
So what? There is a gem you perhaps don’t know about that is getting
invoked in your application stack. The culprit:
This gem does a lot of sanity checking and validation on requests
headed into a Rack stack. It is included in Rails but something is
Rails overrides this particular behavior. Sinatra triggers it, even
mounted on top of Rails. Grepping around revealed this test case