Reverse Proxy Roulette



  •  We have a bunch of old servers hosting a number of client websites, two of which have hung and are throwing 404 and db errors. These two sites are being hosted as an act of good will because these organizations have connections with some of our higher paying clients.

     I ask our hosting provider what would be the best course of action to solve this and they simply say "Find out which server is hosting the two sites and reboot it.", which seemed simple enough.

    So I ssh to the external IP the sites are using, only to find that it's actually a proxy and the piss poor reason for that was because this server posed "security risks" and therefore could not be trusted to host any sites. So my predecessors thought that it would be a bright idea to use it as a reverse proxy for over 25 goddamn sites instead of decommissioning the damn thing.

    Our server documentation seems somewhat helpful at first glance by telling me the server nicknamed "live" is the most likely location. I SSH to the server, find the php and html files of the two sites in /www/sites and reboot the server, although I'm very reluctant to because this server hosts some clients that are much more important. After the reboot, and several other reboots, the sites still hang and then a back and fourth ensues with me and our hosting provider in trying to find another solution. 

     After glancing at our documentation, it becomes apparent to me that these two sites may have been previously hosted on a bunch of other older servers that should have been decommissioned just like that proxy. I ssh to all of these servers (10 of them) and find the exact same fucking thing I found on the server nicknamed "live", the php files that comprised these sites in /www/sites and a corresponding virtual host entry. I have no other choice but to reboot them one by one because of orders from the brass and despite me saying that some of our higher paying client's sites may incur downtime (keep in mind that these two small sites are being hosted as an act of good will).

    I finally reboot the 8th server nicknamed "Old" and that does the trick... despite other clients being angry at the reverse proxy roulette game I was forced to play.

    As a small wtf sidenote, I'm not a linux admin and am not that well versed in apache configurations (the company knows this). I was forced into this role 1 month ago because our company is too cheap to hire someone else.



  • @captainpants said:

     I ask our hosting provider what would be the best course of action to solve this and they simply say "Find out which server is hosting the two sites and reboot it.", which seemed simple enough.
     

    It is - if you have documentation detailing which site is on which server. @captainpants said:

    As a small wtf sidenote, I'm not a linux admin and am not that well versed in apache configurations (the company knows this). I was forced into this role 1 month ago because our company is too cheap to hire someone else.

    Perhaps now they'll look at hiring someone more competant[1] or spending money in skilling you up. Their cheapskate attitude shows how much value they place in quality of service for paying customers.

    [1] experienced. I've probably used the wrong term there, but it sounds like you have core competancies elsewhere and this stuff is outside of your comfort zone.

     



  • @captainpants said:

    As a small wtf sidenote, I'm not a linux admin and am not that well versed in apache configurations (the company knows this). I was forced into this role 1 month ago because our company is too cheap to hire someone else.

    I know that particular pain.



  • Just for future reference, you could check the access logs to see which server is handling your requests.



  • @jpa said:

    Just for future reference, you could check the access logs to see which server is handling your requests.
     

    .. provided they're written in a format that includes the vhost name.

    I'd check the Apache configs to see if any ServerName/ServerAlias matched that FQDN. 



  • @Cassidy said:

    .. provided they're written in a format that includes the vhost name.

    Didn't include the vhost name. The only thing that indicated to me that these sites were being reverse proxied were vhost entries like this.

    <VirtualHost *>
        ServerName www.example.com
            ProxyPass / http://example.com/
            ProxyPassReverse / http://example.com/
    </VirtualHost>

    But maybe there were other snippets of code in some other config file

    @Cassidy said:
    I've probably used the wrong term there, but it sounds like you have
    core competancies elsewhere and this stuff is outside of your comfort
    zone.


    It's only outside of my comfort zone if I'm forced to do some action that will most likely lose our clients some money... like taking down a server and hoping that it will fix two rinky dink websites. And although I have an decent working knowledge of linux and apache and don't mind learning more, even if it's not part of my job description. I just found it very odd that they wouldn't hire someone with much more knowledge.

    Update: So the server hung again and I can't ssh to it... I hope I don't get blamed for the downtime...

     



  • @captainpants said:

    But maybe there were other snippets of code in some other config file
     

    Yeah, meant the Apache configs that determined which directory was serving up content for which site.@captainpants said:

    I just found it very odd that they wouldn't hire someone with much more knowledge.
    It's one of those strange situations where it doesn't make financial sense to have someone with high skills in an area that's rarely used but when is needed there's a lot riding on it. Was there previously a *nix admin and Apache guru at your place that's never been replaced?@captainpants said:
    Update: So the server hung again and I can't ssh to it... I hope I don't get blamed for the downtime...

    Only if you're responsible for keeping them up.

    Anyhow, good luck. And I'm fairly whizzy at Apache stuff if you need help in future.

     

     



  • @Cassidy said:

    @jpa said:

    Just for future reference, you could check the access logs to see which server is handling your requests.
     

    .. provided they're written in a format that includes the vhost name.

    Even if not, you can always send a request to the site and check the access logs of the "backend servers" which one has received it.



  • @captainpants said:

    Didn't include the vhost name. The only thing that indicated to me that these sites were being reverse proxied were vhost entries like this.

    <VirtualHost *>
        ServerName www.example.com
            ProxyPass / http://example.com/
            ProxyPassReverse / http://example.com/
    </VirtualHost>

     

    Well, the second argument to ProxyPass should be exactly what you need - that's the server that will actually service the response.

    Usually that is the internal name of the machine that is going to service the request, otherwise it could be a static IP set up in the system's hosts file; either way sshing to the reverse proxy and pinging the host you see as the second argument should follow the same redirection logic that the proxy itself is using; clearly the proxy must know to where the request is going.



  • @Cat said:

    @captainpants said:

    Didn't include the vhost name. The only thing that indicated to me that these sites were being reverse proxied were vhost entries like this.

    <VirtualHost *>
        ServerName www.example.com
            ProxyPass / http://example.com/
            ProxyPassReverse / http://example.com/
    </VirtualHost>

     

    Well, the second argument to ProxyPass should be exactly what you need - that's the server that will actually service the response.

    Usually that is the internal name of the machine that is going to service the request, otherwise it could be a static IP set up in the system's hosts file; either way sshing to the reverse proxy and pinging the host you see as the second argument should follow the same redirection logic that the proxy itself is using; clearly the proxy must know to where the request is going.

     

    I read the apache documentation and the second argument should be the name of the internal machine but it wasn't. I took your advice and looked inside the hosts file and the EXTERNAL ips of the INTERNAL servers were defined (yes these internal servers are accessible without the proxy, which goes to show you that the damn proxy isn't needed to begin with).

    It never crossed my mind to look inside the hosts file and it seems to me like this is a stupid way of getting your reverse proxy to work.

     



  • @captainpants said:

    It never crossed my mind to look inside the hosts file and it seems to me like this is a stupid way of getting your reverse proxy to work.

     

    Very, very stupid.

    But with what you'd already seen of the setup by this point, did you expect anything less?



  • @Cassidy said:

    [Was there previously a *nix admin and Apache guru at your place that's never been replaced?


    Given the configuration the OP has described, I wouldn't think he was much of an admin and definitely not an Apache "guru".



  • @Vanders said:

    I wouldn't think he was much of an admin and definitely not an Apache "guru".
     

    True - it was one of those "business thinks he walks on water but anyone who can see through his bullshit knows he's completely incompetant" positions on the scale of perception.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.