Windows explorer FTP: One, global webhost: Zero.




  • This is the story of a bug we ran into with our webhost.
    I won't name the host, but I will say they are big. Very big. And proudly claim they are "the world's no.1 web host".

    While working on a part of the website that wasn't critical, I was using FTP to make changes. Via windows explorer.
    I was editing some php files, copying them, testing them. Repeat until they worked :-)

    Then all of a sudden, after one copy-replace, the window refreshed, and every version of each file was listed. Say, 5 versions of each, only differing by size/date.
    Very odd I thought. So I went to refresh
    No reply.
    Reconnect...?
    Time out.

    Check the webpage.
    404.

    !!?

    The server had gone down. Completly. All domains on it too (we had 6) were not repsonding, no doubt the other users on the server box went down too.
    6 hours later it came back up again. From backup too, so all changes to the file system we had made that morning were gone. (This probably included the databases too, but we wern't in production yet, so I don't know for sure).

    At first I didn't click it was their FTP server, until the exact same thing happened a few days later.

    Needless to say this was a major problem to us.
    We could get huge downtime, and lose data, simply by someone using FTP. No doubt it effected every other server too. We are dealing with money on our site, having the possibly of a day's transaction records wiped would be a problem. We couldn't rely on them.

    So we sent their support an email, listing the exact steps to reproduce the problem, and it's significant consequences.

    The responce we got was nothing short of amazing. It was quite long, so I'll paraphrase, basically went like 'Well of course. You shouldn't be using windows explorer, don't you know how insecure microsoft software is. You should be using [insert linux ftp client]'.



  • In truth, you should be using some sort of revision control system, and ask for secure shell (ssh) access.  ssh is encrypted, so no one can sniff your database passwords/etc...

    I personally think that there are a few WTF's here. 

    WTF#1: Certanly the sites shouldn't have died just because of an FTP client. (#1 rule. Server's should rely on client behavior to be stable).

    WTF#2 is the support staff.  They're hosting servers, they should be reliable.

    WTF#3: Lack of source control.  Its generally considered a good idea to have backed-up revision control.  Unless you LIKE losing data (or like using Developmenstruction.)

    WTF#4: managing your code with a Windows FTP client

    WTF#5: Managing your code with FTP at all.  FTP is okay (not great) for flat content, but anything that accesses a database that should be secure (handling money/personal information, etc...) must not be transmitted unencrypted,  lest some hacker should sniff your password.

    WTF#6: PHP.  (j/k, I use PHP sometimes myself. for hobbyist sites only!)
     

    [quote user="Graham"]
    This is the story of a bug we ran into with our webhost.
    I won't name the host, but I will say they are big. Very big. And proudly claim they are "the world's no.1 web host".

    While working on a part of the website that wasn't critical, I was using FTP to make changes. Via windows explorer.
    I was editing some php files, copying them, testing them. Repeat until they worked :-)

    Then all of a sudden, after one copy-replace, the window refreshed, and *every* version of each file was listed. Say, 5 versions of each, only differing by size/date.
    Very odd I thought. So I went to refresh
    No reply.
    Reconnect...?
    Time out.

    Check the webpage.
    404.

    !!?

    The server had gone down. Completly. All domains on it too (we had 6) were not repsonding, no doubt the other users on the server box went down too.
    6 hours later it came back up again. From backup too, so all changes to the file system we had made that morning were gone. (This probably included the databases too, but we wern't in production yet, so I don't know for sure).

    At first I didn't click it was their FTP server, until the exact same thing happened a few days later.

    Needless to say this was a *major* problem to us.
    We could get huge downtime, and lose data, simply by someone using FTP. No doubt it effected every other server too. We are dealing with money on our site, having the possibly of a day's transaction records wiped would be a problem. We couldn't rely on them.

    So we sent their support an email, listing the exact steps to reproduce the problem, and it's significant consequences.

    The responce we got was nothing short of amazing. It was quite long, so I'll paraphrase, basically went like 'Well of course. You shouldn't be using windows explorer, don't you know how insecure microsoft software is. You should be using [insert linux ftp client]'.
    [/quote]



  • [quote user="danielpitts"]I use PHP sometimes myself. for hobbyist sites only![/quote]

    Hey, it's good enough for Yahoo!


    http://messenger.yahoo.com/partners_msn.php

    If the bussiest site on the net can make PHP work for them, it's probably ready for more than just hobbyists.
     



  • PHP is so good my university doesn't even allow php scripts on their web server without specific permission.



  • [quote user="danielpitts"]

    In truth, you should be using some sort of revision control system, and ask for secure shell (ssh) access.  ssh is encrypted, so no one can sniff your database passwords/etc...

    I personally think that there are a few WTF's here. 

    WTF#1: Certanly the sites shouldn't have died just because of an FTP client. (#1 rule. Server's should rely on client behavior to be stable).

    WTF#2 is the support staff.  They're hosting servers, they should be reliable.

    WTF#3: Lack of source control.  Its generally considered a good idea to have backed-up revision control.  Unless you LIKE losing data (or like using Developmenstruction.)

    WTF#4: managing your code with a Windows FTP client

    WTF#5: Managing your code with FTP at all.  FTP is okay (not great) for flat content, but anything that accesses a database that should be secure (handling money/personal information, etc...) must not be transmitted unencrypted,  lest some hacker should sniff your password.

    WTF#6: PHP.  (j/k, I use PHP sometimes myself. for hobbyist sites only!)
     

    [quote user="Graham"]
    This is the story of a bug we ran into with our webhost.
    I won't name the host, but I will say they are big. Very big. And proudly claim they are "the world's no.1 web host".

    While working on a part of the website that wasn't critical, I was using FTP to make changes. Via windows explorer.
    I was editing some php files, copying them, testing them. Repeat until they worked :-)

    Then all of a sudden, after one copy-replace, the window refreshed, and every version of each file was listed. Say, 5 versions of each, only differing by size/date.
    Very odd I thought. So I went to refresh
    No reply.
    Reconnect...?
    Time out.

    Check the webpage.
    404.

    !!?

    The server had gone down. Completly. All domains on it too (we had 6) were not repsonding, no doubt the other users on the server box went down too.
    6 hours later it came back up again. From backup too, so all changes to the file system we had made that morning were gone. (This probably included the databases too, but we wern't in production yet, so I don't know for sure).

    At first I didn't click it was their FTP server, until the exact same thing happened a few days later.

    Needless to say this was a major problem to us.
    We could get huge downtime, and lose data, simply by someone using FTP. No doubt it effected every other server too. We are dealing with money on our site, having the possibly of a day's transaction records wiped would be a problem. We couldn't rely on them.

    So we sent their support an email, listing the exact steps to reproduce the problem, and it's significant consequences.

    The responce we got was nothing short of amazing. It was quite long, so I'll paraphrase, basically went like 'Well of course. You shouldn't be using windows explorer, don't you know how insecure microsoft software is. You should be using [insert linux ftp client]'.
    [/quote]

    [/quote]

    #1 and #2 are the only relevant WTF's here.  It doesn't matter how Graham found out that IE crashed the server, it's the fact that the support team thinks that telling him not to use IE any more is a solution.  What are they going to do, send this email to the entire Internet so that the server won't crash?

    As for the other items you mentioned, nowhere in the original posting does it say that he does or doesn't use a source code control system.  You have to get the files to the production environment somehow, some hosting providers only support FTP.  Same with sending encrypted data.  This simply amounts to attacking the attacker and this method is generally employed by those that know they are wrong and want to divert attention.  I would expect it from the hosting provider (they actually did do it), but not from a poster here.  #s 3 through 6 at most deserve a BTW in a response rather than being in the core WTF list.



  • [quote user="Graham"]
    Check the webpage.
    404.

    !!?

    The server had gone down. Completly.
    [/quote]

    So.. it was down, but still served 404 pages? nice! :)



  • [quote user="Graham"]
    I won't name the host, but I will say they are big.[/quote]

    Why not name them?  I think I'd appreciate knowing to avoid them, and their stellar tech support, in the future. 



  • [quote user="jsmith"]

    As for the other items you mentioned, nowhere in the original posting does it say that he does or doesn't use a source code control system.
    [/quote]

     Are you sure:
    [quote user="Graham"]
    6 hours later it came back up again. From backup too, so all changes to the file system we had made that morning were gone. (This probably included the databases too, but we wern't in production yet, so I don't know for sure).

    [/quote]

    [quote user="jsmith"]You have to get the files to the production environment somehow, some hosting providers only support FTP[/quote]

    If you were #1 hosting provider, you should support more than FTP.  You really should get your own dedicated server or three, if your doing a real e-commerce site. With full access to the system, including SSH etc..

     



  • [quote user="VGR"]

    [quote user="Graham"]
    I won't name the host, but I will say they are big.[/quote]

    Why not name them?  I think I'd appreciate knowing to avoid them, and their stellar tech support, in the future. 

    [/quote]

     My wild guess: 1&1 Internet. From a quick Google, they seem to be the biggest web host calling themselves #1.
     



  • [quote user="danielpitts"]

    [quote user="jsmith"]

    As for the other items you mentioned, nowhere in the original posting does it say that he does or doesn't use a source code control system.
    [/quote]

     Are you sure:

    [quote user="Graham"]

    6 hours later it came back up again. From backup too, so all changes to the file system we had made that morning were gone. (This probably included the databases too, but we wern't in production yet, so I don't know for sure).

    [/quote]

    [/quote]

    I thought that just implied 'gone from the server, so we had to upload them again'. No-one would use the web space as their only copy of the code, surely? ... (or at least admit it on this website ;) ...)



  • [quote user="jesuswaffle"]

     My wild guess: 1&1 Internet. From a quick Google, they seem to be the biggest web host calling themselves #1.
     

    [/quote]

     

    And my guess would be GoDaddy. 



  • Just to clear things up,
    We do use SSH for all our sensitive work, at the time I was screwing around with some non critical parts of the site, so I was doing it via FTP. Mainly because the SSH performance is terrible... 3k/s at best, and changing directory takes a good 10+ seconds.

    As for the 404 thing, this happened to us nearly 6 months ago, so saying '404' I simply meant the site was not responding at all. So it probably wasn't an actual server sent 404 page. Sorry about that



  • Ohh yes and I did mean gone from the server ;-)

    And it's windows explorer not IE :P I'm not totally bonkers


Log in to reply