Security, we've heard of it (and play it in buzzword bingo)...



  • Quite a long story - but consists of bumbling incompetence throughout...

    NB - Mr Site is a web hosting company…

    TRWTF is their last email:

    @Mr Site Support said:

    Unfortunately we cannot discuss the internal workings of our network for security reasons. We have performed the required changes at our end so that your site is accessible to the public and your Mr Site services are available to you. Thank you for your patience.

    This doesn't seem too bad, they won't discuss internal workings for security reasons. That's sane, no point in shouting about network internals.

    However when you work out how we've got to this email, there are a number of WTF's which we'll spot later.

    Background

    I work as a tech support to network managers with specific focus on our DNS range of appliances, and quite often get slightly weird support requests, which turn out to be a third party misconfiguration (blacklisting swathes of IP space which have long since been reallocated, failed glue records or inoperative name servers still being advertised for instance). So I wasn't that surprised when a (technically competent) customer told me that two domains were not being resolved by their caching DNS.

    With a little digging (apologies for the pun) I found that they were both hosted by Mr Site, and advertised ns1 and ns2.hosthost.co.uk as name servers. Neither of these responded, and using one of our recursive tools I noted that one of the domains did not have NS records with the registrar (so was completely unfindable).

    Opening Gambit

    So I did my normal thing and sent off an email to the advertised support address:

    @Me said:

    Hi,

    There are a couple of zones which are hosted on the hosthost.co.uk nameservers (which I believe to belong to you) which are returning odd / no results:

    The first is XXXXXXX, which appears to have no NS records:

    Domain: XXXXXXb

    Dig failures: [('m.gtld-servers.net.', 'zero answers found'), ('j.gtld-servers.net.', 'zero answers found'), ('d.gtld-servers.net.', 'zero answers found'), ('h.gtld-servers.net.', 'zero answers found'), ('l.gtld-servers.net.', 'zero answers found'), ('c.gtld-servers.net.', 'zero answers found'), ('a.gtld-servers.net.', 'zero answers found'), ('g.gtld-servers.net.', 'zero answers found'), ('e.gtld-servers.net.', 'zero answers found'), ('b.gtld-servers.net.', 'zero answers found'), ('k.gtld-servers.net.', 'zero answers found'), ('i.gtld-servers.net.', 'zero answers found'), ('f.gtld-servers.net.', 'zero answers found')]

    The other is YYYYYYYY, one of your nameservers isn't responding:

    Domain: YYYYYYYY

    Records found: Set(['a.b.c.d'])

    Dig failures: [('ns1.hosthost.co.uk.', 'dig exited with status code of 2304, output: connection timed out; no servers could be reached')]

    Do you have any insight into what might be causing these failures?

    Could you supply me with a list of some other zones hosted on your servers that I could test, to see if we can identify any issues?

    They responded quite quickly, with an email that merits several instances of WTF on it's own:

    @Mr Site Support said:

    Thank you for your email. We can update your domain names' dns settings to so that your domain name returns an authoritative response and your site is accessible:

    http://XXXXXXXXXXXX/

    primary name server:ns4.hosthost.co.uk

    secondary name server:ns5.hosthost.co.uk

    http://YYYYYYYYYYYYY/

    primary name server:ns2.hosthost.co.uk

    secondary name server:ns3.hosthost.co.uk

    Currently your dns settings point to a control dns server and may be too busy. Please be advised that if we update your domain name's dns settings your site will be down for approximately 24 hrs. Please let us know if it would be alright to update the settings.

    Count the WTFs:

    • They are running (at least) two live sites from a “control dns server” which “may be too busy”.
    • They are only moving one of the domains off ns1&2, the other is staying on ns2.
    • They are offering to change DNS entries on zones for which I have no affiliation, without any attempt at confirming who I am.
    • They think that migrating name servers will result in 24 hours downtime.

    Any I've missed?

    Not wishing to fall foul of any computer system abuse laws my response was very non committal:

    @Me said:

    If the current name servers are too busy then it would be sensible to shift these domains.

    You can do this without downtime though - simply add the zones to the new name servers, update the NS records, then wait until the ttl has expired before removing the site from the old name servers.

    Subsequent exchange:

    @Mr Site Support said:

    We have updated your domain name's name server information to point to the server your site is on. Don't worry you will not experience any downtime on your site.

    If only that were true:

    @Me said:

    Whilst XXXXXXX has now been moved to ns4 and ns5 (and is working), YYYYYYYY is still on ns0 and ns1, and I am still getting servfail responses from my recursive resolver.

    Can I ask again for some of the other zones hosted on those nameservers, as it would appear that there is something odd in either your configuration or ours which is preventing these zones from being resolved.

    WTF counting again:

    • They've now made changes to the DNS configuration of a zone without any suitable instruction.
    • They've only done one of the two zones which were being discussed in the one support ticket.
    • They think the zone is mine.

    Although at least they seem to have spotted that this doesn't need to result in downtime…

    @Mr Site Support said:

    We have updated your other domain name's name server information (YYYYYYY) to point to the server your site is on. Don't worry you will not experience any downtime on your site.

    Resolved?

    @Mr Site Support said:

    Dear customer,

    We hope you found the answer to your enquiry “DNS oddness…” (number [####]) satisfactory. As we have not received any further communication from you with regards to this matter, we shall assume it has been resolved and will therefore close this ticket. If , however, you are not satisfied with how your enquiry has been answered, simply reply to this message. Please ensure you put the ticket number (including the brackets) in the subject of your email and we will reopen the ticket for you.

    I should really note again that I am not a customer

    I am satisfied with the outcome, concerned with the manner in which the changes were made, so I replied, copying in contacts from the two websites…

    @Me said:

    I'd like to highlight that the support team at Mr Site (HostHost, or whatever they want want to be called) were willing to make sweeping changes to your respective sites without any attempt to check that the request was from someone with the rights to change the site.

    There is a configuration/load issue with at least one of their name servers (ns1.hosthost.com) which was resulting in your sites being unavailable to certain networks.

    When I emailed to ask about the configuration, and try to assist them with their name servers their response was:

    @Mr Site Support said:

    Please be advised that if we update your domain name's dns settings

    your site will be down for approximately 24 hrs. Please let us know if it

    would be alright to update the settings.

    We have actually managed to make the transition to new name servers without downtime, but the support team at Mr Site still seem to be under the impression that I have something to do with either of the domains in question.

    This concerns me for all users of the Mr Site hosting services, as I could probably have easily either taken you offline, or redirected traffic to a malicious site.

    I hope that this episode will prompt a review of process, and that these requests would be authenticated or validated in the future.

    John

    PS - Mr Site, if you need to upgrade your DNS solution to cope with extra load/complexity then do call {company I work for}.

    WTF

    From this I then recieved a further response from Mr Site:

    @Mr Site Support said:

    Your site XXXXXXX now will report an authoritative response when talking to the server. Your domain name dns settings have been updated.

    Huh? I hadn't said anything about either site, they were working fine. Why did you change anything!

    I forwarded this to the site owners (again ccing support):

    @Me said:

    This is the kind of thing I've been getting. DNS changes made without any question being raised!

    And got another response:

    @Mr Site Support said:

    The dns changes we've made will not affect the functionality of your website nor will bring your site offline. We are simply updating the settings so that your site can be viewed using all internet service providers as some companies have made changes in their network that affect the accessibility of Mr Site websites unless your site name's settings are updated.

    Some ISPs have made changes??? You weren't supplying authoritative responses for a zone

    Glad you're not making any changes that affect the functionality of the site, other that making it accessible?

    My response then prompted the email at the top of the page…

    @Me said:

    So why was it ever in a state where it wasn't returning authoritative records?

    You still seem to have missed the point that none of these domains are “my site”. I have nothing to do with either of them!

    TRWTF

    @Mr Site Support said:

    Unfortunately we cannot discuss the internal workings of our network for security reasons. We have performed the required changes at our end so that your site is accessible to the public and your Mr Site services are available to you. Thank you for your patience.

    Let's count:

    • These aren't my sites
      • I've never said they were
      • I've explicitly said they're not
      • They've made multiple changes to the records without any attempt to validate the changes or my right to request them
    • Their original configuration
      • Had possibly overloaded “control” dns servers serving live domains
      • Failed to register NS records with the tld
    • Their initial changes
      • Were made without question
      • Though that switching DNS servers would result in downtime
      • Would only have moved one zone fully off the broken name servers
      • Only touched one zone anyway
      • Failed to return authoritative results
    • Their subsequent changes
      • Were made without question (as to my prerogative)
      • One was made without even an error report, or question being asked
      • Justification was that some companies had made changes to their recursive DNS settings
    • Security
      • As a justification for not talking to me after making these changes - don't make me laugh

    Finally

    • They STILL think I own the domains…
    • <?ul>


  • Bravo!  Wonderfully written story!   Are you still in ongoing communication with them?

    See if you can get them to change the records for their own hosthost.com website. That would be pretty awesome.



  •  Good one!

    If you'd reversed the orde rof the quotes here, you could have produced a Menento type post.

    TRWTF is having to hand craft HTML for a forum...

    Welcome to Community Server! Be advised: it does not support Chrome!

     



  • Hosting providers always seem to say it'll take up to 24 hours to make DNS changes; this is so pervasive that I think it's probably a CYA move, because if they give an actual estimate (like your TTL or something), they'll get picky business customers complaining when the changes haven't taken effect exactly at the estimated time.

     Also, 24 hours is long enough that you probably won't be willing to make changes willy-nilly, which might decrease the load on their servers or something.



  • @Tacroy said:

    Hosting providers always seem to say it'll take up to 24 hours to make DNS changes; this is so pervasive that I think it's probably a CYA move, because if they give an actual estimate (like your TTL or something), they'll get picky business customers complaining when the changes haven't taken effect exactly at the estimated time.

     Also, 24 hours is long enough that you probably won't be willing to make changes willy-nilly, which might decrease the load on their servers or something.

    I work for a hosting company and say exactly the same thing to customers - simply because you DONT KNOW what configuration an intermediate router/DNS server has on it (Crappy home broadband routers :@) .Give an estimate too low and the customer is on phone asking why there site doesn't work when you said it would; give an estimate too high and the customer goes away satisfied.

    Additionally, most websites and DNS providers default to 86400 seconds as the ttl (eg: thedailywtf.com) so upto 24 hours is a fair reply to make.

     


  • Garbage Person

     Proper response is to TELL the customers (not present them with information that should lead them to the decision if they're smart) that their service provider is a threat to their security and that they MUST move.



  • I notice this kind of thing happening with a number of ISPs. I remember sending an email to an ISP from work about my IP address at home - I asked them if they could add a reverse DNS. I offered no clue who I was or why I wanted it (from my works email address and connection), but I got the seemingly common reply of "OK, should start resolving within 24 hours.". I could of quite easily found the IP address of someone I don't like and asked them to resolve the address to "ilikecock.com" and they would have probably done it. 

    What I don't understand however (maybe someone can shed some light on this) is that over 1 year after the name expired, the reverse dns still reported the same information. It seemed to change to  "ip-addr-static.adsl.isp-name-here.co.uk" a few months after I left the ISP, and I'm guessing thats because before all static IPs were nameless (no records whatsoever) and they were doing a global change. 



  • @dhromed said:

    Be advised: it does not support Chrome!
    Chromium works just fine. In fact I'm using it right now.



  • @Mole said:

    What I don't understand however (maybe someone can shed some light on this) is that over 1 year after the name expired, the reverse dns still reported the same information.

    Forward DNS and Reverse DNS are unrelated. You can put absolute gibberish (even invalid names such as "budgerigar" with noTLD) and it will still appear as the rDNS on the connection. In fact, you could set it to something like "mail.coca-cola.com" if you felt like it, though most mail servers these days validate that the rDNS matches with the corresponding fDNS entry.



  • Nice story, should be on frontpage.



  • A few days ago, our office phone system died horribly. As a result, I was asked to call BT and have them forward our number to a mobile phone. Unbelievably, they did. No authorisation, no details requested apart from the phone number of the line to forward, and so-on. If I wanted to leave and steal our clients for myself...



  • @Mole said:

    I notice this kind of thing happening with a number of ISPs. I remember sending an email to an ISP from work about my IP address at home - I asked them if they could add a reverse DNS. I offered no clue who I was or why I wanted it (from my works email address and connection), but I got the seemingly common reply of "OK, should start resolving within 24 hours.". I could of quite easily found the IP address of someone I don't like and asked them to resolve the address to "ilikecock.com" and they would have probably done it.
    The ISPs here require that the e-mail comes from the account that owns the IP, and the domain name must point to that IP before they'll add the PTR. No checking is done afterwards though, and ...@Mole said:
    What I don't understand however (maybe someone can shed some light on this) is that over 1 year after the name expired, the reverse dns still reported the same information. It seemed to change to  "ip-addr-static.adsl.isp-name-here.co.uk" a few months after I left the ISP, and I'm guessing thats because before all static IPs were nameless (no records whatsoever) and they were doing a global change.
    ... two and a half years after we switched the ISP, our old IP still has our PTR.@Kyanar said:
    In fact, you could set it to something like "mail.coca-cola.com" if you felt like it, though most mail servers these days validate that the rDNS matches with the corresponding fDNS entry.
    I tried doing that, but it blocked way too many legitimate e-mails from misconfigured servers.



  • @ender said:

    I tried doing that, but it blocked way too many legitimate e-mails from misconfigured servers.
    Can you configure your spam blocker to treat emails that have that as much less likely to be spam?



  • @XAPBob said:

    They are offering to change DNS entries on zones for which I have no affiliation, without any attempt at confirming who I am.

    Some years back, one of our external DNS guys called our registrar on the way to work from his personal cell phone, to initiate the process of changing the registration to one of our domains to another company that had bought that business unit.  When he got to work, he had an email saying that they'd taken care of everything, and users would see the change as their TTLs expired; it could take up to a day for everyone to see it.

    This was the first time he had ever talked to them.  He has a fairly thick accent, and he's the only person we've ever had on the team with that accent.

    This was also the last time he, or anyone in our external DNS team talked with them; all of our remaining verbal contacts with them were through our lawyers, as we quickly changed to another registrar - one that has thus far abided by our requirement for a complicated process for domain ownership changes.  (Not only do we want verification of the party making the request, but we also want verification that it isn't a single rogue individual.)



  • @apz said:

    I work for a hosting company and say exactly the same thing to customers - simply because you DONT KNOW what configuration an intermediate router/DNS server has on it (Crappy home broadband routers :@) .Give an estimate too low and the customer is on phone asking why there site doesn't work when you said it would; give an estimate too high and the customer goes away satisfied.

    Additionally, most websites and DNS providers default to 86400 seconds as the ttl (eg: thedailywtf.com) so `upto 24 hours` is a fair reply to make.

     

     It's also the default cache for positive DNS responses for Windows XP / 2003.

     While that isn't relevant to this specific change, using "up to 24 hours" for DNS Changes isn't a WTF- It's giving support a single number to remember that should never be too low. 



  • @cdosrun said:

     While that isn't relevant to this specific change, using "up to 24 hours" for DNS Changes isn't a WTF- It's giving support a single number to remember that should never be too low.

    Changing DNS servers is (or should be) a staged process with NO downtime, and no issue with any intermediate DNS caches.

    Any change to DNS servers should be invisible to the end user



  • @XAPBob said:

    @cdosrun said:
     While that isn't relevant to this specific change, using "up to 24 hours" for DNS Changes isn't a WTF- It's giving support a single number to remember that should never be too low.

    Changing DNS servers is (or should be) a staged process with NO downtime, and no issue with any intermediate DNS caches.

    Any change to DNS servers should be invisible to the end user

    The website was down due to incorrect DNS.  Until the DNS change was propagated everywhere it needed to go, the website would continue to be down.  [b]If[/b] the change takes 24 hours to propagate, that's 24 hours of downtime.  Of course, since this issue caused negative caching instead of positive caching, most of the customers only experienced about a minute of continued downtime after they initiated the change.

    Any other questions?



  • @tgape said:

    @XAPBob said:
    @cdosrun said:
     While that isn't relevant to this specific change, using "up to 24 hours" for DNS Changes isn't a WTF- It's giving support a single number to remember that should never be too low.

    Changing DNS servers is (or should be) a staged process with NO downtime, and no issue with any intermediate DNS caches.

    Any change to DNS servers should be invisible to the end user

    The website was down due to incorrect DNS.  Until the DNS change was propagated everywhere it needed to go, the website would continue to be down.  If the change takes 24 hours to propagate, that's 24 hours of downtime.  Of course, since this issue caused negative caching instead of positive caching, most of the customers only experienced about a minute of continued downtime after they initiated the change.

    Any other questions?

    I think you're giving them too much credit.  Read their response again.  They basically say "If we make the change, the site will be down for 24 hours" not "after we make the change it may take up to 24 hours for it to take effect".

     

    I'm not sure negative caching would come into play.  He says "With a little digging (apologies for the pun) I found that they were both hosted by Mr Site, and advertised ns1 and ns2.hosthost.co.uk as name servers.  Neither of these responded..."  So the domains point to name servers that are non-responsive.  Caching servers won't have A records for the domains cached, but they will have erroneous NS records cached and those are what need to be propagated.  Of course, the OP then says "...and using one of our recursive tools I noted that one of the domains did not have NS records with the registrar (so was completely unfindable)."  So.. I don't know what the fuck is going on.


Log in to reply