Bots be bottin'



  • I'm paging through Apache logs of a superseded webserver. DNS points to another server as of yesterday. Some long TTL on a few secondary domain names mean we still see some hits and I wanted to assess whether it's save to turn it off now. I wont hold back for this guy:

    207.46.REDACTED - - [17/Mar/2016:06:41:09] "GET / HTTP/1.1" 200 29227 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:06] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:06] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:06] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:06] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:06] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    207.46.REDACTED - - [17/Mar/2016:06:42:07] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    
    

    Nothing much in robots.txt in case you were wondering.



  • Oooh I just realized what it is. The bot loaded the homepage and got links to a few subdomains which point to the same server. Before querying these it dutifully requested their robots.txt. So, not really a WTF.


  • sockdevs

    That's just the Bing Bot; I wouldn't worry about it



  • The Bing Bot runs Mozilla on an iPhone? I knew Microsoft has little love for their own platform, but that is adding insult to injury. Or is Bing Bot a 'garage' project?


  • sockdevs

    @Hanzo said:

    The Bing Bot runs Mozilla on an iPhone?

    you havent seen the user agents for IE lately?

    the IE10 User Agent String is.......

    Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)

    DUN DUN DUNNNN!



  • :rolleyes:

    It is strange that it's pretending to be an old iPhone though. Is it trying to crawl for a mobile page, too? :wtf:


  • sockdevs

    @rc4 said:

    Is it trying to crawl for a mobile page, too?

    probably, yes.

    google does that too IIRC


  • Winner of the 2016 Presidential Election


  • sockdevs

    Every browser's user agent string starts with 'Mozilla', and usually mentions at least one other browser; Chrome's mentions Safari, I know that.



  • Now ... try to explain that to a group of 6th graders.
    WTF is WRONG with our industry?! o_O


  • Winner of the 2016 Presidential Election

    @RaceProUK said:

    Every browser's user agent string starts with 'Mozilla', and usually mentions at least one other browser; Chrome's mentions Safari, I know that.

    I refer to the link above. It's actually quite clever and explains the issue well :)


  • Winner of the 2016 Presidential Election

    @Vaire said:

    WTF is WRONG with our industry?! o_O

    Backwards compatibility and lazy people.



  • It PAAAAAINS us, precious!
    We wants it to be better!



  • Sadly I'll take 95% backwards compatibility to:

    "Why old-thing no understand new-thing protocol?" : "That is called forwards compatibility" : "Me no understand, if old and new thing talk must be backwards compatibility!" : "That's bizzaro-world compatibility but whatever..."



  • I ... I don't ... was what English?



  • @sloosecannon said:

    WTF is WRONG with our industry?! o_O

    Backwards compatibility and lazy people.

    I saw this and ran-with-it.





  • I thought I was making fun of people that can't tell the difference between forwards and backwards compatibility? I must have swooshed.

    :badger:away folks!




  • Wait... was IE ever better than Netscape?



  • Yes. Specifically, IE4-6 were better than Netscape 4.





  • Expected? When I get home I might try playing with user agent string to see if that does anything.



  • EXCEPT... 5 to 5.5 on Mac OSX.

    Those really sucked.



  • @Vaire said:

    WTF is WRONG with our industry?!

    There are people involved.



  • @Zemm said:

    Expected?

    Yes. Homestar Runner is a terrible web dev.



  • I keep reading this topic title as bottles be bottling



  • I really just wanted to use my ponies, didn't actually care



  • @boomzilla said:

    There are people involved.

    [single teardrop] :sadface:



  • @MathNerdCNU said:

    Sadly I'll take 95% backwards compatibility

    If it was just backwards compatibility, it would be fine. But this is ass-backwards idiocy-to-idiocy compatibility. Because from the start, there should have been a feature list and not a user agent. But user agent is all we've got, so we run with it.


  • Winner of the 2016 Presidential Election

    Even that could be cocked up by an unscrupulous browser



  • That's not what I remember. Anyway, the timeline of web browsers indicates that when IE 6 was released, Netscape 6.2 was also being released, so it's hardly fair to compare IE6 to Netscape 4.

    Anyway, if my memory is correct, Netscape 4 was a lot better than Internet Explorer 5... IE at the time only allowed you to use document.write while the page was loading; after the page layout was completed, you couldn't change anything that would've required recalculating the layout.



  • @anotherusername said:

    Anyway, if my memory is correct, Netscape 4 was a lot better than Internet Explorer 5... IE at the time only allowed you to use document.write while the page was loading; after the page layout was completed, you couldn't change anything that would've required recalculating the layout.

    Pretty sure you have that backwards.

    document.write was the only way Netscape 4 could add text to a page. They even added a special <layer> tag that you could populate using document.write.

    Whereas IE could modify text in existing divs (and possibly other elements) using the innerHTML property.



  • @anotherusername said:

    That's not what I remember. Anyway, the timeline of web browsers indicates that when IE 6 was released, Netscape 6.2 was also being released, so it's hardly fair to compare IE6 to Netscape 4.

    Should have put this in my last post, but a lot of people avoided Netscape 6 because it was crashy as fuck.



  • @Jaloopa said:

    Even that could be cocked up by an unscrupulous browser

    There is a difference. Answering “yes” to “do you support X?” for any X is easy, but listing all Xs to a question “what do you support?” is not.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.