Network WTF
-
This isn't the usual type of wtf; I just spent an afternoon bashing my head to solve a network problem that just doesn't make any sense. Nevertheless, it left me asking WTF?!
I just have to write this down, in the hopes that someday someone can explain to me what the problem could be.
I lost connectivity from my home network to my server hosted at a remote ISP. In the course of investigation, I discovered:
1) No system at home could ping my server; however, all systems could ping all the other IP addresses in the server's subnet that had running machines.
2) I could ping yahoo, google, etc, and get web traffic from both.
3) I drove to colo and logged into server. No connection problems; I can ping everywhere and have no trouble at all. Home network still cannot reach this host. head back home.
4) Other hosts in server subnet have no trouble reaching server, and vice versa.
5) I can ssh from home network to other server in that subnet and then ssh to my own server. I cannot ssh directly.
6) traceroute from home to server show dropped packets at colo's router.
7) traceroute from server to home network show dropped packets at cableone's main router.
8) I powered down all of my home routers and cable modem again (that was my first step)
9) brought up new IP address on server (+1 from previous IP), cannot ping it from home, same symptoms.
10) brought up new IP address on server (+5 from main IP, +4 from previously mentioned IP)... it works!
11) reverified all previous steps, no change on previous IPs, but new iP works.
12) deciding to change my server's main IP, I try to go to registrar to change my DNS host IP; cannot reach registrar's host from home, DNS not found.
13) reboot home routers again. no change.
14) find DNS server address in router from cableone's dhcp. 24.117.164.232 and 24.116.2.34
15) ping 24.117.164.232 works
16) ping 24.116.2.34 no response
17) query 24.117.164.232 for registrar domain.... no server response
18) query 24.116.2.34 for registrar domain .... I get a response! note that IP was down....
19) ping 24.116.2.34 .... works.
20) ping 24.117.164.232 .... no response..... note, it was up a second ago....
21) I pause to pound my head
22) ping both dns servers again, in case I messed up. both work
23) query both dns servers... get responses from both.
24) bang head some more.
25) I can reach registrar website now.
26) on a whim, I tried to ping the server... you guessed it... It pings perfectly now. Problem resolved, all connectivity works as expected now.
There is no logical explanation for this that I can think of.WTF?
-
Mmmmm... networking....
Sometimes I think it's more sorcery than science.
-
It's simple
Clearly, it was the action taken at step 21) that solved the routing problem!
-
I guess all the trouble was caused by some hardware and/or software change at your or the server's ISP. (More probably at your ISP, causing the DNS ip unreachable stuff)
@DaveK said:
Clearly, it was the action taken at step 21) that solved the routing problem!
No, it was the step at 24) that lead to the final solution at 26) ! To solve such problems, replace "some more" with "a lot more, for hours !" and all your IT problems get just solved! I call it computer voodoo. And there is such a thing, you all know...
-
Classic symptoms of a routing glitch. This happens all the time, and it usually takes between 15 and 30 minutes to resolve itself. It's due to the routing protocols used between ISPs being imperfect approximation algorithms (for performance reasons). They're self-correcting, but it takes a little time for that to happen.
-
clearly here you forget a big step.. REBOOT!, and in your case you have to reboot your pc, your server, isp server, isp technician notebook, your girlfriend notebook, and your microwave...
