The game of "Spaghetti Servers"
-
Mass outage. I've heard of those. We just had one of those.
At the same time, we had a mini outage.
Did you know that you can have spaghetti servers? The jobs that went down were on the IBM mainframe. They were FTP jobs, writing files to a server that we will call Samwise.
Samwise runs Linoma GoAnywhere, which is one of those connectivity products that allow you to FTP, SFTP, http, or do just about anything with files. Maybe you can do too many things with files. Maybe you can do things with too many servers. Maybe you can do too many things with GoAnywhere.
So why did the mainframe jobs fail, that was the question. The host was connecting to Samwise, so call GoAnywhere team.
No, Samwise was working fine, but it returned an error when we tried to PUT a file. Maybe we need to know what user it's connecting to on Samwise? Okay, the user was FUDGE. I like fudge.
Maybe we need GoAnywhere guy to look and see how FUDGE is set up. Look at the GoAnywhere profile. Interesting, the home directory for FUDGE is //Gollum/files/in. Seriously, WTF, it's dropping files in a home folder...on another server?!
So, WTF is wrong with Gollum? Gollum is not responding when we try to connect to the file folder. Is it a problem with Gollum's hardware?
That is a simple question, easily answered right? By the team that handles Gollum, right? No, this is the game of Spaghetti Servers. Gollum is a virtual server that runs on the server Mirkwood, which runs many other virtual servers. Is Mirkwood down?
That's another team. No, Mirkwood is working just fine. Gollum team says Gollum seems to be working just fine. Except for the file folder.
Do we have a hard drive failure? What drive is //Gollum/files on? You're kidding, right? We don't do hardware, we do foggy cloud. It's on the SAN!!! (Storage Area Network) Of course it is, should have thought of that, it only makes good sense.
Okay, so where is it located on the SAN? Call SAN team on-call wizard, have wizard consult SAN profiles. Wizard is testy, yells something about bigger problems, but complies. And now we're getting somewhere, because that is the server Mordor. Now we understand, because Mordor is down! It's been down for hours! Major company outage.
So WhyTF are you wasting the time of 5 on-call teams figuring this out, when it's been related to the Mordor problem the entire time? You should have made that connection right away... it's not like there are any layers of obfuscation...
-
@coynethedup Sounds like your company failed its SAN check.
-
One time our cloud wizards team lost my test server. Not deleted it, not broke or misconfigured, just lost it.
They calmly and publicly admitted that the server is 'somewhere there', it works, eats memory and cpu time, but they don't know where it is or how to connect to it. It was completely normal situation for them apparently.
-
-
@dkf said in The game of "Spaghetti Servers":
@mrl said in The game of "Spaghetti Servers":
our cloud wizards team
He said wizards. That is only one. Slacker.
-
-
Suddenly I don't feel quite so bad about putting a folder on my Western Digital NAS which is mounted via NFS on a virtual machine which services it out as an FTP site.
-
A Celtic Network?
Or maybe a Gordian network.
EDIT: Fucking redirects.
-
@mrl said in The game of "Spaghetti Servers":
One time our cloud wizards team lost my test server. Not deleted it, not broke or misconfigured, just lost it.
They calmly and publicly admitted that the server is 'somewhere there', it works, eats memory and cpu time, but they don't know where it is or how to connect to it. It was completely normal situation for them apparently.<erno> hm. I've lost a machine.. literally lost. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
-
@dragnslcr said in The game of "Spaghetti Servers":
@mrl said in The game of "Spaghetti Servers":
One time our cloud wizards team lost my test server. Not deleted it, not broke or misconfigured, just lost it.
They calmly and publicly admitted that the server is 'somewhere there', it works, eats memory and cpu time, but they don't know where it is or how to connect to it. It was completely normal situation for them apparently.This is like @Captain's workplace and all the shit they have in the walls and ceilings.
-
@mott555 said in The game of "Spaghetti Servers":
Suddenly I don't feel quite so bad about putting a folder on my Western Digital NAS which is mounted via NFS on a virtual machine which services it out as an FTP site.
No, no, you're on the right track. Just add a few hundred more servers and you will be right where we are.
-
@coynethedup said in The game of "Spaghetti Servers":
@mott555 said in The game of "Spaghetti Servers":
Suddenly I don't feel quite so bad about putting a folder on my Western Digital NAS which is mounted via NFS on a virtual machine which services it out as an FTP site.
No, no, you're on the right track. Just add a few hundred more servers and you will be right where we are.
Hey, if you have some spare servers, wanna load up a UE4 game client and help us simulate a few thousand users for a day so I can demonstrate how woefully unprepared we are if we actually get a fraction of the userbase we're planning on?
-
@tsaukpaetra said in The game of "Spaghetti Servers":
Hey, if you have some spare servers, wanna load up a UE4 game client and help us simulate a few thousand users for a day so I can demonstrate how woefully unprepared we are if we actually get a fraction of the userbase we're planning on?
Why not to a cloud generated load test... Easy to simulate as much of a load as anyone can rationally want (I have done upwards to 10M requests/sec for short bursts)
-
@thecpuwizard said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
Hey, if you have some spare servers, wanna load up a UE4 game client and help us simulate a few thousand users for a day so I can demonstrate how woefully unprepared we are if we actually get a fraction of the userbase we're planning on?
Why not to a cloud generated load test... Easy to simulate as much of a load as anyone can rationally want (I have done upwards to 10M requests/sec for short bursts)
It's a bit more difficult to generate legitimate UE4 UDP login handshake packets. Already anything that doesn't have the correct magic byte is immediately dropped and forgotten, so on theory it should handle random requests mostly well.
-
@coynethedup said in The game of "Spaghetti Servers":
@mott555 said in The game of "Spaghetti Servers":
Suddenly I don't feel quite so bad about putting a folder on my Western Digital NAS which is mounted via NFS on a virtual machine which services it out as an FTP site.
No, no, you're on the right track. Just add a few hundred more servers and you will be right where we are.
I wanted to route it through my nginx server, too, but I guess nginx can't reverse-proxy FTP sites. (Or I just couldn't figure out how.)
-
@tsaukpaetra said in The game of "Spaghetti Servers":
Hey, if you have some spare servers....
It's a bit more difficult to generate legitimate UE4 UDP login handshake packets.
Not sure why there is a difference between using @coynethedup servers vs. using ones in the cloud....but whatever...
-
@thecpuwizard said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
Hey, if you have some spare servers....
It's a bit more difficult to generate legitimate UE4 UDP login handshake packets.
Not sure why there is a difference between using @coynethedup servers vs. using ones in the cloud....but whatever...
Donated versus paid.
-
@tsaukpaetra said in The game of "Spaghetti Servers":
Donated versus paid.
Often paying cash (money) directly is th cheapest overall ;)
-
Why not take out a malware ad that runs a hidden client in the browser? That'll be free if you make the ad so bad that nobody clicks it.
-
@mott555 said in The game of "Spaghetti Servers":
runs a hidden client in the browser?
Can you legitimately send and receive UDP packets from a browser-based ad?
-
@tsaukpaetra said in The game of "Spaghetti Servers":
Can you legitimately send and receive UDP packets from a browser-based ad?
That strikes me as a very loaded word in context.
-
@boomzilla said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
Can you legitimately send and receive UDP packets from a browser-based ad?
That strikes me as a very loaded word in context.
Well, I mean, I don't want to intentionally utilize CPU flaws to escape sandboxes and start arbitrary code execution glitches just for this simple task...
-
@tsaukpaetra said in The game of "Spaghetti Servers":
@boomzilla said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
Can you legitimately send and receive UDP packets from a browser-based ad?
That strikes me as a very loaded word in context.
Well, I mean, I don't want to intentionally utilize CPU flaws to escape sandboxes and start arbitrary code execution glitches just for this simple task...
WebRTC uses UDP by default, so yes.
-
@ben_lubar said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
@boomzilla said in The game of "Spaghetti Servers":
@tsaukpaetra said in The game of "Spaghetti Servers":
Can you legitimately send and receive UDP packets from a browser-based ad?
That strikes me as a very loaded word in context.
Well, I mean, I don't want to intentionally utilize CPU flaws to escape sandboxes and start arbitrary code execution glitches just for this simple task...
WebRTC uses UDP by default, so yes.
But is it raw "you get to build the datagram from scratch!" UDP? Or "yeah we handle the nitty gritty details for you under an encapsulation"?
