Proper server infrastructure



  • @Bulb said in WTF Bites:

    Our build server just had another of its hiccups. This time it somehow disconnected from the network share where it is uploading the build products and logs. Other times it gets another hiccups, but often also network-related. Usually a couple a day. And that's supposed to be a highly reliable server farm…

    So they fixed it on Monday just for the problems to apparently come back today. Well, the thing is the administrator wrote in the mail that they (hopefully) fixed it:

    […] The build server is an F̌B̈X̱0nnnn server, so these are the ones that are in the myCloud space they are having problems with.

    myCloud, right. That's supposed to be a server.



  • @bulb said in Proper server infrastructure:

    myCloud, right. That's supposed to be a server.

    Apparently the bigger ones do support iSCSI. But yeah, home NAS.


  • Grade A Premium Asshole

    @bulb well...that's a steaming pile of undefined



  • @polygeekery Yep. The project is burning something like $2,000,000 a year on personnel, but does not have a couple thousand for a proper build server. It's almost like the time when (years ago in different company) we couldn't test one feature, at all, because the management couldn't approve purchase of two antennas worth about $2 in the six weeks remaining to the deadline.


  • Grade A Premium Asshole

    @bulb said in Proper server infrastructure:

    Yep. The project is burning something like $2,000,000 a year on personnel, but does not have a couple thousand for a proper build server.

    That's...insane. The build server is running on the myCloud? Or using it as storage? Or an iSCSI target?



  • @polygeekery Using it as a storage. But it's enough to be flaky.

    Actually, the server seems to have its own disk too, but the myCloud seems to be used by the network share where the results are published and the TFS server from where the sources are downloaded.

    Update: yes, it appears to use it as storage.



  • … of course all of that is compounded by the fact that we also need to build for iOS and that can only be done on a Mac, so the Windows build server connects via ssh to a Mac on team leaders desk, downloads the sources from folder shared from the Windows server and builds it. But that's because there are no Apple servers, so it's a completely separate undefined.


  • Grade A Premium Asshole

    @bulb said in Proper server infrastructure:

    Using it as a storage. But it's enough to be flaky.

    They could have bought a Netgear RR2304 for $500-600 without disks and had something suitable for the job. Even if they got the myCloud for free they have already blown their savings.



  • @bulb I would suggest looking at https://help.apple.com/xcode/mac/9.0/#/dev466720061 as that may be hat you want, unless it requires all devs to be running Macs with Xcode too. (As doing iOS development on Windows/Linux is undefined .)



  • @polygeekery said in Proper server infrastructure:

    @bulb said in Proper server infrastructure:

    Using it as a storage. But it's enough to be flaky.

    They could have bought a Netgear RR2304 for $500-600 without disks and had something suitable for the job. Even if they got the myCloud for free they have already blown their savings.

    Yes, even with us el cheapo contractors that's no more than 2 man-days. And a couple of failed builds easily costs that much on lost productivity. But since lost productivity is hard to estimate and account, my experience with big companies so far is that the higher managers simply don't understand it.

    @atazhaia said in Proper server infrastructure:

    @bulb I would suggest looking at https://help.apple.com/xcode/mac/9.0/#/dev466720061 as that may be hat you want, unless it requires all devs to be running Macs with Xcode too. (As doing iOS development on Windows/Linux is undefined .)

    This is multi-platform, so we have to build on both Windows (cross-compile to Android and tests running on Windows) and MacOS (for iOS) and thus have to coordinate the build servers anyway. And XCode is not the problem. The problem is sharing the data between them over the crappy storage.

    Besides, we are tied to TFS—corporate policy—and an old version 2012 of it—several related projects cross-merging make an upgrade really hard to coordinate. So we have to use its integrated build server for gated checkins. Newer version could at least manage an iOS slave itself, but with this old one we have to handle it ourselves. Still wouldn't be big problem if it wasn't for the crappy infrastructure.


  • And then the murders began.

    @bulb said in Proper server infrastructure:

    Besides, we are tied to TFS—corporate policy—and an old version 2012 of it—several related projects cross-merging make an upgrade really hard to coordinate.

    As in, multiple application tier servers that need to be upgraded? Or it's just hard to get everybody to be willing to take an outage window at the same time?


  • Grade A Premium Asshole

    @bulb said in Proper server infrastructure:

    Yes, even with us el cheapo contractors that's no more than 2 man-days.

    I find that hard to believe. I bet the real cost is .5-1.0 man day.

    @bulb said in Proper server infrastructure:

    But since lost productivity is hard to estimate and account, my experience with big companies so far is that the higher managers simply don't understand it.

    Since when is that hard to account? Cost * lost hours. Easy peasy. I mean, yeah, it is impossible to estimate how many you would have. That is why you engineer for zero lost hours when possible. What is possible is to not use cheap as chips consumer grade gear.



  • @polygeekery

    1. Lost hours aren't considered expenses, and if they are they're operational expenses, which is completely different from capital expenses according to idiot MBAs I know.
    2. Lost hours are a sunk cost. From that point. the same value proposition applies as before the hours were lost. Therefore the same decision gets made.
    3. Lost hours are a hidden cost. Managers making purchasing decisions don't factor in any variables other than the initial outlay and any associated service fees, since that's what's on the sales sticker; variables such as reliability and their knock-on effects, even though potentially denominated in dollars, don't appear on invoices.

    Not justifying, just epxlaining the thought process.


  • Grade A Premium Asshole

    @twelvebaud said in Proper server infrastructure:

    Not justifying, just epxlaining the thought process

    Yeah, all of those thought processes are idiotic.

    I am not saying that you are wrong in what you say. I am 100% certain that people have made shitty decisions based on that logic. I am just saying those people are idiots.



  • @polygeekery said in Proper server infrastructure:

    @bulb said in Proper server infrastructure:

    Yes, even with us el cheapo contractors that's no more than 2 man-days.

    I find that hard to believe. I bet the real cost is .5-1.0 man day.

    I am pretty sure it is, but I don't actually know the rate, so I've just put a conservative bound on it.

    @polygeekery said in Proper server infrastructure:

    @bulb said in Proper server infrastructure:

    But since lost productivity is hard to estimate and account, my experience with big companies so far is that the higher managers simply don't understand it.

    Since when is that hard to account? Cost * lost hours. Easy peasy. I mean, yeah, it is impossible to estimate how many you would have. That is why you engineer for zero lost hours when possible. What is possible is to not use cheap as chips consumer grade gear.

    The lost hours is still just a rough estimate. Because, you see, rescheduling the build takes just a couple of clicks and a few minutes to find whether it was a server issue or actually a bug in the submission. The real impact is that it breaks your concentration, but that's just your estimate that the higher managers won't believe you.

    And the other reason it's hard to account is that the time tracking sheets don't have a column for it. You can track it with a special “task” “fighting with broken infrastructure”, but even if you do, it's unlikely that anything more than simple “total hours worked on project” ever reaches the key account manager.

    And then even if it reached them, it would require some forward thinking like “there were many hours lost to infrastructure problems in this project, so let's give the next one bigger budget for infrastructure and lets see whether it will be more efficient.” You can expect that in a small company where these things get to the owner, but not in a large one where even the key account manager is just a grunt with little actual interest in the company efficiency.

    @polygeekery said in Proper server infrastructure:

    I am not saying that you are wrong in what you say. I am 100% certain that people have made shitty decisions based on that logic. I am just saying those people are idiots.

    And I am saying that it is rather typical in large companies, because they are full of people who just don't give a damn and end up doing this all the time.



  • @bulb said in Proper server infrastructure:

    XCode is not the problem

    Now that's unexpected


    Filed under: No spanish inquisition memes please


  • kills Dumbledore

    @ixvedeusi said in Proper server infrastructure:

    No spanish inquisition memes please

    OK

    0_1510921085513_56085ca4-11fa-482a-bd62-900efec4614d-image.png



  • @bulb said in Proper server infrastructure:

    Besides, we are tied to TFS—corporate policy—and an old version 2012 of it

    Pity you are tied to the old version. Modern versions of TFS have build agents that directly run on iOS [and Linux].



  • @ixvedeusi said in Proper server infrastructure:

    @bulb said in Proper server infrastructure:

    XCode is not the problem

    Now that's unexpected

    Don't worry, there is plenty of problems with XCode. It just isn't the thing that usually breaks the build.



  • @bulb said in Proper server infrastructure:

    @ixvedeusi said in Proper server infrastructure:

    @bulb said in Proper server infrastructure:

    XCode is not the problem

    Now that's unexpected

    Don't worry, there is plenty of problems with XCode. It just isn't the thing that usually breaks the build.

    XCode's like cancer. It wasn't a serious killer back in the day because people didn't live that long.




  • Discourse touched me in a no-no place

    @benjamin-hall said in Proper server infrastructure:

    XCode's like cancer.

    No. We might eventually find a cure for cancer.



  • @dkf There's a cure for XCode--nuke Apple from orbit.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.