Curse of A Slow Build™



  • People in our current team are randomly struck by a Curse of A Slow Build. The symptoms are that build that normally takes about 10 minutes (from clean state) suddenly starts taking over an hour for full rebuild. With no obvious change in the environment. It usually affects the Android (NDK, old-style with GNU Make) build, but sometimes also the Windows one (Visual Studio 2013 or 2015). We use Windows 7.

    The usual wisdom is that antivirus might be slowing things down, but everybody who had this tried to turn off the antivirus completely or at least any disk scanning and it didn't help. For some people it had gone away on its own again, but most ended up reinstalling the system.

    The basic performance monitor tells that CPU is far from fully utilised (the graphs jump up and down, so it's hard to say what the average is, but seems below 30%) and the disk is far from fully utilised too (jumping up and down even worse, but the average won't be over 30% either) and memory is pretty full (91%), but not any significant numbers of hard faults there either. The build is run parallelised to as many tasks as they are CPU threads, which should definitely give much better utilisation.

    So it seems to be waiting for some Godot or something before it gets to actually do anything on each file. Does anybody have an idea how to find which Godot it might be?



  • @Bulb said in Curse of A Slow Build™:

    memory is pretty full (91%)

    Did you try turning it off and on again?



  • @JazzyJosh Yes, I did. I've restarted a couple of times since it started and I tried building without starting the memory hogs like browser and Outlook. Did not help. Also, 9% is still over 1GiB.


  • Impossible Mission Players - A

    @Bulb said in Curse of A Slow Build™:

    Does anybody have an idea how to find which Godot it might be?

    Did you scan for bad sectors?



  • @Tsaukpaetra said in Curse of A Slow Build™:

    Did you scan for bad sectors?

    Hm, good idea. The system appears generally increasingly fucked up in other aspects too; that might be.



  • … did not seem to find anything (:wtf: the computer just restarts and says absolutely nothing after the check) and did not help.

    I also tried with nothing started except the build after the restart, memory was almost empty this time, but did not help either.


  • sockdevs

    have you tried looking at Resource Monitor while the slow build is running?

    at this point i'm thinking it's possibly thrashing the disc. check your disc queue lengths, if they're over one then you are bottlenecking on disc access with more requests than resources to service them and everything slows to a crawl.


  • Impossible Mission Players - A

    @Bulb said in Curse of A Slow Build™:

    … did not seem to find anything (:wtf: the computer just restarts and says absolutely nothing after the check) and did not help.

    I also tried with nothing started except the build after the restart, memory was almost empty this time, but did not help either.

    Standard disk check is basically useless in that regard, as most hard disks "hide" bad sectors with replacements reserved for that purpose. You'll need a utility to check S.M.A.R.T. And do a access-rate-based scan of the disk to find areas that are slow but technically not "bad" yet.



  • @accalia said in Curse of A Slow Build™:

    have you tried looking at Resource Monitor while the slow build is running?

    at this point i'm thinking it's possibly thrashing the disc. check your disc queue lengths, if they're over one then you are bottlenecking on disc access with more requests than resources to service them and everything slows to a crawl.

    I did, but wasn't looking at the queue length, so I did it once more. No, queue length rarely exceeds 0.1 (sometimes the scale is just 0.01, sometimes it goes up to 0.1, but not seen it above that).

    @Tsaukpaetra said in Curse of A Slow Build™:

    You'll need a utility to check S.M.A.R.T.

    CrystalDiskInfo shows available reserved space 98% and media wearout 93% (93% good), so no problem.

    @Tsaukpaetra said in Curse of A Slow Build™:

    And do a access-rate-based scan of the disk to find areas that are slow but technically not "bad" yet.

    Hm, not sure how to do that one.


  • sockdevs

    @Bulb said in Curse of A Slow Build™:

    No, queue length rarely exceeds 0.1

    well..... bang goes that theory. disc queues of 0.1 are quite good.


  • I survived the hour long Uno hand

    @Bulb
    Is the scale 0.1, but the line on the chart is going toward 100, so there's actually a queue length of 10? Or when you click on "queue length" in performance monitor, the "Max" numeric value never goes above 0.1?


  • Impossible Mission Players - A

    @Bulb said in Curse of A Slow Build™:

    @Tsaukpaetra said in Curse of A Slow Build™:

    And do a access-rate-based scan of the disk to find areas that are slow but technically not "bad" yet.

    Hm, not sure how to do that one.

    Hddscan for example (though that one has a fairly ugly interface)



  • Ok, case given up, system reinstalled.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.