Good article on the root of Windows quality problems
-
Microsoft’s problem isn’t how often it updates Windows—it’s how it develops it
Buggy updates point at deeper problems.
Pretty interesting article. Some representative snippets.
We can see an example of this in practice in RS5. Last year's October update introduced a new feature for OneDrive: placeholders to represent files that were stored in OneDrive, but not downloaded locally. Whenever an application tries to open the files, OneDrive will transparently fetch the file from cloud storage and save it locally, without the application ever knowing that the file was initially not available locally. RS5 builds on this to optionally purge cloud-replicated files from local storage if disk space is low.
A reasonable expectation is that Microsoft would have a set of tests around this new code to verify that it works correctly: create a file, check it syncs properly, delete the local copy leaving a placeholder, open the file to have the real file retrieved, delete the file entirely, and so on and so forth. There's a handful of basic operations around manipulating files and directories, and in any kind of respectable agile development process, there will be tests to verify all the operations work as expected, and make sure that the API does what it's supposed to do.
Moreover, one would expect any code change that broke those tests to be rejected and not integrated. The code should be fixed, and it should pass its tests, before it's ever merged into the main Windows code—much less shipped to beta testers.
And yet, this is not what happened: many of the preview builds had a bug wherein deleting a directory that was synced to OneDrive crashed the machine. Not only was this bug integrated into the Windows code, it was allowed to ship to end users.
A secondary concern is the nature of the testing being performed. Microsoft used to have a huge number of dedicated testers, with each feature having both developer and testing resources assigned to it. Many of these testers were laid off or reassigned in 2014, with the idea that more of the testing burden be shifted to the developers creating the features in the first place. The Windows Insider program also provides a large amount of informal testing—with many millions of members, it's much bigger than any of the Windows beta programs ever were.
It's not certain that the old approach would have necessarily caught the data loss bug; perhaps the dedicated testers wouldn't have tested the particular scenario needed to cause data to be deleted. But it is clear that Microsoft is struggling to handle the bug reports made by the non-expert testers in the Insider program. The data loss was reported as much as three months before the update shipped. Many reports of the bug appear to be low quality, lacking in necessary detail or using improper terminology, but if the company didn't find the problem in three months, it's not at all obvious that an even longer development period would have made a difference. A longer development period would just mean that the bug was ignored for six months rather than three.
But read it all, the entire article is interesting.
Basically, Microsoft is trying to increase cadence of releases, but their testing methodology is hopelessly out of sync with their ambitions, it seems.
-
What could possibly go wrong with asking developers to QA their own code while accelerating project velocity? That’s how lean, agile startups work and they make billions!
-
TL;DR: Microsoft isn’t testing its stuff well enough. No shit.
Frankly, the article assumes too much stuff IMHO and seems to just say “they’re not unit-testing their code”.
Some things are easy to unit test, but race conditions, drivers, integration between different systems are a different story. I’m not saying unit testing is a bad idea, it just won’t get you 100% covered.Eventually, they will always need real people actually using the build. IMHO, if the software is buggy, they need more testing time, more testers, better testers, or all of that.
-
@WhatYouSay said in Good article on the root of Windows quality problems:
What could possibly go wrong with asking developers to QA their own code while accelerating project velocity? That’s how lean, agile startups work and they make billions!
I was hoping to find a link to the source for that statement in the article. I remember Yahoo did that at some point but I’ve never heard about MS going the same way.
-
@Zmaster said in Good article on the root of Windows quality problems:
Eventually, they will always need real people actually using the build. IMHO, if the software is buggy, they need more testing time, more testers, better testers, or all of that.
Counterpoint: Google can push out ChromeOS with greater velocity and greater stability, but without more manual testers.
-
@WhatYouSay said in Good article on the root of Windows quality problems:
That’s how lean, agile startups work and they make billions!
I'm still waiting for that red number to fade to black, much less read "billions"...
-
-
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
-
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
-
@cartman82 You'd think that someone at Microsoft would think “maybe we ought to figure out how to get good at Local Search and how to get good at Installing Updates” but it never really seems to occur to them.
-
@Benjamin-Hall said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
Yeah, but that's on Microsoft. They should have maintained a modern system and put all their backwards compatibility crap into an optional subsystem where it doesn't affect me.
-
@dkf said in Good article on the root of Windows quality problems:
@cartman82 You'd think that someone at Microsoft would think “maybe we ought to figure out how to get good at Local Search and how to get good at Installing Updates” but it never really seems to occur to them.
Unfortunately, as alluded to in the Reddit quote above, you don't get promotions for minor feature fixes.
-
@anonymous234 said in Good article on the root of Windows quality problems:
@Benjamin-Hall said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
Yeah, but that's on Microsoft. They should have maintained a modern system and put all their backwards compatibility crap into an optional subsystem where it doesn't affect me.
Perhaps I could interest you in an S Mode device? Nothing but Store apps and Edge as far as the eye can see! Never worry about installing any of that old software ever again!
-
two feature updates a year is too many and Redmond should cut back to one, and that Microsoft needs to stop developing new features and just fix bugs
Wait, wait, that sounds kind of familiar...
Number Five. Do you fix bugs before writing new code?
A score of 12 is perfect, 11 is tolerable, but 10 or lower and you’ve got serious problems. The truth is that most software organizations are running with a score of 2 or 3, and they need serious help, because companies like Microsoft run at 12 full-time.
That very article was written in 2000. Times surely have changed since then.
-
@Parody I do like the idea of "UWP only" mode, but there are at least two other conditions that S mode doesn't have:
- Let me have root access and let me install UWP apps from outside the store. It's my computer, dammit, I won't let you go all Apple on it.
- When switching from any established standard to a new one, there needs to be a long transition period where both standards are supported but there is some incentive to develop in the new one. The transition period for something the size of Windows is going to be looong, even in the ideal case.
-
@Benjamin-Hall said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
Not to mention who-knows-how-many-orders-of-magnitude fewer users.
-
@Zerosquare said in Good article on the root of Windows quality problems:
@Benjamin-Hall said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
Not to mention who-knows-how-many-orders-of-magnitude fewer users.
Which is the entire reason there's so many fewer backwards compatibility kludges. Otherwise just think of all the spacebar-CPU-heater kludges that would be needed in a Linux system that actually had users
-
@anonymous234 said in Good article on the root of Windows quality problems:
@Parody I do like the idea of "UWP only" mode, but there are at least two other conditions that S mode doesn't have:
- Let me have root access and let me install UWP apps from outside the store. It's my computer, dammit, I won't let you go all Apple on it.
That defeats the entire point. They want it to be 100% sandboxed so you can't hurt yourself on the sharp edges. If you want to sideload UWP apps, then you're already breaking the sandbox so there's no reason to not just give you the regular Windows and just refrain from installing non-UWP apps.
- When switching from any established standard to a new one, there needs to be a long transition period where both standards are supported but there is some incentive to develop in the new one. The transition period for something the size of Windows is going to be looong, even in the ideal case.
S isn't the 'future' of Windows, it's just a 'safe mode' version.
-
@anonymous234 said in Good article on the root of Windows quality problems:
When switching from any established standard to a new one, there needs to be a long transition period where both standards are supported but there is some incentive to develop in the new one. The transition period for something the size of Windows is going to be looong, even in the ideal case.
In theory we're in that transition period. In practice there isn't much incentive.
-
@Parody said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
When switching from any established standard to a new one, there needs to be a long transition period where both standards are supported but there is some incentive to develop in the new one. The transition period for something the size of Windows is going to be looong, even in the ideal case.
In theory we're in that transition period. In practice there isn't much incentive.
If there was a perceived incentive to upgrade software for new versions, there wouldn't be the big backward compatibility problem. The only thing I've ever seen work is a hard "upgrade or else" deprecation status. And even then...
-
@pie_flavor said in Good article on the root of Windows quality problems:
S isn't the 'future' of Windows, it's just a 'safe mode' version.
The 'safe mode' of Windows is OFF
-
@thegoryone said in Good article on the root of Windows quality problems:
Massive distributed teams of developers, and over-the-top compartmentalisation.
As in, groups of people working all over the world, but individuals don't really work on the same things as those they share an office with? That sounds hard, wouldn't surprise me if that led to less than stellar results.
-
@WhatYouSay said in Good article on the root of Windows quality problems:
That’s how lean, agile startups work and they make billions of bugs!
Our CEO is constantly talking about increasing the velocity of releases. sigh. It's the industry buzz word of the day now...
-
@anonymous234 said in Good article on the root of Windows quality problems:
@Benjamin-Hall said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
@cartman82 ChromeOS has 1/10th of the complexity of Windows.
And 1.0e-6 the backwards compatibility kludges that have to be maintained.
Yeah, but that's on Microsoft. They should have maintained a modern system and put all their backwards compatibility crap into an optional subsystem where it doesn't affect me.
Until they make that change that does affect you - and then you get to bitch-and-moan too!
-
@pie_flavor said in Good article on the root of Windows quality problems:
If you want to sideload UWP apps, then you're already breaking the sandbox so there's no reason to not just give you the regular Windows and just refrain from installing non-UWP apps.
The idea here came from me saying that Microsoft should replace Win32 with "Win32 2.0". Someone suggested Windows "S Mode", which only allows UWP apps (win32 is obviously still there, but as a background thing), the implication being that UWP is "Win32 2.0". I responded that I'd be OK with replacing win32 with UWP, but not in the way S Mode does. So the point is UWP Mode yes, S Mode no.
The problem with Windows that "S Mode" seeks to solve is that it lets user applications touch system files on installation. This is a very, very, very bad thing that causes major breakage when people install the wrong programs. But UWP mode would mostly solve it too (it would allow users to break their systems but only if they ask explicitly).
Side note: websites run sandboxed in my browser, and I don't have to get them through the "Mozilla Store".
@Parody said in Good article on the root of Windows quality problems:
In theory we're in that transition period. In practice there isn't much incentive.
It's still not clear if Win32 is "deprecated" or not. I find both "don't worry, it's not going away" and "stop using it, UWP is the future" claims.
-
@anonymous234 said in Good article on the root of Windows quality problems:
It's still not clear if Win32 is "deprecated" or not.
Only because if they stated that outright, people would panic. They would if they could.
-
@Magus said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
It's still not clear if Win32 is "deprecated" or not.
Only because if they stated that outright, people would panic. They would if they could.
CMD is deprecated, but nobody's panicking there
-
@anonymous234 They've backed off from it, just like they have in the past. Originally Vista was going to be all .NET with Win32 deprecated; Windows 8 and RT ("Windows 8 S Mode") was UWP with Win32 deprecated. Now the major direction is Azure and other SaaSy things with Windows deprecated.
-
@anonymous234 said in Good article on the root of Windows quality problems:
It's still not clear if Win32 is "deprecated" or not. I find both "don't worry, it's not going away" and "stop using it, UWP is the future" claims.
That sounds like it could well be the result of:—
@thegoryone said in Good article on the root of Windows quality problems:
Massive distributed teams of developers, and over-the-top compartmentalisation.
-
@Parody said in Good article on the root of Windows quality problems:
Now the major direction is Azure and other SaaSy things with Windows deprecated.
I assume you're talking about the managed desktop thing.
-
@pie_flavor said in Good article on the root of Windows quality problems:
I assume you're talking about the managed desktop thing.
No. It's about "we haven't quite figured out how keep Windows the cash machine it used to be (don't mind that ad dollar), so screw Windows, we go where the shareholder value is."
-
@Applied-Mediocrity said in Good article on the root of Windows quality problems:
we haven't quite figured out how keep Windows the cash machine it used to be
It's so much harder when you have some competition.
-
@Magus said in [Good article on the root of Windows quality problems]
They would if they could.
The moment they actually remove Win32 compatibility is the moment they alienate a massive part of their customer base. Windows' hegemony on the desktop is not just because people play PC games ; it's also because lots of companies have workflows based on Windows-only software (either commercial applications, or stuff developed in-house). If you read Raymond Chen's blog, you'll find plenty of examples of insane things they had to do to keep older (and buggy) applications working in newer versions of Windows -- and he clearly tells that they did so because they knew their corporate customers wouldn't update otherwise.
I assume they're not quite ready to shoot themselves in the foot that much... yet.
-
@Zerosquare said in Good article on the root of Windows quality problems:
The moment they actually remove Win32 compatibility is the moment they alienate a massive part of their customer base.
It's also probably at the same moment that Wine will become more Windows-compatible than Windows itself
-
@Zmaster said in Good article on the root of Windows quality problems:
Frankly, the article assumes too much stuff IMHO and seems to just say “they’re not unit-testing their code”.
Some things are easy to unit test, but race conditions, drivers, integration between different systems are a different story. I’m not saying unit testing is a bad idea, it just won’t get you 100% covered.Bang on. Also, what happens when Team A creates all of those great unit tests and later Team B commits something that breaks the tests, but no one on Team B has enough in-depth knowledge to update Team A's code (say with new APIs or necessary API changes due to a flawed design earlier)? I've seen that happen before.
Instead of being worried about "speed" and "number of unit tests", the real problem here is that management either doesn't care to stop deployment of highly broken code or those who know about it don't have the authority to say it isn't ready yet. Marketing promised the release of the great new shiny and no matter what the shiny will ship on time.
-
@dcon said in Good article on the root of Windows quality problems:
Our CEO is constantly talking about increasing the velocity of releases.
Increase the velocity of releases by throwing them off the roof of the office.
-
@quijibo said in Good article on the root of Windows quality problems:
Instead of being worried about "speed" and "number of unit tests", the real problem here is that management either doesn't care to stop deployment of highly broken code or those who know about it don't have the authority to say it isn't ready yet. Marketing promised the release of the great new shiny and no matter what the shiny will ship on time.
It might also be over-managed.
-
I liked this, from one of the reader comments on the article:
Moving to Linux is like being stuck in a cage with a tiger, but I figured out, over time, that tigers are predictable. They can maul you with teeth and fangs, but they don't grow wings and carry you up 100 feet and drop you. Windows 10 is like being stuck in a cage with a very shadowy corner that holds a grue. The types of torment and even the shape of the tormentor are completely unknown and seem to strike without notice.
-
@dcon said in Good article on the root of Windows quality problems:
@WhatYouSay said in Good article on the root of Windows quality problems:
That’s how lean, agile startups work and they make billions of bugs!
Our CEO is constantly talking about increasing the velocity of releases. sigh. It's the industry buzz word of the day now...
We're currently on a 2-week release schedule. I think that's insane, but I don't make the decisions here...
-
@WhatYouSay Upvoted for sarcasm.
-
@pie_flavor said in Good article on the root of Windows quality problems:
@Magus said in Good article on the root of Windows quality problems:
@anonymous234 said in Good article on the root of Windows quality problems:
It's still not clear if Win32 is "deprecated" or not.
Only because if they stated that outright, people would panic. They would if they could.
CMD is deprecated, but nobody's panicking there
Partly because Microsoft will never dare to throw it out the window. God knows how many companies have their "nest" of startup script to run, and these are the most revenue generating customers.
Much like Microsoft will never throw VB6 runtime out the window (at least on 32-bit subsystem. The WinXP has created sufficient upgrade inertia that, lots of software exists then go away within that period. With the companies gone and no source code available, the companies can just stuck on whatever Windows that can run VB6 applications).
-
@Scarlet_Manuka said in Good article on the root of Windows quality problems:
I liked this, from one of the reader comments on the article:
Moving to Linux is like being stuck in a cage with a tiger, but I figured out, over time, that tigers are predictable. They can maul you with teeth and fangs, but they don't grow wings and carry you up 100 feet and drop you. Windows 10 is like being stuck in a cage with a very shadowy corner that holds a grue. The types of torment and even the shape of the tormentor are completely unknown and seem to strike without notice.
Come on, moving to Linux has it's own set of "unknown" that can strike you without notice.
Say, if all web server is IIS and all web browser is IE/Edge, we shall never need to kill SSL3.0 in such a hurry and painful manner because Microsoft never include option to select "null cipher" in their implementation.
-
@Tsaukpaetra said in Good article on the root of Windows quality problems:
@dcon said in Good article on the root of Windows quality problems:
@WhatYouSay said in Good article on the root of Windows quality problems:
That’s how lean, agile startups work and they make billions of bugs!
Our CEO is constantly talking about increasing the velocity of releases. sigh. It's the industry buzz word of the day now...
We're currently on a 2-week release schedule. I think that's insane, but I don't make the decisions here...
Netflix pushes new updates to production multiple times per day, and somehow they manage...
-
@Tsaukpaetra said in Good article on the root of Windows quality problems:
We're currently on a 2-week release schedule. I think that's insane, but I don't make the decisions here...
We pretty much are too. But it feels like every week (cause it usually takes a week to get it out). And the release often starts in the middle of a sprint. I no longer understand the relation between sprints and releases. (not that I ever did, but that's neither here nor there)
@masonwheeler said in Good article on the root of Windows quality problems:
Netflix pushes new updates to production multiple times per day, and somehow they manage...
I'm pretty sure they're not a thick-client C++/Win32 program that uses MSI to install.
-
@masonwheeler said in Good article on the root of Windows quality problems:
@Tsaukpaetra said in Good article on the root of Windows quality problems:
@dcon said in Good article on the root of Windows quality problems:
@WhatYouSay said in Good article on the root of Windows quality problems:
That’s how lean, agile startups work and they make billions of bugs!
Our CEO is constantly talking about increasing the velocity of releases. sigh. It's the industry buzz word of the day now...
We're currently on a 2-week release schedule. I think that's insane, but I don't make the decisions here...
Netflix pushes new updates to production multiple times per day, and somehow they manage...
How many more thousands of employees do they have, and behold their architecture is vastly different from ours. For example, they have the money to run enough application instances that randomly crashing them has no impact on the system.
-
@Tsaukpaetra said in Good article on the root of Windows quality problems:
For example, they have the money to run enough application instances that randomly crashing them has no impact on the system.
There's a quality element to that as well as a quantity element. They've carefully, deliberately designed their system for robustness, specifically so that it can survive app instances going down. That wasn't always the case.
-
Hey guys I heard Windows started sucking again. What's up with that?
-
@masonwheeler said in Good article on the root of Windows quality problems:
@Tsaukpaetra said in Good article on the root of Windows quality problems:
For example, they have the money to run enough application instances that randomly crashing them has no impact on the system.
There's a quality element to that as well as a quantity element. They've carefully, deliberately designed their system for robustness, specifically so that it can survive app instances going down. That wasn't always the case.
Well, when we have enough slack in my pipeline I'm going to redesign the game engine to support failover. And while I'm at it I'll make it stateless so you won't be able to tell if a server instance is down!
-
@Captain said in Good article on the root of Windows quality problems:
Hey guys I heard Windows started sucking again. What's up with that?
What do you mean "again"? Did it stop sucking at one point?
Kinda numb down there, I didn't notice.
-
@Captain said in Good article on the root of Windows quality problems:
Hey guys I heard Windows started sucking again. What's up with that?
Welcome to the 21st Century, same as the 20th!
