Is there any big software corporation out there still practicing Waterfall?
-
@jaloopa Nobody said splitting everything in microscopic pieces is efficient.
-
@pleegwat but isn't Agile meant to facilitate getting a quality product out there faster? How does that work if the developers spend 50% of their time reworking things they could have done up front at the expense of being able to demo something?
-
@blakeyrat said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard Look, Scrum is great if you're:
- Working on an already-finished application
- Solving easy problems.
I ALREADY SAID THAT.
The problem is: real software development involves solving difficult problems, and Scrum's insistence on the miniminimiminimimal amount of time a task can take is a HUGE obstacle to that and makes it a poor fit.
The only way to handle that situation is to split your developers working on the hard problem out of the process until they have something Scrum can iterate on. But what's the point then if half your team isn't following the methodology?
Clearly that is your opinion, but it contradicts the experience of thousands of teams who have been quite successful.
-
@jaloopa said in Is there any big software corporation out there still practicing Waterfall?:
@pleegwat but isn't Agile meant to facilitate getting a quality product out there faster? How does that work if the developers spend 50% of their time reworking things they could have done up front at the expense of being able to demo something?
Surprising to many, but the answer is NO, it actually takes significantly less time when all aspects are included.
The concept of "Just in Time" is a major mind shift for most (and therefore uncomfortable, and likely to induce fear).
If you ever have the opportunity, attend a workshop on highly iterative development [most are 2-5 days long]. At worst you get a couple of days away. At best you may be very pleasantly surprised.
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
attend a workshop on highly iterative development [most are 2-5 days long]
Is there a more iterative workshop I can attend?
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
Clearly that is your opinion, but it contradicts the experience of thousands of teams who have been quite successful.
The fact that a team is successful and does X doesn't mean that X is the cause of the team being successful.
It also doesn't mean that the team was successful at solving the problems our team was trying to solve. 700,000 companies using Scrum to make simple REST apps doesn't convince me it can be used to make complicated applications to enforce thousands of Government rules and regulations.
-
@jaloopa said in Is there any big software corporation out there still practicing Waterfall?:
How does that work if the developers spend 50% of their time reworking things they could have done up front at the expense of being able to demo something?
Also don't forget that neither the devs, product owner, business expert, analyst, stake holder or the cleaning lady know how or even what it is that is being made.
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
- True Waterfall rarely exists (as per the definition), there is almost always some capability of a "Change Order" which results in a process that does have feedback loops (which are by definition lacking in Waterfall). Even the originator of the term acknowledged that they did not (commonly) exist.
This is true for ANY project, in and out of software projects. Waterfall resembles other non-software projects that by their very nature need to be rigid. Building construction for example. But even those are going to have complications in the implementation. Perhaps the ground was softer than initial surveys measured and need more piles. Or some material they were originally gong to use was discontinued. Or a new fire code gets passed requires them to change their sprinkler schematics.
The point is, waterfall is a goal and a mindset, just like agile is. Waterfall's goal is to minimize the risk of too many unforeseen changes to the requirements, but Murphy's Law will always get in the way of that goal. There's no way to 100% accomplish that goal but you can give it a 100% effort to accomplish it regardless.
One must deliver exactly what is in the document or FAIL.
...
it literally caught fire and had a smallish explosion.In other words... it FAILed? Is that what proper usage of waterfall means to you? All logic and reason be damned, you'll let a project risk lives in the name of being pure in your workflow? I sure hope you aren't using that philosophy in nuclear reactor software.
Do you have no concept of "ideals" versus "dogma?" One can be a vegetarian, but still accidentally eat jello and not lose that label. One can say they've adopted a policy of never using bleach on their clothes except for that one time their lapdog shit diarrhea all over their pants.
-
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
- True Waterfall rarely exists (as per the definition), there is almost always some capability of a "Change Order" which results in a process that does have feedback loops (which are by definition lacking in Waterfall). Even the originator of the term acknowledged that they did not (commonly) exist.
This is true for ANY project, in and out of software projects. Waterfall resembles other non-software projects that by their very nature need to be rigid. Building construction for example. But even those are going to have complications in the implementation. Perhaps the ground was softer than initial surveys measured and need more piles. Or some material they were originally gong to use was discontinued. Or a new fire code gets passed requires them to change their sprinkler schematics.
The point is, waterfall is a goal and a mindset, just like agile is. Waterfall's goal is to minimize the risk of too many unforeseen changes to the requirements, but Murphy's Law will always get in the way of that goal. There's no way to 100% accomplish that goal but you can give it a 100% effort to accomplish it regardless.
One must deliver exactly what is in the document or FAIL.
...
it literally caught fire and had a smallish explosion.In other words... it FAILed? Is that what proper usage of waterfall means to you? All logic and reason be damned, you'll let a project risk lives in the name of being pure in your workflow? I sure hope you aren't using that philosophy in nuclear reactor software.
Do you have no concept of "ideals" versus "dogma?" One can be a vegetarian, but still accidentally eat jello and not lose that label. One can say they've adopted a policy of never using bleach on their clothes except for that one time their lapdog shit diarrhea all over their pants.
Yes, that is what Waterfall means. A government agency produced a specification for a product. The company I worked for at the time won the contract. Our team realized it could not possibly work (and was dangerous) and informed the government agency. Their response was (paraphrased, it has been 40+ years) "That is not your concern, you must build to the specification without revision. Failure to deliver a product that meets the specification will result in any and all penalties being applied".
This is why true Waterfall is so rare. What is CALLED Waterfall invariably has some type of "Change Order", and is therefore NOT Waterfall, despite what it is being called.
-
@luhmann said in Is there any big software corporation out there still practicing Waterfall?:
Also don't forget that neither the devs, product owner, business expert, analyst, stake holder or the cleaning lady know how or even what it is that is being made.
Hence: "Business people and developers must work together daily throughout the project. " [Principle #4]. With the probable exception of the "cleaning lady" all of the people should be together, making use of what they do know, and revising their plan as more becomes known.
-
@blakeyrat said in Is there any big software corporation out there still practicing Waterfall?:
It also doesn't mean that the team was successful at solving the problems our team was trying to solve. 700,000 companies using Scrum to make simple REST apps doesn't convince me it can be used to make complicated applications to enforce thousands of Government rules and regulations
That is true. But there are plenty of teams (at least in the 10s of thousands) who are making those "complicated applications" and need to conform with said rules and regulations.
-
I have to agree with Blakey on this one. Demos of three new blank white screens are no use to anyone. Admittedly, the team who demo'd those was working with an awful mess, but part of the mess was also Scrum.
Planning out several iterations ahead of time makes it nearly impossible to be accurate, to the point where we had a team that was going to use the next 3-4 sprints on a software upgrade, because to upgrade the framework all the way would take more than a sprint. So they decided to do it one version at a time over several sprints.
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
- True Waterfall rarely exists (as per the definition), there is almost always some capability of a "Change Order" which results in a process that does have feedback loops (which are by definition lacking in Waterfall). Even the originator of the term acknowledged that they did not (commonly) exist.
This is true for ANY project, in and out of software projects. Waterfall resembles other non-software projects that by their very nature need to be rigid. Building construction for example. But even those are going to have complications in the implementation. Perhaps the ground was softer than initial surveys measured and need more piles. Or some material they were originally gong to use was discontinued. Or a new fire code gets passed requires them to change their sprinkler schematics.
The point is, waterfall is a goal and a mindset, just like agile is. Waterfall's goal is to minimize the risk of too many unforeseen changes to the requirements, but Murphy's Law will always get in the way of that goal. There's no way to 100% accomplish that goal but you can give it a 100% effort to accomplish it regardless.
One must deliver exactly what is in the document or FAIL.
...
it literally caught fire and had a smallish explosion.In other words... it FAILed? Is that what proper usage of waterfall means to you? All logic and reason be damned, you'll let a project risk lives in the name of being pure in your workflow? I sure hope you aren't using that philosophy in nuclear reactor software.
Do you have no concept of "ideals" versus "dogma?" One can be a vegetarian, but still accidentally eat jello and not lose that label. One can say they've adopted a policy of never using bleach on their clothes except for that one time their lapdog shit diarrhea all over their pants.
Yes, that is what Waterfall means. A government agency produced a specification for a product. The company I worked for at the time won the contract. Our team realized it could not possibly work (and was dangerous) and informed the government agency. Their response was (paraphrased, it has been 40+ years) "That is not your concern, you must build to the specification without revision. Failure to deliver a product that meets the specification will result in any and all penalties being applied".
This is why true Waterfall is so rare. What is CALLED Waterfall invariably has some type of "Change Order", and is therefore NOT Waterfall, despite what it is being called.
Have you never heard of a spectrum? It's like saying "Fidel Castro isn't a communist because he allowed a few people to own cats." One can say they have Waterfall philosophies, meaning they strive to be as Waterfall as possible while understanding there could be exceptions in extraordinary cases to save the project. Few things in the world are black and white, but it's pointless to pedant about whether someone who philosophically aligns with a particular workflow or methodology breaks the rules a few times they suddenly lose all rights to say they are striving to maintain that methodology as much as possible.
-
Isn't Waterfall supposed to be a strawman?
-
@jaloopa said in Is there any big software corporation out there still practicing Waterfall?:
@pleegwat but isn't Agile meant to facilitate getting a quality product out there faster? How does that work if the developers spend 50% of their time reworking things they could have done up front at the expense of being able to demo something?
No. It means having the option to release what you have after 3 months, rather than waiting the full 6. It also means being more easily able to change course after those 3 months.
-
@pleegwat said in Is there any big software corporation out there still practicing Waterfall?:
It means having the option to release what you have after 3 months, rather than waiting the full 6
But taking 9 before you have the finished product?
-
@jaloopa Fast, cheap, good, flexible. Prioritise as desired.
-
Let me tell you a little secret: Waterfall has always been a strawman. I know I've said this many times before, but it is worth repeating.
The 1970 paper that the "Waterfall diagram" originally came from used it as an example of something that seems plausible, but doesn't actually capture enough of the necessary process - it shows the steps most project need to take, but assumes that backtracking would never be necessary, which is the weakness of the (rather informal) software planning methods of the time that the paper was written to address.
Since then, people have copied that diagram year after year, but while it is often taught in software engineering courses (the one I took in 2008 presented it as the best approach, FFS, though when I challenged the professor he weakly said, "well, it covers all the steps and that's all that matters, and besides there are no new approaches that aren't just fads"), but aside from that, it is mostly used as something to contrast 'new' approaches against.
Occasionally, some particularly clueless MGT will try to mandate it, but, well, it doesn't work, and never did. If you actually look at what is being done, 99% of the time the actual 'methodology' is an ad-hoc, "just get it working and ignore the idiot in the corner office" style that hasn't changed since the 1960s.
No actual software project has ever actually used Waterfall, and damn few have ever used Agile. They may say they do, but they don't, and more often than not, the people saying it don't actually know what those terms mean.
Agile isn't impossible, but like OOP, it is really only applicable to fairly narrow range of programming projects. Both caught on because those types of projects - GUIs for OOP, websites with rapidly changing requirements for Agile - were becoming common, which tended to obscure the faults of them both while encouraging a sense of evangelism about the tool among people who had never really done any of the sorts of projects where they were less suitable. This in turn led to even more determined evangelism by people who had been converted to them but didn't actually understand them, meaning that most of what got propagated were the misunderstandings and dumbed-down versions.
The same phenomenon occurred with Hungarian notation - it was useful for certain things but not everything, and the version most people saw was a corrupted version of the idea that threw away its primary advantages even for the things it was useful for.
Unfortunately, also like OOP, Agile is an approach that is particularly demanding of developer competence; it is impossible to say if the approaches really worked in the first place, because people like Alan Kay and Kent Beck were such good designers that they probably would have succeeded in their projects regardless of the approach taken (actually, given that Kay's goal for Smalltalk was basically a 'scripting language for the masses', it didn't actually succeed in its goals in the first place).
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
Let me tell you a little secret: Waterfall has always been a strawman.
Exactly, and thank you for providing all of the details :) [although as I pointed out a few projects actually have done waterfall - with the only one I was involved in resulting in unmitigated disaster].
Agile isn't impossible, but like OOP, it is really only applicable to fairly narrow range of programming projects
Can you please identify which specific Principles or Value Comparisons from the Manifest do not apply to a specific project type?
-
@thecpuwizard Uh oh, I was editing that some more when you replied. Bad Schol-R, bad, no biscuit!
I'll go over the weaknesses of the Principles later; however, you have to also keep in mind that the Principles themselves are usually misrepresented and misunderstood, meaning that even their strengths are often neutralized in practice.
The main class of projects that it doesn't really apply to are those where, once deployed, there really isn't any way to update it, and those where avoiding failures is itself mission critical - things like embedded hardware for control of automobiles, aircraft, power plants, medical equipment, military hardware, and so forth.
It also tends to have problems in fields where requirements change slowly, and especially those which have long-standing regulatory restrictions, such as banking software.
Database design in general tends to be a poor fit for Agile because changes to an existing database tend to cascade across the whole system. For OOP, there is a similar problem, in that the data tends to be fine-grained - trying to tie it up in a bow and call it an object severs the critical connections that make the data useful.
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard Uh oh, I was editing that some more when you replied. Bad Schol-R, bad, no biscuit!
The main class that it doesn't really apply to are those where, once deployed, there really isn't any way to update it, and those where avoiding failures is itself mission critical - things like embedded hardware for control of automobiles, aircraft, power plants, medical equipment, military hardware, and so forth.
Do not confuse internal processes with external deliveries. I do a fair amount of industrial control and have even done implanted medical - all while following Agile [as per Manifesto] principles and utilizing the Scrum Framework.
It also tend to have problems in fields where requirements change slowly, and have regulatory restrictions, such as banking software.
Database design in general tends to be a poor fit for Agile because changes to an existing database tend to cascade across the whole system. For OOP, there is a similar problem, in that the data tends to be fine-grained - trying to tie it up in a bow and call it an object severs the critical connections that make the data useful.
Again, Which SPECIFIC Principle(s)? http://agilemanifesto.org/principles.html
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
I did actually work on ONE project that was Waterfall in the 1970s. Within a week of the company getting the contract, it was realized that the device would never work. No matter, it was built. It passed the individual tests. When it was turned on [by the client, against written recommendation] it literally caught fire and had a smallish explosion.
@Maciejasjmj @Yamikuronue some inspiration for your thedailywtf front page writing
-
@magus said in Is there any big software corporation out there still practicing Waterfall?:
Isn't Waterfall supposed to be a strawman?
That was my first post in this thread.
-
@blakeyrat said in Is there any big software corporation out there still practicing Waterfall?:
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
Clearly that is your opinion, but it contradicts the experience of thousands of teams who have been quite successful.
The fact that a team is successful and does X doesn't mean that X is the cause of the team being successful.
It also doesn't mean that the team was successful at solving the problems our team was trying to solve. 700,000 companies using Scrum to make simple REST apps doesn't convince me it can be used to make complicated applications to enforce thousands of Government rules and regulations.
We use it very successfully to handle interoping healthcare messages between different vendors and enforcing different federal and state laws about certain medications.
We also aren't strictly Scrum. When our company switched to agile practices, it was pretty rigid scrum. Over the years teams have been allowed to evolve their own processes. Most are some flavor of a Scrum-lite. My team did Kanban for a while (we're an infrastructure team so defining all work up front is difficult). We're now Srumban since having the pseudo-sprints allows our product owner to better track pace and prioritize the work items that are better defined or have an internal due date. The developers and QA still pretty much treat it like Kanban.
-
@mikehurley said in Is there any big software corporation out there still practicing Waterfall?:
We use it very successfully to handle interoping healthcare messages between different vendors and enforcing different federal and state laws about certain medications.
But did you adopt it before or after the core functionality of your product existed?
Again: I am not saying Scrum never works. I'm not saying you can't write healthcare applications using Scrum. I'm not even saying it's impossible to write a complex greenfield healthcare application using Scrum. I like Scrum for maintenance development (which is the majority of all development) and would adopt it in that situation if I ran a team.
All I'm saying is that scrum is a terrible fit for greenfield development of very complex products.
-
@thecpuwizard OK, I can do that. Note, however, that the bigger problem is not with the principles, but with the way they are misunderstood, misused, and misrepresented - the existence of such a Creed (in software or anything else) is a problem itself, even when the statements within it are valid (when sensibly applied) on their own.
In addition, I see them as solving the wrong problems. While they identify the biggest issue at hand - communication between the stakeholders - they try to address the group who is least likely to have a communication problem (though they usually do have one).
Don't get me wrong, Agile is a useful set of tools to have, and a step in the right direction. However, it is only one step in a journey we personally will never see the endpoint of, if it even has one. In the end, Agile is (as I have said before) not a model of software engineering, but a tacit admission that we don't know how to engineer software yet, and probably won't for centuries to come. Just making that admission is a good move forward.
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
This assumes that the requirement changes themselves are rational and have been vetted for effectiveness. In practice, that's the exception, rather than the rule.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
My problem here is that this isn't always possible. There are a lot of projects where you have to swallow the whole bowling ball - not just as a matter of delivery, but during development. How do you intend to iterate in the design of, for example, a life support system for a spacecraft? There is only so much that can be done through testing in those cases.
Business people and developers must work together daily throughout the project.
Feel free to try to convince the non-developer stakeholders of this. IME, the problem isn't exclusively on the developers' side (though it is usually happens that the devs are partly to blame, too), but on that of the managers and clients as well - it becomes a matter of trying educate people who think they know what they're doing, and all too often, think that you don't know what you are doing. Asking for further guidance is more likely to lead to them concluding that you are incompetent (especially if they are themselves incompetent), and becoming less communicative just when you need them to be more communicative. Agile goes against the grain for most business customers even more than it does most programmers.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
This leads to precisely the problem I mentioned about competence - if a team is competent and motivated, then they don't really need a formal methodology (though having one does smooth things out a bit); if they aren't, then no methodology can hope to help them.
The purpose of a formal procedure - again, in or out of IT - is to account for the shortcomings of less than stellar players and ensure consistency. A methodology that only works for stellar players, or discards consistency as a goal, is nearly useless, and Agile does both.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
I see two issues with this. First, it is not always feasible in large projects - you can't have a face to face meeting involving 250 devs on three continents, which means that the problem shifts to one of how you organize the project teams. That is something that no methodology can give guidance on, and even if it could, it would be subject to the whims of managers and political atmosphere of the organization.
The other issue is that is devalues written records and audit trails. While a well-run project will still have them, a poorly run project would use that as a n excuse to slough off that responsibility.
The best architectures, requirements, and designs emerge from self-organizing teams.
This statement is in direct contradiction to the realities of software as a commercial or public enterprise. Most software teams do not have that sort of control over these things, and can't.
Continuous attention to technical excellence and good design enhances agility.
Again, this is a contradiction. If you have technical excellence, you don't need formal methods. Moreover, any large project that relies on excellence is doomed to failure, because there is no way to ensure excellence in any kind of project except by cherry-picking of personnel followed by extensive training and practice, something that simply doesn't scale.
To put it bluntly: formal methods aren't about excellence, they are about mediocrity, which is actually a good thing in a large project - you can be consistently mediocre, but no one can be consistently excellent on a large scale. In large projects, excellence is disruptive - decreasing the Truck Number (in the sense of how many people there are whom the loss of one would cripple the project) isn't just about distributing risk, it is also about maintaining stability.
TL;DR: Perfect is the enemy of good. That a perfectionist like me can see that (even if I don't quite believe it) should tell you something.
I'm not sure if I really answered the question, though. Not sure I actually can, now that I've said all this, so... point conceded, @TheCPUWizard.
-
@blakeyrat said in Is there any big software corporation out there still practicing Waterfall?:
@mikehurley said in Is there any big software corporation out there still practicing Waterfall?:
We use it very successfully to handle interoping healthcare messages between different vendors and enforcing different federal and state laws about certain medications.
But did you adopt it before or after the core functionality of your product existed?
Again: I am not saying Scrum never works. I'm not saying you can't write healthcare applications using Scrum. I'm not even saying it's impossible to write a complex greenfield healthcare application using Scrum. I like Scrum for maintenance development (which is the majority of all development) and would adopt it in that situation if I ran a team.
All I'm saying is that scrum is a terrible fit for greenfield development of very complex products.
We've worked on existing core codebases and several new ones as the company has ramped up the amount of work they need done. My team has spun up from scratch under these processes a moderately complex REST service and various Hadoop/Spark jobs that involved a bunch of research to make sure we do things right or adjust as we learn new things. We do research stories when we don't know enough to write implementation stories.
Chunking up research is really no different than chunking up actual coding.
-
WtfFramework implementations are Waterfragile bordering on Waterfall. Changes are expected but prompt a full reestimation.
WtfFramework itself is pure anarchy. Which we call "agile". There is literally no process. Any attempt I make to impose structure is overridden within the week by someone's need for something RIGHT NOW. I hardly even get to write specs for most things, and when I do they're cartoonish email caricatures of specs.
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
Feel free to try to convince the non-developer stakeholders of this.
At WtfCorp, non-developer stakeholders are fond of literally meeting continuously, all day, every day, inviting developers to all of them, and being incensed when the meetings are ignored by developers.
-
@weng said in Is there any big software corporation out there still practicing Waterfall?:
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
Feel free to try to convince the non-developer stakeholders of this.
At WtfCorp, non-developer stakeholders are fond of literally meeting continuously, all day, every day, inviting developers to all of them, and being incensed when the meetings are ignored by developers.
OK, the developers don't listen, which is an obvious problem with or without Agile methods. Agile tells them to listen, and rightfully so. So: what makes anyone think they will listen to the Agile principles? My experience is that a lot of places give lip service to them without actually following them, or even knowing them.
Also, I am guessing that most of the other stakeholders are talking at each other rather than to each other, and none of them are listening to anybody. This is often the case, and given WtfCorp's history (/me glances over at the smartphone on my desk), it is a sure bet that it's the case there, too.
As an aside, in many places such refusal to listen (on the part of the production team, that is - MGT is another story) would lead to disciplinary measures and a threat of dismissal. My understanding of WtfCorp (IIRC who that actually is) is that they have about the same level of reluctance to sack people that IBM did back in the day. Give them punishment duties, or shift them around between departments and locations until they quit, perhaps, but outright fire them for anything short of criminal behavior, no. In most Japanese companies, a job once obtained is a near-sinecure, and I understand the same is true in the country where (IIRC) WtfCorp resides.
-
@scholrlea You must be confusing us with someone else, I've never heard of anyone getting shuffled or punishment duty. Pig ignorance and incorrectness is rewarded and crimes ignored. I've only heard of firings in sexual harassment cases involving actual genitalia. Which is to say, no, we're not Samsung 💩
You're dead right on the "talking at each other" thing, though. And I challenge you to find any developer on the planet who can be productive wearing a headset having the same idiotic drivel being repeated in their ear for 8 hours every day.
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
This in turn led to even more determined evangelism by people who had
been converted to them but didn't actually understand thembooks to sell, meaning that most of what got propagated were the misunderstandings and dumbed-down versions.FTFC
-
@mzh said in Is there any big software corporation out there still practicing Waterfall?:
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
This in turn led to even more determined evangelism by people who had
been converted to them but didn't actually understand thembooks to sell, meaning that most of what got propagated were the misunderstandings and dumbed-down versions.FTFC
That too.
-
So... I've posted this before, but it seems relevant:
Why “Agile” and especially Scrum are terrible
Note: My first novel, Farisa’s Crossing, will be released in spring 2022. Agility is a good thing, no doubt, and the Agile Manifesto isn’t unreasonable. Compared to a straw-man practice…
-
@masonwheeler I agree with some of the things he said, but I think he's overestimating the number of programmers who actually do reach mastery - by a lot. Unfortunately, corporate programming begets corporate programming practices, and corporate programming practices discourage effort made on either improving skills outside of a constrained range, and experimentation on unauthorized practices. This means it is impossible to say whether the practices are the root of the problem, or the lack of skilled programmers - its something of a chicken-and-egg problem.
More to the point, though, is the question, how much of this can actually be done by unskilled labor, or automated, and how much absolutely requires genuine craftsmanship? Because in the end, that's what we are really talking about - a craft, in the old-time sense of skilled labor that requires decades of training in order to work in it effectively.
Over the past 30 years, the degree of automation has increased greatly, through improved tools such as better compilers that generate better code and emit better error messages, build and project managers such as Ant and Maven, IDEs that do a lot of the scut work, code profilers, form generators, unit testing systems, continuous integration tools, etc. The barriers to entry have similarly decreased through the widespread use of easier to use higher-level languages such as Python, Java, C#, etc., bigger libraries, frameworks, online documentation, online learning communities and so forth.
This sounds like it should be causing the amount of craft aspects of the field to decrease - but it isn't. The thing is, the demand has grown at an even greater rate, driven in part by the existence of exactly those improvements - a clear case of Jevon's Paradox at work. The majority of programming work today existed because it has become easier and cheaper to do the less-skilled aspects of programming.
Thus, we can safely say the the craft aspects of the work - which also means the parts we don't have any engineering understanding of yet, though even some engineering fields such as architecture are also crafts - have also grown, grown so fast that it may well be at the point of a System Collapse - where the cost of adding one more incremental improvement becomes unbearable by the overall infrastructure because it is an improvement. I don't think we're at this point yet, but it could well happen if there is no infrastructure for expanding the number of craftspeople in the field.
Historically, crafts were formed into guilds, and the guilds held a monopoly on their crafts. Adam Smith aside, there was more to this than just greed. As long as these fields required skilled labor to perform, there was a need to pass these skills along, and ensure that the reputation of the craft as a whole remained positive. The training took years or even decades in many cases, and requires a structured system of craft advancement independent of the employment - meaning that the guild masters, established experts of the craft, had to also manage the customers' expectations and requirements. A guild master had to be both a domain expert and an expert communicator - and often a politician as well, in light of the need for their chartered monopolies.
In the late 17th and early 18th century, improvements in methods made it increasingly possible to break down most of the skilled craft work into unskilled trades, where things which would have been done by a single craftsman would now be done by several tradesmen. This was a crucial prerequisite for both the Industrial Revolution and for the rise of laissez faire economic principles - which is why The Wealth of Nations is as much about industrial practices as it is about contrasting mercantilism (which was based on expanding the old monopoly system abroad and gather resources from colonies to be used at home) with laissez faire (which focused allowing the market to "sink or rise to its level in a state of nature" and argued that both import and manufacturing should be competition-based).
Right now, though, programming has reversed that trend, and is doing so at an increasing rate, due to precisely those same kinds of methodological improvements. That leaves us... where? Do we need to go back to a guild system, where the experienced programmers call the shots and the customers are (as that essay put it) managed as much as the junior programmers are? I tend to think this might be the case, but if so, it isn't going to go over well - the strong libertarian/anarchist/Objectivist thread running through programming culture since the late 1960s would find this extremely threatening. While by no means are all or even most programmers in this camp, many of the master-level programmers today are (though they often polarize on the 'free enterprise' vs. 'free exchange of ideas' topics even when they otherwise agree), and a lot of the others tend to be just apolitical - they got into programming in part to escape from such mammalian territorial pissings, and avoid it as much as possible. I see this changing, for reasons having nothing to do with either the Belief Systems or technical prowess, but right now trying to form programmers into One Big Union would make herding cats look trivial.
-
If you do waterfall on
bankingsoftware, you are guaranteed to release software that breaks laws.FTFM.
-
First a short summary before I address ScholRLEA's (very good) post...
I 100% agree that the Agile Principles are "misunderstood, misused, and misrepresented" (as are the Value Comparisons). That being said, I put the fault directly on the people involved in such activities, and not on the principles of the Manifesto itself.
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
First, and foremost, thanks for a great post that raises many valid points :)
Don't get me wrong, Agile is a useful set of tools to have, and a step in the right direction. However, it is only one step in a journey we personally will never see the endpoint of, if it even has one. In the end, Agile is (as I have said before) not a model of software engineering, but a tacit admission that we don't know how to engineer software yet, and probably won't for centuries to come. Just making that admission is a good move forward.
Thus Principle #12: At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
This assumes that the requirement changes themselves are rational and have been vetted for effectiveness. In practice, that's the exception, rather than the rule.
People often mistake "wrong" or "missing" requirements as being included in the above. The Principle is about "rational and vetted" requirement [R] that exists at T0 and then there is (again rational and vetted) change that occurs at a future point in time.
This is similar to people taking the value comparison of "Responding to change over following a plan" to mean that one should not have a plan. Consider if I had a plan [made in May] to go to Houston for pleasure in late August; pretty sure that changing the plan [not "following the plan"] would be appropriate.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
My problem here is that this isn't always possible. There are a lot of projects where you have to swallow the whole bowling ball - not just as a matter of delivery, but during development. How do you intend to iterate in the design of, for example, a life support system for a spacecraft? There is only so much that can be done through testing in those cases.
True, the comprehensive solution can not be done [I have worked on space systems, though not directly related to life support], but one can deliver a piece of working software that reads a given sensor, and continue to make expansions to the scope of what is working.
A key element here (often neglected) is that at no point in time should something which previously worked become broken.Business people and developers must work together daily throughout the project.
Feel free to try to convince the non-developer stakeholders of this. IME, the problem isn't exclusively on the developers' side (though it is usually happens that the devs are partly to blame, too), but on that of the managers and clients as well - it becomes a matter of trying educate people who think they know what they're doing, and all too often, think that you don't know what you are doing. Asking for further guidance is more likely to lead to them concluding that you are incompetent (especially if they are themselves incompetent), and becoming less communicative just when you need them to be more communicative. Agile goes against the grain for most business customers even more than it does most programmers.
Agree with "Agile goes against the grain for most business customers even more than it does most programmers" getting support and buy-in from the CEO on down is a key factor here.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
This leads to precisely the problem I mentioned about competence - if a team is competent and motivated, then they don't really need a formal methodology (though having one does smooth things out a bit); if they aren't, then no methodology can hope to help them.
The purpose of a formal procedure - again, in or out of IT - is to account for the shortcomings of less than stellar players and ensure consistency. A methodology that only works for stellar players, or discards consistency as a goal, is nearly useless, and Agile does both.The principles are NOT a methodology!!!!!
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
I see two issues with this. First, it is not always feasible in large projects - you can't have a face to face meeting involving 250 devs on three continents, which means that the problem shifts to one of how you organize the project teams. That is something that no methodology can give guidance on, and even if it could, it would be subject to the whims of managers and political atmosphere of the organization.
Yes, there are times when there are constraints which require a LESS "efficient and effective method of conveying information to and within a development team" are required. Acknowledging that the "bits per second" will be lower and that a tradeoff has been made is a key point. Also, have you never noticed the difference in dynamics when body language and facial expressions are included/excluded from a communication?
The other issue is that is devalues written records and audit trails. While a well-run project will still have them, a poorly run project would use that as a n excuse to slough off that responsibility.
As you point out, this is a problem of application, not of the principle. Face to Face meetings can still be used to provide all of the artifacts of value ["written records and audit trails"]
The best architectures, requirements, and designs emerge from self-organizing teams.
This statement is in direct contradiction to the realities of software as a commercial or public enterprise. Most software teams do not have that sort of control over these things, and can't.
Hmmm. Then have those teams been "Given them the environment and support they need, and trusted to get the job done"???
Continuous attention to technical excellence and good design enhances agility.
Again, this is a contradiction. If you have technical excellence, you don't need formal methods. Moreover, any large project that relies on excellence is doomed to failure, because there is no way to ensure excellence in any kind of project except by cherry-picking of personnel followed by extensive training and practice, something that simply doesn't scale.
To put it bluntly: formal methods aren't about excellence, they are about mediocrity, which is actually a good thing in a large project - you can be consistently mediocre, but no one can be consistently excellent on a large scale. In large projects, excellence is disruptive - decreasing the Truck Number (in the sense of how many people there are whom the loss of one would cripple the project) isn't just about distributing risk, it is also about maintaining stability.
TL;DR: Perfect is the enemy of good. That a perfectionist like me can see that (even if I don't quite believe it) should tell you something.Again, not reading the Principle accurately... "Continuous Attention" does not imply a goal of achieving!!!!! It is about realizing those places where there is "mediocracy" and understanding the ramification. "Playing Ostrich" is one of the leading causes to out of control technical debt.
Again, I agree with your assessment of how the principles are miss-applied, and admit that my responses are overly simple [this post is long enough!]; but I hope this may provide some value to at least a few of the readers here....
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
Hence: "Business people and developers must work together daily throughout the project. " [Principle #4].
Unfortunately, when that hits reality there's a problem in that the people who actually know the business are frequently unavailable. Often for months on end. Some of them are also impatient assholes (I'll not comment on whether they're a minority or otherwise). Business people who are highly available are often pretty junior, and therefore likely to miss out whole critical parts of what the app actually needs to do.
And yes, I've seen that quite a few times.
-
@dkf said in Is there any big software corporation out there still practicing Waterfall?:
Unfortunately, when that hits reality there's a problem in that the people who actually know the business are frequently unavailable. Often for months on end. Some of them are also impatient assholes (I'll not comment on whether they're a minority or otherwise). Business people who are highly available are often pretty junior, and therefore likely to miss out whole critical parts of what the app actually needs to do.
And yes, I've seen that quite a few times.I don't deny that.. and it is one of the biggest impediments to Agile (as per the Manifesto) adoption. The "cure" is to have management from the CEO down insuring that the "people who actually know the business" are regularly available [e.g. some of them scheduled to physically be with the developers] and realize that "impatient assholes" are not "motivated individuals" (in the sense intended) and that nobody is irreplaceable.
About 5 years ago I was retained by a company looking to adopt agile. The exact problems you just touched on were the main blockers, and could be traced to 3 people high (but not board level) in the organization who had been there "forever". The organization was not willing to take these 3 out of power, and instead decided to defer the adoption [knowing it would be futile] until they left of natural causes [retirement]. Last I checked 2 have retired, and one is likely to be gone within 6 months - they are starting preliminary talks to discuss resuming the adoption.
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
The "cure" is to have management from the CEO down insuring that the "people who actually know the business" are regularly available [e.g. some of them scheduled to physically be with the developers] and realize that "impatient assholes" are not "motivated individuals" (in the sense intended) and that nobody is irreplaceable.
If you end up working for WTF-U or any other big university, you'll never get that sort of directive. The people who know the business are not just sitting there, waiting to have a weekly meeting with you. They're up to their eyeballs in other work that is important, often more important (for the moment) than what you are doing, even if what you are doing is vital for long-term plans.
Is nobody irreplaceable? Sure! We don't even need to keep entire faculties, so of course everyone is replaceable!
-
@dkf said in Is there any big software corporation out there still practicing Waterfall?:
If you end up working for WTF-U or any other big university, you'll never get that sort of directive
University [based on USA experience] is indeed a different world than Corporate. My experience there is somewhat limited, but based on that: At the faculty side I agree 100%; on the "business operations" side I have had some success (large school reworking student administrations and record keeping systems) - but it did take a lot of work....
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
University [based on USA experience] is indeed a different world than Corporate. My experience there is somewhat limited, but based on that: At the faculty side I agree 100%; on the "business operations" side I have had some success (large school reworking student administrations and record keeping systems) - but it did take a lot of work....
Be aware that there's quite a bit of difference between how universities operate in different parts of the world. Some places, they're damn close to being extensions of government. Others… not so much. In the UK, they're technically not profit-making enterprises, but that's just for tax reasons (being an “educational charity” saves a ton of money) and they're mostly run on pretty corporate lines behind a façade of academe.
I know that our IT department runs mostly on Kanban. I suppose that beats the previous ways of working, which seemed to be mostly based on passive-aggressive insolence, (metaphorical) megaphones as a communication mechanism, and intelligence-free ego.
-
@thecpuwizard said in Is there any big software corporation out there still practicing Waterfall?:
I did actually work on ONE project that was Waterfall in the 1970s. Within a week of the company getting the contract, it was realized that the device would never work. No matter, it was built. It passed the individual tests. When it was turned on [by the client, against written recommendation] it literally caught fire and had a smallish explosion.
F-35 v1.0?
-
@the_quiet_one said in Is there any big software corporation out there still practicing Waterfall?:
@gąska said in Is there any big software corporation out there still practicing Waterfall?:
@magus personally, I think the main purpose of unit tests isn't to check if the code is correct, but to check if behavior changed unintentionally during implementation of more or less related feature or during refactoring. No amount of manual testing will ever give you that.
Well, yes there are manual regression tests you can do. You'd need a large QA staff to really make it work but it is possible.
QA won't be able to cover cases that don't exist in the application yet.
-
@gąska errr... That's why you have them on a staging environment with release candidates.
-
@gąska said in Is there any big software corporation out there still practicing Waterfall?:
QA won't be able to cover cases that don't exist in the application yet.
"QA" is probably the most misrepresented entity out there.... There is a difference between quality ASSURANCE and quality CONTROL. "Testing" (of all forms, manual/automated) is a measure of control, and really has nothing to do with assurance.
Bluntly, the member of so-called QA teams (really QC teams) do not have the overall skills to assess quality [hint: functional tests passing may be less than 5% of the total]
-
@thecpuwizard sucks that terms don't mean what they originally meant anymore, right?
I recall a blakeyrant on this topic some months ago, but I can't quite remember what exactly it was about.
-
@gąska said in Is there any big software corporation out there still practicing Waterfall?:
that terms don't mean what they originally meant anymore, right?
There can be both evolution [positive connotation] and devolution [negative]. In this case I think it is definitely the latter.
I remember when people (in certain organizations) actually were honored being selected to be part of QA and transferred out of Engineering [while at the same time people in QC were looking to get into product engineering]
-
@scholrlea said in Is there any big software corporation out there still practicing Waterfall?:
@masonwheeler I agree with some of the things he said, but I think he's overestimating the number of programmers who actually do reach mastery - by a lot. Unfortunately, corporate programming begets corporate programming practices, and corporate programming practices discourage effort made on either improving skills outside of a constrained range, and experimentation on unauthorized practices. This means it is impossible to say whether the practices are the root of the problem, or the lack of skilled programmers - its something of a chicken-and-egg problem.
More to the point, though, is the question, how much of this can actually be done by unskilled labor, or automated, and how much absolutely requires genuine craftsmanship? Because in the end, that's what we are really talking about - a craft, in the old-time sense of skilled labor that requires decades of training in order to work in it effectively.
Over the past 30 years, the degree of automation has increased greatly, through improved tools such as better compilers that generate better code and emit better error messages, build and project managers such as Ant and Maven, IDEs that do a lot of the scut work, code profilers, form generators, unit testing systems, continuous integration tools, etc. The barriers to entry have similarly decreased through the widespread use of easier to use higher-level languages such as Python, Java, C#, etc., bigger libraries, frameworks, online documentation, online learning communities and so forth.
This sounds like it should be causing the amount of craft aspects of the field to decrease - but it isn't. The thing is, the demand has grown at an even greater rate, driven in part by the existence of exactly those improvements - a clear case of Jevon's Paradox at work. The majority of programming work today existed because it has become easier and cheaper to do the less-skilled aspects of programming.
Thus, we can safely say the the craft aspects of the work - which also means the parts we don't have any engineering understanding of yet, though even some engineering fields such as architecture are also crafts - have also grown, grown so fast that it may well be at the point of a System Collapse - where the cost of adding one more incremental improvement becomes unbearable by the overall infrastructure because it is an improvement. I don't think we're at this point yet, but it could well happen if there is no infrastructure for expanding the number of craftspeople in the field.
Historically, crafts were formed into guilds, and the guilds held a monopoly on their crafts. Adam Smith aside, there was more to this than just greed. As long as these fields required skilled labor to perform, there was a need to pass these skills along, and ensure that the reputation of the craft as a whole remained positive. The training took years or even decades in many cases, and requires a structured system of craft advancement independent of the employment - meaning that the guild masters, established experts of the craft, had to also manage the customers' expectations and requirements. A guild master had to be both a domain expert and an expert communicator - and often a politician as well, in light of the need for their chartered monopolies.
In the late 17th and early 18th century, improvements in methods made it increasingly possible to break down most of the skilled craft work into unskilled trades, where things which would have been done by a single craftsman would now be done by several tradesmen. This was a crucial prerequisite for both the Industrial Revolution and for the rise of laissez faire economic principles - which is why The Wealth of Nations is as much about industrial practices as it is about contrasting mercantilism (which was based on expanding the old monopoly system abroad and gather resources from colonies to be used at home) with laissez faire (which focused allowing the market to "sink or rise to its level in a state of nature" and argued that both import and manufacturing should be competition-based).
Right now, though, programming has reversed that trend, and is doing so at an increasing rate, due to precisely those same kinds of methodological improvements. That leaves us... where? Do we need to go back to a guild system, where the experienced programmers call the shots and the customers are (as that essay put it) managed as much as the junior programmers are? I tend to think this might be the case, but if so, it isn't going to go over well - the strong libertarian/anarchist/Objectivist thread running through programming culture since the late 1960s would find this extremely threatening. While by no means are all or even most programmers in this camp, many of the master-level programmers today are (though they often polarize on the 'free enterprise' vs. 'free exchange of ideas' topics even when they otherwise agree), and a lot of the others tend to be just apolitical - they got into programming in part to escape from such mammalian territorial pissings, and avoid it as much as possible. I see this changing, for reasons having nothing to do with either the Belief Systems or technical prowess, but right now trying to form programmers into One Big Union would make herding cats look trivial.
A few potential problems with collectivization/guildifying efforts in re. programming & software development are
- The free access to tools and information (esp. learning and practice material), and
- The general utility of the skills involved.
Both of those are good things, and should not be shorted in service to making the profession stronger. Unlike older guilds and monopolies, such a professional body may be better off expanding knowledge and experience with the basics of the field. This would serve to reduce the amount of low-level work imposed on professionals (not many people need to hire accountants to do their weekly bookkeeping, nor should they need to hire a professional programmer to do basic Excel macros) and, if done well, increase the perceived value of hiring a professional.
I can see definite benefits to having some sort of strong professional body (and not just for this trade).
- Customer expectation management (no, you can't do that in 2 months. Not even with the best-of-the-best.)
- Customer requirements management (no, you don't need a dozen of the the best and the brightest to do that, an apprentice can do it in a week.)
- Verified minimum competencies (a junior/medior/senior type system based on demonstrated ability and history.)
- Professional mentorship & training (juniors = apprentices, and they need to work under masters/seniors until they're qualified to be journeymen/mediors. Also, requiring companies that employ a sufficiently large number of guild members to accommodate this.)
- Collective bargaining (in particular with regards to setting minimum salaries / pay scales / benefits, as well as maximum uncompensated work such as emergency overtime.)
- Ensuring professional growth opportunities (matching sufficiently skilled individuals with jobs that will benefit their career progression, as well as providing guidance on where to go next)
And probably a few other things.
Of course, in this field and this country, all of the potential benefits are likely to be overlooked in the name of freedom and capitalism über alles, and a strong fear of things that smell like socialism or unions.
Incidentally, that one about collective bargaining is a major problem in the trade I'm currently in. Despite the work itself not having gotten significantly easier (manual skilled labor that's not very susceptible to modern automation), rates have gone down a quite a bit after accounting for inflation. There's strong downward pressure on the rates from above, and not a lot of ability to push back absent collective effort.
