Why is programming complicated?

jinpa

Conceptually, programming is relatively simple. Programs perform CRUD operations, they move files around, and they display information to the user and allow the user to perform CRUD operations and move files around. There are exceptions, but that's what most programs do.

So why is it that programming is complicated? Why does developing a program take a thousand hours of programmer time rather than ten minutes?

I figure it has something to do with that there isn't one standard way of doing things. You're brought onto a project, and you have to figure out how to connect one "special" program to another "special" program, and there countless ways for a program to be coded "specially".

Is there more to it than that?

boomzilla

My Sensei likes to joke: "Why is this stuff so difficult? Because it's hard!"

As for programming, I think we underestimate the complexity of requirements. Lots of stuff you don't realize until you're in the middle of it.

Oh, what about this edge case? What should we do then?

And the complexity is fractal within the system.

Jaime

@jinpa Here's a quote from Fred Brooks in "The Mythical Man-Month":

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems

So, fifty years ago we knew that programming wasn't conceptually simple. Why do you believe it is?

Applied Mediocrity

(Philosophically speaking) programming is abstraction of real life. And real life is very, very complicated. You most likely don't even notice how many ostensibly or apparently unimportant things every day you manage to ignore, how many things your "hardware accelerates" without involving your conscious self, because you just know they "make no sense" or have a very low chance of happening.
To that end and in itself, creating a specific way of doing things is, counter-intuitively, not creating something from nothing in an ordered manner, but a reduction of endless complexity bringing certain order to chaos.

Disclaimer: Maybe. I don't entirely know if I said what I meant there, nor I have a clear idea of what I meant in the first place.

Dragoon

@jinpa said in Why is programming complicated?:

Is there more to it than that?

Yes, take something as simple as a single dialog box and a button that will just display the text entered.

How many characters do you accept? What char sets do you support? Foreign characters? How does it respond to null inputs? How does it respond to someone entering the entirety of 'War and Peace'? What font is being used? How does everything look at different screen resolutions and DPI scales? Is is written such that tomorrow you can easily add colors/bold/italics/etc...?

That list goes on and on and no. Finally, you hit a request that your original robust framework can't handle. You can't justify rewriting the mountain of code that satisfies everybody else, so you have to find a way to tack this new thing on. Now the real complexity begins.

boomzilla

For a humorous demonstration of a lot of the stuff being discussed, I offer this classic:

https://www.youtube.com/watch?v=Ct-lOOUqmyY

cvi

@Dragoon This. I was thinking of the same example with moving a file (or several files).

At its simplest, that's a line or two in bash. Unless you start accounting for errors. What if one of the moves fail? If this a short script for myself (mainly), that's not too much of a deal. If somebody else is supposed to use it ... what do you do? Report an error? Maybe retry first, or see if there is an obvious reason why it failed? Maybe roll back all other changes? Etc etc.

jinpa

@Dragoon said in Why is programming complicated?:

@jinpa said in Why is programming complicated?:

Is there more to it than that?

Yes, take something as simple as a single dialog box and a button that will just display the text entered.

How many characters do you accept? What char sets do you support? Foreign characters? How does it respond to null inputs? How does it respond to someone entering the entirety of 'War and Peace'? What font is being used? How does everything look at different screen resolutions and DPI scales? Is is written such that tomorrow you can easily add colors/bold/italics/etc...?

Much of this is a solved problem, though. If we used dialog boxes that already addressed those questions, they would work in almost all cases. But the font: you have hit on an example of another big problem: The desire for individuality.

Applied Mediocrity

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though

The law of leaky abstractions dictates that for non-trivial tasks you cannot fully abstract the reality away. This applies to said dialog box or anything some framework or library or even just a wrapper function may offer you. You can be ignorant of it or, upon having it made known, elect to ignore it such as that, for example, only default fonts at reasonable size are supported and the rest is considered user error. But that's you. Your program doesn't do that by itself.

jinpa

@Applied-Mediocrity said in Why is programming complicated?:

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though

The law of leaky abstractions dictates that for non-trivial tasks you cannot fully abstract the reality away. This applies to said dialog box or anything some framework or library or even just a wrapper function may offer you. You can be ignorant of it or, upon having it made known, elect to ignore it such as that, for example, only default fonts at reasonable size are supported and the rest is considered user error. But that's you. Your program doesn't do that by itself.

No, I'm definitely not depending on ignorance. That's a separate problem which is what I am trying to eliminate. You mentioned fonts, whereas I was referring to the other elements you mentioned - I addressed fonts separately.

And all of those are the same problems in almost all softwares - they have already been solved. It is rare that you actually need a different dialog box solution for one software than another.

Jaime

@cvi Here's a more programmery one:

Someone wants another status to be added to the list of statuses for item that show up on some screen.

Inevitably, someone will chime in with "why do we need a programmer to add an item to a list, just make it data-driven". OK, table created. But, is that status reached only by someone picking it from the list? Do things in that status show up on this other screen? In the daily report of overdue items and SLA performance, is this new status counted as "not started", "in progress", or "done"? And the question come rolling in along with answers that elicit even more questions.

Something that sounded simple like "add a status", is revealed to have, all along, been a task that affects many job roles and many parts of the application. You either end up with a very sophisticated design with data-driven statuses that have their own metadata, customizable triggers for status changes, and other bells and whistles, or realizing that it's a programming task after all. Even if you end up with a data-driven design, you're likely to need the application team to configure a new status properly because of all of the options available and all of the teams that will need to be consulted to set the options correctly.

Eventually, you realize that you've built a workflow product. Or, if you realized that earlier and purchased a workflow product, then you realize that you need a development team to manage the workflow product.

boomzilla

@Jaime said in Why is programming complicated?:

You either end up with a very sophisticated design with data-driven statuses that have their own metadata, customizable triggers for status changes, and other bells and whistles

Alex Papadimoulis / Apr 21, 2006

The Inner-Platform Effect

I'm sure that a lot of you have may heard about "antipatterns." They're more or less the converse of "software design patterns" in that they describe a frequently repeated problem in designing a commonly-occurring solution. I've observed quite of a few of these antipatterns in the real world, but...

Applied Mediocrity

@jinpa said in Why is programming complicated?:

No, I'm definitely not depending on ignorance.

Then I'm afraid I cannot attempt to prove my point any further since it expects that you would have.

jinpa

@Applied-Mediocrity
All of the things mentioned about a dialog box are solved problems.
But you have illustrated the point about ambiguity of language and its nuances.

The example given about the list was more illustrative. But I think there is more to the current complexity of programming than understanding the business rules. That is certainly part of it, but each of those gets multiplied by other factors which relate to the uniqueness of the existing development environment.

Suppose, to use a greatly simplified example, there are two components to a software, and there was only one way of doing each of those. You're a programmer, and you come onto the project, so there's only way you have to understand.

But suppose there's two ways of doing each of those two components. So you're a programmer, you come onto the project, and that means that there's four possibilities of how it was done. Etc.

Dragoon

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though. If we used dialog boxes that already addressed those questions, they would work in almost all cases.

How many ways are there to count to 100?

Arantor

@Dragoon that depends, are we playing Mortimer's Variation, version 3, and as a result I presume that we can't just declare we're going to Mornington Crescent as an opening gambit?

There are more ways to count to 100 than there are valid moves in any configuration of Mornington Crescent.

Gern_Blaanston

@jinpa said in Why is programming complicated?:

@Dragoon said in Why is programming complicated?:

Yes, take something as simple as a single dialog box and a button that will just display the text entered.

How many characters do you accept?
What char sets do you support?
Foreign characters?
How does it respond to null inputs?
How does it respond to someone entering the entirety of 'War and Peace'?
What font is being used?
How does everything look at different screen resolutions and DPI scales?
Is is written such that tomorrow you can easily add colors/bold/italics/etc...?

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though.

But you still have to solve it for YOUR program. You still have to consider all of those questions.

Constructing a 30 story building is also a "solved problem". We know exactly how to do it and have done it many times. But that doesn't change the fact that there are A LOT of steps involved and A LOT of details that you have to pay attention to.

It is the same for any non-trivial piece of software. That is why programming is complicated.

Applied Mediocrity

@jinpa said in Why is programming complicated?:

are solved problems.

I posit that they aren't. For a practical example consider these solved problems regarding user interfaces designed before mass adoption of touchscreens and afterwards.
But in my post I also extended the scope to things other than dialog boxes.

The rest, I agree. The n*m is a simple, but sufficiently acceptable way to demonstrate how the complexity rises astronomically.

Gern_Blaanston

@Gern_Blaanston said in Why is programming complicated?:

@jinpa said in Why is programming complicated?:

@Dragoon said in Why is programming complicated?:

Yes, take something as simple as a single dialog box and a button that will just display the text entered.

How many characters do you accept?
What char sets do you support?
Foreign characters?
How does it respond to null inputs?
How does it respond to someone entering the entirety of 'War and Peace'?
What font is being used?
How does everything look at different screen resolutions and DPI scales?
Is is written such that tomorrow you can easily add colors/bold/italics/etc...?

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though.

But you still have to solve it for YOUR program. You still have to consider all of those questions.

Constructing a 30 story building is also a "solved problem". We know exactly how to do it and have done it many times. But that doesn't change the fact that there are A LOT of steps involved and A LOT of details that you have to pay attention to (if you don't want your building to fall down).

It is the same for any non-trivial piece of software. That is why programming is complicated.

boomzilla

@Applied-Mediocrity said in Why is programming complicated?:

@jinpa said in Why is programming complicated?:

are solved problems.

I posit that they aren't. For a practical example consider these solved problems regarding user interfaces designed before mass adoption of touchscreens and afterwards.
But in my post I also extended the scope to things other than dialog boxes.

The rest, I agree. The n*m is a simple, but sufficiently acceptable way to demonstrate how the complexity rises astronomically.

Yeah, he's ignoring YAGNI. Do we really need to consider all of those things every single time?

Also, it was a super simple seeming example that had a lot of complications. It should be easy for pretty much anyone who posts here to be able to see how this sort of analysis explodes for less trivial tasks.

LaoC

@jinpa said in Why is programming complicated?:

@Applied-Mediocrity
All of the things mentioned about a dialog box are solved problems.

I can't find the thread now, perhaps it hasn't been archived, but in a Usenet newsgroup of yore I've seen this competition of guys trying to write a correct and robust HelloWorld program while all the others tried to come up with conditions that would make it fail in unexpected and/or undetected-as-of-yet ways.
You'd start with the obvious

printf("Hello, world!");
return 0;`

but then you'd have to consider what happens if someone calls hello >>foo and foo is on a file system that is full or you're running into a file size ulimit or something, and the program gets a tad more complicated already.

#include <stdio.h>
#include <string.h>
#include <sysexits.h>

int main(void) {
   static const char hello[] = "Hello, world!\n";
   const int len = printf(hello);
   if(len != strlen(hello)) {
      fprintf(stderr,"write error");
      // TODO: differentiate between errors here?
      return EX_IOERR;
   }
   return EX_OK;
}

You might want to look at argv to make sure there were no arguments that the user expected to be used but that you've been silently ignoring. And what happens if you get a signal during the printf call? There could be plenty. Do you want to abort on SIGINT? Probably, but should there be an error message? What about SIGHUP or SIGPIPE? Perhaps you should make sure these have sane defaults before even starting, who knows what they might have been set to before you were exec'd? Your stdout may be set to non-blocking and you'd get an EAGAIN (I don't even know if printf can recover from that, perhaps you'd have to make sure it, or rather the write(2) restarts on its own). Depending on your use case (is this a college assignment or part of a mission control program for the James Webb telescope?) you may be able to ignore all those, or you could be at a HelloWorld of three pages full of advanced fuckery in no time.

Gurth

@jinpa said in Why is programming complicated?:

So why is it that programming is complicated?

Because you need to tell the computer exactly what to do, and often also how to do it. If you’re explaining something to a person, you can skip steps that are self-explanatory: “put the coffee into this part of the machine” will be perfectly understood even if you don’t first say to “take the ground coffee container from the cupboard, open the container, take the scoop that’s inside the container in your hand, scoop out one scoopful of ground coffee”.

dkf

@jinpa said in Why is programming complicated?:

that's what most programs do

You're showing your biases. Everyone has them, but in this case it's showing that you work in an application domain where simple CRUD typically is the solution. It's not universal.

FWIW, the common happy path often is very simple like you describe. The complexity comes when dealing with what happens when things go wrong, or when something extra needs to be tacked on "just for this time". Which wouldn't be so bad except all the extra bits of complexity can interact with each other.

Arantor

What happens if a CRUD operation fails?

Maybe something consumed all the autoincrement primary keys.

Maybe data corruption broke the index and you have two things with the same primary key (have seen this on MSSQL)

Maybe a constraint breaks your delete.

Even “simple CRUD” can go so very wrong.

dkf

@Arantor said in Why is programming complicated?:

two things with the same primary key

https://youtu.be/EKu7TYWNxqA

Arantor

@dkf bonus points this was a medical application and the relevant key was “patient ID”…

LaoC

@Arantor said in Why is programming complicated?:

@dkf bonus points this was a medical application and the relevant key was “patient ID”…

So it was you who did that

- Doctor, I came to pick up my wife’s results…
- Well… I had a little problem with the results. I accidentally scrambled them with another patient, we don’t know if she has AIDS or alzheimer's.
- So what do we do about it?
- Just leave her in the middle of the forest—if she comes back, don’t have sex with her.

Arantor

@LaoC nope, I only saw the aftermath of the royal mess - and my involvement was on the receiving end of 'hey, have you ever seen this happen before, wild huh?'

GOG

@jinpa said in Why is programming complicated?:

So why is it that programming is complicated? Why does developing a program take a thousand hours of programmer time rather than ten minutes?

Technically speaking, programming is simple.

It's the formalising of the problem that's hard.

You must constrain your solution space, account for all possible inputs, as well as the known (and, possibly, unknown) limitations of your environment (which includes the putative users). Having done all that, you must settle on a solution from your previously determined space, deciding between the various trade-offs involved. While you're doing that, it would probably be useful to give some thought to how the system will evolve. Should you account for much greater future volume of data? Are you expecting to have to modify or extend the system in the future?

Hacking out a quick hack is quick. It's creating a robust, future-proof solution that takes a long time.

Benjamin Hall

@GOG And it's the unknown unknowns that really lurk out there to bite you.

To get working software out the door, sometimes you have to do what my PhD advisor said and "stop writing and publish". "Good enough" and "perfect" aren't the same, and can't be. But figuring out where the acceptable deviation is is hard, fact specific, and not stable under time evolution.

Luhmann

@Arantor
Patient IDs are fun! Try mergers ... Doubles galore!
Or crossing the streams and have test patients turn up in production.

HardwareGeek

@Luhmann said in Why is programming complicated?:

have test patients turn up in production.

It's probably even worse to have real patients turn up in test.

"What are you doing? I didn't consent to any experimental treatment! You'll be hearing from my lawyer!"

Luhmann

@HardwareGeek
No issues there ... Test environment is generally a point in time copy of production, sometimes it's even fed the production patient movements to produce a more up to date environment

Gustav

@Luhmann what's Belgian for "HIPAA violation"?

Mason_Wheeler

@Applied-Mediocrity said in Why is programming complicated?:

(Philosophically speaking) programming is abstraction of real life. And real life is very, very complicated.

Reality has a surprising amount of detail

Dragoon

@Gustav said in Why is programming complicated?:

@Luhmann what's Belgian for "HIPAA violation"?

Nah, there are things you can do to sanitize it before copy.

Luhmann

@Luhmann
But it a great domain to show why software development is fucking hard.

Everything has to operate within the guidelines of the laws / guidelines of the social security bozos. These are often ambiguous, vague or contradicting. And change frequently to follow political whims. And sometimes even change post-factum.

The end user has no idea what they want. Medical professionals are often extremely retarded behind their knowledge field. But the end goal is something physical. Make things faster, safer, cheaper (mostly by reducing meat cost). Even if it just means keeping closer watch on the toilet paper supply.

Everything is connected. Patient and other data flows from one application to the next. Preferably at real time. Patients admitted at the ER desk should be identifiable at the RX or pharmacy level ( many medication is only physically unlocked when it is assigned to a patient to prevent human error and theft).

Everything hospital is the same, yet none are identical because of . Again something that is totally incomprehensible for users. At the other hospital/site/department it works like that, why not here?

Summed up: people are stupid.

Carnage

@Gern_Blaanston said in Why is programming complicated?:

@Gern_Blaanston said in Why is programming complicated?:

@jinpa said in Why is programming complicated?:

@Dragoon said in Why is programming complicated?:

Yes, take something as simple as a single dialog box and a button that will just display the text entered.

How many characters do you accept?
What char sets do you support?
Foreign characters?
How does it respond to null inputs?
How does it respond to someone entering the entirety of 'War and Peace'?
What font is being used?
How does everything look at different screen resolutions and DPI scales?
Is is written such that tomorrow you can easily add colors/bold/italics/etc...?

@jinpa said in Why is programming complicated?:

Much of this is a solved problem, though.

But you still have to solve it for YOUR program. You still have to consider all of those questions.

Constructing a 30 story building is also a "solved problem". We know exactly how to do it and have done it many times. But that doesn't change the fact that there are A LOT of steps involved and A LOT of details that you have to pay attention to (if you don't want your building to fall down).

It is the same for any non-trivial piece of software. That is why programming is complicated.

And construction is famously good at going over budget and going through deadlines like a landwhale through B&J pints. Even solved problems are hard the moment they leave the toy problem arena.

Luhmann

@Gustav

GDPR? Test environments are permitted within the same organization with limited goals. For certain certifications at the hospital level this is even a requirement.

Bulb

@Luhmann Indeed. At the very least the environment we usually call ‘staging’, where the user acceptance tests are done, must get a copy of the production data for the acceptance test, because the users wouldn't be able to verify the things they asked to be fixed are really fixed without being able to look at exactly the same data that were broken in the previous release.