"What's the largest amount of bad code you have ever seen work?"
-
Someone asked HN the above question. The most upvoted answer is…
You guessed it.
Oracle Database 12.2. It is close to 25 million lines of C code.
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
-
Start working on a new bug.
-
Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
-
Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
-
Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
-
Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
-
Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
-
Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
-
Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
-
Finally one fine day you would succeed with 0 tests failing.
-
Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
-
Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
-
After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
See also: discussion on /r/programming
-
-
I may have mentioned this before, but one of the core products at this company is 30 years old and involves printing. It uses a programming language designed in-house to support forms with conditional sections, and creating a UI to input data on DOS and OS/2.
This product has been ported to Windows, through MFC and on to WPF, and supports client-server stuff that normally only a web-app could do - except it did it before web apps could. It has 30 years of laws baked into it, so it cannot be rewritten. It makes obscene amounts of money, and you've never heard of it.
-
The worst piece of code I've seen that was written and maintained by professionals. I may have written about it here before, like, 10 years ago...
Aaanyway, it was written in C++, in the ninties by people that got employed as programmers because they could type on a keyboard.
It was a massive monster of 1.500 000 rows of code, on top of another 500 000 lines of code for a framework that kindof did CORBA stuff, but only for C++ and Java. That bit was written by a university student as university work for the company.
And about 25 000 lines of tcsh-scripts that made up the mess of a build system that broke as often as it actually built it.At the very core of the product was a pair of siamese twin classes, one mammoth of 27000 lines of code, where 90% of that (or more) was two methods, one weighing in at like 3000 lines, and the other the rest. The other class was a lot smaller, but that was simply because everyone had grown afraid of changing anything in the first class and just started on a new class. they both imported the other, so to build the horrible shit you first had to build with a fake of one of them, and then you could build the other, and then rebuild the first again.
These two classes did pretty much every single thing wrong, had memory leaks, spaghetti code, lasagna, copy pasta and variables with nonsensical names. And shoveled around data structures with field names like "reserved0, reserved1, reservedByGnu, reserved3, reserved5 and so on. And there was a lot of bit munging to shove multiple things into the same longlong, or short. Or bools and a short into a char[3] and equally dumb stuff.Me being a stubborn fucktard that does not have enough sense to fear things hacked away in those classes with abandon, and actually started cleaning the shit up. Along with actually daring to touch and eventually making sense of the build system.
This whole mess was so bad that manglement gave up on it and started a new project built from scratch in Java 1.4.2. And got it off to a great start by trying to force the new shit to be written in half a year, with a fair amount of pressure on the poor teams to get shit done. And when someone said they couldn't make the deadlines, manglement just told them to skip testing.
So, the devs simply started copying the C++-files to the java project, renamed them to .java, and hammered them until the compiler accepted them.Two years later that project kindof caught up with the old codebase that I and 12 others kept working on, while about 70-80 people munged up the new and shiny. So we swapped over, and nothing really worked. But after a year of panicked hacking away at the new not quite as horrible mess the code was kindof passable at least.
It's been a decade or so since I worked on it, and as all anecdotes, mine gets better with time. But it's pretty accurate still.
-
@Carnage said in "What's the largest amount of bad code you have ever seen work?":
And about 25 000 lines of tcsh-scripts that made up the mess of a build system that broke as often as it actually built it.
Could be worse. Could be 25 000 lines of makefiles.
-
Windows
-
@Gąska said in "What's the largest amount of bad code you have ever seen work?":
@Carnage said in "What's the largest amount of bad code you have ever seen work?":
And about 25 000 lines of tcsh-scripts that made up the mess of a build system that broke as often as it actually built it.
Could be worse. Could be 25 000 lines of makefiles.
Oh, there were makefiles as well. It's just that there were layers of tsch generating new ones from others, that were also input into more tcsh fuckery. It was really hard to figure out what the hell went on in that mess. The makefiles were pretty normal makefiles, and the shell scripting wizard liked to use no letters at all for anything unless he absolutely had to. deeply nested lists of stuff was his modus operandi.
-
@anonymous234 said in "What's the largest amount of bad code you have ever seen work?":
Windows
That was my first thought, too.
-
@Magus said in "What's the largest amount of bad code you have ever seen work?":
I may have mentioned this before, but one of the core products at this company is 30 years old and involves printing. It uses a programming language designed in-house to support forms with conditional sections, and creating a UI to input data on DOS and OS/2.
This product has been ported to Windows, through MFC and on to WPF, and supports client-server stuff that normally only a web-app could do - except it did it before web apps could. It has 30 years of laws baked into it, so it cannot be rewritten. It makes obscene amounts of money, and you've never heard of it.
So... you work(ed) with @Weng?
-
@M_Adams I don't think so... apparently there is more than one such product.
-
@DCoder I actually replied there.
https://news.ycombinator.com/item?id=18443407
I worked on this crap about 5 years ago now.
-
@DCoder said in "What's the largest amount of bad code you have ever seen work?":
Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
So basically Oracle DB is developed through some sort of developer-driven meat-space neural network?
-
@M_Adams said in "What's the largest amount of bad code you have ever seen work?":
@Magus said in "What's the largest amount of bad code you have ever seen work?":
I may have mentioned this before, but one of the core products at this company is 30 years old and involves printing. It uses a programming language designed in-house to support forms with conditional sections, and creating a UI to input data on DOS and OS/2.
This product has been ported to Windows, through MFC and on to WPF, and supports client-server stuff that normally only a web-app could do - except it did it before web apps could. It has 30 years of laws baked into it, so it cannot be rewritten. It makes obscene amounts of money, and you've never heard of it.
So... you work(ed) with @Weng?
Nah. There are only two WtfFramework apps that haven't had a clean rewrite in their document composition, one of those is scheduled, and the other one can't be described like that. Plus it could be rewritten, fairly trivially. Timeline is the only obstacle.
-
@Zecc said in "What's the largest amount of bad code you have ever seen work?":
So basically Oracle DB is
developedsynthesized through some sort of developer-driven meat-space neural network?
-
@HardwareGeek said in "What's the largest amount of bad code you have ever seen work?":
@anonymous234 said in "What's the largest amount of bad code you have ever seen work?":
Windows
That was my first thought, too.
Is GNU/Linux any better in terms of code quality? Yes, Linus controls the kernel, but there's more to GNU/Linux than that.
-
@Gąska said in "What's the largest amount of bad code you have ever seen work?":
@Carnage said in "What's the largest amount of bad code you have ever seen work?":
And about 25 000 lines of tcsh-scripts that made up the mess of a build system that broke as often as it actually built it.
Could be worse. Could be 25 000 lines of makefiles.
Nah, makefiles are OK. I think there is a project having 25k or more lines of them and it's still maintained, by volunteers, not some refugees on H1-B:
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
@Gąska said in "What's the largest amount of bad code you have ever seen work?":
@Carnage said in "What's the largest amount of bad code you have ever seen work?":
And about 25 000 lines of tcsh-scripts that made up the mess of a build system that broke as often as it actually built it.
Could be worse. Could be 25 000 lines of makefiles.
Nah, makefiles are OK.
Did you know there are six different kinds of assignment? I've seen all six used.
-
@tharpa said in "What's the largest amount of bad code you have ever seen work?":
@HardwareGeek said in "What's the largest amount of bad code you have ever seen work?":
@anonymous234 said in "What's the largest amount of bad code you have ever seen work?":
Windows
That was my first thought, too.
Is GNU/Linux any better in terms of code quality? Yes, Linus controls the kernel, but there's more to GNU/Linux than that.
It will be different though. While all distributions will have acquired cruft over the years there is not so much incentive to keep it in there like there is with Windows, and CADT means that any bad code doesn't expand as much as it is replaced.
-
@JBert said in "What's the largest amount of bad code you have ever seen work?":
any bad code doesn't expend as much as it is replaced.
With even worse code. But I think projects like GNOME which jwz mentions are outliers, and they are usually developed almost entirely by a single company, in this case RedHat. No sane volunteer would work on that, only to see his code discarded because 'muh tablets' or 'confuses users'. The ADHD-afflicted teenagers are probably in fact some corporate MBAs pushing interns, and the FOSS licence is only by accident here. Real open source projects (like PostgreSQL or some compilers) usually move slower, but have much stricter quality controls. But that's at the cost of not being new, shiny and webscale.
-
I've had good fortune to work mostly on green field projects, so I haven't seen a lot of really gnarly legacy stuff.
The worst code base I briefly touched was customer database of a swedish telecom company. There were no id columns at all anywhere in the db. Nothing to uniquely distinguish one record from another. That includes foreign keys or references of any kind. It was like somebody took flat pieces of paper and dumped them into tables.
By the time I left the project, guys were trying to join two of these tables together based on timestamps. Except, timestamps drifted by a few milliseconds here and there, so they had to add rounding to their join conditions. Fun.
-
This is the best one from the linked reddit thread.
I was tasked with fixing an isp billing engine written in perl. The company had just been bought by the company i worked for. Presumably for cheap. They hadn’t billed any customers in 8 months because the billing engine was broken. Upon investigating I found that it was trying to store data to an external usb drive. As you do in a data center! Wtf. How do you decide that’s a good idea?
I asked the new guys we’d got with the company if there used to be a usb drive plugged in to the server and where it was. They said that “oh yeah there was one. I think Tom took it home”
“who is Tom and where is he?”
“He left”.
“About 8months ago?”
“YES! How did you know???!”
Nice one Tom. You bankrupted the company when you left and took that usb drive with you. Took them 8 months to get bought out and have someone figure out their billing problems. It was utter mad perl nonsense. My team rewrote it. It was a success.
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
Nah, makefiles are OK.
They can be OK, or they can be horrible. If they're written out in full without shenanigans, that's not too bad. If they use lots of trickery to get shorter via macros to generate macros to write rules with lots of subtlety with the time everything gets substituted, then they're ghastly. And of course, people who like writing complicated shell scripts will end up producing complicated rules that will make life suck more.
-
@dkf said in "What's the largest amount of bad code you have ever seen work?":
. If they use lots of trickery to get shorter via macros to generate macros to write rules with lots of subtlety with the time everything gets substituted, then they're ghastly.
But the same can be said about C, C++ or really any language with preprocessing or reflection. That doesn't make them inherently bad.
By the way, this boost state machine stuff is a part of, maybe not the biggest piece of bad code I've ever seen, but definitely the slowest to compile. The program in question was a deamon running on an industrial computer connected to the network (with access to the internet) on one end and certain proprietary wireless devices on the other. It was a monolith, taking care of everything: talking to these devices over the radio, writing stuff to an sqlite database, and serving a HTTP API over the network. The whole thing took 20 minutes to compile on the fastest i7 CPU available at the time, producing a binary that fit in about 1 MB of disk space, and capable of running on a tiny ARM with 64MB of RAM. Of course the original author left the company, and after some time a bug was discovered - an indicator LED was indicating error despite no error or something like that. After many attempts, a friend of mine rewrote the state machine as a simple switch statement. Now compilation takes a couple of seconds.
-
Other examples of worst code:
This jQuery spaghetti I was fixing recently: over 3KLOC of jQuery stuff attached to a single giant form. Yes, a single form to be submitted with one request. The form itself is so big that we needed to change default limits in php.ini (sometimes it contains over 10k fields, mostly hidden) for it to pass through. While the form itself is served from the backend where it's built in mostly proper way (Symfony form module), the javascript code modifies it in a lot of ways, storing data in DOM nodes, and triggering events in event handlers handling events triggered by other handlers, and fetching additional data with over a dozen ajax requests.
To make things harder to understand, some click events are handled like this:$('body').click(function(e){ switch(e.target){ case some_div: ...but not all. Some are direct. Or both.
The other funny stuff was The Enterprisey ETL. A write-heavy database was once used for reporting. Things were slow, so the lead dev thought up a second 'data warehouse' database in shape of a typical OLAP cube. Since the reports didn't have to be real-time, an enterprise ETL Tool, which is named somewhat similarly to spanish 'pendejo', was launched every midnight, migrating new data to the warehouse. So far, so good.
Of course it only took a couple of weeks before a new important customer with a lot of cash appeared and demanded the reports to be late by at most one hour. That turned out to be a problem. You see, the Tool had a fancy interface, but under the hood it was rather stupid. Since it was advertised as 'backend-agnostic', it was exporting rows one by one. To process a single record, it asked the database if a row already existed, and then did either insert or update. Since the two databases were on different hosts in different countries, processing mere 50k rows, took about half an hour, vast majority of that time spent on network latency.
I insisted on throwing out the Tool and writing migration scripts by hand in Python or any other language, but both the half-technical CTO and the lead dev opposed. If I remember correctly, the thing was eventually hacked so that the Tool migrated everything (using a bulk insert) to temporary tables, created anew every time, and then ran a single UPSERT statement. Not very strightforward, but it worked.
And then, the important customer wished the on-line reports to be editable, and the editing would change data in the original database. Of course it couldn't be done in a simple manner, since updating the warehouse and migrating stuff backwards would result in data being overwritten by the Tool multiple times in unpredictable order, depending on which migration runs first. So some flags 'x_updated_do_not_touch' were added on the tables, and an increasingly complex conflict resolution logic with ladders of if statements started growing. I quit that job shortly after that, but I think the lead sits there to this day manually fixing corrupted records (he already was spending a lot of time on that by the time I was leaving).
I also strongly suspect that the initial slowness which triggered the cascade of wtf was caused by misconfigured database (the amount of data wasn't that big, and there were very few users), but it belonged to another division and we were not allowed tinker with it, so who knows.
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
And then, the important customer wished the on-line reports to be editable,
"I don't like what it's saying about what happened and I want to rewrite history."
Meanwhile ... I've worked on a project myself that was a Codethulu WTF-fest all up and down. I suppose I should submit it for the front page if I can recall enough details through the traumatic amnesia. The application codebase was – I couldn't get accurate measurements but the raw PHP source code was larger than the C that went into making the web server and dbms combined.
-
@Watson said in "What's the largest amount of bad code you have ever seen work?":
"I don't like what it's saying about what happened and I want to rewrite history."
It was pretty much exactly that. The report was actually about some documents being printed and sent to people, based on data uploaded by the customer. They frequently uploaded wrong data, and realised their mistake five minutes before the postman took the bags of documents. They called our office and demanded withdrawal of some of them. The editing was to make this 'process' easier.
At the same company, for the same customer, I even made a failed attempt to use machine learning to un-mix different kinds of documents, which were being mixed together in an email account for reasons unknown ("that's how we do it here. no, we can't send invoices, insurance claims and tax reports to different accounts, that would require too much from our staff"). It almost worked, but they refused to provide more data for learning, so we threw that away. Eventually, manual labor (interns) was and stil probably is being used to this end.
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
manual labor (interns) was and stil probably is being used to this end.
All you need is a buzzwordy-enough name for this process, and you could launch the Next Big Thing in IT.
-
@Zerosquare
I called it 'bio-robots'. The term originated during the cleanup of Chernobyl disaster: the radiation was so high that integrated circuits in real robots didn't work reliably (yep, 80's TTL logic with huge bipolar transistors), so instead they sent a huge army of 'bio-robots'. A bio-robot runs onto the roof, shovels debris for 40 seconds, so that he doesn't exceed the dose limit, runs away, then next one takes his place. This not only souns cool, but also gives us a picture of staff turnover.
-
@sebastian-galczynski: Not bad. "Biobots" sounds even better.
-
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
@sebastian-galczynski: Not bad. "Biobots" sounds even better.
Biots.
-
@sebastian-galczynski I can't wait for the inevitable TV series where they give the bio-robots/biobots/biots weapons and shove them into an arena.
-
405 megabytes of Hello World in BIT
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
bipolar transistors
Sometimes they were positive, the next moment they'd feel like shit.
-
@HardwareGeek said in "What's the largest amount of bad code you have ever seen work?":
@anonymous234 said in "What's the largest amount of bad code you have ever seen work?":
Windows
That was my first thought, too.
Windows 10 (~50 million LOC)
-
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
manual labor (interns) was and stil probably is being used to this end.
All you need is a buzzwordy-enough name for this process, and you could launch the Next Big Thing in IT.
isn't this the basic idea of the (Amazon) Mechanical Turk ?
its tagline is even cornier: Artificial Artificial Intelligence
-
@marcodave: Almost. But Amazon's biobots are remote, while interns are local, thus subject to less latency*. Plus, they can actually bring you coffee.
* Typical results. Actual interns may exhibit lower performance.
-
For remote biobots there's already TaskRabbit. Someone should post the original cleaning-up task there.
-
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
TaskRabbit
https://www.taskrabbit.com/about
“Its slower-growth approach has brought TaskRabbit to a place most unicorns have never seen: the verge of profitability.”
... Wut
-
@Zecc said in "What's the largest amount of bad code you have ever seen work?":
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
bipolar transistors
Sometimes they were positive, the next moment they'd feel like shit.
And then they switched the little darlingtons to feel'd effect technology and things got worse.
-
@marcodave This reminds me of a story I read a few years ago, apparently there is (was?) a webservice where you could (pay to) send a captcha (in picture form) and almost instantaneously get the decoded text. The work was done by actual humans (more likely the modern equivalent of slaves i.e. poor guys in China...) who typed back the answer. The guy who wrote the article actually tested the service and found it worked reasonably well, both in terms of accuracy and response time.
It's probably been posted somewhere here, but, you know, searching, NodeBB, kneeling warthog...
-
@LaoC said in "What's the largest amount of bad code you have ever seen work?":
@Zecc said in "What's the largest amount of bad code you have ever seen work?":
@sebastian-galczynski said in "What's the largest amount of bad code you have ever seen work?":
bipolar transistors
Sometimes they were positive, the next moment they'd feel like shit.
And then they switched the little darlingtons to feel'd effect technology and things got worse.
You, sir, are a terrible person, and I hate you. Well done.
-
@Magus said in "What's the largest amount of bad code you have ever seen work?":
I may have mentioned this before, but one of the core products at this company is 30 years old and involves printing. It uses a programming language designed in-house to support forms with conditional sections, and creating a UI to input data on DOS and OS/2.
This product has been ported to Windows, through MFC and on to WPF, and supports client-server stuff that normally only a web-app could do - except it did it before web apps could. It has 30 years of laws baked into it, so it cannot be rewritten. It makes obscene amounts of money, and you've never heard of it.
Everything above, except it was car dealership software, but started from a forms printing business.
-
Surprisingly given the topic title, nobody mentioned Discourse yet. (I know, you'd have to use a pretty relaxed definition of "work".)
-
@remi said in "What's the largest amount of bad code you have ever seen work?":
@marcodave This reminds me of a story I read a few years ago, apparently there is (was?) a webservice where you could (pay to) send a captcha (in picture form) and almost instantaneously get the decoded text. The work was done by actual humans (more likely the modern equivalent of slaves i.e. poor guys in China...) who typed back the answer. The guy who wrote the article actually tested the service and found it worked reasonably well, both in terms of accuracy and response time.
If you don't need realtime it's usually cheaper to buy a shitton of cheap pr0n and hide it behind those dynamically embedded captchas where they'll be eventually solved by an army of volunteer slaves to their hormones.
-
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
Surprisingly given the topic title, nobody mentioned Discourse yet. (I know, you'd have to use a pretty relaxed definition of "work".)
Discourse works perfectly, no idea what you're talking about.
-
@Tsaukpaetra said in "What's the largest amount of bad code you have ever seen work?":
...unicorns ...
...WutI think it's BA/VC-speak for a startup valued at more than $1 billion.
-
@pie_flavor said in "What's the largest amount of bad code you have ever seen work?":
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
Surprisingly given the topic title, nobody mentioned Discourse yet. (I know, you'd have to use a pretty relaxed definition of "work".)
Discourse works perfectly, no idea what
you'reI'm talking about.FTFY
-
I knew @pie_flavor would object.
-
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
I knew @pie_flavor would object.
Fuckers gotta fuck after all. ;)
-
@Zerosquare said in "What's the largest amount of bad code you have ever seen work?":
Surprisingly given the topic title, nobody mentioned Discourse yet.
Nobody mentioned Enlightenment yet.
(I know, you'd have to use a pretty relaxed definition of "work".)
Oh ok, I see now.
-
@topspin said in "What's the largest amount of bad code you have ever seen work?":
Nobody mentioned Enlightenment yet.
Topic title says it has to work, not COMPLAIN.
OpenWrt build system – Installation [Old OpenWrt Wiki]
