Before AI....Before Crypto...There Was...BIG DATA
-
This article is over a year old now, but it's interesting to see how an earlier bubble / craze has died out and never lived up to its promises. It's written by one of the guys who arguably started it all, since he was a "founding engineer on Google BigQuery" and the guy who got stuck with going around promoting it.
-
@boomzilla said in Before AI....Before Crypto...There Was...BIG DATA:
This article is over a year old now, but it's interesting to see how an earlier bubble / craze has died out and never lived up to its promises. It's written by one of the guys who arguably started it all, since he was a "founding engineer on Google BigQuery" and the guy who got stuck with going around promoting it.
I questioned one of the evangelists about it when I had to work towards a data lake, and they couldn't understand why I thought that having at least some idea of structure world be good, but no. It was all sock gnomes.
Speaking of, that's the best measure of faddery in it. If there is a magic step in the plan that will just have usefulness emerge, then it's all high caliber bullshit.
This has worked every time a new dumb dad showed up during my time as a code jockey.
-
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
It was all sock gnomes.
Can't miss an opportunity for that...
-
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
a new dumb dad showed up
I was pretty dumb (or at least naive) when I was a new dad.
-
@HardwareGeek I have been dumb before, during, and after becoming a new dad. I have never stopped being dumb. Never!
-
@PotatoEngineer I'm quite sure nobody here has ever said otherwise.
-
But ... isn't AI all about Big Data? They gobble up everything they can find to train their LLMs, including the worst bottom of the barrel shitposts, with the hilarious results we've seen the last few days. So, how is that fad not just under a new umbrella?
Well, at least I haven't heard about
I-don't-know-SQLNoSQL in awhile.
-
@topspin said in Before AI....Before Crypto...There Was...BIG DATA:
Well, at least I haven't heard about I-don't-know-SQL NoSQL in awhile.
NoSQL has been
successfully replaced by NoCode.
-
@topspin we solved the webscale problem with microservices.
-
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
But in classic IT tradition, created new (micro)problems.
-
@loopback0 said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
But in classic IT tradition, created new (m
iacro)problems.
-
@loopback0 Which can obviously be solved by paying consultants substantive amounts.
-
@boomzilla said in Before AI....Before Crypto...There Was...BIG DATA:
has died out and never lived up to its promises
@boomzilla said in Before AI....Before Crypto...There Was...BIG DATA:
Google
there's your problem.jif
-
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
solved But seriously, I've suspected for a long time that much webscale processing/problems was really only caused by putting too much data in one place to start with. Microservices help because they don't require making a huge data dumpster as a first step of achieving anything at all, instead focusing on services that have only part of the picture.
-
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
It was all sock gnomes.
Can't miss an opportunity for that...
- Collect data
Build a big datawarehouseUse it to train AI- ...
- Profit!
-
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
It was all sock gnomes.
Can't miss an opportunity for that...
- Collect data
Build a big datawarehouseUse it to train AI...Fleece venture capitalists- Profit!
-
@dkf but then to do anything useful you have to chain together the output of a dozen services whose reflections of reality are possibly out of sync and whose fundamental models are incompatible.
-
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
solved But seriously, I've suspected for a long time that much webscale processing/problems was really only caused by putting too much data in one place to start with. Microservices help because they don't require making a huge data dumpster as a first step of achieving anything at all, instead focusing on services that have only part of the picture.
I think microservices are probably the right size if you have as many teams than services. If you have a small team maintaining 10, 100, 1000 services you're doing nanoservices, or picoservices or whatever is even smaller than that
-
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
I thought it was with serverless?
-
@CodeJunkie we got a case were we can say "why not both?" seriously
-
@topspin said in Before AI....Before Crypto...There Was...BIG DATA:
But ... isn't AI all about Big Data? They gobble up everything they can find to train their LLMs, including the worst bottom of the barrel shitposts, with the hilarious results we've seen the last few days. So, how is that fad not just under a new umbrella?
Well, at least I haven't heard about
I-don't-know-SQLNoSQL in awhile.AI is the answer to the "now what do a I do with all this data?" question that followed
-
and I can say half-seriously that the next big question will be "what do I do with all this AI?"
will this keep going infinitely or we'll get to the "profit!" step some day?
-
@CodeJunkie said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
I thought it was with serverless?
Serverless came after. And it only solves it until it doesn’t.
-
@CodeJunkie said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
I thought it was with serverless?
That's the solution to microservices. Or as an overpriced contractor... job security.
-
@DogsB said in Before AI....Before Crypto...There Was...BIG DATA:
@CodeJunkie said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
I thought it was with serverless?
That's the solution to microservices. Or as an overpriced contractor... job security.
My experience is that it’s less the solution and more the accelerant for the inevitable dumpster fire.
-
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
My experience is that it’s less the solution and more the accelerant for the inevitable dumpster fire.
I did say job security did I not?
-
@DogsB security for someone at least. Usually the devops types who enjoy that kind of thing.
-
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
solved But seriously, I've suspected for a long time that much webscale processing/problems was really only caused by putting too much data in one place to start with. Microservices help because they don't require making a huge data dumpster as a first step of achieving anything at all, instead focusing on services that have only part of the picture.
I think microservices are probably the right size if you have as many teams than services. If you have a small team maintaining 10, 100, 1000 services you're doing nanoservices, or picoservices or whatever is even smaller than that
FaaS.
-
@Benjamin-Hall said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf but then to do anything useful you have to chain together the output of a dozen services whose reflections of reality are possibly out of sync and whose fundamental models are incompatible.
As opposed to trying to join some tables in a NoSQL database where the data is still out of sync and where the fundamental models are incompatible? To be clear, the incompatibility of the models is non-negotiable (if maddening).
-
@topspin said in Before AI....Before Crypto...There Was...BIG DATA:
But ... isn't AI all about Big Data?
I think so, sort of, yes. IIRC there was Big Data which morphed into Machine Learning which morphed into AI.
I think, going backwards, that before those we had the "internet revolution," the "database age" and even before the "digital everything." Going back might be the origins of IBM and similar with analog computing systems (perforated cards for e.g. census).
They're all avatars of the same "making sense of a lot of data" problem.
Note that the problem is a real one, and that can be seen in the fact that it has always been worked on!
Like TFA says, traditional DB engines are highly optimised for large datasets (and, in fact, were created by people having too much data to fit into "naive" data structures). Machine learning is, at its core, stuff like doing a linear regression (or a PCA for a slightly fancier version) or other statistical analyses, which have been around even before computers were a thing. And AI, when it's not just Machine Learning under a new name, is also something that has been around for decades (expert systems or neural networks).
But while the problem is real, the solutions that currently are in their hype phase are usually not worth it for most people, though after the hype has settled down they end up being bread-and-butter.
Just picking one example (but I think there are examples for all things I mentioned), in the 80's small mom and pop businesses were sold computer systems that only big companies really needed at the time, but nowadays you can't really start a business without one.
So, how is that fad not just under a new umbrella?
I think most reasonable people can see the continuity, and the real problems, through the fads. Sadly most people are not reasonable.
-
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Benjamin-Hall said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf but then to do anything useful you have to chain together the output of a dozen services whose reflections of reality are possibly out of sync and whose fundamental models are incompatible.
As opposed to trying to join some tables in a NoSQL database where the data is still out of sync and where the fundamental models are incompatible? To be clear, the incompatibility of the models is non-negotiable (if maddening).
I suspect that in most cases it's only nonnegotiable now because no one with any sense paid attention previously. Not that there aren't some legitimate cases like you describe, but this industry is full of the wrong solution being applied because it's the latest thing someone learned about.
-
@remi said in Before AI....Before Crypto...There Was...BIG DATA:
I think most reasonable people can see the continuity, and the real problems, through the fads. Sadly most people are not reasonable.
The difference is that while the new fad technology probably has its interesting points and some genuine usefulness (except blockchain), the fad tries to pass it off as a new silver bullet for everything you need.
-
@remi said in Before AI....Before Crypto...There Was...BIG DATA:
analog computing systems (perforated cards for e.g. census).
Tsk. Punch cards are not analogue computing systems. They are non-electronic computing at best (mostly just "tabulation" rather than computation, anyway).
Analogue computing is ... Well, I'll cite the first paragraph of the relevant page on the Unreliable Source:
An analog computer or analogue computer is a type of computer that uses the continuous variation aspect of physical phenomena such as electrical, mechanical, or hydraulic quantities (analog signals) to model the problem being solved.
Among the simplest analogue computers is the humble slide rule...
-
@Steve_The_Cynic said in Before AI....Before Crypto...There Was...BIG DATA:
Tsk. Punch cards are not analogue computing systems.
You're right, I was conflating the two but they're two different things. There was (is) both analog computing systems, and non-electr[on]ic computing systems (such as perforated cards).
They were (are) both used since... at least 200 years for perforated cards. And they both are part of trying to make sense of too much data.
Interestingly, the oldest example I can think of perforated cards also generated all the hype, fear and rejection that AI currently does. And while it did end up replacing a lot of jobs, it actually only expanded the size of the industry and the number of people working in it (including new jobs related to the cards, but also types of jobs that did exist before).
-
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
solved But seriously, I've suspected for a long time that much webscale processing/problems was really only caused by putting too much data in one place to start with. Microservices help because they don't require making a huge data dumpster as a first step of achieving anything at all, instead focusing on services that have only part of the picture.
I think microservices are probably the right size if you have as many teams than services. If you have a small team maintaining 10, 100, 1000 services you're doing nanoservices, or picoservices or whatever is even smaller than that
FaaS.
fuck as a service?
-
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
FaaS.
fuck as a service?
happy
noises
-
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
@sockpuppet7 said in Before AI....Before Crypto...There Was...BIG DATA:
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
solved But seriously, I've suspected for a long time that much webscale processing/problems was really only caused by putting too much data in one place to start with. Microservices help because they don't require making a huge data dumpster as a first step of achieving anything at all, instead focusing on services that have only part of the picture.
I think microservices are probably the right size if you have as many teams than services. If you have a small team maintaining 10, 100, 1000 services you're doing nanoservices, or picoservices or whatever is even smaller than that
FaaS.
fuck as a service?
Nah, that's a really old profession.
FaaS in IT is usually Functions as a Service. So a swarm of functions doing stuff™.
-
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
So a swarm of functions
assumed to be doing stuff™.
-
@loopback0 said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
But in classic IT tradition, created numerous new (micro)problems.
-
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@loopback0 said in Before AI....Before Crypto...There Was...BIG DATA:
@Arantor said in Before AI....Before Crypto...There Was...BIG DATA:
@topspin we solved the webscale problem with microservices.
But in classic IT tradition, created new (m
iacro)problems.Nah, they're still microproblems, but now there are webscale amounts of them
-
@hungrier including a non-enumerable amount of quantum entanglements.
-
@BernieTheBernie said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
So a swarm of functions
assumed to be doing stuff™.Oh, they're doing stuff. Whether the stuff they're doing is useful or correct is where you leave the realm of reality and enter the Assumption Zone.
-
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
FaaS in IT is usually Functions as a Service.
I think I was hoping more for Foolishness as a Service. We've got that well covered!
-
@dkf said in Before AI....Before Crypto...There Was...BIG DATA:
@Carnage said in Before AI....Before Crypto...There Was...BIG DATA:
FaaS in IT is usually Functions as a Service.
I think I was hoping more for Foolishness as a Service. We've got that well covered!
Or Fuckery as a Service, but there's none of that happening on these forums
Analog computer - Wikipedia
