WTF Bites
-
@Bulb You need to fork if you're dealing with long-running connections. If you're serving a 10-minute download, and you don't fork, then nobody else can access your server while the download is in progress. If we assume the bandwidth constraint is with the client, not the server, it may even be viable for multiple users to fetch the download at the same time.
In a traditional forking server, the parent process is listening to the socket, and each time a connection is made it spawns a new child process to handle that connection. This is not something a shell script can do. Systemd might though; I know it can open your server socket on your behalf (which may avoid the need for your service to start as root) and I would not be entirely surprised if it was also able to do the listen loop for you.
What you are probably thinking of is forking to go into the background. Doing that properly (as I was taught) is a rather straightforward process, but availability of the required bits in shell varies. If you know you'll only ever be run as a service, you can skip it, but generally doing it twice doesn't hurt.
-
@Bulb You need to fork if you're dealing with long-running connections. If you're serving a 10-minute download, and you don't fork, then nobody else can access your server while the download is in progress.
Ah, you mean this … well, no, you usually clone rather than fork for this. Or not even that, just run an event loop if you are I/O bound anyway. Event loops tend to be faster than many threads in practice, even though the actual reasons are rather non-obvious (the obvious reasons usually state are somewhat obviously wrong).
-
@Bulb The concept of a forking daemon in this sense predates threads. Using threads instead has less overhead, but means you need to pay more attention to resource management.
Event loops require non-blocking IO which significantly increases the amount of code required when working in a systems programming language, that's definitely not suitable for a 'look how simple this is' example. Higher-level languages (particularly javascript with its async capabilities) can hide this complexity from you but it can get quite gnarly in C and I do not expect rust to do any better in this regard.
-
and I do not expect rust to do any better in this regard.
Rust has async. You need to pull in an additional library, because the standard library only has the interfaces, not the event loop¹ and async versions of the I/O functions, but then the code is just few keywords and annotations on top of the synchronous one.
¹ Big part of the reason is that there are multiple implementations of the event loop for different use-cases. One that is large and complex, optimized for high-performance network servers, one smaller to reduce bloat when you only need some concurrency, and some special ones for embedded context that may not even have full OS-level threads.
-
I'd still say 25 lines of raw C would be small, but not excessively so, even when sticking to the standard library.
Yeah, that's my estimate too. 25 LOC would be small, mainly because there's a bit more legwork involved in setting up a listening socket and accepting incoming connections. Then, dealing with buffers and strings is a bit less elegant. You could probably get rid of the latter again in C++ to some degree, but there's no modern sockets in the stdlib (yet).
Event loops require non-blocking IO which significantly increases the amount of code required when working in a systems programming language
It wouldn't be terrible still, though, not for the amount of features in the video at least.
(I'm somewhat tempted at trying a C++ coroutines version, see if that improves things over the simple FSM it normally ends up as.)
-
@PleegWat HTTP may not be that hard to parse, but it still sucks nonetheless for lacking a fixed
sizefield... Instead it has (or not)Content-Lengthwhich moves around depending on the size of the URL and any headers before it, and whose own size is variable.
On the traditional forking server: I'd say it's fine for stuff like FTP servers or good old static HTML, but the moment you're expected to allow clients to interact with each other (i.e. a chat program), your forking server finds itself having to deal with inter-process communication.
-
On the traditional forking server
Also if you want that, there are several ready-made implementations that will do the listening, accepting and forking and run your binary separately for each client, so utilizing that you can make even simpler http server.
-
I'm somewhat tempted at trying a C++ coroutines version, see if that improves things over the simple FSM it normally ends up as.
Yes but no. It will improve the graph (having coroutines makes complex protocols like HTTP much easier for example), but the C++ coroutine support is horrifying. Leave it to the Flying Spaghetti Monster for now.
-
@PleegWat HTTP may not be that hard to parse, but it still sucks nonetheless for lacking a fixed
sizefield... Instead it has (or not)Content-Lengthwhich moves around depending on the size of the URL and any headers before it, and whose own size is variable.As I recall many (older) applications would do line buffering and assume a 1kb line length which is not to be found anywhere in the RFC.
On the traditional forking server: I'd say it's fine for stuff like FTP servers or good old static HTML, but the moment you're expected to allow clients to interact with each other (i.e. a chat program), your forking server finds itself having to deal with inter-process communication.
Yeah, if a forking server makes you need IPC then the solution is definitely worse than the cure. I'd favour an event loop for most real-world applications.
A simple chat server might be a fun toy problem. Simply accept connections, and send each message you receive to all peers (possibly excluding the sender).
On the traditional forking server
Also if you want that, there are several ready-made implementations that will do the listening, accepting and forking and run your binary separately for each client, so utilizing that you can make even simpler http server.
That's how cgi scripts work.
-
On the traditional forking server
Also if you want that, there are several ready-made implementations that will do the listening, accepting and forking and run your binary separately for each client, so utilizing that you can make even simpler http server.
That's how cgi scripts work.
CGI is already behind a HTTP server, while I'm talking about handling TCP—and perhaps forking a process that handles one HTTP connection. That is, (
x)inetd,tcpsvd, orsocatwith antcp-listen:…,forkandexec:… address pair.Of course both cases do work by forwarding the data to a stdin and back from stdout of a forked process, but CGI isn't a method for implementing a generic server, because it is already behind a HTTP server—and gets the headers already parsed and stashed in the environment.
-
allow clients to interact with each other (
i.e.e.g. a chat program)"i.e." means "saying the same thing again but with different words". There are other applications than chat programs that also allow clients to interact with each other, so that can't be what you meant.
"e.g." means "here's an example of what I just said", which is what you probably meant.
Sorry, just had to get that off my chest, feeling better now!
Carry on...
-
@cvi Please don't make me think again of the simile @Polygeekery used.
You're welcome.
-
Kubernetes API uses
POSTrequests for creating resources, andPUTrequests for updating them. But neither can be used as create-or-update.The command-line client has
apply, which does a create-or-update, but I'm trying to use the Python API instead (since I need to work with complex-enough data to nullify advantage of the shell—well, at this point mostly to learn it, because the project is being cancelled anyway) and that only hascreate_* (sendingPOST) andreplace_* (sendingPUT) and neither works in both cases. So I can't simplify my code to try read, else construct default object, then fill in what I need to fill, and apply it. I have to prepare everything, then read-and-replace or create.
-
@ixvedeusi You're right.
I also had another example where a forking server would likely cause more problems than it solves: A database server.
-
I love how I bring up all of the shit that was obfuscated behind JFM in the example and then I am told that I am wrong and then several people proceed to argue about all of the things that were obfuscated in the example given.
Fair, I said libraries when most of what was used is in stdlib, but I still feel like my point stands. You can absolutely write simplistic examples of things in small line counts, and it can be very readable and understandable, but most of what is actually being done still remains as JFM.
Once again, I am not saying that this is inherently a bad thing, but it does lead to situations where developers end up using too much JFM and then when things go away they may not even know where to start to attempt to diagnose the problem.
On the other hand, I fully realize that we work in an industry where it is literally impossible for any person to know everything so such obfuscation is a necessity for virtually any of us to do our jobs and is certainly a necessity for us to have any level of productivity.
Wasn't expecting to be "defending" the person in the video, but in this case, no, I wouldn't say he's pulling in that much library code. (Do you call the kernel that deals with the underlying sockets, IO and drivers a library? I guess one might
Not a library, pedantically speaking, but it is certainly a layer of obfuscation. We work in an industry with many, many, many levels of obfuscation and abstraction between the work that we do and the work that actually gets done. We write code that gets run through an interpreter or compiler, that then runs on an operating system and that operating system is now frequently a container, that is running within an operating system, that will probably be running on a hypervisor (that has its own hardware abstraction layer) that is running on yet another operating system, that finally gets to the bare metal. And along all of those steps there are many layers of obfuscation and abstraction that I glossed over.
I feel like I came across as too harsh on the video and the person who was in it. I really did not mean to be. He's not wrong, but those 25(ish) lines of code he wrote and produced a minimalistic web server ignores the thousands of lines of code that are referenced and utilized in those few lines and characters. Much of what we do, even for the most learned among us, may still be JFM if we are being honest about it.
It reminds me of many years ago I ran across a webpage about a class that built a gigantic computer out of little more than wire and relays. By studying their posted schematics and going over their videos about this super simple but gigantically complex and huge and power hungry and noisy clattering computer I learned tons in a short period of time. It became tangible and understandable in a way that nanometer scale transistors never can be.
Or marble computers. Even simpler yet and fundamentally they work in what amounts to the same way as the computers we use every day, even if the mechanisms are vastly different.
I've always meant to build one of those with my kids. I should get busy with that.
-
@Polygeekery said in WTF Bites:
It reminds me of many years ago I ran across a webpage about a class that built a gigantic computer out of little more than wire and relays. By studying their posted schematics and going over their videos about this super simple but gigantically complex and huge and power hungry and noisy clattering computer I learned tons in a short period of time. It became tangible and understandable in a way that nanometer scale transistors never can be.
Sounds similar to NandGame (or rather, NandGame is an interactive version of that).
-
@Polygeekery said in WTF Bites:
those 25(ish) lines of code he wrote and produced a minimalistic web server ignores the thousands of lines of code that are referenced and utilized in those few lines and characters
Of course you can go deeper, and I wouldn't be surprised if computerphile has done TCP basics at some point. But you cannot cram all of that in a 20-minute video aimed at a high school level background.
Doing this with similarly huge assumptions out of raw network frames is indeed quite a bit more code. But the only major difference I see between the rust APIs he's using and the linux kernel APIs I'm familiar with from C is that rust combines the
socket(),bind(), andlisten()steps into a single call. And I'm honestly not sure why posix bothered splitting those up.If I'm bored, I might write it up some time. Doing that off
socket(AF_PACKET, SOCK_RAW, ETH_P_ALL)(to grab all raw frames) rather thansocket(AF_INET, SOCK_STREAM, 0)(to use the TCP stream interface) could indeed easily take thousands of lines of code.In the meantime, RFC 793 is quite readable.
-
@Polygeekery said in WTF Bites:
I've always meant to build one of those with my kids. I should get busy with that.
As long as you're careful. You don't want to lose your marbles.
-
Kubernetes API uses
POSTrequests for creating resources, andPUTrequests for updating them. But neither can be used as create-or-update.The command-line client has
apply, which does a create-or-update, but I'm trying to use the Python API instead (since I need to work with complex-enough data to nullify advantage of the shell—well, at this point mostly to learn it, because the project is being cancelled anyway) and that only hascreate_* (sendingPOST) andreplace_* (sendingPUT) and neither works in both cases. So I can't simplify my code to try read, else construct default object, then fill in what I need to fill, and apply it. I have to prepare everything, then read-and-replace or create.How come you're able to predict the URL of the resource that doesn't exist? Either you've been told the name, in which case it should exist, or you haven't, when you should create it. Clients typically shouldn't be minting resource names in ReST. (
POSTcan do create-or-update in general; it's a generic "do something non-idempotent" method and isn't constrained by ReST to be a create at all. Kubernetes might be fussier.)The exception to this is when working with something that is pretending to be a filesystem (e.g., WebDAV) when clients can predict resource names ahead of time.
-
I also had another example where a forking server would likely cause more problems than it solves: A database server.
That's exactly how (at least) both Postgres and Oracle work...
-
If I'm bored, I might write it up some time. Doing that off
socket(AF_PACKET, SOCK_RAW, ETH_P_ALL)(to grab all raw frames) rather thansocket(AF_INET, SOCK_STREAM, 0)(to use the TCP stream interface) could indeed easily take thousands of lines of code.Raw ethernet frames are horrible. Proper masochists-only level stuff, as you need to handle all the non-TCP/IP bits and pieces of networking as well. Low level networking really is best left to the kernel and its drivers if you possibly can...
-
@dkf My experience is with packet inspection, meaning we don't have anything to do with generating the stuff.
-
Either you've been told the name, in which case it should exist
I am defining the name. So obviously I know it.
-
@Polygeekery said in WTF Bites:
I love how I bring up all of the shit that was obfuscated behind JFM in the example and then I am told that I am wrong and then several people proceed to argue about all of the things that were obfuscated in the example given.
I'm going to tell you that it wasn't obfuscated. More like abstracted away. Though to be fair I haven't looked at the library code so maybe it really is obfuscated.
-
@dkf Damn, that's pretty counter-intuitive to me...
Well I suppose that could work with extensive amounts of mutexes and other interprocess synchronization mechanics, but that also means no possibility of caching when two clients access the same table (unless the IPC includes shared memory). Wait that's not nearly snarky enough and way too reasonable an answer for WTDWTFThis explains so much.
-
@Medinoc I happen to know that is why it is important to close connections to Oracle DBs correctly — it's caused some of our IT people much trouble when a misconfiguration of Apache left DB connections open for days — and I read an article about how Postgres works a year or two ago. (Yes, shared memory is involved. No, I don't remember exactly what they were doing.)
-
Clients typically shouldn't be minting resource names in ReST.
Au contraire, they should. That's the only way to make the interface idempotent, and being able to make it idempotent is kinda the main reason for using ReST-style interface.
(
POSTcan do create-or-update in general; it's a generic "do something non-idempotent" method and isn't constrained by ReST to be a create at all. Kubernetes might be fussier.)The exception to this is when working with something that is pretending to be a filesystem (e.g., WebDAV) when clients can predict resource names ahead of time.
Well, Kubernetes does use, meaningful-to-humans, names. All or most resources do get UUIDs, but referencing is by name, both in the resource URLs and between resources.
They really managed to snatch defeat from the jaws of victory here. They could have made PUT the idempotent create-or-update method it is supposed to be in ReST, but they designed the API as if the client didn't know the name when creating the resource even though it does and specifies it in the body.
Though it's true that the merging logic (added later) might be making it non-idempotent anyway.
-
it is important to close connections to Oracle DBs correctly
Why? Do you get charged by the
hournanosecond?
-
@Zerosquare said in WTF Bites:
it is important to close connections to Oracle DBs correctly
Why? Do you get charged by the
hournanosecond?The worker processes wouldn't shut down and release their resources. You would end up with a database server unable to serve any more clients after a bit.
-
a database server unable to serve
It's Oracle, so that's probably for the best.
-
I happen to know that is why it is important to close connections to Oracle DBs
I solve this problem by avoiding opening connections to Oracle in the first place.
-
Status: Excel I fucking HATE you...
-
Go home Firefox, you're drunk.
-
The Great Canadian Pottery Throw Down | Shows | CBC Gem
The Great Canadian Pottery Throw Down is a competition series that revels in the remarkable creativity of Canada’s best amateur potters and their beautiful creations.
Canadians .... throwing pottery?
Doesn't it break?
-
-
Self-Pay Gas Station Pumps Break Across NZ As Software Can't Handle Leap Day - Slashdot
An anonymous reader quotes a report from Ars Technica: Today is Leap Day, meaning that for the first time in four years, it's February 29. That's normally a quirky, astronomical factoid (or a very special birthday for some). But that unique calendar date broke gas station payment systems across New...
-
Article @izzion posted said in WTF Bites:
meaning that for the first time in four years, it's February 29.
Written as if that was an astonishing or unexpected occurence....
-
@Tsaukpaetra said in WTF Bites:
Article @izzion posted said in WTF Bites:
meaning that for the first time in four years, it's February 29.
Written as if that was an astonishing or unexpected occurence....
Leap Year / Feb. 29 has only been a thing for hundreds of years. So definitely unexpected, and can't possibly be able to handle it.
-
@Gern_Blaanston said in WTF Bites:
The Great Canadian Pottery Throw Down | Shows | CBC Gem
The Great Canadian Pottery Throw Down is a competition series that revels in the remarkable creativity of Canada’s best amateur potters and their beautiful creations.
Canadians .... throwing pottery?
Doesn't it break?
No. Because in context, “throw” is the beginning of the making process—you throw the wet clay onto the wheel to spin it. The term also seems to extend to mean a custom hand-spun pottery piece.
-
@Gern_Blaanston said in WTF Bites:
The Great Canadian Pottery Throw Down | Shows | CBC Gem
The Great Canadian Pottery Throw Down is a competition series that revels in the remarkable creativity of Canada’s best amateur potters and their beautiful creations.
Canadians .... throwing pottery?
Doesn't it break?
No. Because in context, “throw” is the beginning of the making process—you throw the wet clay onto the wheel to spin it.
-
I have an e-mail account on Microsoft's outlook.live.com because .... raisins.
E-Mail from Microsoft, promoting Microsoft Co-Pilot, is marked as spam. On Microsoft's own e-mail system.
-
@Gern_Blaanston said in WTF Bites:
I have an e-mail account on Microsoft's outlook.live.com because .... raisins.
E-Mail from Microsoft, promoting Microsoft Co-Pilot, is marked as spam. On Microsoft's own e-mail system.
The Outlook.com team are smarter than the Copilot team.
-
@Tsaukpaetra said in WTF Bites:
Article @izzion posted said in WTF Bites:
meaning that for the first time in four years, it's February 29.
Written as if that was an astonishing or unexpected occurence....
There's an old Polish joke we like to tell every December. "Municipal officials were caught off guard by the snowfall."
-
@Tsaukpaetra said in WTF Bites:
Article @izzion posted said in WTF Bites:
meaning that for the first time in four years, it's February 29.
Written as if that was an astonishing or unexpected occurence....
There's an old Polish joke we like to tell every December. "Municipal officials were caught off guard by the snowfall."
Sänk ju for traveling wif Deutsche Bahn.
-
List of 2024 Leap Day Bugs
Well, it's 2024 and leap day has come once again. As I've done in prior leap years, I've captured as many bug reports and outages as I can, along with links to the source where possible. For those have been following along, you'll notice these have been organized a bit
-
There's an old Polish joke we like to tell every December. "Municipal officials were caught off guard by the snowfall."
This seems to be a universal thing.
Weather services: We expect snow next week
Weather services: We expect snow on Xday.
Weather services: Remember, snow tomorrow!
Public transport: HOLY SHIT! IT LIKE SNOWED! This unprecedented weather phenomenon never before observed in a Nordic country in winter took us by complete surprise.
Public transport: *collapses*
-
"Municipal officials were caught off guard by the snowfall."
Sänk ju for traveling wif Deutsche Bahn.
As if they need snowfall to have major delays.
-
This seems to be a universal thing.
Indeed, same thing here. Also applies to leaves on train tracks in Autumn.
-
@Zerosquare said in WTF Bites:
This seems to be a universal thing.
Indeed, same thing here. Also applies to leaves on train tracks in Autumn.
Such things, being yearly occurrences, are entirely impossible to predict.
-
Public transport: HOLY SHIT! IT LIKE SNOWED! This unprecedented weather phenomenon never before observed in a Nordic country in winter took us by complete surprise.
To be fair, the problem isn't that they didn't expect it, the problem is that there isn't much that could actually be done in advance. When snow is forecast, they have the trucks and ploughs loaded with salt and the drivers on stand-by, but they can't just salt the roads a day in advance, because the salt would just get blown off the still dry road. So they can only start salting when it starts to snow, maybe an hour before, but not more, and then it takes some time to salt everything, because they only have so many maintenance vehicles available.
When it starts to snow while there's still some salty slush on the road, it isn't such a big problem, both because now they can add salt in advance and it will stay in that salty slush, and because the first snow that touches that salty slush will melt, so they don't have to get additional salt there as quickly.
Common Gateway Interface - Wikipedia
