The absolute state of web storage protocols
-
@cvi said in The absolute state of web storage protocols:
@PleegWat said in The absolute state of web storage protocols:
@Gustav Guess it was going to be that or "Of course you hardcode all your translations in the binary."
One binary per language.
I know of two apps that do exactly that (with 6+ languages).
-
@cvi said in The absolute state of web storage protocols:
@PleegWat said in The absolute state of web storage protocols:
@Gustav Guess it was going to be that or "Of course you hardcode all your translations in the binary."
One binary per language.
*cries in Windows*
-
@PleegWat said in The absolute state of web storage protocols:
@Bulb How the hell does that work with translation?
Do you really localize your log messages? That would seem to be
to me.Or are you asking this as a more general question?
-
@ixvedeusi some systems do, yes. Though oftentimes what they’re really logging is the localised error message being sent back to the user.
I’d love it if folks localised their
error: process completed successfullymessages, just for shits and giggles so everyone can see how ass-backwards this industry is.
-
@ixvedeusi said in The absolute state of web storage protocols:
@PleegWat said in The absolute state of web storage protocols:
@Bulb How the hell does that work with translation?Do you really localize your log messages? That would seem to be
to me.Or are you asking this as a more general question?
Everything has an end user interface.
-
@PleegWat Indeed; but not everything is an end user interface. If the log output is part of it, and intended to be understood by said end user, I maintain that you're
Maybe it makes sense for some applications, just seems to me like at that point a typical logging library would be the wrong tool for the purpose and / or you'll miss out on having logs that are actually useful for investigating issues with the program.
EDIT To be clear: no judgement from me, I'm just genuinely confused about how you'd reconcile the differing needs.
-
@ixvedeusi depends on what you're logging. Are they entirely dev logs that won't help the end user with anything, or are they logs that the user can actually extract useful info from and fix the issue on their own?
A good example of the latter are access logs. If you're already translating the whole admin panel to, say, French, there's no reason why the record of who logged in when and touched what things should remain in English.
-
@ixvedeusi said in The absolute state of web storage protocols:
@PleegWat Indeed; but not everything is an end user interface. If the log output is part of it, and intended to be understood by said end user, I maintain that you're
Maybe it makes sense for some applications, just seems to me like at that point a typical logging library would be the wrong tool for the purpose and / or you'll miss out on having logs that are actually useful for investigating issues with the program.
EDIT To be clear: no judgement from me, I'm just genuinely confused about how you'd reconcile the differing needs.
I figured the mentioned templates applied to all templates, not just the ones in dev logs. While it is generally discouraged to substitute strings inside translatable strings, substituting numbers is sometimes unavoidable.
-
@PleegWat said in The absolute state of web storage protocols:
I figured the mentioned templates applied to all templates, not just the ones in dev logs.
And you did understand correctly. It does apply to all templates that are in the standard library.
It should be noted that there are no translation tools in that standard library.
-
@Bulb said in The absolute state of web storage protocols:
It should be noted that there are no translation tools in that standard library.
Obviously not. Rust standard library developers are Americans!
-
@Bulb said in The absolute state of web storage protocols:
@PleegWat said in The absolute state of web storage protocols:
I figured the mentioned templates applied to all templates, not just the ones in dev logs.
And you did understand correctly. It does apply to all templates that are in the standard library.
It should be noted that there are no translation tools in that standard library.
Of course not. A universal translation engine (with sentence composition) is probably beyond them. And anything less would be unacceptable.
-
@Bulb said in The absolute state of web storage protocols:
are no translation tools in that standard library.
Translations (and human languages in general) are not unambiguous and provably correct, so they must not be implemented.
Filed under: Logjam is not a human language
-
@dkf said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
It should be noted that there are no translation tools in that standard library.
Obviously not. Rust standard library developers are Americans!
Not all of them.
@PleegWat said in The absolute state of web storage protocols:
Of course not. A universal translation engine (with sentence composition) is probably beyond them. And anything less would be unacceptable.
It's more about whatever gets in the standard library has to continue being supported for foreseeable and unforeseeable future. And it's not like other standard libraries have anything either. Well, Unix does have catgets, but I never heard of anybody actually using that.
-
@Bulb said in The absolute state of web storage protocols:
it's not like other standard libraries have anything either
It's not like Rust would look beyond C and C++ for inspiration. There are internationalisation features in the standard libraries of many languages, but the Rust team tends to ignore far too many of them. That's the #1 thing wrong with Rust, not learning from other languages that they don't feel like trying to compare with, and that's leading them to make all sorts of assertions about how to do things that frankly look foolish to people with exposure to a wider range of languages/systems.
But this isn't the room for an argument; that's down the hall. This is the room for insults.
-
..
-
@Gustav said in The absolute state of web storage protocols:
WHICH languages is Rust not learning from
Well, as a starter try Java and Python. They're pretty well known. Yes, they've got a bunch of crud in their standard libraries, but there's still definitely many things that could be learned. (C# would be another, but I don't know that so well either.)
Javascript is weaker in this regard, as a side effect of starting in a browser environment where that sort of thing is handled by machinery that is out of its scope (the web application delivery side of things) so message catalogs are a third party thing for it.
-
@Gustav It's not learning from PHP, I can tell you that.
-
@Arantor They've also not learned a thing from Lisp or other homoiconic languages. The depth of lack of scholarship is quite thorough.
-
@dkf I think his point is less 'which languages' and more 'specifically what about these languages'.
Like, I know JavaScript, I've dabbled in both Java and Python, and I've touched Rust and outside of comprehensions from Python I'm not sure what Rust could pick up (and even that is syntactic sugar for things that can be expressed in other ways)
Also note that JavaScript is terrifyingly common outside of a browser environment in the form of Node, so perhaps the machinery you're thinking of is less relevant there?
-
@Arantor said in The absolute state of web storage protocols:
Also note that JavaScript is terrifyingly common outside of a browser environment in the form of Node, so perhaps the machinery you're thinking of is less relevant there?
My comment was specifically to do with how the history of the language has meant that built-in support for message catalogs wasn't a priority until the likes of Node came on the picture, so language standard library support for them is weak. It's a curiosity of the history of the language.
-
..
-
..
-
@Gustav said in The absolute state of web storage protocols:
can you name even one thing they could learn?
You seem to be assuming they're capable of learning something.
-
@dkf said in The absolute state of web storage protocols:
@Gustav said in The absolute state of web storage protocols:
WHICH languages is Rust not learning from
Well, as a starter try Java and Python. They're pretty well known. Yes, they've got a bunch of crud in their standard libraries, but there's still definitely many things that could be learned. (C# would be another, but I don't know that so well either.)
And basically everybody who worked on the Rust standard library knows Python pretty well. After all, the build system has been written in Python until recently (and some bits are going to stay). They know it, and know why they don't want to go the same route. It mostly works for Python, but Python does not target resource constrained environments (and when it's used there, most of the standard library usually isn't available anyway), and the clean up with the release of Python 3 is a cautionary tale.
Java is also good counter-example. The old versions of the APIs still need to be supported, so there is now several variants for many things in the Java standard library—and then most programmers end up using some other libraries anyway, because the standard component is still not really sufficient.
It's better to provide the base that is known to be supportable for foreseeable future, and constrained by how the underlying operating systems work anyway, and provide an easy way to pull separate components to build a more ergonomic layer on top of that.
Javascript is weaker in this regard, as a side effect of starting in a browser environment where that sort of thing is handled by machinery that is out of its scope (the web application delivery side of things) so message catalogs are a third party thing for it.
JavaScript was, by that nature, forced to do things differently and it turned out to be better and that's why Rust places such importance on cargo, while trying to fix the main issues with npm like always using the lock file and still providing removed versions if they were fixed in the lock file.
By the way, I didn't see message catalogs in Python anyway. There is even less localization support than in the standard C library, because python does not support formatting numbers by locale in the normal
str.formatandstr.__mod__(%). And see, it already grew two different formatters. Three, because there is also the template module. No, it's a conscious decision by the Rust developers to not include things that are likely to change in future in the standard library.
-
@dkf said in The absolute state of web storage protocols:
@Arantor They've also not learned a thing from Lisp or other homoiconic languages. The depth of lack of scholarship is quite thorough.
That's something I would like. I would like the next language to have the homoiconic approach to generics from Zig integrated with the borrow checked from Rust.
And maybe slightly better async, though the better generics would probably take care of the parts that are not fixed by the presence of the borrow checker. Because Rust did not really have much choice in how to implement stackless coroutines.
And Zig contexts and fully capability-based standard library built on them with always scoped threads and filesystem and network access.
-
@dkf said in The absolute state of web storage protocols:
But this isn't the room for an argument; that's down the hall.
This is the room for insults.I'm sorry, but this is abuse!I've got that one memorized, I used to work as abuse@isp.
-
@Bulb said in The absolute state of web storage protocols:
By the way, I didn't see message catalogs in Python anyway. There is even less localization support than in the standard C library, because python does not support formatting numbers by locale in the normal
str.formatandstr.__mod__(%). And see, it already grew two different formatters. Three, because there is also the template module. No, it's a conscious decision by the Rust developers to not include things that are likely to change in future in the standard library.You need two parts, a way to look up the string templates, and a way to format them (that doesn't require values to be inserted in a specific order). The latter is
str.format()in Python and there's no precompilation stuff there to work around; you just need the catalog part (fiddly but not interesting code).
-
@dkf And the third part, the catalog maintenance, which is on the other hand very complex and interesting code, though that's tooling, not a library. Still, python does not have standard either.
-
..
-
@Gustav It's just how the translation toolkits usually call them.
But there isn't really anything like “regular” translation file. And some of them, e.g. the json usually used by front-ends, are not really well suited for maintenance. Because implementing the runtime translation is simple. It's the maintenance that is complex, as you need to make sure, as the applicätion evolves, you translate all the new strings, retranslate all the changed ones and occasionally drop the obsolete ones without wasting effort on translating everything again for each release.
Also if you want to support some grammar adaptation like plural forms, the “regular” translation file is already somewhat more involved than a simple table.
-
..
-
@Benjamin-Hall said in The absolute state of web storage protocols:
To me, websockets seem like they break the stateless nature of HTTP generally--they assume there's some running context.
Ever since the invention of the cookie, people have been trying to find ways to
breakwork around the stateless nature of HTTP, because state is crucial to getting anything done.
-
@Mason_Wheeler said in The absolute state of web storage protocols:
@Benjamin-Hall said in The absolute state of web storage protocols:
To me, websockets seem like they break the stateless nature of HTTP generally--they assume there's some running context.
Ever since the invention of the cookie, people have been trying to find ways to
breakwork around the stateless nature of HTTP, because state is crucial to getting anything done.But, for the price of some extra payload with each request, cookies manage to preserve the fault tolerance of the stateless protocol. When you re-connect the state in each request using the session cookie, you don't need any special handling for a request randomly dropping out, because the next will have the cookie needed to reconnect the state again. In contrast, websockets require explicit reconnections and state restoration.
-
@dkf said in The absolute state of web storage protocols:
Templates should be from trusted sources only. So too should log messages
That "should" is doing a lot of work...
-
@Gustav said in The absolute state of web storage protocols:
WHICH languages is Rust not learning from
All of the object-oriented ones, just for starters.
In the 21st century, OO is table stakes. And Rust just looks ridiculous for not including it.
-
@dkf said in The absolute state of web storage protocols:
@Arantor They've also not learned a thing from Lisp or other homoiconic languages.
Toby Faire, Lisp's homoiconicity is a crutch, a relic of a massively dumbed-down parser that dates to the Dark Ages before the Chomsky Hierarchy gave us a proper conceptual framework for writing parsers. The single most important thing to learn from homoiconicity is gratitude that we don't have to do such primitive things anymore.
-
@Bulb said in The absolute state of web storage protocols:
@Mason_Wheeler said in The absolute state of web storage protocols:
@Benjamin-Hall said in The absolute state of web storage protocols:
To me, websockets seem like they break the stateless nature of HTTP generally--they assume there's some running context.
Ever since the invention of the cookie, people have been trying to find ways to
breakwork around the stateless nature of HTTP, because state is crucial to getting anything done.But, for the price of some extra payload with each request, cookies manage to preserve the fault tolerance of the stateless protocol. When you re-connect the state in each request using the session cookie, you don't need any special handling for a request randomly dropping out, because the next will have the cookie needed to reconnect the state again. In contrast, websockets require explicit reconnections and state restoration.
Yeah. I'm 100% ok with the client maintaining state and passing along the critical bits with each call. I'm somewhat ok with the server maintaining very simple state about a connection, of the "does this authentication token match something I can retrieve from my database" (ie this person is still logged in and the cookie is valid) variety. Long-lived socket connections are a whole-different ball of snakes.
We have a "server" that has a bi-directional socket connection with the "workers" that it manages, established on startup. Which also gets panicky if it loses connection--if the worker doesn't reconnect in time, it redistributes its load elsewhere (which causes clients to forcibly reconnect). Which means we can't replicate the server--every server has total ownership of a set of workers, and every worker expects to talk to the same server for the duration of its life. This makes the system extremely fragile and annoying to deploy to, with mandatory downtime everytime we do.
-
@Mason_Wheeler said in The absolute state of web storage protocols:
In the 21st century, OO is table stakes. And Rust just looks ridiculous for not including it.
Actually, in the 21st century, OO is dead as a doornail and language designers are realizing how big of a mistake it was.
Most of the languages designed in 21st century don't have inheritance. Just some of them have some delegation crutch to cover the “simple” cases, but in Rust they never agreed on which cases are simple enough for a generic tool while being non-trivial enough to actually need it.
-
@Bulb then there’s PHP which didn’t sprout OO until the 21st Century, didn’t get usable OOP until the second decade (post 2010, obvs) and now it’s obsolete. Well, fuck.
-
@Benjamin-Hall said in The absolute state of web storage protocols:
We have a "server" that has a bi-directional socket connection with the "workers" that it manages, established on startup. Which also gets panicky if it loses connection--if the worker doesn't reconnect in time, it redistributes its load elsewhere (which causes clients to forcibly reconnect). Which means we can't replicate the server--every server has total ownership of a set of workers, and every worker expects to talk to the same server for the duration of its life. This makes the system extremely fragile and annoying to deploy to, with mandatory downtime everytime we do.
The problems come from the workers with a command server pattern, not really from how the sockets are set up. If it was http polling, the requirement for the client to only connect to one server would still be there, because as long as the worker does not manage itself, you need one place that manages that client.
Still, you could replicate the server. You'd have to make the state shared, presumably by putting it in some distributed storage like etcd, tikv, dqlite, surrealdb etc. (or a full blown distributed database if you have it in SQL DB anyway), so when the client connects to a different node than previously, it's new master could read the state, report that it now commands it and pick up where the previous master stopped. Which would eventually also allow you rolling updates if you made sure both the state and protocol will handle the version skew.
Whether such refactoring would be worth the trouble of course depends on how big the installations are and how big issue the downtimes are.
-
@Bulb said in The absolute state of web storage protocols:
@Benjamin-Hall said in The absolute state of web storage protocols:
We have a "server" that has a bi-directional socket connection with the "workers" that it manages, established on startup. Which also gets panicky if it loses connection--if the worker doesn't reconnect in time, it redistributes its load elsewhere (which causes clients to forcibly reconnect). Which means we can't replicate the server--every server has total ownership of a set of workers, and every worker expects to talk to the same server for the duration of its life. This makes the system extremely fragile and annoying to deploy to, with mandatory downtime everytime we do.
The problems come from the workers with a command server pattern, not really from how the sockets are set up. If it was http polling, the requirement for the client to only connect to one server would still be there, because as long as the worker does not manage itself, you need one place that manages that client.
Still, you could replicate the server. You'd have to make the state shared, presumably by putting it in some distributed storage like etcd, tikv, dqlite, surrealdb etc. (or a full blown distributed database if you have it in SQL DB anyway), so when the client connects to a different node than previously, it's new master could read the state, report that it now commands it and pick up where the previous master stopped. Which would eventually also allow you rolling updates if you made sure both the state and protocol will handle the version skew.
Whether such refactoring would be worth the trouble of course depends on how big the installations are and how big issue the downtimes are.
Really, the workers could basically manage themselves, as long as they can ask someone for a list of channels they're supposed to handle and be able to ask "is this person valid". And those two bits of state are already stored very much in a database, so any number of replicas could respond to those requests without any need to "manage" specific nodes.
The one concern is needing to know if a node has gone AWOL (in which case the responsibility for those channels needs to go somewhere else). But that can be handled via a more dedicated heartbeat mechanism or via an outward ping.
Or we could go crazy and allow multicast UDP within a swarm of worker nodes, so each client wouldn't have to be on the same worker as others in that same voice channel. But that increases latency, which is one of our big selling points.
And yes, downtime is a big deal. Basically, we're capped at 4 9s uptime if nothing goes wrong just by deploying once per week. And for real-time audio supplementing (not entirely replacing) first-responder radio communications...that's not particularly great.
-
@Benjamin-Hall said in The absolute state of web storage protocols:
Really, the workers could basically manage themselves, as long as they can ask someone for a list of channels they're supposed to handle and be able to ask "is this person valid". And those two bits of state are already stored very much in a database, so any number of replicas could respond to those requests without any need to "manage" specific nodes.
If the reconnect logic stored the “waiting for the client to reconnect since X” status in the shared storage, and then the redistribution read that from the shared state, I think it should be refactorable to handle both the client reconnecting, in the timeout, via another server node, and also redistributing the channels to workers connected to different server node at the time (with proper transaction handling).
The one concern is needing to know if a node has gone AWOL (in which case the responsibility for those channels needs to go somewhere else). But that can be handled via a more dedicated heartbeat mechanism or via an outward ping.
Outward ping only checks the computer, but not the service, so a heartbeat is definitely preferable.
Or we could go crazy and allow multicast UDP within a swarm of worker nodes, so each client wouldn't have to be on the same worker as others in that same voice channel. But that increases latency, which is one of our big selling points.
That's probably more refactoring than worth it anyway.
And yes, downtime is a big deal. Basically, we're capped at 4 9s uptime if nothing goes wrong just by deploying once per week. And for real-time audio supplementing (not entirely replacing) first-responder radio communications...that's not particularly great.
That makes it sound like it'd be worth considering. The sockets themselves can probably stay—the code is hairy, but it already works—the key is storing the state “waiting for the worker since X” in the shared storage so other server can take over, and adding some code to the worker that will try different servers from the pool when the connection breaks.
-
@Bulb said in The absolute state of web storage protocols:
@Benjamin-Hall said in The absolute state of web storage protocols:
Really, the workers could basically manage themselves, as long as they can ask someone for a list of channels they're supposed to handle and be able to ask "is this person valid". And those two bits of state are already stored very much in a database, so any number of replicas could respond to those requests without any need to "manage" specific nodes.
If the reconnect logic stored the “waiting for the client to reconnect since X” status in the shared storage, and then the redistribution read that from the shared state, I think it should be refactorable to handle both the client reconnecting, in the timeout, via another server node, and also redistributing the channels to workers connected to different server node at the time (with proper transaction handling).
Probably. In the long term, we're moving away from this particular server code because it does lots of screwy things (never trust code you get from the R&D folks, especially when they didn't actually have requirements and made it "flexible"...by which we mean "over-abstracted but not actually flexible in any useful way, just hard to deal with"). When we do, the sockets aren't staying as such.
The one concern is needing to know if a node has gone AWOL (in which case the responsibility for those channels needs to go somewhere else). But that can be handled via a more dedicated heartbeat mechanism or via an outward ping.
Outward ping only checks the computer, but not the service, so a heartbeat is definitely preferable.
I probably used the wrong term. I meant that the workers would contact the server saying "hey, I'm still alive" instead of the server trying to contact the worker to determine that.
Or we could go crazy and allow multicast UDP within a swarm of worker nodes, so each client wouldn't have to be on the same worker as others in that same voice channel. But that increases latency, which is one of our big selling points.
That's probably more refactoring than worth it anyway.
Likely. But sounds like the sort of thing our CTO (aka architecture astronaut #1) would insist on.
And yes, downtime is a big deal. Basically, we're capped at 4 9s uptime if nothing goes wrong just by deploying once per week. And for real-time audio supplementing (not entirely replacing) first-responder radio communications...that's not particularly great.
That makes it sound like it'd be worth considering. The sockets themselves can probably stay—the code is hairy, but it already works—the key is storing the state “waiting for the worker since X” in the shared storage so other server can take over, and adding some code to the worker that will try different servers from the pool when the connection breaks.
Yeah, except we're dropping the sockets because they don't really serve much of a purpose except to get in the way. There are a lot of really stupid abstractions here that make it way more painful to disentangle, but we're trying to move in that direction. If product lets us...
-
@Bulb said in The absolute state of web storage protocols:
@Mason_Wheeler said in The absolute state of web storage protocols:
In the 21st century, OO is table stakes. And Rust just looks ridiculous for not including it.
Actually, in the 21st century, OO is dead as a doornail and language designers are realizing how big of a mistake it was.
Most of the languages designed in 21st century don't have inheritance. Just some of them have some delegation crutch to cover the “simple” cases, but in Rust they never agreed on which cases are simple enough for a generic tool while being non-trivial enough to actually need it.
Don't be ridiculous. There's a reason OOP conquered the world while languages without it remain relegated to minor niches: The class/inheritance model is the single best tool we've ever developed for managing complexity.
Is it perfect? No. But it's significantly better than anything else out there, and people pointing to problems and saying "we should be using my preferred language because it doesn't have this problem" reek of sour grapes. OOP has triumphed decisively in the marketplace of ideas.
-
@Mason_Wheeler said in The absolute state of web storage protocols:
@dkf said in The absolute state of web storage protocols:
Templates should be from trusted sources only. So too should log messages
That "should" is doing a lot of work...
Two independent ones. Basic message templates mostly come from the logging package itself or the application, or are configured at installation time, and state things like "logging messages have a timestamp and a logging level and a locus component name and a message" and so on. They're very cacheable. (What they should be depends on the application; for example, logging the thread context is vital in some use cases, and pointless overhead in others.) This is the part where you might want some recursive expansion... but I'm not convinced it is needed.
By contrast, the message comes from the program code itself, and really ought to be a compile time constant. Sometimes it isn't, but that's often a sign of a programming error. Don't just log a generated string, log a message that says "
the string is this: {}" and that has the real string of interest as a parameter. My coworkers used to hit that problem quite a bit with logging sets when the sets were empty and the result looked like a template parameter...
For completeness, the parameters (that get substituted into the messages before that goes into the template) should be able to be things things that aren't trusted, such as the username that failed to log in; the substitution engine probably should add quoting for "weird" characters by default...
-
@Benjamin-Hall said in The absolute state of web storage protocols:
Or we could go crazy and allow multicast UDP within a swarm of worker nodes,
That would match the definition of "go crazy" for sure. The nice nurses will be along shortly to help, with that special jacket with the long sleeves...
-
@Bulb said in The absolute state of web storage protocols:
Actually, in the 21st century, OO is dead as a doornail and language designers are realizing how big of a mistake it was.
I note that very few of the languages that decry OO have also made much of an effort to produce GUI tools (either native or via something like Electron). Or at least haven't done so seriously. The basic issue there is that having some kind of local context to manage the state for a UI component (such as a button or entry field) is pretty much essential; there's a lot of bits of stuff in there that the wider world doesn't care about normally. OO helps hugely with that; instead of thinking of all the different bits of drawing state, you just think about the overall component and the notifications it delivers when "something happens" (the button is clicked, etc).
If you say "but we have that, we just don't do inheritance/derivation" then I'd observe that inheritance is by far the least important part of OO, and indeed is an optional part, and that you instead have limited OO. Delegation is more important, where you pass a task on to another object you know about so that the caller doesn't need to know everything, and even that is less important than encapsulation...
-
..
-
@Gustav It's impossible to do GUI nicely without it. But it's possible.
My stepdad wrote an entire "paint program" in QBASIC (not even QuickBASIC) with mouse support (hooking into assembly). Also, not using the FUNCTIONS and SUBS that QBASIC has, because he doesn't like how QBASIC hides those so the entire thing is one very long, very globally scoped thing with GO SUBs everywhere.
-
