The absolute state of web storage protocols
-
@Bulb said in Azure bites:
Storage services have stopped supporting directory listings a … well, as far as I know none of the cloud ones does.
[…]
That said, I would prefer if the protocol was compatible with webdav rather than a custom http-based api.So I thought webdav should be the obvious http-based storage protocol … until I went to search through documentation and didn't find any way to chunk an upload. A single PUT request is fine for a 100 kB or 1 MB document, but I have a use-case where the normal size of upload is in the ballpark of 20 GB and trying to do that with a single PUT request has a lot of problems.
The Azure blob storage supports uploading either by block or by page. By block means PUTting the blocks (max 4 GiB) with some extra query parameters and then putting (or is it posting, not sure) the list of blocks to assemble as the file. And by page it means with some range parameter, but I believe for PUT it's a custom range parameter. These work for the use-case, but are not standard.
It seems to me we are in the
stage—there is a bunch of variations, none of which handles all the use-cases.
Or (honest question) is there some http-based (to be usable from the browser and also to traverse proxies) filesystem protocol that has something that could be called a specification, an open-source reference implementation, and covers all of browsing files, large file uploads by blocks, and preferably also some form of change notification?
-
@Bulb I don’t think so, sadly.
So here’s a fun concept, you’ve seen how Azure does it, but SharePoint/OneDrive can also do this, except with the ability to chunk but it’s extremely pissy about how you do it. You can even query the list of content via HTTP-with-Bearer token but the whole thing is a fucking shambles to actually use. (And of course, OneDrive/SharePoint isn’t designed for the sorts of volume Blob Storage is, though it will take files up to 60GB I believe. And the API is, naturally, different.)
The saddest part is that outside of Azure, the solution everyone has coalesced towards is using S3’s API as a reference - so there’s several vendors who will give you S3-compatible storage.
At heart I think they’ll all accept HTTP with Bearer token for requests, but the rest of your needs might be more or less hard to match because these services don’t really do “browsing” in the truest sense, though “list this bucket” and “list the bucket using the following prefix” do exist, plus you can get hold of URLs that are time-limited signed so you can share files outside your immediate network, which should be proxy-safe because it’s a URL served by S3 or whatever service with a token URL parameter.
-
@Bulb I wonder if a
PUTwith aRange:header would work, at least conceptually. (In practice, you do what you gotta do with the providers you have.)searches
Well that was truly pointless. I found a total lack of understanding of what is meant by idempotency and advocacy of a method (
PATCH) that nobody uses. Hooray.
-
@dkf PUT with Range is how OneDrive does it, as long as you already opened the session to say where you're writing to, and it's pissy about the chunk size you're writing while you're doing it (the manual warbles on about chunks being multiples of 320KB in size)
I think it's how S3 does it but honestly, I use the SDK there and never touch it directly.
-
Let me just add to the astonishment that a kindergarten level problem like this is still a problem in this day and age. Truly, the Ouroboros of this reeking collective stupidity of an industry never ceases to amaze.
-
@Arantor said in The absolute state of web storage protocols:
The saddest part is that outside of Azure, the solution everyone has coalesced towards is using S3’s API as a reference - so there’s several vendors who will give you S3-compatible storage.
That's actually … reasonable given the clusterfuck. They still do the directory listing as a query on
/like azure blobs though.@Arantor said in The absolute state of web storage protocols:
SharePoint/OneDrive […] the API is, naturally, different.
I was under the impression, possibly incorrect, that SharePoint/OneDrive is actually based on WebDAV.
-
@Bulb if it is based on WebDAV, the HTTP interface doesn't quite smell like it, at least not via the Graph API. But that's a whole different can of fucking worms anyway.
-
See also the
stage.
-
There really ought to be some sort of file transfer protocol.
-
@Bulb said in The absolute state of web storage protocols:
http-based
That is where you stop. In general, imo, it was never intended for large file transfer of any kind, and all the chunked whatsis thonks were hacks that never got properly codified.
You can pretend using a hacked-together thing yourself, but that means you naturally have to abuse the protocol, just like everyone else.
-
@Zecc said in The absolute state of web storage protocols:
There really ought to be some sort of file transfer protocol.
Better make sure the file transfer protocol is secure.
-
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
http-based
That is where you stop. In general, imo, it was never intended for large file transfer of any kind, and all the chunked whatsis thonks were hacks that never got properly codified.
The dumb thing is that it is actually mostly very good at it, better than most other alternatives (I've seen the data). The problems are because people want complicated things of their file transfer systems. And you've got the usual fun of what happens when two systems simply don't agree on what a filename is in the first place; that's not the protocol's fault.
The other respectable option is SSH; scp and rsync sit on top of that typically.
-
@Watson said in The absolute state of web storage protocols:
See also the
stage.
Most e-mail providers have somewhere to store documents too, and support sending a link with some hash that allows downloading it. Whether you have GoogleDrive or OneDrive or whatever-Apple-calls-their-cloud-storage, almost everybody has some place to upload things for sharing now. Even mailfence supports document storage and sending download links.
And for this purpose, plain HTTP is good enough. It doesn't support listing anything, but we have that thing called zip for when we want to send more than one file.
-
@dkf said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
http-based
That is where you stop. In general, imo, it was never intended for large file transfer of any kind, and all the chunked whatsis thonks were hacks that never got properly codified.
The dumb thing is that it is actually mostly very good at it, better than most other alternatives (I've seen the data). The problems are because people want complicated things of their file transfer systems. And you've got the usual fun of what happens when two systems simply don't agree on what a filename is in the first place; that's not the protocol's fault.
The trickiest part is consistency. You can have multiple clients writing the same file, or you can have a file open on one client and another client might rename it and then the stateless nature of http allows precious few ways out of the problem. That's why NFS so weirdly assigns handles server-side even though it complicates the implementation quite a few bits.
The other respectable option is SSH; scp and rsync sit on top of that typically.
SSH is just a secure session, basically a TLS + authentication. Scp is a very simple protocol with a header indicating which file to read or write and the data terminated by closing the connection. Sftp is a more complex protocol over the same session, and rsync is even more complex protocol over the same session.
But ssh is not really relevant, because the concept of security of browser applications is based around domains and http headers and they can only connect with http or protocols that are upgrade from that (websockets), plus half the companies only allow access to internet through a http proxy, so for a lot of internet applications, ssh is not usable.
Filed under: http is the new tcp
-
-
@Bulb said in The absolute state of web storage protocols:
The trickiest part is consistency. You can have multiple clients writing the same file, or you can have a file open on one client and another client might rename it and then the stateless nature of http allows precious few ways out of the problem. That's why NFS so weirdly assigns handles server-side even though it complicates the implementation quite a few bits.
The funny bit is that "write bytes at a specific offset into the file" is an idempotent operation, but "append bytes to the file" is not.
And file locking is a pain as soon as you have a network simply because lock holders can arbitrarily vanish and you can't stop that from happening.
-
@Arantor said in The absolute state of web storage protocols:
multiples of 320KB in size
Slowly backs off to my area of computer stuff, which has its own share of WTFs, but at least they're not in the cloud...
-
@topspin this is the same API whose documentation can't keep it straight if a single atomic PUT (versus a chunked one) will allow 3MB or 4MB as a maximum upload when the actual observed maximum is approximately 5.5MB.
-
@Arantor must be metric vs
binaryimperial MB.
-
@topspin it is Microsoft so the correct theme will be playing in the background.
-
@Arantor I always figured the Microsoft theme was Yakety Sax
-
@loopback0 Sort of like this? https://www.youtube.com/watch?v=fJo7zqO4Bcg
-
@dkf said in The absolute state of web storage protocols:
you've got the usual fun of what happens when two systems simply don't agree on what a filename is in the first place; that's not the protocol's fault.
Which is why S3 gave up that pretense.
-
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
Cumming soon to you: iFuck.
-
@Bulb said in The absolute state of web storage protocols:
Filed under:
httpwebsockets is the new tcpI can't wait to see what the industry does next!
-
@Tsaukpaetra and yet the number of people using websockets is far smaller than you'd think (and using regular polling or long-polling), partially because websockets are hard, partially because websockets "aren't reliable" and partially because servers are cheap enough that just polling is good enough in most cases.
-
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
-
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
Your meme is not old enough and you should feel bad.
-
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
ClownCar.
-
@Tsaukpaetra said in The absolute state of web storage protocols:
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
Your meme is not old enough and you should feel bad.
The SuperDrive predates any and all iShit. Behold my massive 
but not too closely.
-
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
Your meme is not old enough and you should feel bad.
The SuperDrive predates any and all iShit. Behold my massive but not too closely.Besides, why would they reuse old names?
-
@Tsaukpaetra said in The absolute state of web storage protocols:
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@LaoC said in The absolute state of web storage protocols:
@Tsaukpaetra said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
whatever-Apple-calls-their-cloud-storage
iCloud. What did you think it would be called? Seriously? That's like their meme.
CloudDrive or VaporCapsule like all their storage stuff so far?
Your meme is not old enough and you should feel bad.
The SuperDrive predates any and all iShit. Behold my massive but not too closely.Besides, why would they reuse old names?
Remind me again how many iMac iterations there have been, how many iPhone models, how many iPad models…
-
@Arantor said in The absolute state of web storage protocols:
partially because websockets are hard
I never understood why the websocket protocol insists on internally splitting the stream into “messages”—most of the time you are running another protocol over it that already has its own framing anyway.
partially because websockets "aren't reliable"
They are as reliable as any TCP. Which means … you need to send something around every 30 seconds or will flow control itself into oblivion, especially if the line has a bit of noise … and if a proxy of any kind is involved, the something has to be data and not just a keepalive packet.
and partially because servers are cheap enough that just polling is good enough in most cases.
Long poll and websocket take similar amount of resources on the server. You have the connection open in both cases, with the associated server-side state, and you have to refresh it occasionally with an empty message to avoid it being dropped from proxies and masquerades.
… the main point of websockets is really tunnelling some other, existing, protocol through them.
-
@Tsaukpaetra said in The absolute state of web storage protocols:
Better make sure the file transfer protocol is secure.
Make sure that the file transfer protocol is secure, or that it's a secure file transfer protocol?
-
Now, is you'll excuse me, I need to consider joining either the Judean People's Front or the People's Front of Judea.
-
@Bulb said in The absolute state of web storage protocols:
They are as reliable as any TCP.
No, no, you misunderstand me. The use of scare quotes was very specific.
I've recently bumped into a bunch of, let's call them developers, who finally got websockets working in their app and were dismayed at how unreliable the whole thing seemed to be.
Apparently no-one had told them that 'if the connection stops you have to reconnect' and/or 'you need to have some kind of active keepalive'.
… the main point of websockets is really tunnelling some other, existing, protocol through them.
Yes but that's not what people have been sold on as the concept. People have been sold on it being an open bidirectional connection that either side can use to do things. You know, in the 'server push' mentality because 'we have to send user notifications AS SOON AS SOMETHING HAPPENS'.
Majority of the time from what I've seen people are going from HTTP to websockets solely to be able to have server push over HTTP in lieu of a long polling or regular polling strategy. And that actually the framing stuff never comes up because the people implementing it are all using libraries that hides all that away which means the moment they're not doing this in dev, the real world smacks them upside the head because they have no idea what the protocol is actually doing and what they have to do to actually make it work anyway.
-
@Arantor said in The absolute state of web storage protocols:
doing this in dev
I'm rather peeved at the amount of internal websites I use on the daily that are literal dumps of the dev workspace. Yes I know that if it's "cached" it "loads fast the second time". Point is I shouldn't have to wait two minutes to start using a site to begin with, mothafucka!
-
@Arantor said in The absolute state of web storage protocols:
a bunch of, let's call them developers, ... they have no idea
-
@Arantor said in The absolute state of web storage protocols:
'if the connection stops you have to reconnect'
What do you need to do to establish a websocket connection? (I could Google, but
.) Guess a TCP handshake, probably SSL as well and then some other stuff on top of it?I guess the question is ... how many round trips?
-
Here's a question about long polling and web-sockets:
How well do they handle replicated servers? That is, where you may have multiple replicas of the same server behind a load balancer. To me, websockets seem like they break the stateless nature of HTTP generally--they assume there's some running context. Sure, each time they break, you could re-establish it with a new server. But then you're paying the overhead.
The context here is that we have a server-to-server structure dreamed up by R&D guys who were in love with socket connections. We've got one "manager" server, which acts as an HTTP server and authentication server and coordinator managing a bunch of "nodes", who handle the grunt work of RTP connections. When the nodes start up, they establish a long-running socket connection (not even a proper websocket, this is a hand-rolled library) with the manager. And they're coded to expect one and only one manager--they reconnect if a connection drops, but both sides expect the identities to stay the same indefinitely. And that's really bad, because it means we can't do any of our normal tricks with deploying replicated servers (rolling restarts, etc). It means our reliability is only as good as that single container.
And the only real traffic on those connections is every 5 minutes or so, plus when someone logs in (which could be handled with a simple token). It feels like they were trying to do some kind of poor-man's service discovery mesh...badly.
To me, simple polling would work. What I'm not sure is how long-polling plays with this.
So the question, again--does long polling work reasonably well when every connection might hit a different (replicated) server? When the client has to assume that the servers are ephemeral and have no recollection of any previous connections? Is there a way to get websockets to work under those conditions?
-
@cvi said in The absolute state of web storage protocols:
@Arantor said in The absolute state of web storage protocols:
'if the connection stops you have to reconnect'
What do you need to do to establish a websocket connection? (I could Google, but
.) Guess a TCP handshake, probably SSL as well and then some other stuff on top of it?I guess the question is ... how many round trips?
I haven't gone deeply into it - PHP isn't exactly kind to long running processes and the like - but as I understand it, you start with an HTTP request to the server to establish the connection, you get a response in HTTP to say 'yes, cool' and then you converse over WS(S) instead of HTTP(S), which has its own framing, but it'll sit on top of TCP at that point.
Which also means if you let it lapse, you're back to the HTTP-over-TCP kickoff.
But most people never see the underlying part because they're all building on languages that abstract a lot of this away.
-
@Arantor said in The absolute state of web storage protocols:
But most people never see the underlying part because they're all building on languages that abstract a lot of this away.
So, the real problem is that the abstraction doesn't abstract this away too.
If documentation ever says something along the lines of "You have to do this with the component because of some internal implementation quirk.", then you have broken encapsulation. The client library component could just as easily contain the code that "The developer should have written".
-
@Benjamin-Hall technically WS is a separate protocol to HTTP, you just negotiate the protocol change from HTTP to WS at which point you’re no longer pretending it’s a stateless connection with magic helpers to signify any pretence of state; you explicitly have a stateful connection with all that implies.
Long-polling is a bastardisation of the same concept but without the protocol change. You open a connection, and you hold it open notionally for the server to push to the client. Things get wonky depending on exactly how you’re doing the keep alive (because proxies, including reverse proxies like nginx in front of Apache, will want to shut that down after a given timeframe with the gateway timeout error) and it certainly used to happen that a long poll was “I’m listening, send me something when there’s something to send me” and then the server would send something and then close the request.
That at least is more in keeping with the transactionality model that HTTP is supposed to have but never underestimate engineers abusing a protocol in all sorts of unexpected ways for “neat functionality”.
In a replication model the theory is the same: you hold the connection open somewhere and when it fails and is reestablished, one of the sides needs a source of truth to backfill whatever might have been missed in the middle. It’s one of the reasons pub/sub is frequently used in this fashion, because you just re subscribe back to the channel and the server should have the subscription events in a backing store and not just on whichever front facing node happened to hit it.
But of course this implies some actual thinking about distributed design and state management which already elevates it above many of those who work with it…
tl:dr; ideally the node you’re creating the long poll (or web socket) with isn’t solely responsible for the data in the session but is a relay to/from some slightly more permanent store than “the session”.
-
@Jaime you assume these people ever read the documentation. Or that the people writing the library documented it. Or even thought of it.
Web dev is a fucking wasteland of bad documentation that few write, and even fewer read. And is why in no small part ChatGPT is getting a foothold because inexplicably it’s less awful at it.
-
@Benjamin-Hall said in The absolute state of web storage protocols:
So the question, again--does long polling work reasonably well when every connection might hit a different (replicated) server? When the client has to assume that the servers are ephemeral and have no recollection of any previous connections? Is there a way to get websockets to work under those conditions?
With a websocket, you have one TCP connection that switches from handling HTTP (typically inside TLS) to the websocket protocol (still inside TLS). The connection doesn't hop between back-end services (normally; I bet someone has somehow made that work). It's common to layer STOMP over that so you then are dealing with JSON messages and a message bus (conceptually); you could have the designated receiver service spread those messages to the back end via any mechanism you care to implement.
When I did WS, I did something totally different, but I was really proxying binary messages over UDP (which randomly sucks over the open Internet for non-DNS use...) The nature of the per-message overhead was such that it was unnoticeable (i.e., nothing got split across Ethernet frames).
Long polling (which we also use) also ties up some server resources, but is better suites for when you have something that has a low data rate most of the time (perhaps because you're waiting for an event). The aim is to at least not tie up a whole thread on the server while waiting.
-
@Arantor said in The absolute state of web storage protocols:
But of course this implies some actual thinking
-
@Jaime said in The absolute state of web storage protocols:
@Arantor said in The absolute state of web storage protocols:
But most people never see the underlying part because they're all building on languages that abstract a lot of this away.
So, the real problem is that the abstraction doesn't abstract this away too.
If documentation ever says something along the lines of "You have to do this with the component because of some internal implementation quirk.", then you have broken encapsulation. The client library component could just as easily contain the code that "The developer should have written".
Since it's a potentially stateful connection, the reconnection cannot be abstracted away at this level. It might be possible to abstract at the level of the protocol you tunnel through the websockets (e.g. AMQP or MQTT libraries do), but a websocket is just a socket with some extra bits to make it usable from a browser, and you don't have automatic reconnection of TCP sockets either.
And some aspects on the network can't really be abstracted away anywhere. For example acknowledge always has to be done on the application level, because the application needs to know whether the message was processed. Which is why TCP sockets don't expose the information up to where the stream was acknowledged on the TCP level—it would mislead coders to assume the data reached the other application when the TCP acknowledge only means they reached the other computer.
-
@Bulb protocols are hard, let's go shopping!
-
@Arantor said in The absolute state of web storage protocols:
@Bulb said in The absolute state of web storage protocols:
They are as reliable as any TCP.
No, no, you misunderstand me. The use of scare quotes was very specific.
I think I mostly did.
I've recently bumped into a bunch of, let's call them developers, who finally got websockets working in their app and were dismayed at how unreliable the whole thing seemed to be.
Apparently no-one had told them that 'if the connection stops you have to reconnect' and/or 'you need to have some kind of active keepalive'.
I don't have experience with your specific cow-orkers, but mine made the same mistake (and were told so by the back-end dev recently). We use GraphQL between the back-end and the front-end and the ChillyCream framework comes with websocket subscription support, so that's what we use. Long poll would probably be more suitable exactly because you have to restart it after every notification—or timeout—so it would not require special handling for reconnects.
But when I said “as reliable as any TCP”, I said it because TCP is not reliable, and I've seen quite a few devs make that mistake too. With TCP you also have to handle reconnects and acknowledgement and a lot of devs fail to.
Including those on the older version of the back-end in the same project. That version used ZeroMQ (the newer uses RabbitMQ), which is a relatively thin layer over sockets that handles service discovery and reconnection. But it is “as reliable as any TCP”, which means it is not, and they kept complaining that it's problematic, mostly because they didn't properly understand the nature of network and what the library can and cannot abstract for them. Notably that you have to be ready to re-send any message until your component on the other side confirms it handled it.
… the main point of websockets is really tunnelling some other, existing, protocol through them.
Yes but that's not what people have been sold on as the concept. People have been sold on it being an open bidirectional connection that either side can use to do things. You know, in the 'server push' mentality because 'we have to send user notifications AS SOON AS SOMETHING HAPPENS'.
Majority of the time from what I've seen people are going from HTTP to websockets solely to be able to have server push over HTTP in lieu of a long polling or regular polling strategy. And that actually the framing stuff never comes up because the people implementing it are all using libraries that hides all that away which means the moment they're not doing this in dev, the real world smacks them upside the head because they have no idea what the protocol is actually doing and what they have to do to actually make it work anyway.
… which is dumb, yeah. For notifications, long-polling is as good (or bad—if your server does thread per connection, stick with short-polling) as a websocket for resource use and in most cases simpler to implement.
Though, never underestimate webdevs, and JavaScripts, ability to miss events. If your event handler throws (up), your chain of handlers will break and there's generally nothing that could come around and restart it.
-
@Bulb said in The absolute state of web storage protocols:
Though, never underestimate webdevs, and JavaScripts, ability to miss events.
I am in the minority precisely because I do not underestimate these two things and their capacity to do the most inane and stupid shit.
But when I said “as reliable as any TCP”, I said it because TCP is not reliable, and I've seen quite a few devs make that mistake too. With TCP you also have to handle reconnects and acknowledgement and a lot of devs fail to.
That's the thing about webdevs, TCP is a level almost never touch, that's handled one level below us so having to deal with TCP reconnects and acknowledgements is almost exclusively a library problem. Which is why for webdevs this is a whole class of problem they're unfamiliar with.
mostly because they didn't properly understand the nature of network and what the library can and cannot abstract for them.
This is certainly the problem this group of people I mentioned ran into. I wouldn't describe them as coworkers because they're not, they're a group of #buildinpublic wingnuts on Twitter who declare that the technology is broken because they don't know how to use it correctly.
For notifications, long-polling is as good (or bad—if your server does thread per connection, stick with short-polling) as a websocket for resource use and in most cases simpler to implement.
Yup, though very often even how to implement long-polling correctly is too hard (because they assume it will run like a full duplex stateful connection until they realise it isn't), and end up implementing short polling until the backend folks start complaining about how much extra load it produces with the constant reconnections/re-establishing of state etc.
