WebSocket connection dropping for reasonably large messages

[Zecc]

We have an integration which phones home using WebSockets. A specific server started disconnecting a lot, after moving to a new location.

I've pinpointed it to the connection being dropped with status 1006 after I try to send messages larger than 1440~ characters.
All I could find about this status is what RFC6455 has to say:

    1006 is a reserved value and MUST NOT be set as a status code in a
          Close control frame by an endpoint.  It is designated for use in
          applications expecting a status code to indicate that the
          connection was closed abnormally, e.g., without sending or
          receiving a Close control frame.

So it looks to me like the underlying TCP connection is being closed without the websocket being explicitly close in either side.

My tests have boiled down to creating a super simple echo server in nodejs in our machine, and in their machine using Chrome's devtools to repeatedly create a new WebSocket object and calling ws.send('x'.repeat(n)) with increasing n until I no longer saw my echo server logging the message and Chrome getting it back.

What bothers me is:

• No error handler is called. Both sides eventually see the connection closed, after 10 seconds or so, which I guess is how long it takes for both sides to realize heartbeats have stopped.
• I can RDP to their server just fine, so it's not like their internet connection sucks.
• There's an exact maximum size of message I can send. Up to that size, all the messages are received 'instantaneously'. But add 1 more character and the message doesn't go through. This makes me think of network packet sizes, which is a weird thing to think about when dealing with a protocol that's built on HTTP(S) and TCP. The maximum size varies a bit between ws:// or wss://, but it is constant for a given protocol.
• This is the only client for which this happens. If I do the exact same test from other client machines it can send multi-megabyte messages without any problems.
• When I configured the client to connect to a different server in our side it worked for a bit, but after a day or so it started disconnecting again. This happened twice. Could there be a "clever" firewall or antivirus or the like "learning" to block traffic? But RDP from the same server to the client works fine. Shouldn't TLS mean network traffic can't be told apart?

I haven't tried sending the client messages of different sizes, come to think of it, so I don't know what would have happened there.

I haven't tried using Wireshark or the like either because I'm not very experienced with it (though I definitely think it's something I should learn at some point) and these aren't my machines to be putting third party software in.

I've been told this is not my problem , but it still rubs me the wrong way not knowing what might be causing this kind of behaviour. Any ideas?

[CHUDbert]

Network switch doing something and failing? Doubtful... Maybe some kind of security plug in?

[PleegWat]

@Zecc said in WebSocket connection dropping for reasonably large messages:

Shouldn't TLS mean network traffic can't be told apart?

Only if it's on the same TCP port.

I suspect your websocket message is being split over 2 ethernet frames. This means the first half of the message arrives before the second does. Perhaps for some reason it tries to parse the partial message, and fails?

[Kamil Podlesak]

The behavior and the cutoff value sound both very, very familiar. I haven't had this problem in a while, but it used to be very common 20 years ago.

Check your MTU settings in all places you can and if possible, try setting lower values (ie lower than 1400).

[dkf]

@Zecc said in WebSocket connection dropping for reasonably large messages:

I've pinpointed it to the connection being dropped with status 1006 after I try to send messages larger than 1440~ characters.

Ethernet MTU is usually 1500 and you need quite a bit for the various message headers and framing. That being about 60 bytes would be... unsurprising.

But what would be highly surprising is the MTU mattering for a protocol that's tunnelling over a TCP socket. What's supposed to happen is that the message gets split up over several TCP messages and reassembled on the other end, and the whole damn point of using TCP is that all that shit should be hidden away from you. If you were using UDP to route things then I could understand it, but even with a UDP VPN that should be all transparent. This points to something deciding to be a complete ass, and because it is on one machine only, it's probably not on the server side. Does that client machine have an unusual firewall setup, or something with antivirus? Almost all the truly crazy things I've heard of in this general area have come from AV deciding that it only needed to implement a subset of OS semantics...

[Zecc]

@PleegWat said in WebSocket connection dropping for reasonably large messages:

Only if it's on the same TCP port.

Wouldn't it be always the same port during the lifetime of a connection?

@PleegWat said in WebSocket connection dropping for reasonably large messages:

I suspect your websocket message is being split over 2 ethernet frames. This means the first half of the message arrives before the second does.

Shouldn't TCP handle this?
(inb4 committing sudoku is a way of handling it. Not a good way, but a way)

@Kamil-Podlesak said in WebSocket connection dropping for reasonably large messages:

The behavior and the cutoff value sound both very, very familiar. I haven't had this problem in a while, but it used to be very common 20 years ago.

Check your MTU settings in all places you can and if possible, try setting lower values (ie lower than 1400).

I'm not a network person, and in particular I'm not their network admin, but I might take a look just because.

@dkf, that's exactly what I was thinking and why I find this whole thing so confusing. But I'm not in their IT department (or even the same country) so I don't really know their infrastructure. Presumably it's not anything software because the only thing they've done was physically move the server from one location to another, and the ISP is still the same.

Sigh. I guess I only wanted reassurance that I wasn't missing something obvious and that it's them, not me.

The deciders have decided this is Not My Problem, so I'm not sure if I'll ever learn the end to this story, unless they suddenly decide to put it on me again. We'll see.

[PleegWat]

@Zecc said in WebSocket connection dropping for reasonably large messages:

Wouldn't it be always the same port during the lifetime of a connection?

Yes, it would be. I must have misread the situation.

@Zecc said in WebSocket connection dropping for reasonably large messages:

Shouldn't TCP handle this?

Depends on how the receiving code is written. In non-blocking mode, it is quite possible for it to have received a 1500-byte length specifier with 1440 bytes of payload in the first frame, and throw or crash rather than block when trying to reference the last 60 bytes of the payload.

@Zecc said in WebSocket connection dropping for reasonably large messages:

I'm not a network person, and in particular I'm not their network admin, but I might take a look just because.

It is something which is supposed to be handled by path MTU autodiscovery. But in misconfigured networks, it might not be.

[dkf]

@PleegWat said in WebSocket connection dropping for reasonably large messages:

Depends on how the receiving code is written. In non-blocking mode, it is quite possible for it to have received a 1500-byte length specifier with 1440 bytes of payload in the first frame, and throw or crash rather than block when trying to reference the last 60 bytes of the payload.

But the websocket core on the server side should handle the message reassembly at that level. Websocket streams are internally split into messages, which can be text or binary, but it is usually an option for whether you want to handle partial messages in app code or not; if you're using STOMP inside that websocket then you'll typically not want partial messages delivered to higher levels. (You only really ever want partial messages when dealing with multi-megabyte messages. With smaller stuff, reassembly is just too useful in terms of simplifying the higher levels of your code.)

Asking for partial messages but not handling the consequences is like asking for UDP but being surprised when messages go missing or are reordered. Dumb! Dumb! Dumb! Please don't be that idiot.

[BernieTheBernie]

@Zecc said in WebSocket connection dropping for reasonably large messages:

I've been told this is not my problem , but it still rubs me the wrong way not knowing what might be causing this kind of behaviour. Any ideas?

And likely, StackOverflow is too toxic for such a question...
With C# WCF, you can use a configuration file, and there the transferMode can be configured. What about a wrong configuration in this place with streamed vs. buffered?
See e.g.

saravanakumar's WCF tutorial

[Zecc]

@BernieTheBernie said in WebSocket connection dropping for reasonably large messages:

And likely, StackOverflow is too toxic for such a question...

I'm not sure I remember what email address I used to create an account, let alone the password. Apparently it predates my password manager.

@BernieTheBernie said in WebSocket connection dropping for reasonably large messages:

With C# WCF, you can use a configuration file

That's great, although I'm not sure how it will help me with Chrome connecting to NodeJS. :-J

[BernieTheBernie]

@Zecc said in WebSocket connection dropping for reasonably large messages:

Chrome connecting to NodeJS

And there, what can be configured there? No transferMode?

[Zecc]

Their guys have now fixed this situation.

Their solution was — :drumroll: — to move the machine back to its previous physical location.

Edit: fixed there/their mistake

[dkf]

@Zecc said in WebSocket connection dropping for reasonably large messages:

Their solution was — :drumroll: — to move the machine back to its previous physical location.

Ok, so it was probably some weird firewall appliance (many of which have severe problems with websockets) and they're now not connecting via that. I wish I was surprised.

[BernieTheBernie]

@Zecc said in WebSocket connection dropping for reasonably large messages:

move the machine back to its previous physical location

That can be a very efficient mode of data transfer, too.

[aitap]

I know I'm late to the party, but this sounds like an overzealous firewall that blocks the ICMP messages required to discover a safe MTU value for a TCP connection.

A workaround for that is called MSS clamping, which sets maximum packet size for a connection based on a value from the TCP header, not using ICMP in the process. There's a lot of HOWTOs on the Internet if you search for --clamp-mss-to-pmtu.

[Zecc]

@aitap Thanks. I'm not looking at this at the moment or the predictable future, but this sort of information is invaluable.

Or valuable? Stupid English language.

[Unperverted Vixen]

@Zecc Either one works. “Invaluable” is a superlative form of “valuable”.

[PleegWat]

@Zecc Some things are priceless. For the rest...

[Zecc]

@Unperverted-Vixen said in WebSocket connection dropping for reasonably large messages:

@Zecc Either one works. “Invaluable” is a superlative form of “valuable”.

I know. I was being facetious. (ˈfeɪsʃəs)

[Unperverted Vixen]

@Zecc Sorry - I know we have multiple non-native English speakers here so I made the mistake of trying to be helpful. I’ll make sure to not do that again.

[error]

@Unperverted-Vixen said in WebSocket connection dropping for reasonably large messages:

the mistake of trying to be helpful.

What category is this, anyway?

[Zecc]

@Unperverted-Vixen said in WebSocket connection dropping for reasonably large messages:

@Zecc Sorry - I know we have multiple non-native English speakers here so I made the mistake of trying to be helpful. I’ll make sure to not do that again.

Since it's difficult to transmit tone in writing, and I honestly can't tell whether you were being sarcastic or not, let me be perfectly clear: as non-ESL, I will _always appreciate corrections even when I'm making silly jokes (which might have happened once or twice).

I try to make my jokes obvious, but I understand it's not always the case.

And the same goes for typos.

[Applied Mediocrity]

@Zecc said in WebSocket connection dropping for reasonably large messages:

it's difficult to transmit tone in writing

If your intonations are pure, just write down the fundamental frequency.

[HardwareGeek]

@Applied-Mediocrity

[Unperverted Vixen]

@Zecc I was not being sarcastic, so thanks for the clarification. In this case, your joke(s?) went entirely over my head.