NAT traversal and stuff

[aitap]

I stumbled upon a nice work Peer-to-Peer Communication Across Network Address Translators, Bryan Ford, Pyda Srisuresh, and Dan Kegel. USENIX Annual Technical Conference, April 10-15, 2005 and decided to reproduce some of its results.

The first thing you notice when reading it is that TCP NAT traversal requires multiple sockets bound to the same address (one for the introducer server - should be kept alive, lest the NAT table entry is removed - one for the outgoing connection, one for the incoming connection - in practice, sometimes it's possible to get by without the third socket, but the theory says that the third socket still may be needed). It is not normally allowed in Berkeley sockets API to have multiple sockets bound to the same local address, but one can set SO_REUSEADDR and/or SO_REUSEPORT options (depending on the OS) to fix that.

I would like to ask you to compile and run the following code and tell me if it produces any errors. I am also interested whether it would produce any errors if you build it with #define NO_REUSEPORT on non-Windows.

Now, I know how bad it looks (running someone else's code written in C of all languages and letting it connect to stuff? and call listen()? what he thinks he's doing? lives have been ruined this way), but the program is really small (164 LoC), does not have any outside dependencies, and I tried to comment it so you could read it and see that it only creates three sockets, connects one of them to example.org:80 and binds the other two to the same address the first one is bound to - I just need to know how portable is it to bind multiple sockets to the same address, which options are required to do that and which are not. I am particularly interested in results from Apple-verse, since I don't have access to the hardware myself.

#ifdef _WIN32
	#define NO_REUSEPORT
	#include <winsock2.h>
	#include <ws2tcpip.h>
	#define close closesocket
	/* XXX: link with Ws2_32.lib */
#else
	#define _POSIX_C_SOURCE 201112L /* request Berkeley sockets API when strict C standard required */
	#define _BSD_SOURCE /* NI_MAX... visible with _GNU_SOURCE on new glibc, but requires _BSD_SOURCE explicitly on BSD */
	#define _GNU_SOURCE
	#include <errno.h>
	#include <netdb.h>
	#include <sys/socket.h>
	#include <sys/types.h>
	#include <unistd.h>
	typedef int SOCKET;
	enum { INVALID_SOCKET = -1 };
#endif

#include <stdio.h>

/* In real world use, this would be the introducer server. For the purposes of
 * this test, any server you could connect to will suffice. */
static const char *hostname = "example.org", *port = "80";

/* SO_REUSEADDR seems to be required for this to work. Not sure about
 * SO_REUSEPORT: it may be required on Linux (but is only available on Linux
 * > 3.9); it has been available on BSDs (I haven't checked yet) and is not
 * needed on Windows, where SO_REUSEADDR already provides necessary changes. */
static int setup_socket(SOCKET sock) {
	int ret = -1;
#ifdef _WIN32
	BOOL
#else
	int
#endif
		opt = 1;

	printf(" setsockopt(SO_REUSEADDR");

	if (setsockopt(
		sock, SOL_SOCKET, SO_REUSEADDR,
#ifdef _WIN32
		(const char *)
#endif
		&opt, sizeof opt)
	)
		goto cleanup;
#ifndef NO_REUSEPORT
	printf(",SO_REUSEPORT");
	if (setsockopt(sock, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof opt))
		goto cleanup;
#endif
	ret = 0;

cleanup:
		printf(")");
		return ret;
}

int main() {
	/* C89 requires declarations before code. */
	char hostaddr[NI_MAXHOST], localserv[NI_MAXSERV];
	SOCKET socket_conn = -1, socket_bind = -1, socket_listen = -1;
	struct addrinfo hints = {0}, *server = NULL;
	struct sockaddr_storage connaddr;
	socklen_t addrlen = sizeof connaddr;
#ifdef _WIN32
	WSADATA wsa;
	{
		int result = WSAStartup(MAKEWORD(2,2), &wsa);
		if (result) {
			printf("WSAStartup failed, %d\n", result);
			return -1;
		}
	}
#endif

	setbuf(stdout, NULL); /* Disable printf() buffering. */

	/* In real world use, first socket is connected to the introducer server
	 * to learn the address of the other party. All connections to the other
	 * party should be made from the same local port as the connection to the
	 * introducer server in the hope of getting the same outbound port on the
	 * other side of NAT. */
	printf("First socket:");

	hints.ai_family = AF_UNSPEC;
	hints.ai_socktype = SOCK_STREAM;
	{
		int result;
		printf(" getaddrinfo(%s)", hostname);
		result = getaddrinfo(hostname, port, &hints, &server);
		if (result) {
			printf(" failed, %d\n", result);
			goto cleanup;
		}

		/* This is not required, but I wanted to print the resolved address. */
		printf(" getnameinfo()");
		result = getnameinfo(server->ai_addr, server->ai_addrlen, hostaddr, sizeof hostaddr, NULL, 0, NI_NUMERICHOST);
		if (result) {
			printf(" failed, %d\n", result);
			goto cleanup;
		}
	}

	/* XXX: We are assuming that the first address in the list returned by
	 * getaddrinfo() is viable. In real world usage, we should iterate over it,
	 * retrying both socket() and connect() calls until we get a result. */
	printf(" socket()");
	socket_conn = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
	if (socket_conn == INVALID_SOCKET) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	/* For this trick to work, all participating sockets should allow address
	 * and port reuse. */
	if (setup_socket(socket_conn)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" connect(%s)", hostaddr);
	if (connect(socket_conn, server->ai_addr, server->ai_addrlen)) {
		printf("failed, %d\n", errno);
		goto cleanup;
	}

	printf(" ok\n");
	/* By the way, this connection should stay open, lest the NAT would stop
	 * translating the incoming packets for us. */

	/* In real world usage, the second socket is used to connect() to the
	 * address given to us by the introducer, while the other party tries to
	 * connect() to us. Sometimes it even works, but see below.
	 * The connection attempt should stem from the same source port, hence the
	 * setsockopt() shenanigans. */
	printf("Second socket:");
	printf(" getsockname()");
	if (getsockname(socket_conn, (struct sockaddr *)&connaddr, &addrlen)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	{
		/* Again, this is not required, but I wanted to print the local port. */
		int result;
		printf(" getnameinfo()");
		result = getnameinfo((struct sockaddr *)&connaddr, addrlen, hostaddr, sizeof hostaddr, localserv, sizeof localserv, NI_NUMERICHOST|NI_NUMERICSERV);
		if (result) {
			printf(" failed, %d", result);
			goto cleanup;
		}
	}

	/* We are trying to "reproduce" the first socket as close as we can, so no
	 * iteration should be going on here. */
	printf(" socket()");
	socket_bind = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
	if (socket_bind == INVALID_SOCKET) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	if (setup_socket(socket_bind)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" bind(%s,%s)", hostaddr, localserv);
	if (bind(socket_bind, (struct sockaddr *)&connaddr, addrlen)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" ok\n");

	/* Despite double-connect() trick seems to work on both Windows and Linux
	 * for some people, it is not guaranteed to work. Instead, we should also
	 * create a listening socket on the same port as above and wait for
	 * incoming connections there. Only one of the two sockets will ever get a
	 * working connection, since the tuple { protocol, local address, local
	 * port, remote address, remote port } uniquely identifies a connection
	 * and we got the first three to be the same for all three sockets.
	 * We may have to use SO_REUSEPORT for the third socket. */
	printf("Third socket:");
	printf(" socket()");
	socket_listen = socket(server->ai_family, server->ai_socktype, server->ai_protocol);
	if (socket_listen == INVALID_SOCKET) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	if (setup_socket(socket_listen)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" bind(%s,%s)", hostaddr, localserv);
	if (bind(socket_listen, (struct sockaddr *)&connaddr, addrlen)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" listen()");
	if (listen(socket_listen, 1)) {
		printf(" failed, %d\n", errno);
		goto cleanup;
	}

	printf(" ok\n");

cleanup:
	if (socket_conn != INVALID_SOCKET) close(socket_conn);
	if (socket_bind != INVALID_SOCKET) close(socket_bind);
	if (socket_listen != INVALID_SOCKET) close(socket_listen);
	if (server) freeaddrinfo(server);
#ifdef _WIN32
	{
		int result = WSACleanup();
		if (result)
			printf("WSACleanup failed, %d\n", result);
	}
#endif
	return 0;
}

[topspin]

Missing includes on macOS.

topspin@Macbook-Air:~> uname -v
Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64
topspin@Macbook-Air:~> clang++ --version
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
topspin@Macbook-Air:~> clang++ nat.cpp
nat.cpp:51:35: error: use of undeclared identifier 'SO_REUSEPORT'
        if (setsockopt(sock, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof opt))
                                         ^
nat.cpp:63:16: error: use of undeclared identifier 'NI_MAXHOST'
        char hostaddr[NI_MAXHOST], localserv[NI_MAXSERV];
                      ^
nat.cpp:63:39: error: use of undeclared identifier 'NI_MAXSERV'
        char hostaddr[NI_MAXHOST], localserv[NI_MAXSERV];
                                             ^
nat.cpp:151:108: error: use of undeclared identifier 'localserv'
  ...addrlen, hostaddr, sizeof hostaddr, localserv, sizeof localserv, NI_NUME...
                                                           ^
nat.cpp:151:90: error: use of undeclared identifier 'localserv'
  ...sockaddr *)&connaddr, addrlen, hostaddr, sizeof hostaddr, localserv, siz...
                                                               ^
nat.cpp:172:35: error: use of undeclared identifier 'localserv'
        printf(" bind(%s,%s)", hostaddr, localserv);
                                         ^
nat.cpp:201:35: error: use of undeclared identifier 'localserv'
        printf(" bind(%s,%s)", hostaddr, localserv);
                                         ^
7 errors generated.

[topspin]

@topspin said in NAT traversal and stuff:

Missing includes on macOS.

Actually the culprit was _POSIX_C_SOURCE on line 8.
From their <sys/socket.h>:

#if !defined(_POSIX_C_SOURCE) || defined(_DARWIN_C_SOURCE)
#define SO_REUSEPORT    0x0200          /* allow local address & port reuse */
#define SO_TIMESTAMP    0x0400          /* timestamp received dgram traffic */
#define SO_TIMESTAMP_MONOTONIC  0x0800  /* Monotonically increasing timestamp on rcvd dgram */
#ifndef __APPLE__
#define SO_ACCEPTFILTER 0x1000          /* there is an accept filter */
#else
#define SO_DONTTRUNC    0x2000          /* APPLE: Retain unread data */
                                        /*  (ATOMIC proto) */
#define SO_WANTMORE     0x4000          /* APPLE: Give hint when more data ready */
#define SO_WANTOOBFLAG  0x8000          /* APPLE: Want OOB in MSG_FLAG on receive */


#endif  /* (!__APPLE__) */
#endif  /* (!_POSIX_C_SOURCE || _DARWIN_C_SOURCE) */

Program output (macOS Mojave 10.14.4):

First socket: getaddrinfo(example.org) getnameinfo() socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) connect(93.184.216.34) ok
Second socket: getsockname() getnameinfo() socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) bind(172.20.10.6,60164) ok
Third socket: socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) bind(172.20.10.6,60164) listen() ok

[aitap]

@topspin said in NAT traversal and stuff:

Actually the culprit was _POSIX_C_SOURCE on line 8.

Thanks a lot! This is really helpful. I had expected problems with NI_MAX... (which is not the point anyway), but not with SO_REUSEADDR being invisible.

[Tsaukpaetra]

Oh yeah, I was going to try it on FreeBSD. Thanks for the reminder! I'll... Put a pin on it, since I'm still in bed.

[PJH]

$ uname -a
Linux pjh-Aspire-ES1-533 5.2.9-050209-generic #201908160940 SMP Fri Aug 16 09:42:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/issue
Ubuntu 19.04 \n \l

$ gcc --version
gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ make sock
cc     sock.c   -o sock
In file included from /usr/include/errno.h:25,
                 from sock.c:11:
/usr/include/features.h:185:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [-Wcpp]
 # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE"
   ^~~~~~~

$ ./sock 
First socket: getaddrinfo(example.org) getnameinfo() socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) connect(93.184.216.34) ok
Second socket: getsockname() getnameinfo() socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) bind(192.168.1.65,44380) ok
Third socket: socket() setsockopt(SO_REUSEADDR,SO_REUSEPORT) bind(192.168.1.65,44380) listen() ok

After introducing #define NO_REUSEPORT,

First socket: getaddrinfo(example.org) getnameinfo() socket() setsockopt(SO_REUSEADDR) connect(93.184.216.34) ok
Second socket: getsockname() getnameinfo() socket() setsockopt(SO_REUSEADDR) bind(192.168.1.65,44388) ok
Third socket: socket() setsockopt(SO_REUSEADDR) bind(192.168.1.65,44388) listen() ok

[aitap]

@PJH Thank you! So far it looks like I won't need SO_REUSEADDR except on *BSD and macOS.

[Tsaukpaetra]

Yeah, just slapping the code from OP in...

Changing it to SO_REUSEADDR apparently succeeds the compile, but fails the execute:

[aitap]

@Tsaukpaetra Thanks!

Hmm, so SO_REUSEPORT is only available if __BSD_VISIBLE is true, and the latter is only defined if one does not request standard C or POSIX. And there doesn't seem to be a feature test macro to enable FreeBSD symbols back, unlike _BSD_SOURCE in OpenBSD. Curiouser and curiouser!

I guess it is my fault for requesting POSIX compatibility when I explicitly need an option not defined in POSIX. I shouldn't ask for it at all if I detect *BSD or __APPLE__.

[aitap]

Are there any books or HOWTOs on development of application-level network protocols? Specifically, the serialization part? I've been obsessing over this question and on-and-off binge-searching the Web for a few years now, but haven't started implementing anything for fear of getting things wrong. I would like the protocol to be implementable with (1) as little as possible potential for getting things wrong and (2) little to no dependencies. A person who taught me an elective on back-end C++ recommended me to (1) read RFCs for inspiration, (2) separate semantics from serialization in the protocol description and (3) read Stevens' TCP/IP Illustrated. Which kinda helped to build understanding, but didn't get me anywhere further.

From the looks of it, a simple binary protocol looks hard to get wrong: read one-byte header off the line, determine whether there's a need to read a payload, do stuff; while parsing a text-based protocol involves, well, a parser, a grammar, maybe even a buffer for variable-length strings - all those things that programmers get wrong and/or introduce corner cases in. One could say "Oh, it's just JSON (over HTTPS)", but such a framework is too general and makes it possible to describe so much more stuff different from what I need, which means that a peer expecting something simple along the lines of {"type":"INCOMING", "peer":"c30fde008b867f14ab04cd449bdce5585c5ffb26bed05928a7ebe7d6384d6096"} must be ready to receive a big structured turd (like a MongoDB dump - valid JSON, just not the kind we need) instead and still not crash. And while everyone speaks HTTPS, it would present another dependency with its own attack surface. It is, of course, possible to do that securely, but might be harder. Or not?

On the other hand, I've just (some people would say ill-advisedly) finished esr's The Art of UNIX Programming and read his post about the virtues of self-describing data. And it's even possible to parse simple JSON without building a tree of it, so maybe receiving a War and Peace equivalent in JSON world should not present such a problem for our hypothetical JSON-speaking peer. (Still, microjson only works with complete JSON representations, which might require buffering data and/or limiting the message size.) Also, JSON does not solve the problem of representing values like IP addresses and key fingerprints as strings and parsing them back: while it's easy to describe an easily comparable binary representation for them (IPv6 address: 16-byte unsigned integer, network order. Done!), there may be multiple ways to make strings of them (upper case or lower case hexadecimal digits? full or simplified representation of IPv6 address?), which all must be handled (harder to implement) or only one way to stringify the data should be presented (harder to describe the protocol).

And exchanging IP addresses and certificate fingerprints is the easy part. Later I hope to design another protocol to merge Merkle trees across a previously established P2P connection.

Perhaps a type-length-value format (Bencode?) with binary payloads is a better idea, but that means hand-parsing, which is hard to get right. There is, indeed, a whole spectrum of already existing formats, some of them human-readable, others binary; some requiring a schema, others fully self-describing; implemented as libraries or code generators. Maybe I ought to choose one of them and stop listening to my NIH syndrome - dependencies, after all, mean code reuse and less work for the implementor - but it's so hard to choose the right one!

[Tsaukpaetra]

@aitap said in NAT traversal and stuff:

development of application-level network protocols?

So what I'm hearing is "Should I roll my own or reuse something that I can find that 'fits the best'".

The answer of course, is another set questions/considerations:

• How much time do you want to invest in this?
• Do you need other applications to use it?
• How resilient should this protocol be (disconnects, replay attacks, impersonation, MITM, encryption)

Getting a naive protocol up and running is pretty simple if you make a lot of assumptions, but has a lot of potential tripping points that you'd have to be aware of or discover yourself. For example, Hypatia's voice protocol was built ground-up and has not-a-few places of weakness, but it works well for our purposes and took a year to develop in-house across three people.

@aitap said in NAT traversal and stuff:

Maybe I ought to choose one of them and stop listening to my NIH syndrome

Maybe, but you're just shifting the time sink from the development of the actual protocol to the adaptation of your application to the library's way of doing things, which may be simple or hard, but it still not-negligible.

[dkf]

@aitap said in NAT traversal and stuff:

a simple binary protocol looks hard to get wrong

Bwahahahahahahaaaa! Nurse! I want to go back to my padded cell, at once!

But seriously, binary protocols have their own special set of shit sandwiches. For one thing, they're really quite difficult to debug to start with; text-based protocols can at least be examined easily in the protocol dump (with protocols layered on top of HTTP, you probably don't even need a custom debugging tool to see what is going on). For another, if you get a read wrong, you get total garbage and have no real chance of resetting things into a known state. I've done far too much debugging of binary protocols and their implementations over the past few years, and it's really difficult and yucky.

For your first version, sending JSON over HTTP is relatively easy to make work, especially if you assume that all clients are non-malicious. It might not be great, but it at least you get started and can then think about where you want to go from there. The description object trees (i.e., the things that you serialize as JSON) that you encode to send over should be just what makes sense for your application. And JSON serialization libraries for many programming languages are actually quite good, and HTTP implementations are debugged; not having to debug those parts will save you quite a lot of hair. Or virtual hair, if you're already bald.

[aitap]

@Tsaukpaetra said in NAT traversal and stuff:

• How much time do you want to invest in this?

Well, is "about as much as required to get it right" a meaningful answer? Is it even possible to "get right" a protocol?

• Do you need other applications to use it?

Ideally, I want it to be easy to write an independent implementation of the protocol for someone else in a language I don't even know about, but chances are, no-one ever will be interested in it at all.

• How resilient should this protocol be (disconnects, replay attacks, impersonation, MITM, encryption)

[D]TLS with key pinning shall be the inner layer of the protocol, so I'm assuming I've got the security aspect covered. Disconnects? Yes, one of the purposes of this exercise is to recover after partial transfers.

For example, Hypatia's voice protocol was built ground-up and has not-a-few places of weakness, but it works well for our purposes and took a year to develop in-house across three people.

Thanks, that's an important data point.

@dkf said in NAT traversal and stuff:

I've done far too much debugging of binary protocols and their implementations over the past few years, and it's really difficult and yucky.

Thank you for the warning. I needed it.

For your first version, sending JSON over HTTP is relatively easy to make work, especially if you assume that all clients are non-malicious.

I really don't want to make assumptions like that, but perhaps it's unavoidable in a hobby project.

not having to debug those parts will save you quite a lot of hair

That's true. Again, perhaps only people with infinite time can afford not having dependencies. And it's certainly not the case for a hobby project.

[PJH]

@aitap said in NAT traversal and stuff:

application-level network protocols

One I researched, and contemplated, but never ended up using (project cancelled) was:

Protocol Buffers

[dkf]

@aitap said in NAT traversal and stuff:

I really don't want to make assumptions like that, but perhaps it's unavoidable in a hobby project.

Again, with a lot of experience, you won't be able to avoid making assumptions until you have several groups of people all writing independent implementations of both sides of the communication. The amount of experience required to reliably get this sort of thing right with your first attempt is probably more than any ordinary programmer can get across their whole career. It's a lot easier to make mistakes and then go back and change things when you find you got it wrong, even if that takes quite a bit of work.

Also, be ultra-clear in your docs/design notes about those assumptions that you're aware of making. You'll miss lots, but it's dangerous to entrust that sort of thing wholly to memory. (I'm totally guilty of this. Don't make my mistakes; make your own!)

But it's not a horrible mistake to use JSON (or XML) as a transport encoding over HTTP (it's the basic network protocol) since they're pretty mature technologies and you can get good quality implementations as libraries instead of writing your own. The combination will largely reduce the communication challenge down to “am I preparing the right documents” and “am I sending them to the right place” and that's a great win.

[Deadfast]

@PJH said in NAT traversal and stuff:

@aitap said in NAT traversal and stuff:

application-level network protocols

One I researched, and contemplated, but never ended up using (project cancelled) was:

Protocol Buffers

Alternatively:

FlatBuffers: FlatBuffers

Unless you're writing the protocol for educational purposes I would strongly suggest using this. One useful feature is that you can easily toggle between binary and JSON.

[aitap]

@PJH @Deadfast Thanks! I've seen protobufs used in strangest places (e.g. home-grown machine learning framework for search engine design elective). I should try them, too.

Sending raw Protocol Buffers / FlatBuffers / JSON over TLS does not seem to be the way, since most parsers don't offer a way to read a partial message and ask for more, retaining the state (or stop when one message ends and allow to treat trailing bytes as another message later).

@dkf said in NAT traversal and stuff:

The amount of experience required to reliably get this sort of thing right with your first attempt is probably more than any ordinary programmer can get across their whole career.

I guess this explains the lack of books/courses on application protocol design.

@dkf said in NAT traversal and stuff:

But it's not a horrible mistake to use JSON (or XML) as a transport encoding over HTTP

But pick up a too high level abstraction, and I wouldn't be able to perform the low-level TLS and socket fuckery I want. I'm also afraid of ending up like XMPP and spending more traffic on auxiliary information than messages themselves. The way forward is trying it and seeing what fails. I'll try to get a prototype working soon.