Wtfbuild: A build system for the next decade.

PleegWat

The only time I had to deal with cmake (and I do believe it was cmake) was when I had an open-source utility on ubuntu (pdwtags) and wanted a binary on RHEL5. The build system in question wasn't available on RHEL5, couldn't be built for it, so I had to magic up the compile commands myself. Took me at least an hour or so.

I still use that binary; it can be a very useful tool.

blakeyrat

@Bulb said:

Macos

powerlord

@blakeyrat said:

If it's AWARE of them, that's better than most tools.

Even though you've never managed to get TortoiseGit working, it's supported Submodules for years.

Although, I have noticed it doesn't seem to have an option to remove submodules, only to add or update them.

@asdf said:

I wouldn't be so sure. Python and Java builds may be complicated in different ways than the average C++ build.

I have a... 14 project Java Maven build that includes signing some of the jar files it builds.

It still takes only 5 minutes to build.

Speaking of Java builds, Java devs worth their salt are likely already using Maven, Ivy, Gradle, or something like them. Maven even has a standard directory layout that it encourages devs to use. Makes it much easier to figure out where a project's files are. I know Gradle copied this layout wholesale, not sure about other Java build environments.

insert jellypotato here

@Kian said:

@ben_lubar said:
What system do you control which version of the dependencies get used with?

Well, wtfbuild will manage that for me. In ten years, give or take.

Great... the build system I use for my Java projects does that already. As does the less insane system I considered replacing it with.

Get on that, chop chop!

Kian

Well, miniz is pretty handy for extracting zip files. Pleasantly surprised by how straightforward the interface is. Specially for a single file C library.

sloosecannon

@powerlord said:

Speaking of Java builds, Java devs worth their salt are likely already using Maven, Ivy, Gradle, or something like them. Maven even has a standard directory layout that it encourages devs to use. Makes it much easier to figure out where a project's files are. I know Gradle copied this layout wholesale, not sure about other Java build environments.

QFT!!!!!!!!!!!

Filed Under: mvn clean test package

Bulb

@asdf said:

They recently added support for building Visual Studio solutions. A quick Google search also showed that there's an XCode plugin.

I still don't understand what exactly it is you're trying to do, though. A better explanation might certainly help me answer your questions.

I simply need to build a C++ application for all the mentioned platforms (Windows Phone 8, Windows Universal, iOS, Windows CE and Android), properly packaged (xap, appx, ipa, apk), and so that the debugger works.

So I need some projects for the respective IDEs. And custom build step-based ones did not work well for Windows CE when I tried some years ago with scons. So so far, generating them for the native build system was the only option that worked.

OffByOne

@PleegWat said:

I had an open-source utility on ubuntu (pdwtags)

$ apt-cache search pdwtags
dwarves - set of advanced DWARF utilities

Paging @ben_lubar

asdf

@Bulb said:

I simply need to build a C++ application for all the mentioned platforms (Windows Phone 8, Windows Universal, iOS, Windows CE and Android), properly packaged (xap, appx, ipa, apk), and so that the debugger works.

Ok, the latest Gradle version should(TM) support all that:

Note that I personally don't have experience with generating Visual Studio solutions from Gradle or with the Gradle XCode plugin. None of my projects supports Windows Phone, Windows CE or iOS.

OffByOne

@tar said:

You don't have any control over the continued availability of that server.

@blakeyrat said:

us old codgers who keep our products ENTIRELY on our own servers

I see my pre-emptive
@OffByOne said:

or another repo on your company git server

failed...

@tar said:

The point of version control is to store the entirety of your project, as far as I'm concerned. Anything less than that is adding unwarranted complexity.

IMO it depends on your use case. If you're talking about a work project, keeping the entire source for your project and its dependencies in the repo for your project might be the best option.
I don't see any harm in using submodules (or whatever equivalent mechanism your VCS of choice offers) to point to those dependencies instead of including the sources directly.

What about projects where you can choose to link them to either the system-wide version of a library or the included (either directly or through the submodule pointer)?
It makes sense to me to not include the source of dependencies which might very well not be used, but at the same time make it easy (if you know how to work with submodules, which isn't that straightforward to be honest) to pull those sources if desired.

Bulb

@asdf said:

should(TM)

It certainly looks Promising™.

Not that I am rewriting the beast that is our build system anytime soon, because that thing is huge (mess).

asdf

@Bulb said:

Not that I am rewriting the beast that is our build system anytime soon, because that thing is huge (mess).

Yeah, last time I did something like that (complete rewrite of a complex build), it took me 3 months until everything worked like it did before. But gradually rewriting a messy build can be a good idea, and Gradle is certainly not the worst choice.

Bulb

@asdf said:

But gradually rewriting a messy build

Yeah, problem is that it is all tied in one huge knot.

I actually started to make the libraries (the 3rd party ones that are known to be independent first) compilable independently, but didn't have time to finish even that yet.

Kian

Another weekend, another progress update. As long as it keeps my interest, anyway.

I was pleasantly surprised by how clear MSDN's tutorial for Winsock2 is, and how straightforward opening a socket and listening on it was. Never had to do it before, so I was a bit unsure of how it would interact with a firewall, for example. Apparently that was all handled without me having to worry about it. First time I ran the code with the networking bits added, windows asked me if I wanted to let it through the firewall and it all just worked. I could then connect through telnet and direct the application.

Up to now, when I ran the app it read its config (still hardcoded), got the latest commit hash from git, tried to build that revision, and shut down. With the network code, the app now waits for a connection, waits for a commit hash to be sent through (right now just plain text), and that triggers the app to build that revision. It can handle multiple connections and handle the requests simultaneously, which is pretty neat. I have been connecting to localhost, though, so when I have a bit more time I have to copy it and make it run in another computer and see if it works as expected.

Also, I don't know how to kill it now that it runs in a loop listening for connections. I need to learn what an asynchronous procedure call is, and how to schedule one when I want to shut down. For a clean shutdown, anyway. Obviously the task manager and Ctrl+c work.

PleegWat

Do you need to build your own RPC call for that? I'd expect if you set it up as a windows service that is handled for you.

Kian

@PleegWat said:

Do you need to build your own RPC call for that?

"That" what, exactly? The shutdown thing? Actually, after looking a bit it's much easier than that.

The issue is that after I start listening for connections, the accept function blocks ( https://msdn.microsoft.com/en-us/library/windows/desktop/ms737526(v=vs.85).aspx ). There's a note there that says:

Note When issuing a blocking Winsock call such as accept, Winsock may need to wait for a network event before the call can complete. Winsock performs an alertable wait in this situation, which can be interrupted by an asynchronous procedure call (APC) scheduled on the same thread. Issuing another blocking Winsock call inside an APC that interrupted an ongoing blocking Winsock call on the same thread will lead to undefined behavior, and must never be attempted by Winsock clients.

However, a more straightforward way of handling the situation appears to be to close the listening socket. That will cause accept to fail, and I can then break out of the loop, stop accepting connections and wait for the current requests to finish before quitting. I think, I'll have to test it later.

I don't want to set it up as a Windows Service, at least not quite yet, for two main reasons. One, this is in part a learning exercise. Resorting to frameworks that hide away all the interesting bits would be counter-productive. And two, I want to try porting it to nix eventually, and everything I off-loaded on a framework has to be redone anyway. Although I haven't looked into how much setting it up as a service would cost, and how much it gives in return.

Kian

Also, I know this isn't news to anyone, but I'm surprised by what huge dicks people answering questions on the internet are. Not surprisingly, "how do I interrupt a blocking accept call" is a fairly common question (always for the same reason, wanting to shut down cleanly). The common answer from shitty programmers is "don't use accept, use select or some other polling method". Which is stupid, and doesn't answer the question.

Closing the listening socket (if it works) would be the cleanest solution, since the first thing I want to do when I start shutting down is stop listening for new connections. The proposed alternative, using some polling method, is more cumbersome and requires having a pipe you write a byte to to wake up early and I don't know how many more rube-goldberian contortions.

PleegWat

Ah, I guess I misread your question. And was only taking a potshot anyway since I don't really have windows experience. Sorry.

Kian

I didn't mean the part about people answering on the internet about anyone here, in case that's what you were apologizing for. I meant that I looked for the answer, found a lot of people asking that same question, and a lot of people answering unhelpfully.

PleegWat

Well, on linux, this is actually messy, and I think you do need pselect or one of its friends to avoid race conditions.

The thing is you're usually shutting down based on a signal (could be an intra-process one). But you don't know when that signal arrives. Illustrative example, forgive me getting callback signatures wrong:

static exit = 0;

void sigterm_handler()
{
    exit = 1;
}

/* in some function */
while(1)
{
    if( !exit )
    {
        accept(...);
    }
}

The above code contains an important race condition: although the shutdown signal interrupts the accept, there is a real risk it arrives between the check on exit and the call to accept. If it does, the shutdown will not be detected until after the next session arrives, or another signal (with a handler) is received. However, in typical scenarios that next signal will be SIGKILL because you did not shut down when asked to.

The only solution (that I know of) is to block the shutdown signal, and use pselect to unblock it only while waiting for the signal to come in.

This may work differently on windows.

Bulb

@Kian said:

Closing the listening socket (if it works) would be the cleanest solution, since the first thing I want to do when I start shutting down is stop listening for new connections. The proposed alternative, using some polling method, is more cumbersome and requires having a pipe you write a byte to to wake up early and I don't know how many more rube-goldberian contortions.

In majority of cases, the proposed alternative, using a polling function, is simpler.

You can only close the socket from a different thread. That is fine if you are receiving the shutdown request over the network protocol and you are already handling it in a separate thread.

However, most servers don't do that. On Unix, the only options are another socket, pipe or signal and on Windows there is another options of window message or event. And appropriate polling function will wake up to any of these—select for pipes and sockets, pselect for signals on Unix and on Windows MsgWaitForMultipleObjects (together with WSASelect to get associate the socket with a HANDLE) can wait on sockets, events and windows messages. So you just handle the terminate request in the loop and don't need another thread. That is simpler in most cases, especially since the thread waiting for the terminate request would need to use similar polling function anyway (except unix signals).

@PleegWat said:

The above code contains an important race condition: although the shutdown signal interrupts the accept, there is a real risk it arrives between the check on exit and the call to accept.

I think the easier way is to longjmp out of the signal handler and the loop. That does not have a race condition. Which shows that accept can, actually, be interrupted.

@PleegWat said:

This may work differently on windows.

Totally differently. Windows don't have signals.

PleegWat

@Bulb said:

I think the easier way is to longjmp out of the signal handler and the loop.

This has its own problems, as you never actually exit the signal handler, and you may still be holding locks without realizing it. For example, if the thread was writing to its log file (using stdio) when the interrupt fired, and you jump away, the next attempt to access the log file descriptor will deadlock.

Kian

@Bulb said:

You can only close the socket from a different thread. That is fine if you are receiving the shutdown request over the network protocol and you are already handling it in a separate thread.

Shutting it down from the network is one possibility I would like to be able to handle, and handling sockets in separate threads is already what I'm doing. I mean, is there some other way of handling multiple connections?

PleegWat

Nonblocking with select().

blakeyrat

I dunno modern C++, but the old-fashioned C way was using select().

@PleegWat said:

Nonblocking with select().

^- yah like that

PleegWat

You could probably pull of CPS in C++. Though you may have to write more support code than it's worth...

blakeyrat

I assume select() doesn't poll on modern systems, right? I guess it depends on the implementation. I can't find any documentation saying one way or the other.

PleegWat

The manpage doesn't say; I strongly suspect it depends on the implementation.

In addition to select/pselect there is also poll/ppoll, which are the same functionality under a different API. Although I'm pretty sure poll does use polling at least for the first millisecond or so - it sure spends enough CPU for it.

blakeyrat

The presence of the similar function poll() makes it impossible to search for "does select() poll?" in Google and get useful results.

This annoys me.

swayde

@blakeyrat said:

it impossible to search for "does select() poll?" in Google

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740141(v=vs.85).aspx

If TIMEVAL is initialized to {0, 0}, select will return immediately; this is used to poll the state of the selected sockets.

Behind the scenes ? No one seems to know...

ben_lubar

You could just close whatever listener is being given to accept from the sigterm_handler. That way, assuming you do error checking, you wouldn't need the data race.

Bulb

@PleegWat said:

This has its own problems, as you never actually exit the signal handler, and you may still be holding locks without realizing it.

True. It should be better with exceptions, but I suspect there is actually some trouble there too.

To be honest, signals are a gross hack.

@Kian said:

Shutting it down from the network is one possibility I would like to be able to handle, and handling sockets in separate threads is already what I'm doing. I mean, is there some other way of handling multiple connections?

You can do it in one with select/poll/MsgWaitForMultipleObjects.

Also don't forget you have other things to wait for like the compiler processes. Waiting on child processes is also very different between Windows and Unix. Most operations you will do is waiting for something. So you can either have a really large number of threads, or you'll have to have some event loops.

I would choose the event loop route, because locking is more error-prone and there is nothing CPU-intensive in the build server itself to take advantage of loading multiple cores.

I'd pick some library with decent portable event loop (GLib(mm), QtCore, Boost.Asio or something) and write it on top of that. Because as the discussion shows, some of these things are very different between Windows and Unix, and C++11 only fixes a very small subset of them.

@blakeyrat said:

I assume select() doesn't poll on modern systems, right? I guess it depends on the implementation. I can't find any documentation saying one way or the other.

The operation is, traditionally, called polling. But actually implementing it by periodically waking up and checking is so absurdly insane that I doubt even the early unices did that. They definitely don't now.

blakeyrat

@Bulb said:

They definitely don't now.

I can't find any documentation saying so. Can you?

Kian

So, if I'm understanding this correctly, the options are basically to wait in one thread for many events, or to wait in many threads for one event each. The threaded approach is easier to design and reason about (I don't have to identify the event and call a handler and do all that bookkeeping), but perhaps having a ton of threads waiting, while not CPU intensive, could be eating other resources?

Bulb

@blakeyrat said:

I can't find any documentation saying so. Can you?

It is so obvious that no documentation bothers to mention. It simply contradicts the observed performance. Hundreds of threads would totally hog the CPU if they were waking up periodically, but it is absolutely no problem, therefore it is not polling.

Of course, for Linux, just use The Source, Luke.

@Kian said:

So, if I'm understanding this correctly, the options are basically to wait in one thread for many events, or to wait in many threads for one event each. The threaded approach is easier to design and reason about (I don't have to identify the event and call a handler and do all that bookkeeping), but perhaps having a ton of threads waiting, while not CPU intensive, could be eating other resources?

It takes more memory, since each thread needs a stack, but it is that bad these days. But the threaded approach still does have its problems like that you have to cleanly shut down all the threads, not just the master.

blakeyrat

@Bulb said:

It is so obvious that no documentation bothers to mention.

It's not obvious to me.

@Bulb said:

Of course, for Linux, just use The Source, Luke.

Yeah, uh, if you can't answer the question just don't bother, k?

Kian

@Bulb said:

like that you have to cleanly shut down all the threads, not just the master.

Well, shutting down cleanly essentially means "finish doing what you were doing".

I suppose I have two situations to consider: one where shutdown means "stop accepting requests, finish the ones still outstanding, and quit" and another that means "abort any abortable operations and quit immediately but cleanly." In the first case, the program wouldn't exit until I close every connection from the client side (but no new connections would be made). In the second, the server would shut down the connections.

Bulb

Yes. Finish what you were doing is easy. But the connection might have a long outstanding queue of commands, or the compilation command might for some reason be stuck or something.

Kian

So, if I want to do something more elaborate, I could start a timer, give the threads a chance to finish, and then force them to finish. But then I have to make the threads smarter, and I'm back to waiting for multiple events. And if I'm doing that, might as well do it in one thread in the first place....

But for now, I think the simpler approach will work best while I work on everything else. After all, I already have the instant kill from the OS if something takes too long.

Bulb

Yes.

I am used to using event loops, so I'd probably go with one, but either approach will work. Just be aware of the pitfalls. Both approaches have them.

And I agree that "clean" exit is overrated. Operating system will clean up everything that matters if you just exit()/ExitProcess(). And you can't rely on always going through the "clean" exit anyway, because there are situations (e.g. power failure) where the process will simply cease to exist.