./configure and the horrors within

Minor WTF this time. This particular issue is found in several Apache products, notably including apr, apr-util and log4cxx. It has to do with the configure executable it comes with; this is there to make the install process simple ( ./configure; make; make install ), and it's great for that. Really great. Compilation is a breeze and finishes surprisingly quickly.

That's when something breaks. You go to inspect what's going on, but you're not quite sure what compiler flags are getting passed correctly; the output seems to indicate that CPPFLAGS is blank, but you know it isn't because you just set it, and echo $CPPFLAGS tells you exactly what you expect. What do you do? You dive into configure to find out why it seems to not be passing things correctly.

But beware, friend, for you have stumbled into one steaming morass of code. The configure tool turns out to be a shell script, which makes perfect sense. What doesn't make sense is that these files weigh in at a hefty 50,000 lines of code. And while large parts may be shared between projects, it appears that the files for apr, apr-util, and log4cxx are quite different.

Sigh. Maybe the WTF is on me this time; I've got a suspicion that digging into this beast is a world of trouble...but then, I do need it to compile. Once more, unto the breach!

smxlong

You know, the GNU toolchain expects CXXFLAGS, not CPPFLAGS...

I do actually know that, and that is what we tried first. But if you read ./configure --help, or look inside any of the configure files, you'll note that there are actually 0 occurrences of CXXFLAGS, total. I label this "Corrollary WTF 1.1".

kpu

configure is usually automatically generated by autoconf. In fact, it typically has a comment at the top e.g. "Generated by GNU Autoconf 2.62".

arty

First off, don't read./configure, it's the output of autoconf. Since it's really horribly generated code, made to run on every possible build of every bourne-like shell ever made, it's just ugly, has too much repetition and is just plain evil code to work on. If there's an autogen script, edit the autoconf input and rebuild configure.

The real WTF is perhaps that autoconf is so widely used yet so poorly documented, and has such horrible rules (even the ones not related to m4 cleanliness), such as the mingling of enable and with settings.

kpu

You might also read ./configure --help. Quoth:

Some influential environment variables:
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir=""> if you have libraries in a
              nonstandard directory <lib dir="">
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    C/C++/Objective C preprocessor flags, e.g. -I<include dir=""> if
              you have headers in a nonstandard directory <include dir="">
  CC          C compiler command
  CFLAGS      C compiler flags
  CPP         C preprocessor
  CXXCPP      C++ preprocessor
  F77         Fortran 77 compiler command
  FFLAGS      Fortran 77 compiler flags
</include></include></library></lib></lib>

DaveK1

You need this: the 'goat' book.

Configure scripts look scary, and that's because they are. But if you look at the corresponding configure.in, you'll find it's a lot smaller. Most of that spew is autogenerated recursively expanded macros generated from the m4 processor. At one point you refer to the shell script as an "executable", but actually that's a good way of looking at it, and if you wanted to debug an executable, you wouldn't go about it by ploughing through a hex dump, you'd go and start reading the source. So treat configure the same. Once you start to see it as blocks of code and groups of components and start seeing how the structure of the generated file relates to the macros invoked in the source, it'll seem a lot less scary.

Still fugly, though!

Oh, and yeah. CFLAGS = flags for compiling C, CXXFLAGS = flags for compiling C++, CPPFLAGS = flags for running the C pre-processor stand-alone.

Hmm, this submission ended up strikingly helpful and informative. I wondered whether it may have been an auto-generated file, but I'm just a junior part-time developer and I was following instructions from my immediate supervisor (who is technically attached to a different project at the moment...) to delve into configure. Upon further inspection, I'm not sure how I got it into my head that CXXFLAGS didn't occur anywhere, as cat configure | grep CXXFLAGS yielded pages and pages of occurrences.

Good call on looking at configure.in though, I wouldn't have known what was going on without that. The documentation I'm looking at (http://apr.apache.org/compiling_unix.html) is kinda sparse.

Overall, build processes here are kinda special. Apparently some of the other teams have dedicated "build teams", whose entire responsibility is to maintain the build scripts - and their 4-person team is apparently quite backlogged. I'm about 95% certain that this should terrify me, at least partially because the overflowed work from the build team flowed onto my supervisor, leaving her spending 6 weeks to incorporate a 4-day code modification into the build process.

Thanks, TDWTF forumers!

DaveK1

@DemonWasp said:

Hmm, this submission ended up strikingly helpful and informative. I wondered whether it may have been an auto-generated file, but I'm just a junior part-time developer and I was following instructions from my immediate supervisor (who is technically attached to a different project at the moment...) to delve into configure. Upon further inspection, I'm not sure how I got it into my head that CXXFLAGS didn't occur anywhere, as cat configure | grep CXXFLAGS yielded pages and pages of occurrences.

Oh, BTW, don't do this:

[code]$ export CXXFLAGS="-g -O2"
$ ./configure[/code]

or this:

[code]CXXFLAGS="-g -O2" ./configure[/code]

Do it like this:

[code]./configure CXXFLAGS="-g -O2[/code]

flaquito

@DemonWasp said:

Hmm, this submission ended up strikingly helpful and informative. I wondered whether it may have been an auto-generated file, but I'm just a junior part-time developer and I was following instructions from my immediate supervisor (who is technically attached to a different project at the moment...) to delve into configure. Upon further inspection, I'm not sure how I got it into my head that CXXFLAGS didn't occur anywhere, as cat configure | grep CXXFLAGS yielded pages and pages of occurrences.

smxlong

@DaveK said:

Do it like this:

./configure CXXFLAGS="-g -O2

Then leave for lunch as your program "configures." When you return, the shell is happily awaiting your entry of the terminating double-quote.

Qwerty

You do know that this forum software eats double-quotes for lunch? It is also quite fond of square brackets.

DaveK, thanks so much for this:

./configure CXXFLAGS="-g -O2"

Solved the issue I was having immediately before I left. I'm still running into other issues which may be related, but at least I know what I'm doing a little better now. Thanks for your help.

@flaquito said:

@DemonWasp said:
Hmm, this submission ended up strikingly helpful and informative. I wondered whether it may have been an auto-generated file, but I'm just a junior part-time developer and I was following instructions from my immediate supervisor (who is technically attached to a different project at the moment...) to delve into configure. Upon further inspection, I'm not sure how I got it into my head that CXXFLAGS didn't occur anywhere, as cat configure | grep CXXFLAGS yielded pages and pages of occurrences.
tag: Filed under useless use of cat award.

Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.

As a side note, this is part of the reason I fucking hate the tags, especially when people make a point in their tags. When I quote your text it doesn't make any sense because the tag doesn't get quoted.

arty

@DaveK said:

You need this: the 'goat' book.

Thanks for pointing that out. I take back what I said about poor documentation.

flaquito

@tster said:

Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.

As a side note, this is part of the reason I fucking hate the tags, especially when people make a point in their tags. When I quote your text it doesn't make any sense because the tag doesn't get quoted.

Whoa, sorry. Just trying to introduce some humor. After all, is it that hard to type 'grep sometext somefile.txt'?

As for making a point using tags, I've seen it done enough times on the forums, without vitriolic responses, that I assumed it was acceptable posting etiquette.

@Qwerty said:

You do know that this forum software eats double-quotes for lunch? It is also quite fond of square brackets.

You just need to know how to cook 'em. And use preview, of course.

tdb

@tster said:

@flaquito said:
@DemonWasp said:
Hmm, this submission ended up strikingly helpful and informative. I wondered whether it may have been an auto-generated file, but I'm just a junior part-time developer and I was following instructions from my immediate supervisor (who is technically attached to a different project at the moment...) to delve into configure. Upon further inspection, I'm not sure how I got it into my head that CXXFLAGS didn't occur anywhere, as cat configure | grep CXXFLAGS yielded pages and pages of occurrences.
tag: Filed under useless use of cat award.

Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.

As a side note, this is part of the reason I fucking hate the tags, especially when people make a point in their tags. When I quote your text it doesn't make any sense because the tag doesn't get quoted.

But but, don't you know that cat sandboxes each process, acting as a buffer and preventing them from messing with each other's encapsulation? Instead of something like:

grep Invalid auth.log | sed 's/^.*from //' | sort | uniq -c

You should do:

cat auth.log | grep Invalid | cat | sed 's/^.*from //' | cat | sort | cat | uniq -c

To be really safe, you can duplicate the cats for double buffering:

cat auth.log | cat | grep Invalid | cat | cat | sed 's/^.*from //' | cat | cat | sort | cat | cat | uniq -c

dabean

@tdb said:

cat auth.log | cat | grep Invalid | cat | cat | sed 's/^.*from //' | cat | cat | sort | cat | cat | uniq -c

TwelveBaud1

@flaquito said:

@tster said:
Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.
As a side note, this is part of the reason I fucking hate the tags, especially when people make a point in their tags. When I quote your text it doesn't make any sense because the tag doesn't get quoted.
Whoa, sorry. Just trying to introduce some humor. After all, is it that hard to type 'grep sometext somefile.txt'?
As for making a point using tags, I've seen it done enough times on the forums, without vitriolic responses, that I assumed it was acceptable posting etiquette.

It is acceptable posting etiquette here, especially for the use you put it to. tster just hasn't learned to copy and paste the tag line yet, although what really needs to happen is for someone to make a CS add-on that lets you pick whether to quote tags (defaulting to no so we don't get double the "Larry Ellison... herpes" fun) and give it to Alex.

DaveK1

@smxlong said:

@DaveK said:
Do it like this:

./configure CXXFLAGS="-g -O2

Then leave for lunch as your program "configures." When you return, the shell is happily awaiting your entry of the terminating double-quote.

Did I tell you to press enter yet? Did I? DID I?

No?

Well why did you do it then?

;-)

Emphyrio

@flaquito said:

After all, is it that hard to type 'grep sometext somefile.txt'?

It's not hard to type, but though I've been using various flavors of Unix since 1983, I still have to consciously think "Is it grep somefile sometext or grep sometext somefile?" It only takes a second to remember (now) that the order is backwards from what I would expect, but it's still a distraction from what I'm trying to think about. Typing "cat file | grep text" is cognitively easier. Plus I frequently have to search gzipped files, so I'm used to typing "zcat foo.gz | grep bar".

DaveK1

@tster said:

Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.

No it isn't. You really think it's easier to type "cat |" than absolutely nothing at all?

@Your way said:

cat filename | grep pattern

@My way said:

grep pattern filename

Even in more complex situations, "cat |" is five keypresses, and "<" is one.

What kind of ludicrous code have you seen that's an even more pointlessly overcomplex way of saying "<" than "cat |" already is?

DaveK1

@Emphyrio said:

@flaquito said:
After all, is it that hard to type 'grep sometext somefile.txt'?

It's not hard to type, but though I've been using various flavors of Unix since 1983, I still have to consciously think "Is it grep somefile sometext or grep sometext somefile?"

I have a sort-of-mnemonic way of remembering it. I just remind myself that there can be many file arguments, but only one pattern. That makes it intuitively obvious (to me, at any rate) that the pattern has to be first and the variable-length list of files last.

@Emphyrio said:

Plus I frequently have to search gzipped files, so I'm used to typing "zcat foo.gz | grep bar".

Well, that's why there's no such thing as a useless use of zcat award for a good reason!

@DaveK said:

@Emphyrio said:
Plus I frequently have to search gzipped files, so I'm
used to typing "zcat foo.gz | grep bar".

Well, that's why there's no such thing as a useless use of zcat award for a good reason!

Except that zgrep is usually available.

andrewm

@DaveK said:

@tster said:
Who gives a fuck seriously? It's easier to type "cat |" than some of the alternatives that to cat that I've seen.
No it isn't. You really think it's easier to type "cat |" than absolutely nothing at all?

Yes, it can be. Sometimes you know which file you want to search before you decide on the pattern, and it's easier to type "cat filename | grep" while you think of the appropriate pattern and then add it on at the end. Typing "grep filename" and then navigating back to the gap between "grep" and "filename" to fill in the pattern doesn't flow as well and takes more keystrokes.

Using the "cat filename | grep pattern" is also great for re-using command lines. Sometimes you start out just looking at a file (say, "cat filename"). If you then decide to filter for a pattern, it can be easier to just up-arrow and tack on "| grep pattern" than to start from scratch by re-typing "grep pattern filename". Likewise, having the pattern at the end in the "cat filename | grep pattern" form makes it easier to re-run the command with a different pattern: just up-arrow and edit the command, which will be immediately to the left of the cursor. In the "grep pattern file" form, you have to first navigate the cursor to the pattern.

Fister

Autoconf is the spawn of the devil. It purports to make software "portable," but it can only do that if you put a bunch of crap macros into your code, that you could put there without autoconf. And it only ports between slightly different flavors of UNIX. (You mean, I can compile this NetBSD program for OpenBSD? How about the 94% of my users who use Windows?)

Also, Autoconf is hardcoded to use recursive make. Recursive make is a terrible idea, because it makes builds slow, and re-does a lot of file scanning and dependency work for each directory it recurses into. When you have 2 directories, that's not so bad. When you have 200, it's bad. It's totally possible to build a make system that uses some templates and source inspection to build full knowledge of the entire system, and build everything that you need, with full dependencies, and it often builds substantially faster than autoconf based recursive make.

The funny thing is that the main open source thought leaders are just blind to these problems. They don't understand why it's even a problem.

I love make. I think GCC is a quite competent compiler. And I worked on portable applications long before Linux 1.0 was released. But, you know, autoconf is under the GPL and used by the Open Source gurus, so clearly it's the Right Way to do things...

morbiuswilters

@Fister said:

Autoconf is the spawn of the devil. It purports to make software "portable," but it can only do that if you put a bunch of crap macros into your code, that you could put there without autoconf. And it only ports between slightly different flavors of UNIX. (You mean, I can compile this NetBSD program for OpenBSD? How about the 94% of my users who use Windows?)
Also, Autoconf is hardcoded to use recursive make. Recursive make is a terrible idea, because it makes builds slow, and re-does a lot of file scanning and dependency work for each directory it recurses into. When you have 2 directories, that's not so bad. When you have 200, it's bad. It's totally possible to build a make system that uses some templates and source inspection to build full knowledge of the entire system, and build everything that you need, with full dependencies, and it often builds substantially faster than autoconf based recursive make.
The funny thing is that the main open source thought leaders are just blind to these problems. They don't understand why it's even a problem.
I love make. I think GCC is a quite competent compiler. And I worked on portable applications long before Linux 1.0 was released. But, you know, autoconf is under the GPL and used by the Open Source gurus, so clearly it's the Right Way to do things...

I agree that autoconf sucks (I've had too many bugs between different versions that broke compiles of software that wasn't even mine, the solution being to modify someone else's source or even the damn autoconf source itself) but I also think it's not a problem for a lot of unix users. I'm not sure it's so much autoconf being GPL as it is that most end-users don't compile and unix C is oftentimes enough black magic that autoconf's bitchiness is just a drop in the bucket. If you're already writing C that will work on the BSDs, OSX and Linux you are jumping through enough hoops that working around autoconf's shortcomings isn't that bif of a deal. In theory, end-users should just install binaries helpfully provided by the OS distributor. In practice, you will eventually end up in DLL Hell this way but generally the solution is just to reinstall the whole distro rather than compiling. Most of the binary Linux distros I've used made compiling a massive pain in the ass. Although the learning curve is steeper, source-based distros like Gentoo tended to be more adaptable in the long run but it seems users are put off by them and it seems Gentoo is a dying project. I would consider myself a very advanced user but too often I've run into unpredictable and difficult-to-solve compatibility problems with binary distros that just don't happen when compiling from source. Part of this is just Linux and the GPL, though, as a significant part of the ideology consists of rejecting binary compatibility and even outright trying to prevent it.

tdb

@morbiuswilters said:

In theory, end-users should just install binaries helpfully provided by the OS distributor. In practice, you will eventually end up in DLL Hell this way but generally the solution is just to reinstall the whole distro rather than compiling. Most of the binary Linux distros I've used made compiling a massive pain in the ass. Although the learning curve is steeper, source-based distros like Gentoo tended to be more adaptable in the long run but it seems users are put off by them and it seems Gentoo is a dying project. I would consider myself a very advanced user but too often I've run into unpredictable and difficult-to-solve compatibility problems with binary distros that just don't happen when compiling from source. Part of this is just Linux and the GPL, though, as a significant part of the ideology consists of rejecting binary compatibility and even outright trying to prevent it.

I have found Debian's package management to work pretty well. Libraries are installed when needed, and uninstalled when not needed anymore. I'm using the unstable distribution, and even there verseion conflicts are rare. Problems that the package manager can't solve don't occur even once a month. I wonder what distrubutions you have tried and when?

It is unfortunate that different GNU/Linux distributions are not very binary compatible. However, I wouldn't attribute this to the Linux kernel and definitely not GPL. It's mostly due to different distros compiling libraries with different options and having different combinations of packages. It's not like they are purposefully trying to prevent interoperability, it just isn't the number one or even number two priority.

TwelveBaud1

@Fister said:

(You mean, I can compile this NetBSD program for OpenBSD? How about the 94% of my users who use Windows?)

My [code]uname[/code] is "[code]Interix 4.0[/code]" Autoconf doesn't seem to have a problem. GCC doesn't have a problem. And with the new CC, even Visual Studio 2008's command line compiler doesn't have a problem. (Under Windows XP with Interix 3.5, you had to use the VS6 compiler, or tweak the "Supports [code]-G[/code]" code in [code]./configure[/code] and include a few Windows NT libs to support stack cookies.) All I did was check "Subsystem for Unix Applications" in Add/Remove Features.

Fister

Don't get me started on Linux and binary packages. Compared to Linux, the Windows "DLL hell" is a nice vacation.

Suppose I'm running Loonix 11 with AVT (advanced versioning technology) and LTS (long trauma support).

Suppose it shipped with GCC version 4.0, but I now need version 4.2, because the C++ constitution/bible was magically re-interpreted, and suddenly all code on the planet is broken. Suppose Loonix uses the latest and greatest "loop" package manager. (could be rpm, dpkg, apt or any other binary manager).

Turns out, the distro maintainers were all so excited about releasing version 13, that they built GCC 4.2 against the version of libc that ships with 13. If I try to just "loop upgradeizeate gcc-4.2" then I'll get back a note that I need to upgrade libc. And if I try to upgradeizeate libc, it turns out that needs a new kernel. Which needs new drivers. Which break compatibility with the old applications. Meanwhile, I was using Long Trauma Support to avoid having that kind of problem -- if it works, don't break it.

Imagine what Windows would work like if you had to buy "Adobe Photoshop for Windows XP SP2," or "Adobe Photoshop for Windows XP SP3," or "Adobe Photoshop for Windows Vista Home US Language Build 6603," depending on what OS you were running. That's insane!

The root problem is autoconf, which is too aggressive at detecting "dependencies" on "versions" of libraries, where no such dependency actually physically exists, and that in turn pushes out to all the package managers.

Linux on the desktop? Yeah, right.

(Btw: I've been a Linux user since 1.3 or so; doing LFS until recently when I jumped on the Ubuntu bandwagon)

DaveK1

@Fister said:

The root problem is autoconf, which is too aggressive at detecting "dependencies" on "versions" of libraries, where no such dependency actually physically exists, and that in turn pushes out to all the package managers.

Uh, no it doesn't - does it? Autoconf doesn't have anything to do with detecting dependencies, at least as far as I've discovered; can you post an example?

Zemm

@andrewm said:

Typing "grep filename" and then navigating back to the gap between "grep" and "filename" to fill in the pattern doesn't flow as well and takes more keystrokes.

grep fil[tab][esc]b'pattern'[space][enter]

@andrewm said:

Sometimes you start out just looking at a file (say, "cat filename")

FWIW I usually use less for looking a file, which has built-in searching through the / key.