G++ linking error (solved)

mott555

I am totally stumped on an issue here at work, just throwing this out here to see if anyone has had something like this before.

Project #1 is a small C library compiled to a static library. Project #2 is a command-line C++ application which utilizes Project #1. They compile and work flawlessly on Windows.

I'm getting things fixed up for various Linuxes and having some very strange issues. Project #1 gets compiled with gcc (I've also tried g++ but no difference), and Project #2 gets compiled with g++. However the linker fails on #2 with messages like this (function names anonymized):

main.cpp:(.text+0x3a70): undefined reference to `project1Function1'
main.cpp:(.text+0x3b8b): undefined reference to `project1Function2'
main.cpp:(.text+0x4337): undefined reference to `project1Function3'
main.cpp:(.text+0x44e9): undefined reference to `project1Function4'
main.cpp:(.text+0x4633): undefined reference to `project1Function2'

At first it looks like it's not finding the output from Project #1, but if I delete the static library I get a totally different set of linker errors. It can find the library but it doesn't know how to link into it!

All my web trawling has turned up is that the Project #1 headers need to be included in an extern "C" block in the C++ project, which it is!

extern "C"
{
    #include "project1HeaderFile.h"
}

I'm stumped. I've even gone so far as to check the encodings of all the source files in case one of them was a weird encoding and that actually carried into the compiled library, causing the linker to not realize the names are the same, but everything is "ANSI as UTF-8" according to Notepad++.

dkf

Just checking: you've not got the library order wrong to the linker? Unix linkers tend to be pretty stupid (by design), so you need to put libraries that are depended on by other code later on the link line.

Aside from that, get familiar with ldd and nm -g… :(

PJH

readelf -Ws libProject1.a

or

readelf -Ws libProject1.so

How are project1FunctionX being exported?

mott555

The Makefiles look correct and I've had a couple other people here at the office review them. In any case they're auto-generated via MPC which is what we use for project control.

I forgot to mention there is also a Project #3 which is just a unit test project, but it's also unable to link into Project #1, so I'm assuming there's an issue with Project #1 and not the others.

powerlord

Did you try compiling with -Wall to see if it's tossing any warnings?

FrostCat

@PJH said:

How are project1FunctionX being exported?

Is it possible the functions aren't being exported? My first thought was "name-mangling fail" too, but extern "C" should've fixed that--and the fact that project1's being compiled with g++ suggests that's not the problem either.

FrostCat

@powerlord said:

Did you try compiling with -Wall to see if it's tossing any warnings?

Also try -pendantic. Not necessarily because it will help, of course.

mott555

Here's what I get, didn't bother anonymizing this time (it's a .so btw, guess I was wrong about it being a static library).

Symbol table '.dynsym' contains 18 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000658     0 SECTION LOCAL  DEFAULT    9
     2: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     3: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fseek@GLIBC_2.2.5 (2)
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen@GLIBC_2.2.5 (2)
     6: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.2.5 (2)
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fread@GLIBC_2.2.5 (2)
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND feof@GLIBC_2.2.5 (2)
    10: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fclose@GLIBC_2.2.5 (2)
    11: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fwrite@GLIBC_2.2.5 (2)
    12: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __gxx_personality_v0@CXXABI_1.3 (3)
    13: 0000000000203500     0 NOTYPE  GLOBAL DEFAULT  ABS _end
    14: 00000000002033d0     0 NOTYPE  GLOBAL DEFAULT  ABS _edata
    15: 00000000002033d0     0 NOTYPE  GLOBAL DEFAULT  ABS __bss_start
    16: 0000000000000658     0 FUNC    GLOBAL DEFAULT    9 _init
    17: 0000000000002e38     0 FUNC    GLOBAL DEFAULT   12 _fini

Symbol table '.symtab' contains 78 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000190     0 SECTION LOCAL  DEFAULT    1
     2: 00000000000001b8     0 SECTION LOCAL  DEFAULT    2
     3: 00000000000001f0     0 SECTION LOCAL  DEFAULT    3
     4: 00000000000003a0     0 SECTION LOCAL  DEFAULT    4
     5: 00000000000004b6     0 SECTION LOCAL  DEFAULT    5
     6: 00000000000004e0     0 SECTION LOCAL  DEFAULT    6
     7: 0000000000000520     0 SECTION LOCAL  DEFAULT    7
     8: 0000000000000598     0 SECTION LOCAL  DEFAULT    8
     9: 0000000000000658     0 SECTION LOCAL  DEFAULT    9
    10: 0000000000000670     0 SECTION LOCAL  DEFAULT   10
    11: 0000000000000700     0 SECTION LOCAL  DEFAULT   11
    12: 0000000000002e38     0 SECTION LOCAL  DEFAULT   12
    13: 0000000000002e46     0 SECTION LOCAL  DEFAULT   13
    14: 0000000000002e54     0 SECTION LOCAL  DEFAULT   14
    15: 0000000000002ed0     0 SECTION LOCAL  DEFAULT   15
    16: 0000000000203148     0 SECTION LOCAL  DEFAULT   16
    17: 0000000000203158     0 SECTION LOCAL  DEFAULT   17
    18: 0000000000203168     0 SECTION LOCAL  DEFAULT   18
    19: 0000000000203170     0 SECTION LOCAL  DEFAULT   19
    20: 0000000000203178     0 SECTION LOCAL  DEFAULT   20
    21: 0000000000203358     0 SECTION LOCAL  DEFAULT   21
    22: 0000000000203370     0 SECTION LOCAL  DEFAULT   22
    23: 00000000002033c8     0 SECTION LOCAL  DEFAULT   23
    24: 00000000002033e0     0 SECTION LOCAL  DEFAULT   24
    25: 0000000000000000     0 SECTION LOCAL  DEFAULT   25
    26: 0000000000000700     0 FUNC    LOCAL  DEFAULT   11 call_gmon_start
    27: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    28: 0000000000203148     0 OBJECT  LOCAL  DEFAULT   16 __CTOR_LIST__
    29: 0000000000203158     0 OBJECT  LOCAL  DEFAULT   17 __DTOR_LIST__
    30: 0000000000203168     0 OBJECT  LOCAL  DEFAULT   18 __JCR_LIST__
    31: 0000000000000720     0 FUNC    LOCAL  DEFAULT   11 __do_global_dtors_aux
    32: 00000000002033e0     1 OBJECT  LOCAL  DEFAULT   24 completed.6349
    33: 00000000002033e8     8 OBJECT  LOCAL  DEFAULT   24 dtor_idx.6351
    34: 00000000000007a0     0 FUNC    LOCAL  DEFAULT   11 frame_dummy
    35: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    36: 0000000000203150     0 OBJECT  LOCAL  DEFAULT   16 __CTOR_END__
    37: 0000000000003140     0 OBJECT  LOCAL  DEFAULT   15 __FRAME_END__
    38: 0000000000203168     0 OBJECT  LOCAL  DEFAULT   18 __JCR_END__
    39: 0000000000002e00     0 FUNC    LOCAL  DEFAULT   11 __do_global_ctors_aux
    40: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS statistics.c
    41: 00000000000007cc    42 FUNC    LOCAL  DEFAULT   11 _ZL8getFieldPhjh
    42: 00000000000007f6   118 FUNC    LOCAL  DEFAULT   11 _ZL12getSubSystemjtjjP14IntHSNNetStats
    43: 000000000000086c   134 FUNC    LOCAL  DEFAULT   11 _ZL7getPortjjjP20IntHSNSubSystemStats
    44: 00000000000008f2    95 FUNC    LOCAL  DEFAULT   11 _ZL10getMessagejP13IntHSNVlStats
    45: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS conversion.c
    46: 0000000000203400   255 OBJECT  LOCAL  DEFAULT   24 _ZL11errorString
    47: 0000000000000d5f   448 FUNC    LOCAL  DEFAULT   11 _ZL13writeRawFrameP10_DataFrameP8_IO_FILE
    48: 000000000000139a   280 FUNC    LOCAL  DEFAULT   11 _ZL13loadNextFrameP10_DataFrameP8_IO_FILE
    49: 00000000002033c8     8 OBJECT  LOCAL  DEFAULT   23 DW.ref.__gxx_personality_v0
    50: 0000000000001b0c  1260 FUNC    LOCAL  DEFAULT   11 convertToRawFromPcapNg
    51: 0000000000203370     0 OBJECT  LOCAL  DEFAULT  ABS _GLOBAL_OFFSET_TABLE_
    52: 00000000000014b2  1626 FUNC    LOCAL  DEFAULT   11 mergeRawDataFiles
    53: 0000000000000951   444 FUNC    LOCAL  DEFAULT   11 parsePcapFrame
    54: 0000000000000b0d   578 FUNC    LOCAL  DEFAULT   11 parseDataFrame
    55: 0000000000000f1f  1147 FUNC    LOCAL  DEFAULT   11 convertToRawFromPcap
    56: 0000000000203170     0 OBJECT  LOCAL  DEFAULT   19 __dso_handle
    57: 0000000000203160     0 OBJECT  LOCAL  DEFAULT   17 __DTOR_END__
    58: 0000000000001ff8  2185 FUNC    LOCAL  DEFAULT   11 convertToPcapNg
    59: 0000000000002881  1401 FUNC    LOCAL  DEFAULT   11 convertToPcap
    60: 0000000000203178     0 OBJECT  LOCAL  DEFAULT  ABS _DYNAMIC
    61: 0000000000000d50    15 FUNC    LOCAL  DEFAULT   11 getError
    62: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
    63: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _Jv_RegisterClasses
    64: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fseek@@GLIBC_2.2.5
    65: 0000000000002e38     0 FUNC    GLOBAL DEFAULT   12 _fini
    66: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen@@GLIBC_2.2.5
    67: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@@GLIBC_2.2.5
    68: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@@GLIBC_2.2.5
    69: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fread@@GLIBC_2.2.5
    70: 00000000002033d0     0 NOTYPE  GLOBAL DEFAULT  ABS __bss_start
    71: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND feof@@GLIBC_2.2.5
    72: 0000000000203500     0 NOTYPE  GLOBAL DEFAULT  ABS _end
    73: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fclose@@GLIBC_2.2.5
    74: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fwrite@@GLIBC_2.2.5
    75: 00000000002033d0     0 NOTYPE  GLOBAL DEFAULT  ABS _edata
    76: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __gxx_personality_v0@@CXXABI_1.3
    77: 0000000000000658     0 FUNC    GLOBAL DEFAULT    9 _init

PJH

@powerlord said:

Did you try compiling with -Wall to see if it's tossing any warnings?

-Wall tends to be compile time (into object files) - this is link-time after all compiling has happened (and objects are being linked to each other and any libraries...)

mott555

The Makefile uses both -Wall and -pedantic already. I see one warning that says "'typedef' was ignored in this declaration" but unfortunately our codebase has taught me to ignore warnings because normally there are like 928346347691234857y123984 of them.

PJH

@mott555 said:

Here's what I get, didn't bother anonymizing this time (it's a .so btw, guess I was wrong about it being a static library).

Are the functions concerned (1) appearing in the Name column and if they are (2) ~~unadorned with @ signs~~ spelt exactly as they are in the source file without any extra @'s, _'s or any other characters?

mott555

Yes, the functions I'm having trouble with are convertToRawFromPcapNg, convertToRawFromPcap, convertToPcapNg, convertToPcap, and getError. The spelling looks correct, I don't see any name mangling going on.

Hmm, the binding is local, what's the difference between local and global?

PJH

@mott555 said:

Hmm, the binding is local, what's the difference between local and global?

What are the meanings of the columns of the symbol table displayed by readelf?

Could someone explain the columns shown of the symbol table using readelf?

Bind = GLOBAL binding means the symbol is visible outside the file. LOCAL binding is visible only in the file. WEAK is like global, the symbol can be overridden.

tarunik

Smells like an ELF visibility issue to me -- I'd double-check this.

PJH

Are your functions declared static in your .c source? If so, remove it.

mott555

So if I'm understanding the data correctly, my problem is that the function bindings are LOCAL they aren't visible outside of the library?

mott555

They are not declared static in the source.

boomzilla

@mott555 said:

(it's a .so btw, guess I was wrong about it being a static library).

How is it being passed to the linker? Should be something like -lproject1 and not just passed like a regular object file (i.e., how you'd do a static library).

mott555

@boomzilla said:

Should be something like -lproject1

It is -l"project1"

PJH

@mott555 said:

They are not declared static in the source.

Only other thing I've found while googling is: is

-fvisibility=hidden

being specified when compiling the lib? (From https://www.technovelty.org/code/why-symbol-visibility-is-good.html and various searches after that.)

mott555

Interesting, that is in the Makefile. I'm going to remove it and see what changes.

That was it. The functions are now global according to readelf and Project #2 is linking successfully.

Thanks a lot guys. Now I just need to figure out why MPC is sticking that into the Makefile.

PJH

Workaround from my link if you can't change the Makefile - stick

__attribute__((visibility("default")))

between the return type and function name of every function you want GLOBALly visible.

(typedef it to something simpler and #ifdef according to platform if it needs to be x-compilable.)

mott555

Found it, in the top-level MPC file there's genflags += -fvisibility=hidden

Now to ask around and see if that's needed, if so I can probably remove it in the project's MPC file.

EvanED

Incidentally, if you have a reasonable number of such functions, there's a good chance you'll find -fvisibility=hidden and explicitly exporting the functions you want to export via __attribute__((visibility("default"))) is the better option, especially if you are compiling with optimization and are interested in performance.

mott555

Actually, the ultimate problem was that Project #1 was supposed to be a static library but was accidentally being built as a shared object instead (in Linux-land, was doing the right thing in Windows). Function visibility doesn't matter when a static library is being imported.