Inversion of Control with plugins?

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

and the plug-in calls frobnicate

And this is my question: How does the plugin know how to call frobnicate? What is the method by which it is being told how to do this call?

That's all I want to know.

Nobody has been able to tell me this one simple thing.

@Kian said in Inversion of Control with plugins?:

The thing is, in Unix land an elf executable and shared library are basically the same thing, so you can link to an executable just as easily as you can to a shared library.

This is (as far as I'm aware), very possible in Windows land as well, with some extra incantations to export functions as if it were a shared library.
But then, that's not my question.

Thanks for trying though.

Kian

@Tsaukpaetra said in Inversion of Control with plugins?:

And this is my question: How does the plugin know how to call frobnicate? What is the method by which it is being told how to do this call?
That's all I want to know.

Far as I know, the same way you tell any code to call a shared library. You have a header, which provides a the declaration of the function, and then at link time you point the linker at the executable (-lexecutable, might be a slightly different flag), which should have that function as one of the exported symbols, and the linker puts in all the required magic.

Or do you mean in the sense of, how does the plug-in author know that frobnicate exists?

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

@Tsaukpaetra said in Inversion of Control with plugins?:

And this is my question: How does the plugin know how to call frobnicate? What is the method by which it is being told how to do this call?
That's all I want to know.

Far as I know, the same way you tell any code to call a shared library. You have a header, which provides a the declaration of the function, and then at link time you point the linker at the executable (-lexecutable, might be a slightly different flag), which should have that function as one of the exported symbols, and the linker puts in all the required magic.

Right, and @pie_flavor said he can't do that because reasons, but on Linux he can.

Or do you mean in the sense of, how does the plug-in author know that frobnicate exists?

No, we know that frobnicate exists, and what it has for parameters and returns, and {magic} is expected to happen that the plugin knows where and how to call it when dynamically loaded, simply by dint of having that function declared inside the plugin itself (somehow). This is what I want explained, because as far as I know, this is not how it works, or should work, or would work/

Kian

@Tsaukpaetra said in Inversion of Control with plugins?:

This is what I want explained, because as far as I know, this is not how it works, or should work, or would work/

Hmm, not sure I understand exactly your question, but I made a simple example and maybe we can work out which part confuses you?

// exe.cpp

#include <iostream>
#include <string>
#include <dlfcn.h>

void Print(std::string_view msg) {
	std::cout << msg << '\n';
}

typedef void (*funcPtr_t)();

int main() {
	Print("Call function from executable");

	auto sharedLib = dlopen("libplugin.so", RTLD_LAZY);
	auto pluginFunc = reinterpret_cast<funcPtr_t>(dlsym(sharedLib, "func"));
	pluginFunc();

	return 0;
}

I hope even for people not familiar with C++ this is easy to follow. We have two functions here, Print and main, and a typedef to make dealing with function pointers easier.

Main calls Print, since it's just a function declared inside the executable. We then load a dynamic library, and ask it for the address of the symbol "func", which we know is a function that doesn't return anything and doesn't have any parameters. Then we call the function and exit. Nothing too strange here.

To build this, we use the command line: clang++ -rdynamic -std=c++17 exe.cpp -ldl -o exe
-rdynamic is there to export the functions. Otherwise they can't be found by the plug-in later.

Next, the source for the plug-in:

// plugin.cpp

#include <string>

extern "C" {

void func();

}

void Print(std::string_view msg);

void func() {
	Print("Calling a function in the exe from the plug-in.");
}

We use extern C on the declaration of func so we don't mangle the name when we export it. That way the executable can look for the symbol easily. We then just declare that a function called Print exists (normally this would be taken from a header provided by the person that wrote the executable), and in func, we call that function.

We build this with this command line: clang++ -shared -std=c++17 -fPIC plugin.cpp -L. -l:exe -o libplugin.so
-shared to tell it we're building a shared library, -fPIC so the code is relocatable (required for shared libraries), -l:exe to tell it to link against our executable.

Finally, we do LD_LIBRARY_PATH=$(pwd) ./exe and the output is:

Call function from executable
Calling a function in the exe from the plug-in.

LD_LIBRARY_PATH is there so the exe will look for the .so in the current directory. I didn't want to put the .so in a real path for an example, or use some of the other mechanisms for telling the loader how to find it.

So, the executable loads the plug-in, calls a function in the plug-in, and the function in the plug-in calls a function defined in the executable. Are any of this steps giving you trouble, or is something else that you don't understand?

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

We then just declare that a function called Print exists (normally this would be taken from a header provided by the person that wrote the executable), and in func, we call that function.

This part. This is that part I don't understand.

How is the plugin able to know where Print() is? Or, in other words, how does the main program tell the plugin where Print() should point to?

This is the magic I'm talking about.

Edit: Ah, I see this here:

@Kian said in Inversion of Control with plugins?:

-l:exe to tell it to link against our executable.

This is something that @pie_flavor says he cannot do, but expects it to still work. That's the bigger highlight here.

Kian

@Tsaukpaetra Right, I'm describing the situation (as I understand it) in linux where it works.

@Tsaukpaetra said in Inversion of Control with plugins?:

How is the plugin able to know where Print() is? Or, in other words, how does the main program tell the plugin where Print() should point to?

Ah, it's not the main program that tells the plug-in, it's the OS (specifically, the loader). Same as with all shared libraries. So you compile your code, which calls a function that has a declaration but no definition. The output of this is an object file that basically has all the binary code for the functions you defined, and a list of all the symbols that were declared and used, but not defined.

After the compiler is done and created a bunch of these object files, the linker grabs all the object files and tries to match symbols defined in one object file with all the uses of that symbol in other object files. If there are any left over, it then looks through the libraries you told it to use to resolve missing symbols, and resolves those dependencies not by writing the address of the actual symbol, which is only known at runtime, but by leaving a note for the loader that says "When you load this binary, get the address of this symbol from this library and put it here".

Then at runtime, while loading, the loader sees that the linker left a note to look for the Print function in the exe binary, and puts it in where the plug-in needs it to work.

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

it then looks through the libraries you told it to use to resolve missing symbols

Ah, and that's the part that @pie_flavor says he doesn't need to do, but still manages to get it to work (so far as I understand what he wrote).

@pie_flavor if this is not true, and in fact you are providing it the actual main program to link against, please say so, and we can close this thread as a misunderstanding.

pie_flavor

@Kian said in Inversion of Control with plugins?:

@Tsaukpaetra said in Inversion of Control with plugins?:

And this is my question: How does the plugin know how to call frobnicate? What is the method by which it is being told how to do this call?
That's all I want to know.

Far as I know, the same way you tell any code to call a shared library. You have a header, which provides a the declaration of the function, and then at link time you point the linker at the executable (-lexecutable, might be a slightly different flag), which should have that function as one of the exported symbols, and the linker puts in all the required magic.

Or do you mean in the sense of, how does the plug-in author know that frobnicate exists?

You don't have to do the bit with the linker. You just have to say that the function exists.

Tsaukpaetra

@pie_flavor said in Inversion of Control with plugins?:

@Kian said in Inversion of Control with plugins?:

@Tsaukpaetra said in Inversion of Control with plugins?:

And this is my question: How does the plugin know how to call frobnicate? What is the method by which it is being told how to do this call?
That's all I want to know.

Far as I know, the same way you tell any code to call a shared library. You have a header, which provides a the declaration of the function, and then at link time you point the linker at the executable (-lexecutable, might be a slightly different flag), which should have that function as one of the exported symbols, and the linker puts in all the required magic.

Or do you mean in the sense of, how does the plug-in author know that frobnicate exists?

You don't have to do the bit with the linker. You just have to say that the function exists.

See! And that's the question I'm asking: How does it work if you don't tell it it needs to find that function elsewhere?

pie_flavor

@Tsaukpaetra Beats me. But it works. You can read the documentation here. https://hexchat.readthedocs.io/en/latest/plugins.html

Tsaukpaetra

@pie_flavor said in Inversion of Control with plugins?:

@Tsaukpaetra Beats me. But it works. You can read the documentation here. https://hexchat.readthedocs.io/en/latest/plugins.html

So, in this case, you're to include a header file that includes a struct that holds pointers to functions, which the loading executable (in this case HexChat) provides the plugin by calling a specific function in the plugin and providing it that handle to the struct.

This isn't magic, and Iseveral people including myself mentioned that's another way to do it.

In other words: You lied and caused confusion and dismay. Thanks a lot.

Kian

@Tsaukpaetra Ah, I think I remember that. Right, shared objects don't need to have all the symbols resolved by the linker. They can be left "missing", and then at load time the loader will try to match it with any already existing symbols in the process memory. If it can find it, it will use that, if it can't find it your program crashes. Since this is a symbol defined in the executable that is calling you, the symbol will always exist. If it was a symbol defined in some other library that may or may not be loaded, your process may or may not crash. Providing the library at link time ensures that the loader will know what library to use in case it wasn't already loaded, but it is not strictly necessary (just tested too, by removing the -l:exe flag in the above example, still works).

Apparently on Windows, either the loader or the linker are not smart enough to provide this option of a "missing symbol", and building the dll will complain of missing references, maybe? Not sure the exact error he was seeing.

pie_flavor

@Tsaukpaetra On Windows.
On Lunix you don't need to do that. It provides it in case you wanted to do it anyway, but on Windows the hexchat_do_x functions are macros that refer to a plugin handle named ph, and on Lunix they're just the regular functions.

Caveat: plugins compiled on Win32 must have a global variable called ph, which is the plugin_handle, much like in the sample plugin above.

pie_flavor

@Kian That's it, yes. The Windows linker doesn't do unresolved symbols. It has to know where everything is from at compile time.

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

the loader will try to match it with any already existing symbols in the process memory.

This seems incredibly dangerous and prone to weird, wild, and wacky non-determinism that I'm not sure why this is a thing...

I am a bit more saddened by the knowledge I have gained this day.

Kian

@Tsaukpaetra I'm happy to "help". Everyone should suffer as I suffer.

Tsaukpaetra

@Kian said in Inversion of Control with plugins?:

@Tsaukpaetra I'm happy to "help". Everyone should suffer as I suffer.

I'm just going to sit out here, in the rain, pondering, for a while...

Edit: Turns out, capacitive screens don't work well when wet. This was a bad idea....

dkf

@Zenith said in Inversion of Control with plugins?:

I thought that was how plug-ins worked (although I've never done it in C where it's probably far more difficult).

It's not much more difficult. The reflection tools available are a bit more primitive (hey, it's C; everything is more primitive) but that's a detail.

dkf

@pie_flavor said in Inversion of Control with plugins?:

You don't have to do the bit with the linker. You just have to say that the function exists.

That depends on the options given to the dynamic library loader. This is an area where exact details matter very much and hand-waving just confuses; the description as provided works on some operating systems, and fails on others. The issue has to do with how link libraries are managed; on Linux, those are a built-in feature of the ELF library/executable format so no special action is required, whereas on Windows they're generally separate files (actually they're static libraries, but that's an unimportant detail here). I don't know why COFF works that way, but it does; it makes the DLLs a bit smaller, but makes linking without the library really awkward. (macOS doesn't use ELF, but the library linking works as if it did. At a high level.)

The alternative is dynamic binding, where as part of DLL startup a pointer to a table of API functions in the executable is provided (this is the executable explicitly doing the call) and the DLL then effectively replaces FooFunc() calls with funcTable->FooFunc() calls. Or the DLL uses the dlopen()/dlsym() API with the right options to look at the global symbol table and finds the functions that way (this is a horrible horrible thing, but works).

Or you can use weak binding in some executable types to get things plugged together for you, provided you allow for the possibility that the functions don't exist. This is a bit tricky, but is quite useful in some cases.

The important thing to note is that the dynamic, runtime binding cases all have more failure modes than the static, compile-time binding case. This is normal when adding flexibility.

JazzyJosh

@pie_flavor said in Inversion of Control with plugins?:

@dkf Yes, exactly. That's the solution I said I was using.

THEN WHAT THE FUCK IS THE PROBLEM

> I can't call functions by name because of how C works
> Use function pointers
> I can't call functions by name because of how C works
>

dkf

@JazzyJosh Is it that he's using Rust and so can't use function pointers because they might be unsafe?

PleegWat

@dkf He's interacting with C code. Unsafe comes with the territory.

HardwareGeek

@PleegWat said in Inversion of Control with plugins?:

@dkf He's interacting with C code. Unsafe comes with the territory.

Many words have been used to describe C. Safe has never been one of them.

ObjectMike

Why would the plugin need to call something in the executable? I'd expect handy bits like that to be extracted into a TheApp-guts.dll that both the exe and plugin link against. The dll would only be loaded one time into the process (memory wise) so frobnicated state can be shared between plugin and app. I'm not seeing the problem. If the problem is the guts dll doesn't exist and plugins really need to share that functionality, that sounds like a crappy plugin implementation in that app. I've seen this pattern used for both .NET and native C++ apps with plugins.

sloosecannon

@mikehurley this is also how I have seen this done (and implemented myself).
Granted, this is .NET and took about 5 minutes to implement, without worrying about the deep dark secrets of the linker and compiler but I assume the same can be done in C/C++/Rust

pie_flavor

@JazzyJosh said in Inversion of Control with plugins?:

@pie_flavor said in Inversion of Control with plugins?:

@dkf Yes, exactly. That's the solution I said I was using.

THEN WHAT THE FUCK IS THE PROBLEM

> I can't call functions by name because of how C works
> Use function pointers
> I can't call functions by name because of how C works
>

You forgot "on Windows".
Anyway, I was just complaining. It was @Tsaukpaetra that did a thread about it.

JBert

@sloosecannon said in Inversion of Control with plugins?:

@mikehurley this is also how I have seen this done (and implemented myself).
Granted, this is .NET and took about 5 minutes to implement, without worrying about the deep dark secrets of the linker and compiler but I assume the same can be done in C/C++/Rust

Please do not bring .NET into a C / C++ / Rust discussion like this.

.NET might spit out DLL files but any native x86 code in there is for compatibility reasons only and has nothing to do with this topic. Then the actual CIL assembly doesn't even get looked at by the dynamic linker in the OS, it's an entirely different one in the .NET runtime.

dkf

@mikehurley said in Inversion of Control with plugins?:

Why would the plugin need to call something in the executable?

Because it makes the executable a lot easier to deploy if it doesn't need to have lots of extra required files placed beside it.

ObjectMike

@JBert said in Inversion of Control with plugins?:

@sloosecannon said in Inversion of Control with plugins?:

@mikehurley this is also how I have seen this done (and implemented myself).
Granted, this is .NET and took about 5 minutes to implement, without worrying about the deep dark secrets of the linker and compiler but I assume the same can be done in C/C++/Rust

Please do not bring .NET into a C / C++ / Rust discussion like this.

.NET might spit out DLL files but any native x86 code in there is for compatibility reasons only and has nothing to do with this topic. Then the actual CIL assembly doesn't even get looked at by the dynamic linker in the OS, it's an entirely different one in the .NET runtime.

The way I'd break apart code and deploy dll/so files is the same. I've worked on C++ systems that worked this way.

ObjectMike

@dkf said in Inversion of Control with plugins?:

@mikehurley said in Inversion of Control with plugins?:

Why would the plugin need to call something in the executable?

Because it makes the executable a lot easier to deploy if it doesn't need to have lots of extra required files placed beside it.

Why are those files a big deal?

Medinoc

As far as I know, on Windows it's pretty much just as easy to use "load-time dynamic linking" to functions in an EXE as it is to functions in a DLL (at least, when the dependency isn't circular, and in @pie_flavor's case it isn't because loading the plug-in DLL uses run-time dynamic linking): In both cases, you need to provide the "import library" that says "the function you're loading, it's in this DLL"

From what I understand, it's easier on Linux, because you don't need import libraries, and you can have circular dependencies transparently, and you can even have a symbol defined in multiple libraries in a "fallback" mechanic (or maybe I'm mixing up static and dynamic libraries here).

Regardless, the verdict is, @pie_flavor wants to use load-time dynamic linking towards his exe, which is a breeze in Linux, and has whined (incorrectly) that you can't do it on Windows, whereas the actual conclusion is you can do it on Windows, but it takes work. And so far, @pie_flavor has fallen back on methods more akin to run-time dynamic linking, such as the good old "pass it a bunch of callback function pointers". Which is also the approach favored by object-oriented code (including COM).

Did I get it right?

dkf

@mikehurley said in Inversion of Control with plugins?:

Why are those files a big deal?

Not usually, but it does limit your deployment options.

Also, using import libraries Windows-style ends up with needing the exact names (and sometimes even versions) of things being bound into the plugins, which is a bit rigid. It's a PITA for everyone to have to rebuild their plugins just because the executable creator puts out a new patch release.

Tsaukpaetra

@dkf said in Inversion of Control with plugins?:

rebuild their plugins just because the executable creator puts out a new patch release.

Which is why you tie the bindings to an interop, right?

Kian

@Medinoc said in Inversion of Control with plugins?:

In both cases, you need to provide the "import library" that says "the function you're loading, it's in this DLL"

No, this is the thing he doesn't need to do in Linux. I put an example above, but basically Linux let's you do load time linking without saying where the function is from.

dkf

@Tsaukpaetra said in Inversion of Control with plugins?:

Which is why you tie the bindings to an interop, right?

Pretty much. You define API/ABI interop layers and, if you're careful, you can upgrade fairly extensively without breaking anything that uses the old bindings. It's a touch tricky, but works quite well with only modest discipline.

Some programmers hate it, of course. They're not the people who have to suffer the consequences of random upstream breakage being forced on them…

Medinoc

@Kian said in Inversion of Control with plugins?:

@Medinoc said in Inversion of Control with plugins?:

In both cases, you need to provide the "import library" that says "the function you're loading, it's in this DLL"

No, this is the thing he doesn't need to do in Linux. I put an example above, but basically Linux let's you do load time linking without saying where the function is from.

Which is precisely what I was saying. "both" here didn't mean "windows and Linux", it meant "dynamically link to EXE and to DLL".

jinpa

@The_Quiet_One said in Inversion of Control with plugins?:

This thread takes me back to a job that had 3 hour long "scrum" meetings full of these kind of discussions. Only the CTO had a short fuse so there was a lot of loud screaming.

I hated that job.

I assume you didn't have the standing there to suggest that any tangents more than a couple of minutes be forked into a meeting with just the people involved. BTDT.

The_Quiet_One

@jinpa It was a very small team of like 4 people. And we were often involved/responsible for the same stuff. Our meetings were SINO.

jinpa

@The_Quiet_One said in Inversion of Control with plugins?:

@jinpa It was a very small team of like 4 people. And we were often involved/responsible for the same stuff. Our meetings were SINO.

Though overall, I would prefer a scrum team of 4 people than 14 people. At least a higher percentage of what's said has something to do with you.

PleegWat

@The_Quiet_One said in Inversion of Control with plugins?:

@jinpa It was a very small team of like 4 people. And we were often involved/responsible for the same stuff. Our meetings were SINO.

We're in a similar boat at times, though without the screamy manager. 4 man and our twice-weekly daily scrums tend to get timeboxed by lunch, which is after 1.5 hours.

Buddy

@Tsaukpaetra said in Inversion of Control with plugins?:

So, in this case, you're to include a header file that includes a struct that holds pointers to functions, which the loading executable (in this case HexChat) provides the plugin by calling a specific function in the plugin and providing it that handle to the struct.

I'm interested to know how you parsed the original post to mean something other than that:

@Tsaukpaetra program A has plugin compatibility. so any DLLs in a particular folder, including DLL B, get loaded at runtime and a particular exported symbol is called. This is an inverse of how DLLs are supposed to work; the program provides the library functions to the DLL instead of the other way around. This is fine on non-Windows systems, because externally-defined functions can be referenced even if you don't specify the DLL name as long as the function exists at runtime. Windows however does not have an 'amorphous blob' model of functions, and requires that you say what DLL each function comes from. This means that this plugin compatibility model is impossible in Windows.
Unless of course you pass a struct to that particular exported symbol containing all your library function pointers. Which is the solution I'm currently running with.

Was it the ranty interjection that threw you? If we cut that out we get

@Tsaukpaetra program A has plugin compatibility. so any DLLs in a particular folder, including DLL B, get loaded at runtime and a particular exported symbol is called.
you pass a struct to that particular exported symbol containing all your library function pointers. Which is the solution I'm currently running with.

which to me seems to be saying the same thing as your summary above.

Tsaukpaetra

@Buddy said in Inversion of Control with plugins?:

@Tsaukpaetra said in Inversion of Control with plugins?:

So, in this case, you're to include a header file that includes a struct that holds pointers to functions, which the loading executable (in this case HexChat) provides the plugin by calling a specific function in the plugin and providing it that handle to the struct.

I'm interested to know how you parsed the original post to mean something other than that:

@Tsaukpaetra program A has plugin compatibility. so any DLLs in a particular folder, including DLL B, get loaded at runtime and a particular exported symbol is called. This is an inverse of how DLLs are supposed to work; the program provides the library functions to the DLL instead of the other way around. This is fine on non-Windows systems, because externally-defined functions can be referenced even if you don't specify the DLL name as long as the function exists at runtime. Windows however does not have an 'amorphous blob' model of functions, and requires that you say what DLL each function comes from. This means that this plugin compatibility model is impossible in Windows.
Unless of course you pass a struct to that particular exported symbol containing all your library function pointers. Which is the solution I'm currently running with.

Was it the ranty interjection that threw you? If we cut that out we get

@Tsaukpaetra program A has plugin compatibility. so any DLLs in a particular folder, including DLL B, get loaded at runtime and a particular exported symbol is called.
you pass a struct to that particular exported symbol containing all your library function pointers. Which is the solution I'm currently running with.

which to me seems to be saying the same thing as your summary above.

I don't understand the question?