Help Bites

[remi]

@dcon It's the program I'm developing, not a third-party one. Turned out I messed up when copying stuff around for tests.

But my question wasn't so much about fixing one specific occurrence of that message, but rather about how to try and make that message less useless when it happens for whatever reasons.

That specific instance was OK(-ish) because I have various tools (if cumbersome, since they list all dependencies rather than just failed ones) to solve it (Dependency Walker or dumpbin), but in the past we've had the same thing happen to build tests/testers/users because e.g. we forgot to bundle something somewhere, and it's a pain to debug without the DLL name.

By contrast, on Linux if I do the same mistake (forget one library), I end up with several lines of messages telling me all the places it looked for it and failed (this is the message we get from QPluginLoader::errorString(), itself bubbled up from dlopen(), I guess). While this is too verbose in that case (though is useful when debugging search path issues!), at least it clearly states (several times!) the name of the library that it failed to load, so just looking at whatever log file is used allows me to pinpoint the issue quickly.

[dcon]

@remi said in Help Bites:

But my question wasn't so much about fixing one specific occurrence of that message, but rather about how to try and make that message less useless when it happens for whatever reasons.

Unless you're specifically LoadLibrarying the DLL, I don't think that's possible. As it's the OS loading the program that fails. (I'm not sure exactly what happens when you link the DLL via delay loading)

I keep thinking I've seen traces somewhere that did show what DLL. But my brain isn't pulling up any references. And a quick google pretty much just said 'use depends'.

[Gąska]

@remi said in Help Bites:

When a DLL/EXE fails to load because it's missing some DLLs, is it possible to get Windows to tell you which module is missing rather than the cryptic "The specified module could not be found"?

Doesn't it do it by default? I think I remember Windows always telling me which DLL is missing. It looks something like this:

Though this one is for "statically declared" DLLs only (the ones declared in executable file's import symbols section). If the program dynamically loads extra DLLs beyond those, then the only error is the one returned by WinAPI call. If you don't get the DLL name in the error dialog, it's likely that the DLL is being loaded dynamically and whoever wrote the dynamic loading part fucked up/didn't care about error reporting and just blindly dumped the default text for the error code into the dialog box - in which case, the solution is to fix your fucking program.

[topspin]

@Gąska said in Help Bites:

@remi said in Help Bites:

When a DLL/EXE fails to load because it's missing some DLLs, is it possible to get Windows to tell you which module is missing rather than the cryptic "The specified module could not be found"?

Doesn't it do it by default? I think I remember Windows always telling me which DLL is missing. It looks something like this:

It used to do that in older versions. On our Windows 10 server I now get no message at all when things fail to load due to missing DLLs. Considering I sometimes run into the same problem as @remi where I need to figure out what file I missed, this is extremely annoying.

I think it's somehow configurable, since I've complained about it to IT before and they fixed it, but it keeps going back to failing silently. Not sure if it's idiotic IT or idiotic MS updates. ()

[Gąska]

@topspin I knew it showed that dialog on 7 but wasn't sure on 10, and had no way to check quickly. But I googled and it seems it works the same (you can see the image in my post has W10 window style.)

Unless something fucks up the settings, that is; as you pointed out.

Edit: still, no dialog and a dialog with the default message is quite different. I still believe @remi's problem is in his own program (or one of the libraries).

[Gąska]

@remi said in Help Bites:

this is the message we get from QPluginLoader::errorString()

Only saw that now. Yes, someone definitely fucked up dynamic loading. Either you or authors of QPluginLoader.

[remi]

@Gąska I'll blame QPluginLoader rather than my code because my code is perfect I can't see where else I could get the information from it apart from errorString() (which, on Linux, returns the wall of text that I mentioned above which does include the lib name).

But that last point (Linux is OK), and the fact that as a rule Qt developers are not total morons (not saying they're perfect, but I'd say they're "above average"), makes me wonder whether QPluginLoader can actually do better on Windows. The fact that so many people, including developers, have the same problem as I do and I haven't found any discussion saying "you should call function GetMissingDLLNameEx() to get the full string" is also a strong hint that it's not probably not that easy.

But again, I'm asking the question hoping to be proven wrong, so if you know what is the magic function call that would give me this missing DLL name, I'd be happy to hear it.

[Gąska]

@remi I don't know how you're using QPluginLoader exactly but it has a fileName() method which returns the file name of the DLL it tries to load. But then, to load a DLL you first have to set the file name yourself in the constructor or the setFileName() method, so you should already know which file it's loading anyway.

[Gribnit]

@remi said in Help Bites:

Qt developers are not total morons

[remi]

@Gąska That would work if the file passed to QPluginLoader was the missing one, but that's a pretty obvious case (and I'm not sure I'd get this error in that case). The missing file is a DLL that is required by the file passed to QPluginLoader.

All I can do (well, again, if I'm wrong please correct me...) is calling QPluginLoader::load(), see that it returns false and call errorString(). Which says "The specified module could not be found", nothing more.

(ETA: in fact the message becomes misleading to the user because we compose the message as something like fileName() + ": " + errorString() so it looks to the user as if the "specified module" is fileName() but it's not... but that's not the issue here)

[remi]

@Gribnit said in Help Bites:

@remi said in Help Bites:

Qt developers are not total morons

Any juicy story to tell? The "New Thread" button is ...

[Gribnit]

@remi sorry, no, just general principles.

[topspin]

@remi said in Help Bites:

@Gąska That would work if the file passed to QPluginLoader was the missing one, but that's a pretty obvious case (and I'm not sure I'd get this error in that case). The missing file is a DLL that is required by the file passed to QPluginLoader.

All I can do (well, again, if I'm wrong please correct me...) is calling QPluginLoader::load(), see that it returns false and call errorString(). Which says "The specified module could not be found", nothing more.

(ETA: in fact the message becomes misleading to the user because we compose the message as something like fileName() + ": " + errorString() so it looks to the user as if the "specified module" is fileName() but it's not... but that's not the issue here)

From a quick glance I actually don't see anything in LoadLibraryEx, which I assume is what QPluginLoader must be using behind the scenes, that reports which dependency is missing either. But obviously there is some way to get this information as the dialog box in @Gąska's post above shows.

E: Relevant code link.

[Gąska]

@remi said in Help Bites:

The missing file is a DLL that is required by the file passed to QPluginLoader.

Oh okay, that makes more sense. I did some digging and indeed, it seems WinAPI just doesn't provide this information, at all. There's no way to even know that there are any dependencies either.

So to answer your question: lolgf.

[Gąska]

@topspin said in Help Bites:

But obviously there is some way to get this information as the dialog box in @Gąska's post above shows.

That's from Windows internals. Windows knows which file is missing. It just won't tell you. It only tells you when the error was a direct result of double-clicking an executable - in all other cases, in particular LoadLibrary function call, there's just no way to know what happened.

[dcon]

@topspin said in Help Bites:

I think it's somehow configurable, since I've complained about it to IT before and they fixed it, but it keeps going back to failing silently. Not sure if it's idiotic IT or idiotic MS updates. ()

Or, IT fixes it. But leaves it in the group policy so it reverts on the next group update.

[remi]

@Gąska said in Help Bites:

So to answer your question: lolgf.

Thought so, and that was also the result of my searches. I just hoped that, for once, I was TR and not Microsoft and I had missed something. But no.

Windows knows which file is missing. It just won't tell you. It only tells you when the error was a direct result of double-clicking an executable - in all other cases, in particular LoadLibrary function call, there's just no way to know what happened.

Yeah, that's the annoying thing. It's not that there is some theoretical deep reason why Windows doesn't tell you, just that... it doesn't. And the error message just rubs it in. "The specified module". Which one? The one you specified, of course. Which one was that? Ah, well, it wouldn't be fun if I told you, right?

Anyway, thanks again for double-checking and confirming my research!

[dcon]

@remi I just renamed a dll for an app I built - and got the "can't start because xxx.dll is missing" message. I guess whether Windows tells you or not is "it depends". (I did that with a different app, and it gave a generic answer - but that one may have been a launcher the starts the real app, so that would be "different")

[remi]

@dcon See @Gąska's post above, it seems like you've experimentally confirmed what he said?

Anyway, even if some cases trigger a more helpful message, what matters for me is that when something triggers the unhelpful one, there is no way to get more information.

[dcon]

@remi said in Help Bites:

@dcon See @Gąska's post above, it seems like you've experimentally confirmed what he said?

Yup

Anyway, even if some cases trigger a more helpful message, what matters for me is that when something triggers the unhelpful one, there is no way to get more information.

I wonder - you mentioned this was an internal issue (testers) as I remember - is there any way to create a "stupid" program that simply links all those plugins so you could trigger the (slightly) more informative message?

[remi]

@dcon said in Help Bites:

I wonder - you mentioned this was an internal issue (testers) as I remember - is there any way to create a "stupid" program that simply links all those plugins so you could trigger the (slightly) more informative message?

That's an interesting idea. I guess I could do that... of course it wouldn't work if the error comes from a plugin that wasn't built at the same time as the main executable (which is the point of having a plugin system!), but there are quite a few plugins (most, actually) that comes bundled with the application, so at least for those it would help.

Can I just link the libraries in an executable, and let dlopen()/LoadLibrary() find them when needed (i.e. not changing anything in the code itself, just in the linker), or do I need to make it so that the executable actually does something with those libraries from the start? The former is easy as it just requires editing the link command but I wonder if the linker will not be smart enough to see that I'm not using any symbols from the DLL and just ignore it (I could check myself, but it's easier to let people here do the work ), the latter means changing the code and that's a bit more complicated. Not infeasible, of course, but enough work (add includes, add call to the plugin, remove call to loading the plugin as a plugin...) that I would question whether it's worth it (because those errors happen only when someone fucked up something in either building the plugin or distributing it, which isn't supposed to happen every day!).
Wait, I got it all wrong.

You meant writing a simple executable that links with all plugins, doing nothing at all with them (maybe just calling whatever function is in there to ensure they are indeed linked), just so that launching the exe fetches all DLLs directly.

Yes, that could work.

[dcon]

@remi said in Help Bites:

You meant writing a simple executable that links with all plugins, doing nothing at all with them (maybe just calling whatever function is in there to ensure they are indeed linked), just so that launching the exe fetches all DLLs directly.

Exactly.

(Its WinMain could be as simple as a messagebox! "Yes! All (known) dependencies found!")

[Gribnit]

@remi said in Help Bites:

@dcon See @Gąska's post above, it seems like you've experimentally confirmed what he said?

Anyway, even if some cases trigger a more helpful message, what matters for me is that when something triggers the unhelpful one, there is no way to get more information.

Eh, the data is in memory somewhere, just go peek at it, you just need to calculate where it is vs having the API spoon-feed it to you. It'll be in a protected segment but w/e, it's windows, circumventing memory protection is left as an exercise for the reader.

[boomzilla]

@remi said in Help Bites:

But that last point (Linux is OK), and the fact that as a rule Qt developers are not total morons (not saying they're perfect, but I'd say they're "above average"), makes me wonder whether QPluginLoader can actually do better on Windows.

Probably depends on if the plugin (or whatever) is the thing that's missing or one of its linked dependencies.

[Gąska]

@remi said in Help Bites:

You meant writing a simple executable that links with all plugins, doing nothing at all with them (maybe just calling whatever function is in there to ensure they are indeed linked), just so that launching the exe fetches all DLLs directly.
Yes, that could work.

For bonus points, you could generate that executable at runtime and launch it just before loading the plugin, so you get a nicely formatted error dialog at the right time.

[Gribnit]

@Gąska said in Help Bites:

@remi said in Help Bites:

You meant writing a simple executable that links with all plugins, doing nothing at all with them (maybe just calling whatever function is in there to ensure they are indeed linked), just so that launching the exe fetches all DLLs directly.
Yes, that could work.

For bonus points, you could generate that executable at runtime and launch it just before loading the plugin, so you get a nicely formatted error dialog at the right time.

Anything to avoid a sane linking mechanism.

[remi]

@Gąska said in Help Bites:

For bonus points, you could generate that executable at runtime and launch it just before loading the plugin, so you get a nicely formatted error dialog at the right time.

That would only work on a dev's machine, I guess, since you need a compiler (and linker). So unlikely to work on Windows. Though it would have the advantage to be able to link with all the plugins that are present at that time, including those that might not have been compiled at the same time as the main executable.

I did something similar for fun and giggles on Linux some years ago. We had a bit of code that included a parser for user-specified equations and of course applying that equation billions of times was slow. The syntax of the equation was already C-like, enough so that I wrote some code that pasted that equation in a C function, compiled it as a library and dlopen()'ed it on the fly. That worked because on Linux it's less of a stretch to expect gcc to be installed, though to make it truly generic I would have had to be careful about versions (of... everything, basically). That still could have worked in my case because all the target machines were corporate installs of the same Linux distrib (so essentially I was guaranteed a fixed setup, identical to my work machine). In any case, it was far too wonky to ever make it to production code, but it was a fun couple of hours tinkering with that.

[dkf]

@Gąska said in Help Bites:

I always wondered how people pronounce that

My brain autocarrots SCOTUS to “scrotum”. I'm pretty glad I'm not based in the US.

[dkf]

@Gąska said in Help Bites:

If the program dynamically loads extra DLLs beyond those, then the only error is the one returned by WinAPI call.

The problem is that it's converted to just an error code at some point and those are just numbers. There's simply nowhere for any sort of extended error information (such as which damn file was it looking for) to be described. It's literally an API flaw. LoadLibraryEx doesn't fix it.

[topspin]

@remi said in Help Bites:

We had a bit of code that included a parser for user-specified equations and of course applying that equation billions of times was slow. The syntax of the equation was already C-like, enough so that I wrote some code that pasted that equation in a C function, compiled it as a library and dlopen()'ed it on the fly. That worked because on Linux it's less of a stretch to expect gcc to be installed, though to make it truly generic I would have had to be careful about versions (of... everything, basically).

Are we co-workers?

I've had exactly that use case and almost done that before, but then I decided I definitely won't be shipping clang along for this so if it's slow then tough luck.
Tangentially, for something similar I also made some Python bindings for internal use which allow nice scripting of stuff, but I'm also really wary of figuring out how to ship that.

[aitap]

@topspin See also: Rcpp and Cling. The latter is especially impressive because it presents the user with a C++ REPL.

[Gąska]

@remi said in Help Bites:

@Gąska said in Help Bites:

For bonus points, you could generate that executable at runtime and launch it just before loading the plugin, so you get a nicely formatted error dialog at the right time.

That would only work on a dev's machine, I guess, since you need a compiler (and linker).

Not really. A relatively simple template would suffice. PE format isn't terribly complicated, and all you'd need to do is replace one string. The program doesn't even need a runtime, it may exit immediately. The bigger concern is how not to look like malware (you know, generating new executables on the fly and running them...)

[Gribnit]

@Gąska said in Help Bites:

@remi said in Help Bites:

@Gąska said in Help Bites:

For bonus points, you could generate that executable at runtime and launch it just before loading the plugin, so you get a nicely formatted error dialog at the right time.

That would only work on a dev's machine, I guess, since you need a compiler (and linker).

Not really. A relatively simple template would suffice. PE format isn't terribly complicated, and all you'd need to do is replace one string. The program doesn't even need a runtime, it may exit immediately. The bigger concern is how not to look like malware (you know, generating new executables on the fly and running them...)

Maybe memoize the parsed functions instead of trying to make the parsing faster.

[remi]

@Gąska I see what you mean but really, at that point the bigger concern is just / .

It's not quite complicator's gloves because there isn't really a readily-available other way (manual or other) to get this exact information, but it's not that far because there are still things like depends. And for something that happens once in a while, it doesn't really seem worthwhile to develop and maintain a specific thing just for that.

Still, finding complicated solutions that are likely to end up as front-page articles is a part of what this site is about, so I appreciate the effort!

[Gribnit]

@remi said in Help Bites:

@Gąska I see what you mean but really, at that point the bigger concern is just / .

It's not quite complicator's gloves because there isn't really a readily-available other way (manual or other) to get this exact information, but it's not that far because there are still things like depends. And for something that happens once in a while, it doesn't really seem worthwhile to develop and maintain a specific thing just for that.

Still, finding complicated solutions that are likely to end up as front-page articles is a part of what this site is about, so I appreciate the effort!

If the parser produces an executable in-memory representation of whatever tree it parses from yon function, and if that tree can be parameterized again and again, it seems that it only needs to be parsed the first time the string is encountered, then could be retrieved based on the string's hash. But of course, I don't know what these functions are or if the parser is coupled tightly to the execution.

[remi]

@Gribnit said in Help Bites:

But of course, I don't know what these functions are or if the parser is coupled tightly to the execution.

Also, you've mashed together two (barely related) things and as a result make even less sense as usual.

[Gribnit]

@remi said in Help Bites:

@Gribnit said in Help Bites:

But of course, I don't know what these functions are or if the parser is coupled tightly to the execution.

Also, you've mashed together two (barely related) things and as a result make even less sense as usual.

They're related, I've left out too many steps. You're parsing expressions? formulae sound like expressions. Those usually parse to an in-memory representation like a tree of operators whose traversal performs the execution. If you can keep hold of yon in-memory representation and re-execute it, vs reconstruct it, it's likely to be rather faster.

As far as tightly coupling parsing to execution, that's what happens if there's no intermediate reusable in-memory representation of the expression. It would have been simpler to say "if it's not a real parser, just some hack job" maybe.

[remi]

@Gribnit said in Help Bites:

They're related, I've left out too many steps.

No they're not, or at least not for what @Gąska's talking about.

You're parsing expressions?

Not in the sub-thread where you started talking about memoization, no. Please try and read what you're replying to.

Now if you were writing all that as answers to the post where I actually mentioned parsing, that would make a bit more sense. And I could have told you from your first post that we were already doing that and it didn't help. But since you wrote all that as reply to @Gąska's subthread, which was clearly not about parsing, I won't tell you.

[Gribnit]

@remi said in Help Bites:

@Gribnit said in Help Bites:

They're related, I've left out too many steps.

No they're not, or at least not for what @Gąska's talking about.

You're parsing expressions?

Not in the sub-thread where you started talking about memoization, no. Please try and read what you're replying to.

Now if you were writing all that as answers to the post where I actually mentioned parsing, that would make a bit more sense. And I could have told you from your first post that we were already doing that and it didn't help. But since you wrote all that as reply to @Gąska's subthread, which was clearly not about parsing, I won't tell you.

I had no idea that French was so different. I would have expected Joffrey, or similar. Nice to meet you, Mr. Atwood!

[Zerosquare]

@Gribnit said in Help Bites:

Nice to meet you, Mr. Atwood!.

https://youtu.be/J3424sOpYjg

[remi]

@Gribnit said in Help Bites:

I had no idea that French was so different.

Yes, we have this weird thing where when someone responds to something, we assume that the discussion that follows is about this something, not another thing that was said before and that the response wasn't about.

Though I understand why it seems weird to you, as it requires actually reading other people's posts.

Sorry not sorry.

[Gribnit]

@remi said in Help Bites:

@Gribnit said in Help Bites:

I had no idea that French was so different.

Yes, we have this weird thing where when someone responds to something, we assume that the discussion that follows is about this something, not another thing that was said before and that the response wasn't about.

Though I understand why it seems weird to you, as it requires actually reading other people's posts.

Sorry not sorry.

Oh. I followed the links back and the parsing stuff was mentioned in the same post where this complete-compiler-for-a-limited-regular-language monstrosity was being discussed. Using the parser as designed rather than horrifically abusing it, came to mind. However, there is always a place in history for another Chaucat.

[remi]

@Gribnit said in Help Bites:

the same post where this complete-compiler-for-a-limited-regular-language monstrosity was being discussed

I... don't even know what you're talking about. I can't see a way to read the discussion to get to this idea.

[Gribnit]

@remi ah. by followed the links back, I meant that I followed them back, to verify whether the thread was crossed or just cycling back to a common root, I came to Help Bites where it seemed like a compiler was being reached for to parse up some expressions to where, I suppose, something could evaluate them. But there are all sorts of places where code generation might be used instead of a more appropriate dynamism, so who knows.

[HardwareGeek]

@remi said in Help Bites:

make even less sense as usual.

Impossible.

[Tsaukpaetra]

@Captain said in Help Bites:

Is there any way to write a stored procedure that accepts a block as an argument? I end up writing a ton of boiler plate all day every day. Help me abstract that away. T-SQL.

You mean like dynamic SQL?

Ranga Babu  /  Aug 15, 2019

Dynamic SQL in SQL Server

This article will provide a review on how to construct and execute dynamic SQL in SQL Server using EXEC command and sp_executesql extended stored procedure.

[Tsaukpaetra]

@Zerosquare said in Help Bites:

@remi said in Help Bites:

it finds the machine (there is no DNS on my LAN so how? no idea, but it does).

NetBIOS over TCP/IP - Wikipedia

Basically, name resolution via network broadcasts, for small networks that don't have a proper name server.

And of course Microsoft has deprecated that and changed it to user Simple Service Discovery Protocol to "advertise" that a particular computer is a file server....

[Tsaukpaetra]

@remi said in Help Bites:

When a DLL/EXE fails to load because it's missing some DLLs, is it possible to get Windows to tell you which module is missing rather than the cryptic "The specified module could not be found"?

I've often used Process Monitor to see file system accesses (and failures).

Usually when a library is being loaded it will fail to be loaded from several locations before finally being found and read, but oftentimes you'll find a series of failures with no success. This in and of itself isn't always the one-shot deal, but a good pointer to possibilities.

@remi said in Help Bites:

By contrast, on Linux if I do the same mistake (forget one library), I end up with several lines of messages telling me all the places it looked for it and failed (this is the message we get from QPluginLoader::errorString(), itself bubbled up from dlopen(), I guess). While this is too verbose in that case (though is useful when debugging search path issues!), at least it clearly states (several times!) the name of the library that it failed to load, so just looking at whatever log file is used allows me to pinpoint the issue quickly.

This is essentially the same thing, but not using anything built-in to the actual runtime library.

As for actually having the program itself tell you? Yeah that's going to be a non-starter, as reading the resultant discussion has revealed.

In theory if you had debugging rights the process could launch a watcher app specifically targeting the main app to watch the process in the manner I described above, but that's probably a bit too deep...

[remi]

@Tsaukpaetra said in Help Bites:

I've often used Process Monitor to see file system accesses (and failures).

Thanks, that's a good one! It's (unsurprisingly) very verbose and somewhat hard to pinpoint the exact failure point because as you say even for successful loads there are often some failures from other locations first, but (surprisingly) the filters seem to work reasonably well, so by filtering by the application name and then the error type it's easy to whittle down the list to a manageable size. It's basically looking at strace output, which isn't ideal, but would help in this case.

The best thing is that procmon can be installed (and doesn't even need an install, just being copied) on any computer, even a non-dev one, so for tricky cases I can tell the user to grab it and look at it (or send me the events). Of course it requires local admin rights (you're going to poke into every process internals, after all!), so maybe not all users will be able to run it, but it's still far better than a dev-only tool.

As for actually having the program itself tell you? Yeah that's going to be a non-starter, as reading the resultant discussion has revealed.

Yeah, that would have been the ideal solution, but that's not going to happen.

[loopback0]

@Tsaukpaetra said in Help Bites:

@Captain said in Help Bites:

Is there any way to write a stored procedure that accepts a block as an argument? I end up writing a ton of boiler plate all day every day. Help me abstract that away. T-SQL.

You mean like dynamic SQL?

Ranga Babu  /  Aug 15, 2019

Dynamic SQL in SQL Server

This article will provide a review on how to construct and execute dynamic SQL in SQL Server using EXEC command and sp_executesql extended stored procedure.

What could possibly go wrong...