WTF Bites

I tried to use good old Dependency Walker to check that some DLLs all depend on the right (and same) runtime and Qt libraries. What used to work fine for the same file and the same version of depends.exe, now takes half a million years on a different machine and then gives me this:

Bildschirmfoto 2022-12-15 um 12.20.09.png

It always had the warnings about the delay load stuff and a bunch of weird dependencies on whatever that API sets stuff is (WhyTF would user32.dll depend on all that shit?). But now apparently it detects a circular dependency where kernel32 depends on it and traverses that a billion layers deep.
But hey, at least it shows an error that it detected a circular dependency, even if it didn't actually bail out.

Zerosquare

If a library has a circular dependency, how could it get loaded in the first place?

Bulb

@Zerosquare I am not sure how exactly the resolution works in Windows, but if the resolution is lazy, and I think it is, there is no problem, really. First library gets mapped, then the second gets mapped, it is found that the first already is and now all the symbols can be found when needed.

Carnage

@dcon Here, we just burn all of it. Carton and milk boxes and stuff is for when there is already a pretty decent fire going in the furnace. When it slows down, just jam some plastic in there for a bit of a boost. I guess it's nice to the workers to have things somewhat sorted, so they can do that though.

sebastian.galczynski

There was once a dollar sign in the password. That password was passed from Gitlab CI to a shell variable to yaml to another shell variable. It didn't work.

4 hours wasted.

PleegWat

@sebastian-galczynski That is very likely a shell code injection vulnerability. Depending on who would be able to set this password, that can be a high-severity security bug. This kind of bad coding practice indicates it is likely there are more similar bugs in this integration.

LaoC

@sebastian-galczynski You could try setting your password to /bin/bash (iterate sh, /usr/bin etc.) and see whether ${SHELL} works for authentication.

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski That is very likely a shell code injection vulnerability. Depending on who would be able to set this password, that can be a high-severity security bug.

In this case this are the same people that can break everything anyway, so it's not an issue.

This kind of bad coding practice indicates it is likely there are more similar bugs in this integration.

I'll try to reproduce this in a clean environment. There's an additional variable substitution somewhere, probably in docker-compose (they have issues with that), but when we replaced the dollar with 2 dollars, one container still failed to reach the thing secured with the password, so there may be even more places.

sebastian.galczynski

OK, so the additional substitution happens in docker-compose (and in docker compose):

seba@bojo:~/dollar-bug$ cat docker-compose.yml                                                                                                                                                             
version: "3.9"
services:
    foo:
        image: alpine:latest
        environment:
            DUPA: "${DUPA}"
        command: 
            - 'sh'
            - '-c'
            - 'echo $DUPA'

seba@bojo:~/dollar-bug$ export DUPA='ABC$DEF'
seba@bojo:~/dollar-bug$ echo $DUPA
ABC$DEF

seba@bojo:~/dollar-bug$ docker compose up
[+] Running 1/0
 ⠿ Container dollar-bug-foo-1  Created                                                                                                                                                                   0.0s
Attaching to dollar-bug-foo-1                                                                                                                                                                                 
dollar-bug-foo-1  | ABC
dollar-bug-foo-1 exited with code 0


seba@bojo:~/dollar-bug$ docker-compose up                                                                                                                                                                  
Recreating dollar-bug-foo-1 ... done
Attaching to dollar-bug_foo_1
foo_1  | ABC
dollar-bug_foo_1 exited with code 0

sebastian.galczynski

Ok, now can someone explain is happening here? Where does that '1' come from?

seba@bojo:~/dollar-bug$ export DUPA='ABC$$DEF'

seba@bojo:~/dollar-bug$ docker-compose up
Recreating dollar-bug_foo_1 ... done
Attaching to dollar-bug_foo_1
foo_1  | ABC1DEF
dollar-bug_foo_1 exited with code 0


seba@bojo:~/dollar-bug$ docker compose up                                                                                                                                                                  
[+] Running 1/1
 ⠿ Container dollar-bug_foo_1  Recreated                                                                                                                                                                 0.1s
Attaching to dollar-bug-foo-1                                                                                                                                                                                 
dollar-bug-foo-1  | ABC1DEF
dollar-bug-foo-1 exited with code 0

seba@bojo:~/dollar-bug$ echo $DUPA                                                                                                                                                                         
ABC$$DEF

PleegWat

@sebastian-galczynski The bash variable $$ holds the current process's PID. In this context that is probably init's PID, which is always 1.

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski The bash variable $$ holds the current process's PID. In this context that is probably init's PID, which is always 1.

Hmm, that means the additional expansion doesn't happen in YAML, but in some shell. Let's try '\$'.

seba@bojo:~/dollar-bug$ export DUPA='ABC\$DEF'
seba@bojo:~/dollar-bug$ docker compose up
[+] Running 1/1
 ⠿ Container dollar-bug-foo-1  Recreated                                                                                                                                                                 0.2s
Attaching to dollar-bug-foo-1                                                                                                                                                                                 
dollar-bug-foo-1  | ABC$DEF
dollar-bug-foo-1 exited with code 0

Yep, it works. But how exactly did this substitution happen? Docker-compose passed some unquoted arguments to docker?

PleegWat

@sebastian-galczynski Since the pid you're getting is 1, it is almost certainly happening inside a docker container.

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski Since the pid you're getting is 1, it is almost certainly happening inside a docker container.

OK, but inside container the variable either holds 'ABC$$DEF', or was substituted before being passed to the container. But if it was substituted, it should hold another pid than 1, so how come?
I think I need to take a break.

PleegWat

@sebastian-galczynski Something inside the container must be expanding the ABC$$DEF

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski Something inside the container must be expanding the ABC$$DEF

Something, but what? It's not that it simply passed it like this:

seba@bojo:~/dollar-bug$ docker run -e "DUPA=ABC$$DEF" alpine env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=a986afc38c9b
DUPA=ABC1200742DEF
HOME=/root

seba@bojo:~/dollar-bug$ echo $$
1200742

This way it shows the host's shell PID, not container's PID 1.

And if I use single quotes, it simply doesn't substitute at all.

PleegWat

@sebastian-galczynski So will echo "ABC$$DEF". Compare echo 'ABC$$DEF', with single instead of double quotes.

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski So will echo "ABC$$DEF". Compare echo 'ABC$$DEF', with single instead of double quotes.

With single quotes it doesn't substitute at all. I tried various combinations of slashes, and in no way I can get 'ABC1DEF'.

PleegWat

@PleegWat And these won't substitute the $$ either:

test='ABC$$DEF'
echo $test
echo "$test"

error

A bug which obviously can't happen is happening. I put in logging statements to see is going on, and of course, the bug stopped happening.

PleegWat

@sebastian-galczynski A bash -c inbetween (or another wrapper which calls a program using system() instead of using the exec() family) may cause double expansion.

dkf

@sebastian-galczynski said in WTF Bites:

I think I need to take a break.

QFT &

sebastian.galczynski

@PleegWat said in WTF Bites:

@sebastian-galczynski A bash -c inbetween (or another wrapper which calls a program using system() instead of using the exec() family) may cause double expansion.

Bash -c with double quotes works.

$ export test='ABC$$DEF'
$ bash -c 'echo $test';
ABC$$DEF
$ bash -c "echo $test";
ABC1224150DEF
$ echo $$
1200742

So it double-quotes the args in the command list?

BernieTheBernie

@error said in WTF Bites:

A bug which obviously can't happen is happening. I put in logging statements to see is going on, and of course, the bug stopped happening.

Compiler or processor optimizations?
Delayed execution with side effects?

dkf

@PleegWat It's always "fun" when you run across code that blithely uses system(). It's so very often a source of trouble.

It also sounds like credentials are being passed via command line arguments. That's horribly unsafe (or ) as command line arguments are publicly-visible information about a process; anything on the system can get them using ps with the right options. Environment variables are no better (they just require a different option). Now maybe things will be OK inside the docker container, but it does mean that if you have a vulnerability on the inside then the credentials are very exposed in that context, making it much easier for the attacker to expand the attack surface.

sebastian.galczynski

@dkf said in WTF Bites:

It also sounds like credentials are being passed via command line arguments.

It's something more mysterious. The double dollar worked for one container, and this container doesn't pass credentials via arguments anywhere. I even looked into the code and nope, it only uses node's builtin Http(s).
It's also also possible that the double dollar broke the other container (python-based), which would otherwise work correctly. We ended up creating another user with a better password (the existing root password couldn't be changed quickly, because admin's a ).

HardwareGeek

@error said in WTF Bites:

I put in logging statements to see is going on, and of course, the bug stopped happening.

It works.

BernieTheBernie

@dkf said in WTF Bites:

That's horribly unsafe (or ) as command line arguments are publicly-visible information about a process; anything on the system can get them using ps with the right options. Environment variables are no better (they just require a different option).

Customer complaint: Your dictation system logs username and password of our RIS (radiology information system).
Fact: the dictation system logged all environment variables at startup. Those RIS credentials were...
(A true story! Not made up for this post)

izzion

@BernieTheBernie said in WTF Bites:

(A true story! Not made up for this post)

Applied Mediocrity

@PleegWat said in WTF Bites:

may cause double expansion.

Dieting threads are

Zecc

Watching a tutorial on a thing.

"There are two methods. First let me show you the second method"

Okay, if no one's numbering the alternative ways of doing the same thing except you, why are you arbitrarily saying you're going to show the second method first?

Applied Mediocrity

@Zecc Perhaps not arbitrarily. I know I've used similar language. Suppose that:

the first method is more compatible, widely used, official, but kind of crap.
the second method is what we prefer (or what we'd like to sell, anyway)

I'd start with the second method, because that's what sticks into people's minds. Once you show something that works good enough,
alternatives are rarely considered. But the written material (slides etc.) will still show that we've put the official one first.

Benjamin Hall

As usual, the WTF is xCode.

Apple has very strong requirements for build numbers--they must be both unique and monotonically increasing. No uploaded build can ever have the same number as a previous build, and you can't jump around either. And all targets MUST have the same version, especially if you've got an Apple Watch extension (ie an Apple Watch app as a target).

Ok, fine. We can script that, right? Apple provides a nice tool: agvtool. Except...it can only be used

from the command line (not as a build-time script)
with xcode closed.

Otherwise, it aborts the build because it changes the project file and xcode panics. xCode does support command line builds, but only kinda barely. And it's fragile.

So people have come up with all sorts of tools to do this automatically. All of which are fragile (especially since xCode is constantly changing where things live) and have to be set up just right.

Bulb

@Benjamin-Hall said in WTF Bites:

As usual, the WTF is xCode.

Apple has very strong requirements for build numbers--they must be both unique and monotonically increasing. No uploaded build can ever have the same number as a previous build, and you can't jump around either. And all targets MUST have the same version, especially if you've got an Apple Watch extension (ie an Apple Watch app as a target).

Ok, fine. We can script that, right? Apple provides a nice tool: agvtool. Except...it can only be used

from the command line (not as a build-time script)

Wait, . I thought the point of command-line interface is calling it from a scri…

with xcode closed.

Oh, you mean you can't call it from a step in the project. Well, you are holding it wrong anyway. The release build—you are doing this on a build server, ain't you?—needs to do steps before and after calling xcodebuild to build the packages themselves anyway, so it can call this tool too.

Normally my release builds—in various languages—follow the same general pattern: 1) determine the version, update the project files and commit them locally, 2) build and publish the packages and 3) mark the project files as pre-release again, commit and push to the source control. This is the process for mvn release, but it applies to all projects where the version is written in the build files.

So you'd use the agvtool in step 1, xcodebuild in step 2 and maybe agvtool again in step 3 (I don't know whether there is any pre-release state in this case).

Benjamin Hall

@Bulb said in WTF Bites:

@Benjamin-Hall said in WTF Bites:

As usual, the WTF is xCode.

Apple has very strong requirements for build numbers--they must be both unique and monotonically increasing. No uploaded build can ever have the same number as a previous build, and you can't jump around either. And all targets MUST have the same version, especially if you've got an Apple Watch extension (ie an Apple Watch app as a target).

Ok, fine. We can script that, right? Apple provides a nice tool: agvtool. Except...it can only be used

from the command line (not as a build-time script)

Wait, . I thought the point of command-line interface is calling it from a scri…

with xcode closed.

Oh, you mean you can't call it from a step in the project. Well, you are holding it wrong anyway. The release build—you are doing this on a build server, ain't you?—needs to do steps before and after calling xcodebuild to build the packages themselves anyway, so it can call this tool too.

Normally my release builds—in various languages—follow the same general pattern: 1) determine the version, update the project files and commit them locally, 2) build and publish the packages and 3) mark the project files as pre-release again, commit and push to the source control. This is the process for mvn release, but it applies to all projects where the version is written in the build files.

So you'd use the agvtool in step 1, xcodebuild in step 2 and maybe agvtool again in step 3 (I don't know whether there is any pre-release state in this case).

Yeah. But I have to make sure that our (slightly WTF) release process can properly handle this. Because xcode build servers...aren't really a thing anymore. At least without shelling out to someone to have it in a cloud somewhere, since our main deployment pipeline runs in a Jenkins container on a linux host somewhere in AWS.

My strong preference is to do this

Only on actual releases (not ad-hoc builds we send around to testers for individual tickets)
As part of building the release itself
In a way that can be scripted so that, eventually when we do have a proper build server, it can do this all automatically.

Which is as you say. A script that calls agvtool and then xcodebuild (we don't need a 2nd agvtool step, I don't think) and then let the build tool do the related git operations automatically on the right branch.

Rhywden

WTF of my day: So, I've got this (secondary) 2 TB SATA-SSD from Western Digital. And it looked like the thing was dying - I was getting regular timeouts / reset events in the Windows event viewer (whereupon the SSD simply would not respond anymore). Though it might also have been a driver issue. So I updated the relevant drivers with the one from my motherboard's vendor.

And the thing then promptly shat the bed completely, as in "A device has been suddenly removed" entry in the Event viewer (along with the SSD vanishing completely from the whole system). Even after a reboot it'd vanish in the same manner after a few seconds, even if I simply let it sit there. Linux exhibited similar symptoms, sometimes even being unable to mount the volume outright.

Well, okay. My motherboard has 3 NVMe slots with the 1st one already taken. So I ordered another NVMe SSD and put it into the 2nd slot. According to the manual, if NMVe_2 is in use, SATA_2 should not be used. That's what the (dying) SATA SSD was connected to.

But I still had the hope of getting at least some data off this disk (nothing essential, just faster than redownloading everything). So I switched it from SATA_2 to SATA_3.

And since then, no problems.