Linux locks and a kinder, gentler Linus
-
-
@izzion said in Linux locks and a kinder, gentler Linus:
that's..... surprisingly calm and rational response from him...
I guess he hasn't broken his new years resolutions yet this year?
-
@Vixen said in Linux locks and a kinder, gentler Linus:
@izzion said in Linux locks and a kinder, gentler Linus:
that's..... surprisingly calm and rational response from him...
His arguments reminds of the sorts of things I have to explain to people. Different domain, of course, but basically shooting down their simplified view of things and explaining why their seemingly obvious plan is doomed to fail.
-
Good read. (The original blog post and Linus' replies, that is, not the Phoronix summary.) I guess this is a nice reminder that NUMA is hard and that you shouldn't make any assumptions about the OS scheduler, even if you're a game developer and really, really want to.
-
I'll be sure to be totally surprised when they find the actual problem is caused by Google Stadia
-
@hungrier said in Linux locks and a kinder, gentler Linus:
I'll be sure to be totally surprised when they find the actual problem is caused by Google Stadia
If you read through the whole discussion, it's pretty obvious that the problem has nothing to do with Stadia, but only with game developers being used to the Windows scheduler and its guarantees.
-
@dfdub I've been reading the thread attached to that article (not the original one that I've seen referenced where Linus's posts actually come from), mostly to see everyone dunking on the idiot that would have preferred Linus to use softer language
-
@hungrier
If you have some time, the interesting parts are:- The original blog post (mostly for the background and the spinlock implementation)
- The reply to Linus' original post by the author of the blog post
- Linus' long reply in which he explains the wrong assumptions and why the Linux schedulers (have to) work differently
The shortest tl;dr is "don't use spinlocks and don't implement them in userspace".
The slightly longer tl;dr is "you cannot expect the scheduler to schedule the thread you want if you don't tell it what thread that is, even if all your other threads are telling the scheduler to do something else". Apparently, Windows (potentially only in game mode?), Xbox and PS4 do that, but it doesn't make sense for a general-purpose scheduler to behave this way.
-
@dfdub Even there, using yield() is highly dubious, pretty much for the same reasons Linus points out in the extended discussion.
IMO, so are handwritten spinlocks most of the time - the underlying primitives (CRITICAL_SECTION, and IIRC some flavours of pthread_mutex) typically already spin for a short period in userspace before going for a syscall.
-
@dfdub I'm not sure if I'll have time to read all of those but I've bookmarked your post and will hopefully at some point remember to check it out.
-
@hungrier said in Linux locks and a kinder, gentler Linus:
@dfdub I'm not sure if I'll have time to read all of those but I've bookmarked your post and will hopefully at some point remember to check it out.
"Okay Google, Have Cortana remind me later this week to remind @hungrier about that post what they want to read."
"Who is Cortana? Are you cheating on me?!"
-ut-oh-
-
@Vixen I actually managed that recently:
Hei Alexa... oh, damn
: Now, that's a bit embarassing...
-
@dfdub said in Linux locks and a kinder, gentler Linus:
@hungrier said in Linux locks and a kinder, gentler Linus:
I'll be sure to be totally surprised when they find the actual problem is caused by Google Stadia
If you read through the whole discussion, it's pretty obvious that the problem has nothing to do with Stadia, but only with game developers being used to the Windows scheduler and its guarantees.
I ,as a game developer, have no idea about any guarantees the Windows Scheduler makes.
I however, also don't trust the Scheduler and assume things can complete immediately or in forever cycles and program accordingly.
-
@Tsaukpaetra said in Linux locks and a kinder, gentler Linus:
any guarantees the Windows Scheduler makes.
according to Raymond Chen...... it doesn't make any promises above and beyond running your threads in as efficeint a manner as it can figure out.
he's also be annoyed that the game developers did shit like this instead of just doing the right thing.
he's be just as acerbic as Tovalds, but much less...... pyrotechnic.
-
Oh well, this will all be a non-problem in 3 months when Stadia shuts down anyway...
-
@levicki mostly that seems idiotically off-topic.
Otherwise it might be bait to get Linus to lose his temper again, but he’s not going to fall for that.
-
@dfdub said in Linux locks and a kinder, gentler Linus:
@hungrier
If you have some time, the interesting parts are:- The original blog post (mostly for the background and the spinlock implementation)
- The reply to Linus' original post by the author of the blog post
- Linus' long reply in which he explains the wrong assumptions and why the Linux schedulers (have to) work differently
The shortest tl;dr is "don't use spinlocks and don't implement them in userspace".
The slightly longer tl;dr is "you cannot expect the scheduler to schedule the thread you want if you don't tell it what thread that is, even if all your other threads are telling the scheduler to do something else". Apparently, Windows (potentially only in game mode?), Xbox and PS4 do that, but it doesn't make sense for a general-purpose scheduler to behave this way.
Linus:
Yes, it turns out that certain simple schedulers get exactly the behavior you want. The best way to get exactly your behavior is to have a single run-queue for the whole system, and make 'sched_yield()' always put the thread at the back of that run-queue, and pick the front one instead.
IOW, for your [use case], the optimal scheduler is a stupid one that does not try to take any kind of CPU cache placement into account, does not try to at all optimize the run-queues to be thread-local, and just basically treats the scheduling decision as if we were still running one single CPU core, and that CPU had no cache locality issues.
Heh, that's been ck's point since the beginning. Linux schedulers are overwrought and not very good for desktop computing.
-
@Atazhaia said in Linux locks and a kinder, gentler Linus:
Oh well, this will all be a non-problem in 3 months when Stadia shuts down anyway...
We can only hope...
-
@Buddy said in Linux locks and a kinder, gentler Linus:
Heh, that's been ck's point since the beginning. Linux schedulers are overwrought and not very good for desktop computing.
I don't think that's an accurate representation of the tradeoffs involved here. It sounds like Windows only behaves the way it does to please buggy software that expects it to do just that.
I don't see a good reason to have a single run queue for all CPUs in a modern scheduler (or emulate one).
-
@dfdub said in Linux locks and a kinder, gentler Linus:
@Buddy said in Linux locks and a kinder, gentler Linus:
Heh, that's been ck's point since the beginning. Linux schedulers are overwrought and not very good for desktop computing.
I don't think that's an accurate representation of the tradeoffs involved here. It sounds like Windows only behaves the way it does to please buggy software that expects it to do just that.
I don't see a good reason to have a single run queue for all CPUs in a modern scheduler (or emulate one).
Con Kolivas did see a reason for that. His single run queue scheduler does perform better for interactive tasks on a desktop workstation than the stock Linux scheduler. But that's not the workload that Linux is designed for.
-
@Buddy
I don't have any personal experience with BFS, so I'll take your word for it.
-
@dfdub said in Linux locks and a kinder, gentler Linus:
buggysoftware made to a bad manual that expects it to do just thatBingo.
There was once a tutorial, from Microsoft, that had a delay(0) kind of spinlock in it. Recommended for real-time performance. IIRC.
They've had to deal with it since.
It also kinda explains why multi-core machines boot Windows exponentially faster than single-core ones.If it were me, I'd deal with this by silently increasing the delay from 0 to 1 ms when the call comes from userspace. If it wants to spin, then it shall yield for longer than 0 while waiting for lock.
Edit:
added clarification
-
@Vixen said in Linux locks and a kinder, gentler Linus:
according to Raymond Chen...... it doesn't make any promises above and beyond running your threads in as efficeint a manner as it can figure out.
he's also be annoyed that the game developers did shit like this instead of just doing the right thing.Yeah well, no one cares what promises Microsoft makes. Because we know they'll keep backward compatibility at any cost. So I don't have to care about "undefined behavior" or "doing the right thing" or stuff like that, if it works it works.
Like, you should have accepted that by now, Mr. Chen. Pretending you won't have to deal with that stuff in the future is literal delusion.
-
@acrow said in Linux locks and a kinder, gentler Linus:
If it wants to spin, then it shall yield.
You never want to yield inside a spinlock, that's just plain dumb.
Essentially, the only reason you'd use a spinlock on a modern (multi-core) desktop is because you want the low latency/cost and because you have very little contention already (and everybody only holds the lock very briefly).
If that's not the case, just use the OS provided synchronization primitives. With the OS provided primitives, you end up with at most one or two syscalls (e.g., if you couldn't get the lock immediately), and you give the OS additional information about when to schedule your thread again (for example, don't schedule it as long as the lock is held elsewhere).
With a yield in a spinlock, you end up with a syscall each iteration. That gives you the worst of both worlds - you pay for a lot of syscalls, but you don't gain anything (in fact, you make the load worse because you're still polling).
-
@cvi said in Linux locks and a kinder, gentler Linus:
@acrow said in Linux locks and a kinder, gentler Linus:
If it wants to spin, then it shall yield.
You never want to yield inside a spinlock, that's just plain dumb.
Of course not. I never suggested to do that?
Oh. I see. You didn't understand my point. Let me state it more clearly then.
This is (paraphrased) from an ancient Microsoft programming tutorial:
do
{
delay(0);
}
while (getLock() == false);
And it's still followed by all kinds of idiots.
My suggestion for a band-aid:
Treat all delay(0) system calls as delay(1). This way, the idiots still making this kind of locks will hopefully take notice.Edit:
added clarification
-
This post is deleted!
-
@Buddy said in Linux locks and a kinder, gentler Linus:
Con Kolivas did see a reason for that. His single run queue scheduler does perform better for interactive tasks on a desktop workstation than the stock Linux scheduler. But that's not the workload that Linux is designed for.
System that's spent 3 decades utterly failing to break into the desktop market was never designed with the technical needs of a desktop computer in mind! News at 11!
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@Buddy said in Linux locks and a kinder, gentler Linus:
Con Kolivas did see a reason for that. His single run queue scheduler does perform better for interactive tasks on a desktop workstation than the stock Linux scheduler. But that's not the workload that Linux is designed for.
System that's spent 3 decades utterly failing to break into the desktop market was never designed with the technical needs of a desktop computer in mind! News at 11!
What do you mean? I've seen it in desktop use for a decade already.
Just not in any place that needs to handle Office, Skype, Adobe, ...
-
@acrow said in Linux locks and a kinder, gentler Linus:
What do you mean?
The perpetual predictions that "this year will be the year of the Linux desktop", followed by its market share not hitting the S-curve tipping point that year, and in fact barely budging at all.
-
@Mason_Wheeler Well, if it didn't happen this year, then it will not happen ever. Win7 EOL and all that.
Speaking of EOL-7. I'll push Ubuntu Linux on an unsuspecting grandma later this month. Let's see how long it'll take before she coughs up which SkypeJavaSyphillis she actually needs installed, in addition to "just a bit of e-mail and banking". And then I'll cough up for that Win10 license for her.
-
@acrow said in Linux locks and a kinder, gentler Linus:
Speaking of EOL-7. I'll push Ubuntu Linux on an unsuspecting grandma later this month. Let's see how long it'll take before she coughs up which SkypeJavaSyphillis she actually needs installed, in addition to "just a bit of e-mail and banking". And then I'll cough up for that Win10 license for her.
Get her a tablet or chromebook. Except for people actively creating a lot of content, such devices are entirely adequate.
-
@dkf said in Linux locks and a kinder, gentler Linus:
@acrow said in Linux locks and a kinder, gentler Linus:
Speaking of EOL-7. I'll push Ubuntu Linux on an unsuspecting grandma later this month. Let's see how long it'll take before she coughs up which SkypeJavaSyphillis she actually needs installed, in addition to "just a bit of e-mail and banking". And then I'll cough up for that Win10 license for her.
Get her a tablet or chromebook. Except for people actively creating a lot of content, such devices are entirely adequate.
plus if she really needs Skype there's an APK for it that'll run on both of those no problem.
Can't help with the STI or the Syphillis.... That's her problem......
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@acrow said in Linux locks and a kinder, gentler Linus:
What do you mean?
The perpetual predictions that "this year will be the year of the Linux desktop", followed by its market share not hitting the S-curve tipping point that year, and in fact barely budging at all.
Which says nothing about the fact that it actually performs just fine. I dunno...perhaps there was a time when the minutiae of the scheduler mattered in a noticeable way for a desktop experience. That's not today. That you think "not hitting the S-curve" matters in this discussion is pretty funny considering your opinion of Apple.
I suppose if I had to run intrusive virus scanners on Linux like I have to on Windows then the experience on Linux might be as clunky as it is on Windows today.
-
@boomzilla Wait, I'm confused. What is one of the most enthusiastic contributors to the thread about cheap Linux IoT devices getting pwned en masse because of their abysmal security doing trotting out the old "Linux is secure and doesn't need malware countermeasures" argument that's been thoroughly discredited by said IoT security disasters?
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@boomzilla Wait, I'm confused. What is one of the most enthusiastic contributors to the thread about cheap Linux IoT devices getting pwned en masse because of their abysmal security doing trotting out the old "Linux is secure and doesn't need malware countermeasures" argument that's been thoroughly discredited by said IoT security disasters?
I literally have no idea what you're talking about here.
-
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@boomzilla Wait, I'm confused. What is one of the most enthusiastic contributors to the thread about cheap Linux IoT devices getting pwned en masse because of their abysmal security doing trotting out the old "Linux is secure and doesn't need malware countermeasures" argument that's been thoroughly discredited by said IoT security disasters?
well....... if the applications that these IoT devices run were designed to themselves be secure that wouldn't happen. but when you put an admin page on every IP address, and have a default username/password that no user is ever going to change..... or any number of a thousand oand five security vulnerabilities in your application.... can you really be surprised that your application can be pwned, and because you made you rapplication root to make programming easier for you.... once your app is pwned your entire device is pwned.
Linux wasn't what was pwned. Your application was.
And sure you say, this was all done because the device promised it was secure, but i have my doubts that even if that "promise" wasn't there that the applications would have been designed with security in mind. If security was a consideration you wouldn't be able to open IoT locks with a credit card in the door jam, or a pin into a exposed reset port, or be able to disassemble the lock while it is locked, or -blathers on for ten minutes putting the audience to sleep entirely-
-
@acrow said in Linux locks and a kinder, gentler Linus:
My suggestion for a band-aid:
Treat all delay(0) system calls as delay(1). This way, the idiots still making this kind of locks will hopefully take notice.Ok, fair -- I can get behind that in fact. (s/delay/yield/, but, yeah, whatever).
-
@cvi said in Linux locks and a kinder, gentler Linus:
@acrow said in Linux locks and a kinder, gentler Linus:
My suggestion for a band-aid:
Treat all delay(0) system calls as delay(1). This way, the idiots still making this kind of locks will hopefully take notice.Ok, fair -- I can get behind that in fact. (s/delay/yield/, but, yeah, whatever).
No, @acrow really means delay. Make each iteration really wait a short period, the idea being to make that kind of tomfuckery painful and expensive. It will also guarantee that if there's a lot of contention, sometimes someone (selected at random by the Mistress of Pain) will get starved for a long time.
-
@Steve_The_Cynic said in Linux locks and a kinder, gentler Linus:
Mistress of Pain
do you have her number? Is she taking new clients?
asking for a.... friend... yes a friend.
-
@Steve_The_Cynic The original post mentions delay(0), which I assume was a roundabout way of yielding the thread's current time slice (rather than being a nop). So, delay(1) would yield the current time slice and make sure the thread isn't scheduled for one unit of time (alas, make that unit a second rather than a millisecond ).
However, sure, I can also get behind making it not yield, but actually busy waiting for the amount of time. That would be even more evil^Wgood.
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
Who cares? What are you talking about?
-
@boomzilla OK, laying it out step by step:
The reason IoT is a massive dumpster fire of security holes is because they're almost universally running on Linux, which has never taken security seriously. All throughout the 90s and 00s, the discussion went like this:
: Look at with all their need for antivirus. We don't have that!
: Yeah, because nobody's targeting your systems because no one's using them. Should that change at some hypothetical future point, and you actually get enough market share to be a worthwhile target, you'll be the biggest security mess of all, because you've never had to take it seriously.And then some brillant people went and invented the IoT, and that's exactly what happened. People built devices, using Linux to save money, and ended up getting exactly what they paid for as these devices got hacked en masse to create botnets an order of magnitude larger than anything the world had ever seen before.
You're active in our thread about the IoT being a massive mess.
Therefore, you're presumably aware of all this already.
So why are you still trotting out an outdated talking point that got irrevocably blown to smithereens years ago?
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@boomzilla OK, laying it out step by step:
The reason IoT is a massive dumpster fire of security holes is because they're almost universally running on Linux, which has never taken security seriously.
Which has exactly nothing to do with what I said, so why are you talking about that here replying to me instead of over in the IoS thread?
You're active in our thread about the IoT being a massive mess.
Yes.
Therefore, you're presumably aware of all this already.
Duh.
So why are you still trotting out an outdated talking point that got irrevocably blown to smithereens years ago?
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
So why are you still trolling
if you're going to try and get in a flame war...... could you at least do it outside? I'd like to avoid scorching the hardwood floors..... again.
-
@boomzilla said in Linux locks and a kinder, gentler Linus:
Which has exactly nothing to do with what I said
@boomzilla said in Linux locks and a kinder, gentler Linus:
I suppose if I had to run intrusive virus scanners on Linux like I have to on Windows then the experience on Linux might be as clunky as it is on Windows today.
...you were saying?
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
@boomzilla said in Linux locks and a kinder, gentler Linus:
Which has exactly nothing to do with what I said
@boomzilla said in Linux locks and a kinder, gentler Linus:
I suppose if I had to run intrusive virus scanners on Linux like I have to on Windows then the experience on Linux might be as clunky as it is on Windows today.
...you were saying?
Exactly what I said. All this nonsense about IoS is your own invention and I can't be held responsible for it.
-
@levicki said in Linux locks and a kinder, gentler Linus:
@boomzilla said in Linux locks and a kinder, gentler Linus:
Which says nothing about the fact that it actually performs just fine. I dunno...perhaps there was a time when the minutiae of the scheduler mattered in a noticeable way for a desktop experience. That's not today.
I disagree. Try using Linux for DAW and let me know how it goes. Without at least changing the default scheduler you won't get far even if you are a guru who knows how to set up the Linux audio stack.
:sigh: Is DAW anything like CP? I have no idea what you're talking about, either, unfortunately, but that's just down to unfamiliar acronyms, not shoulder aliens.
-
@boomzilla said in Linux locks and a kinder, gentler Linus:
@levicki said in Linux locks and a kinder, gentler Linus:
@boomzilla said in Linux locks and a kinder, gentler Linus:
Which says nothing about the fact that it actually performs just fine. I dunno...perhaps there was a time when the minutiae of the scheduler mattered in a noticeable way for a desktop experience. That's not today.
I disagree. Try using Linux for DAW and let me know how it goes. Without at least changing the default scheduler you won't get far even if you are a guru who knows how to set up the Linux audio stack.
:sigh: Is DAW anything like CP? I have no idea what you're talking about, either, unfortunately, but that's just down to unfamiliar acronyms, not shoulder aliens.
apparently Digital Audio Workstation?
NFC what that is or why it would be relevant, but it's the top result in a google search...... so..... that's something?
-
@Mason_Wheeler said in Linux locks and a kinder, gentler Linus:
The reason IoT is a massive dumpster fire of security holes is because they're almost universally running on Linux, which has never taken security seriously.
Most IoT security is shit because it is made for least cost by someone who doesn't care other than to get the money (yours or VC, it doesn't matter). Security isn't a feature that most buyers of IoT devices look for. It is therefore mostly not done at all or only done badly. If nobody will pay extra for it and stuff without it sells, why would anyone bother?
The devices could run Windows, except then they'd cost quite a lot more and the market is definitely price sensitive. See above.