The Official Status Thread
-
@Dreikin said in The Official Status Thread:
@heterodox said in The Official Status Thread:
Python "multiprocessing"
Single-threaded multiprocessing isn't really going to save you time unless you have long waits. One moment, I've got a video or two for that...
More related:
https://www.youtube.com/watch?v=Obt-vMVdM8sLess related:
https://www.youtube.com/watch?v=MCs5OvhV9S4
-
@Dreikin said in The Official Status Thread:
Single-threaded multiprocessing isn't really going to save you time unless you have long waits. One moment, I've got a video or two for that...
I'm using the multiprocessing library, which is supposed to spawn child processes and avoid the global interpreter lock. It's not going to make much of a difference on a 12-second task, but I'm looking at O(n^2) operations next which are going to be a lot more of a pain in the ass/infinite when run serially.
I'd say I'm getting overhead from synchronization but I don't think it's a coincidence that I'm spawning four processes and getting 4x the runtime. I'm doing something wrong.
-
@heterodox said in The Official Status Thread:
@Dreikin said in The Official Status Thread:
Single-threaded multiprocessing isn't really going to save you time unless you have long waits. One moment, I've got a video or two for that...
I'm using the multiprocessing library, which is supposed to spawn child processes and avoid the global interpreter lock. It's not going to make much of a difference on a 12-second task, but I'm looking at O(n^2) operations next which are going to be a lot more of a pain in the ass/infinite when run serially.
I'd say I'm getting overhead from synchronization but I don't think it's a coincidence that I'm spawning four processes and getting 4x the runtime. I'm doing something wrong.
Got some essential, frequently accessed data synchronized that shouldn't be, maybe?
-
@Dreikin said in The Official Status Thread:
Got some essential, frequently accessed data synchronized that shouldn't be, maybe?
Yes. I've got the queue of tasks; each process pulls a task, performs it, then loops (until it's terminated by one of the processes having found the correct result). The tasks individually are very fast, which might be causing the synchronization overhead to dominate. But my attempt to split the list of tasks into chunks didn't work out very well. I probably should try it again.
-
@heterodox said in The Official Status Thread:
@Dreikin said in The Official Status Thread:
Got some essential, frequently accessed data synchronized that shouldn't be, maybe?
Yes. I've got the queue of tasks; each process pulls a task, performs it, then loops (until it's terminated by one of the processes having found the correct result). The tasks individually are very fast, which might be causing the synchronization overhead to dominate. But my attempt to split the list of tasks into chunks didn't work out very well. I probably should try it again.
Maybe try a Pool instead?
Edit:
@heterodox said in The Official Status Thread:
until it's terminated by one of the processes having found the correct result
Oh, hm. Not sure if you can do that with a pool..
-
@Dreikin said in The Official Status Thread:
Maybe try a Pool instead?
I would love to use a Pool, but it didn't look right for my use case. I have a long list of items to evaluate and they'll all boil down to one result. Not one result per item, and I want the task to short circuit when that one result is found.
I wish I were using anything other than Python.
-
@heterodox said in The Official Status Thread:
I clearly don't understand Python "multiprocessing" and accompanying queues. Using get() and put() from the queue:
Elapsed time: 43.97640013694763 seconds
Using get_nowait() and put_nowait():
Elapsed time: 68.29680013656616 seconds
Fuck off, Python. Fuck you.
I wouldn't trust a language that reports times several degrees of magnitude larger than a normal CPU would be capable of actually measuring.
-
Split into chunks of 100. Elapsed time: 3.33840012550354 seconds. THERE we go.
Thanks for being my rubber duck, @Dreikin. I certainly was convinced it wasn't synchronization overhead. <3
-
Japanese lets plays
Lizardon and Metamon
Translations are dumb
-
@heterodox said in The Official Status Thread:
@Dreikin said in The Official Status Thread:
Maybe try a Pool instead?
I would love to use a Pool, but it didn't look right for my use case. I have a long list of items to evaluate and they'll all boil down to one result. Not one result per item, and I want the task to short circuit when that one result is found.
I wish I were using anything other than Python.
Okay, so this is heavily cribbed from here, and probably looks like crap to someone who knows what they're doing, but:
from multiprocessing import Pool, Manager # Worker function used to get and check result def calculate(func, arg, target, event, output): if not event.is_set(): result = func(arg) print(result) if result == target: output.append(result) event.set() # Function that actually does the calculation def fib(n): if n == 0 or n == 1: return n return fib(n-1) + fib(n-2) # Pool management and printing of final result def pool_test(): testval = fib(5) manager = Manager() event = manager.Event() retval = manager.list() print("starting pool") with Pool(processes=4) as pool: for i in range(10): pool.apply_async(calculate, (fib, i, testval, event, retval)) print("waiting on pool") event.wait() pool.terminate() print(retval) if __name__ == '__main__': print("running test") pool_test()
-
@heterodox said in The Official Status Thread:
Split into chunks of 100. Elapsed time: 3.33840012550354 seconds. THERE we go.
Thanks for being my rubber duck, @Dreikin. I certainly was convinced it wasn't synchronization overhead. <3
Cool, glad to be of help :)
-
Status: Over the weekend, some friends and I started Vindictus again. I still like Fiona best, but I've been trying out Delia, who is really quite cool.
One of them is playing as Evie and going for Scythes, and the other is Lynn, who uses a glaive. Scythes get a good dodge and some nicely explosive skills, so I think it'll go well. I'm more worried about the guy who picked Lynn, since while she's great at dealing damage, she's really vulnerable.
In any case, feels good to be playing it again. Though with the Path of Exile update so close, we'll probably abandon this for a while again.
-
Status: WTF, am I missing something (other than No such thing as graphics in a VM)?
m
-
@heterodox said in The Official Status Thread:
@Dreikin said in The Official Status Thread:
Maybe try a Pool instead?
I would love to use a Pool, but it didn't look right for my use case. I have a long list of items to evaluate and they'll all boil down to one result. Not one result per item, and I want the task to short circuit when that one result is found.
I wish I were using anything other than Python.
Why not split them into groups of 100 or 1000 tasks instead of 1 task per process? Or if you can predict the tasks beforehand, just have each process take every nth task.
-
@Tsaukpaetra said in The Official Status Thread:
Status: WTF, am I missing something (other than No such thing as graphics in a VM)?
m
-
@cheong said in The Official Status Thread:
@Tsaukpaetra said in The Official Status Thread:
Status: WTF, am I missing something (other than No such thing as graphics in a VM)?
m
@cheong said in The Official Status Thread:
@Tsaukpaetra said in The Official Status Thread:
Status: WTF, am I missing something (other than No such thing as graphics in a VM)?
m
No, turns out that was the "nice" error message for "could not load xinput1_3.dll".
Smashed a copy of that into the image and all is well.
Now I just need to figure out what needs to be done to enable ODBC with the SQL Server 11.0 adapter...
-
status
Holiday caning is done. Pasta sauce, beef soup, turkey soup, and hot sauce.
Baking is next. Got a head start with a 6x batch of Pfeffernuesse.
Fake edit: Yes, I know what that means, @Jarry =)
-
@Lorne-Kates said in The Official Status Thread:
Holiday caning is done
-
@heterodox CPython?
-
@ben_lubar goddamn English language.
-
Status: The DirecTV box seems to have experienced an encoder issue.
-
Status: Thank you ASP.NET MVC output caching for caching a partial view on a route that never yields a partial.
-
@HardwareGeek said in The Official Status Thread:
@Jaloopa said in The Official Status Thread:
"I'm
thirstysleepy. Time for a cup of tea"FTFM. Typically, one of those tends to happen around afternoon tea, but sadly without the associated accouterments.
To wake you up or help you sleep? My wife is the latter, which I've never understood.
Status: wondering where the view raw button went
-
@heterodox said in The Official Status Thread:
I clearly don't understand Python "multiprocessing" and accompanying queues.
While Python is well known to be TRWTF when it comes to threading anyway, parallel programming is difficult. In particular, you shouldn't expect to understand what's going on easily, and it is very easy to forget that shared resource contention is a key cause of performance trouble. What's more, there's many different levels of resource contention possible; I forget the best way of diagnosing this sort of thingโฆ
-
@Jaloopa view raw is in the three dots menu now.
-
@ben_lubar said in The Official Status Thread:
@Lorne-Kates said in The Official Status Thread:
Holiday caning is done
Or maybe he made an excursion to Singapore.
-
@RaceProUK said in The Official Status Thread:
Status: Thank you ASP.NET MVC output caching for caching a partial view on a route that never yields a partial.
Well, that's sorted out now.
Status: To be honest, given how light my workload is now... a bit bored
-
@Jarry said in The Official Status Thread:
@heterodox CPython?
Don't have a choice of interpreter, so.
@dkf said in The Official Status Thread:
@heterodox said in The Official Status Thread:
I clearly don't understand Python "multiprocessing" and accompanying queues.
While Python is well known to be TRWTF when it comes to threading anyway, parallel programming is difficult. In particular, you shouldn't expect to understand what's going on easily, and it is very easy to forget that shared resource contention is a key cause of performance trouble. What's more, there's many different levels of resource contention possible; I forget the best way of diagnosing this sort of thingโฆ
Yeah, I'm used to it in C++ and Java, just not familiar with Python's paradigms. Of course the issue turned out not to be language-specific, but that's the way the diagnostic process goes.
-
@RaceProUK said in The Official Status Thread:
Status: To be honest, given how light my workload is now... a bit bored
Yeah... turns out using the output cache in ASP.NET is more complicated than I thought.
Status: Crossing all her digits, hoping her latest attempt works properly.
-
Status:
-
-
@TimeBandit it's almost like changing circumstances and a new CEO can change the direction of a company. Hilarious
-
@Jaloopa said in The Official Status Thread:
@TimeBandit it's almost like changing circumstances and a new CEO can change the direction of a company. Hilarious
Where is MS Office for Linux ?
Where is Visual Studio for Linux ?You're right: they love Linux.
-
-
@TimeBandit said in The Official Status Thread:
Where is Visual Studio for Linux ?
Bonus points: It's built on NodeJS. Whether that's +ve or -ve is your interpretation.
-
@RaceProUK said in The Official Status Thread:
Visual Studio Code is not Visual Studio
Bonus points: It's built on NodeJS.
-
@TimeBandit said in The Official Status Thread:
Cancer is great, if you're a flesh-eating parasite. It's like a self-replicating buffet!
-
@Jaloopa said in The Official Status Thread:
To wake you up or help you sleep?
Caffeine is my best friend. It's what allows me to remain employed, rather than being fired for falling asleep on the job. Usually; some days, it's a struggle even with caffeine.
-
@Lorne-Kates said in The Official Status Thread:
Cancer is great
-
Status: WAOW COMPLETELY MEANINGLESS NUMBER HITS ALL TIME HIGH!
-
@MathNerdCNU said in The Official Status Thread:
Status https://youtu.be/aRM2YcGpmxg
Damn onebox plugin!
In order to onebox now, links just be on their own line so links mid-paragraph don't fuck the paragraphs.
-
@JazzyJosh said in The Official Status Thread:
Status: WAOW COMPLETELY MEANINGLESS NUMBER HITS ALL TIME HIGH!
How many times a joke I made fell flat?
-
@Tsaukpaetra The DJIA
-
Status: why do we have keurig cups? We don't have one of their machines...
-
@Tsaukpaetra said in The Official Status Thread:
Status: why do we have keurig cups? We don't have one of their machines...
Even if you did, milk chocolate: . "Dark Chocolate Sensation" FTW!
-
@Tsaukpaetra said in The Official Status Thread:
@MathNerdCNU said in The Official Status Thread:
Status https://youtu.be/aRM2YcGpmxg
Damn onebox plugin!
In order to onebox now, links just be on their own line so links mid-paragraph don't fuck the paragraphs.
Do you want me to install Discourse so we can run the onebox plugin? Because it's not available for NodeBB, and the iframely plugin is configured not to embed YouTube.
-
@JazzyJosh said in The Official Status Thread:
Status: WAOW COMPLETELY MEANINGLESS NUMBER HITS ALL TIME HIGH!
-
@ben_lubar said in The Official Status Thread:
Do you want me to install Discourse
You fuckin maniac !
-
Status: The office Christmas decorations have appeared
-
@cartman82 said in The Official Status Thread:
TIL:
You can use github as a piracy platform.
https://github.com/rmsadik/ebooks
It even gives you a nice PDF reader experience, so you don't have to bother downloading books to your computer.
That is kind of genius. You can replace DMCA takedown requests with pull requests. Nifty.