Imports (Was: {brace yourselves} the import is coming {Spoiler Alert: Not all of it} [i.e. blakeyrat was not utterly wrong for the first time ever] Filed under: append-only titles.)
-
Huh. Your username is made of all letters people hit when they want to make a body have some valid in it...
And… Your point is?
-
-
I might ask you the same question?
I just saw my username and wanted to make a lame joke by replying to that post. That's all.
-
Can I just draw everyone's attention to this?
Why the fuck not‽
Because regression testing means that as you test, your brain regresses. Can't have that now, can we?
-
I just saw my username and wanted to make a lame joke by replying to that post. That's all.
It was a pretty good lame joke. I'd like to celebrate it by summoning every user who's name starts with @asdf:
Filed under: shit I wish I did before the import
-
...@mention limit...
-
-
It looks like it finished.
It may not be running any more, but it definitely didn't finish importing everything. Looking at the new Sidebar category, it looks like we got to about January 2008.
-
So... them emails... Not that I miss them that much personally, but why are they still off if the script isn't running?
-
but why are they still off if the script isn't running?
probably because:
It may not be running any more, but it definitely didn't finish importing everything.
putting emails back on seems like a post-import action. And since the import keeps dying
gracefullylike a dwarf facing a dragon it never reaches that stage.Filed Under: I - personally - accept the "no emails will be sent" message as the new header of what.tdwtf
-
Looks like @accalia left that out of the spec.
Intentionally so.
If discourse let's you view the scorecard of anyone then the shadowmod query is redundant.
Otherwise you can only check your own status.
-
And since the import keeps dying
The Forgotten Beast Import has come! A huge shell script with poor error handling. It has a knobby CSV file and it is slavering. Beware its deadly SQL statements!gracefullylike a dwarf facing a dragon it never reaches that stage.
-
+@ - a like is not enough
-
Is this running again? Because... Server Cooties™ are fucking back.
Fuck You Dischorse.
-
Is this running again?
Has been for the last four or five hours; been watching the numbers in grey bubbles go down a few times today.
-
Grey bubbles?
-
-
How does those going down relate to an import happening?
I are confused.
-
I think the import- oh, wait, I have cyberparts that read old posts for me…
-
-
Is this running again? Because... Server Cooties™ are fucking back.
i'm perriodically getting crash PMs from my bots.... so there's that.
-
i'm perriodically getting crash PMs from my bots
Oddly, I'm not… OK, I only have one, but still…
-
OK, I only have one, but still…
i think i'm possibly tripping on rate limiting from IP.... i really gotta get the IPV6 instructions to @apapadimoulis so i can move the bots to IPv6 and eliminate that issue.
-
i'm perriodically getting crash PMs from my bots.... so there's that.
I'm not.
Oh, wait, mine don't send them
-
i think i'm possibly tripping on rate limiting from IP.... i really gotta get the IPV6 instructions to @apapadimoulis so i can move the bots to IPv6 and eliminate that issue.
I'm assuming one IP per bot?
-
-
I'm assuming one IP per bot?
that would be my solution, yes. but i'm not paying $10/mo for more IPv4 ips (i can get a whole server for $5/mo) i'm much rather use the 16 IPv6 ips that i get for free from digital ocean.
-
-
they would if you updated them.
But I wouldn't need them to. If I'm receiving the PM, the bot is up. If it's up, I don't need a notification.
-
The PM is telling you that the bot is restarting though… although, I'm not actually sure how useful that is, come to think of it…
*thinks about making those PMs optional*
-
The PM is telling you that the bot is restarting though
So what if it's restarting? I only care if it's up, and it's only up if I get the PM.
I don't have the servers at work notify me that they were down, I have another service entirely notify me that the server is down.
although, I'm not actually sure how useful that is, come to think of it…
Quite.
-
thinks about making those PMs optional
i do mean to eventually make them not happen at all...
i just have to find where the messagebus and notifications go off the rail.
-
Quite.
they make sense if the bot isn't on an upstart script or something to automatically respawn it.
-
grep -A1 'Exception' import.log > exceptions.log
-
they make sense if the bot isn't on an upstart script or something to automatically respawn it.
They do?
So the bot can't connect to Dischorse, won't get magically restarted, so it sends a PM to a server it can't talk to to say "HELP"?
-
sort exceptions.log | uniq -c
[snip a bunch of lines that only differ in IDs] 94 Couldn't find User without an ID 10139 EXECABORT Transaction discarded because of previous errors. 352 MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 1 Script Timed Out
-
Hmmm...I've seen some "user deleted because they never responded to email" messages in the logs. I don't think that's what's going on here, but without usernames, it's impossible to tell.
@ben_lubar, can you look up the usernames for some of those?
-
They do?
well if it can't contact entirely then yeah that's a bad, ususally though they've just run off the rails thanks to a bug i still havent tracked down in messageBus.js that causes it to stop polling messages/notifications.
-
cat find_missing_users.rb
require 'csv' missing_posts = Hash.new(false) IO.popen("xzcat import.log.xz | tr '\\r' '\\n' | grep -B1 without") do |f| f.each_line.with_index do |l, i| missing_posts[l[/post (\d+)\. Skipping\./, 1]] = true if i % 3 == 0 end end missing_users = Hash.new(0) CSV.open('tdwtf-posts.csv', headers: 'id,parent,topic,author,category,title,created_at,raw,tags') do |posts| posts.each do |p| missing_users[p['author']] += 1 if missing_posts[p['id']] end end CSV.open('tdwtf-users.csv', headers: 'id,email,name,created_at,avatar') do |users| users.each do |u| puts "#{u['name']}\t#{missing_users[u['id']]}" if missing_users[u['id']] != 0 end end
ruby find_missing_users.rb
Anonymous 89
-
Hmmm...I've seen some "user deleted because they never responded to email" messages in the logs. I don't think that's what's going on here, but without usernames, it's impossible to tell.
@ben_lubar, can you look up the usernames for some of those?
Well that's fun …
-
Anonymous 89
Ah, I think that was enabled for a while. So...either lets create a dummy user or ignore those posts?
-
Of the posts that didn't have associated users, 89 of them were from the Anonymous account that didn't get added because it had no email address associated with it. That's only 5 posts left unaccounted for. Those posts didn't even have an associated user in tdwtf-users.csv.
-
So are we now done with the whole importing of posts from CS thing?
edit: Presumably not as the topics in the old Sidebar category only go to 2009.
-
Well, I could run the import script again and hopefully it'll pick up some of the 10492 posts that failed due to timeouts or whatever. Should I do that now or over the weekend?
-
-
#over the weekend
Seriously, stick to when the forum is quieter. Please. For everyone's sanity.
-
-
Works for me; I'll be in bed
-
-