So is the migration thing still happening?
-
My favorite thing about docker... you don't need a divining rod to debug OPS issues.
-
UNICORN_SIDEKIQS
I thought Stack Exchange was unicorns and Discourse was rainbows.
Filed under: Unicornfused
-
On the basis that unicorns can be navigated by way of the pointy end and rainbows can't?
-
-
Filed under: WTF? Why can't I post this?
There's a test on posts. One of which, that cannot be overridden, is "does it contain all caps and nothing else".
&nsbp;
in the post somewhere evades the test (and a couple of others.)
-
@PaulaBean is murdering my computer:
I totally called it!
Filed under: Melting is a kind of murder...
-
There's a test on posts. One of which, that cannot be overridden, is "does it contain all caps and nothing else".
Oh, no. @blakeyrat was right after all....slashdot is leaking into the forum!
Filed Under: Lameness filter is lame
-
If you can evade such a test in such a trivial fashion, that would suggest there is something amiss. Perhaps automatically condensing all entities back to characters before performing the test (and discarding the result to avoid chance of more XSS joy)
-
Or, perhaps, get rid of the test altogether. Or at least allow it to be configurable.
-
If you can evade such a test in such a trivial fashion, that would suggest there is something amiss.
You really expect majority of (l)users to know how to insert a HTML entity of any kind?
-
You really expect majority of (l)users to know how to insert a HTML entity of any kind?
If they'd been using CS with Chrome, they'd have needed to!
-
You really expect majority of (l)users who are not here on TDWTF to know how to insert a HTML entity of any kind?
FTFM
-
Only CS users with Chrome.
-
You really expect majority of (l)users to know how to insert a HTML entity of any kind?
Discourse seems to expect users to...
-
9855 / 263866 ( 3.7%) Error creating post 75834. Skipping.
Script Timed Out
9900 / 263866 ( 3.8%) Error creating post 75911. Skipping.
Script Timed Out
10007 / 263866 ( 3.8%) Error creating post 76294. Skipping.
Script Timed Out
10114 / 263866 ( 3.8%) Error creating post 76714. Skipping.
Script Timed Out
10192 / 263866 ( 3.9%) Error creating post 76936. Skipping.
Script Timed Out
10225 / 263866 ( 3.9%) Error creating post 77021. Skipping.
Script Timed Out
10231 / 263866 ( 3.9%) Error creating post 77044. Skipping.
Script Timed Out
10251 / 263866 ( 3.9%) Error creating post 77082. Skipping.
Connection timed out@sam is there a way to check the number of sidekiq tasks that are waiting and (separately) to disable (or extend) the timeout on post creation? I'd like to do something like
while sidekiq_has_more_than_10_tasks_waiting sleep for a second end
before giving the post data back to the base import script.
-
Pretty sure you can do that with their API checkout the sidekiq wiki.
-
Ok, I think it's
Sidekiq::Stats.new.enqueued
.@sam how about the timeout on whatever script is timing out? (The import base just prints the message, so I can't tell you the stack trace.)
-
That number in parentheses is the number Sidekiq::Stats.new.enqueued returned. Oh dear.
-
The real question that enquiring minds need to know: will those & entities be shown literally as-is or not?
-
This user also failed to import (because the name suggester throws out everything after @)
@sam can you make the name suggester check to see if a name starts with @ and then throw out the @ instead of the whole thing?
creating users 331 / 138275 ( 0.2%) Skipping user id 333 because email is blank 604 / 138275 ( 0.4%) Skipping user id 1000 because email is blank 605 / 138275 ( 0.4%) Skipping user id 1001 because email is blank 28723 / 138275 ( 20.8%) Failed to create user id redacted@ymail.com: ["Username can't be blank", "Username must be present"] 37256 / 138275 ( 26.9%) Failed to create user id redacted@hotmail.com: ["Username can't be blank", "Username must be present"] 138275 / 138275 (100.0%) created: 138229 users failed: 5 uploading avatars 1872 / 1872 (100.0%) /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:320:in `raise_record_not_found_exception!': Couldn't find Category with 'id'=10 (ActiveRecord::RecordNotFound) from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:420:in `find_one' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:404:in `find_with_ids' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:68:in `find' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/querying.rb:3:in `find' from communityserver.rb:46:in `block in execute' from communityserver.rb:45:in `each' from communityserver.rb:45:in `execute' from /var/www/discourse/script/import_scripts/base.rb:43:in `perform' from communityserver.rb:129:in `<main>'
And that's when I realized that I forgot to restore from backup before I started the import over and I just tried to import TDWTF into an empty discourse forum. Wheeeee
-
7662 / 263866 ( 2.9%) Error creating post 67823. Skipping.
Script Timed Out
7762 / 263866 ( 2.9%) Error creating post 68110. Skipping.
Script Timed Out
7814 / 263866 ( 3.0%) Error creating post 68298. Skipping.
Script Timed Out
7827 / 263866 ( 3.0%) Error creating post 68359. Skipping.
Connection timed out
7861 / 263866 ( 3.0%) Error creating post 68464. Skipping.
Script Timed Out
7948 / 263866 ( 3.0%) Error creating post 68883. Skipping.
Script Timed Out
Error creating post 68885. Skipping.
Connection timed out
7951 / 263866 ( 3.0%) Error creating post 68887. Skipping.
Script Timed Out
7969 / 263866 ( 3.0%) Error creating post 68949. Skipping.
Script Timed Out
8134 / 263866 ( 3.1%) Error creating post 69539. Skipping.
Script Timed Out
8239 / 263866 ( 3.1%) Error creating post 69804. Skipping.
Script Timed Out
8243 / 263866 ( 3.1%) Error creating post 69825. Skipping.
Script Timed Out
8312 / 263866 ( 3.2%) Error creating post 70146. Skipping.
Script Timed Out
8315 / 263866 ( 3.2%) Error creating post 70153. Skipping.
Script Timed Out
8322 / 263866 ( 3.2%) Error creating post 70171. Skipping.
Script Timed Out
8328 / 263866 ( 3.2%) Error creating post 70179. Skipping.
Script Timed Out
Error creating post 70183. Skipping.
Script Timed Out
8391 / 263866 ( 3.2%) Error creating post 70576. Skipping.
Script Timed Out
8414 / 263866 ( 3.2%) Error creating post 70645. Skipping.
Script Timed Out
8473 / 263866 ( 3.2%) Error creating post 70928. Skipping.
Script Timed Out
8513 / 263866 ( 3.2%) Error creating post 71174. Skipping.
Script Timed Out
8564 / 263866 ( 3.2%) Error creating post 71342. Skipping.
Script Timed Out
8634 / 263866 ( 3.3%) Error creating post 71627. Skipping.
Script Timed Out
8642 / 263866 ( 3.3%) Error creating post 71638. Skipping.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:222:inrescue in io': Connection timed out (Redis::TimeoutError) from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:220:in
io'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:228:inread' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:96:in
block in call'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:201:inblock (2 levels) in process' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:309:in
ensure_connected'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:191:inblock in process' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:270:in
logging'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:190:inprocess' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:96:in
call'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:1294:inblock in smembers' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:37:in
block in synchronize'
from /usr/local/lib/ruby/2.0.0/monitor.rb:211:inmon_synchronize' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:37:in
synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:1293:insmembers' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-namespace-1.4.1/lib/redis/namespace.rb:352:in
method_missing'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:30:inblock in queues' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in
with'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq.rb:70:inredis' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:29:in
queues'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:50:inenqueued' from communityserver.rb:77:in
block (2 levels) in execute'
from /var/www/discourse/script/import_scripts/base.rb:218:inblock in create_posts' from /usr/local/lib/ruby/2.0.0/csv.rb:1716:in
each'
from /var/www/discourse/script/import_scripts/base.rb:217:increate_posts' from communityserver.rb:76:in
block in execute'
from /usr/local/lib/ruby/2.0.0/csv.rb:1266:inopen' from communityserver.rb:75:in
execute'
from /var/www/discourse/script/import_scripts/base.rb:43:inperform' from communityserver.rb:129:in
<main>'
-
So what happens when you try and import a post by a user on CS and DC but wasn't migrated across systems? As much as I would love to see all like, 6 posts of mine on CS, I'd rather not my account blowup because of 6 posts.
Maybe transform any 'account' on CS that wasn't in the migration batch to append _WTF_LE_OLD to de-conflict? Works until someone reads this post and does WTFery as to be expected.
-
If you used the same email address, the posts will be attributed to you. Otherwise, it'll be a separate account with a name based on the name suggester. I'm only doing the migration locally to test my code. When this is finished, it'll be a fresh migration from the current databases.
-
@sam it's going to be really important to disable email jobs during the import since there are 263866 posts from 138271 users. That's a total of thirty-six billion, four hundred eighty-five million, fifteen thousand, six hundred eighty-six emails or about one email for every 8 stars in the galaxy.
Edit: It seems that the flood of jobs is from this commit. It's funny how a commit named "FEATURE:" removed more code than it added.
Filed under: Me too!
-
That is just for people who have mailing list mode on, are you enabling mailing list mode for everyone?
-
And if so I strongly recommend you do not do that @ben_lubar . Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.
Filed under: I'm going off the rails on a crazy train
-
No, I'm not setting any preferences on the imported users, but each post is generating somewhere around 300 jobs.
-
How many users in the db have mailing list mode on ?
That line is only enqueing one job, can you debug through to find the loop that is pumping jobs in?
-
[1] pry(main)> User.where(mailing_list_mode: true).count => 2
Weird, I wonder what's been causing all the jobs then. I'll have to take another look when it finishes processing the users for my fixed avatar code.
-
Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.
You must be new her... we already did this, didn't we?
-
Well, there are only two crazy people on this forum, which seems a bit low to me.
@Lorne_Kates @ender you might want to turn off crazy people mode when the import actually goes through.
-
Bug: Discourse causes sanity. Sanityzation should be optional and defaulted to OFF
-
@ben_lubar that is crazy town, just disable it for the users at the begining of the import and re-enable after done. Or perhaps a PR to temporarily disable various mail queuing that we can use during imports.
-
- FeatureTopicUsers runs for every post and then again for every topic at the end of the import.
- ProcessPost runs for every post, but if I recall correctly, posts get re-cooked if the system notices they're out of date, so that could probably also be skipped.
- The other jobs I could find were all notification/email jobs, which we can skip because this is an import.
Unless I'm mistaken, I can just turn off jobs during the import and everything will be fine.
@sam, am I mistaken?
-
Its tricky, you can disable feature topic users, but you will need to walk through all topics and re-run it, after the import.
ProcessPost is kind of important though, it takes care of resizing images so you get light boxes and so on. See CookedPostProcessor.
-
Won't this call CookedPostProcessor?
-
Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.
No. Not if you'd like to read the boards via email. Like CS lets you. One post per email. Without needless cruft.
Whatever DC is doing with emails is intolerable and unreadable, and I ended up disabling them all within the first couple of days.
Well almost all - I still get an occasional "Digest for MM-DD-YYY, Here's a brief summary of the discussion on What the Daily WTF? since we last saw you on <some time/date that has no bearing on when I last visited the website...>" which I can't figure out how to disable:
-
Er.. what? I can't understand this UK "English".
You have mailing list mode on, yes or no? Because it sounds like you are singing the praises of one-email-per-post in the top part, yet your screenshot at the bottom shows all email settings disabled.
And if that's the case, we definitely should not be sending you an email digest, and I suggest opening that as a new bug.
-
Whatever DC is doing with emails is intolerable and unreadable, and I ended up disabling them all within the first couple of days.
One moment there @PJH can you elaborate on the "intolerable and unreadable" thingy
What in particular are you referring to?
Keep in mind, we send stuff in the same email without html shananigens if that suites you better:
-
It is rather clear:
- On CS it was possible to follow the forum by using e-mail only
- On DC the setting exists but the resulting mails are "intolerable and unreadable"
- Hence he disabled all mail settings as in the screenshot
- But he occasionally still receives a digest mail.
Vous comprenez maintenant? Begrijp je het nu?
Filled under: English isn't even my second language
-
On DC the setting exists but the resulting mails are "intolerable and unreadable"
I am really not following, what about the emails above ^^^ is intolerable and unreadable?
Also, if digest is disabled and you are still getting it, its a bug and we will fix it.
-
Excellent translation.
@pjh I bet you have multiple accounts, and that digest is coming from the other account. Remember the days of the 3 char username and the alt account?
-
Er.. what? I can't understand this UK "English".
Which part of
I ended up disabling them all within the first couple of days.
did you have difficulty parsing Jeff?
Could you perhaps suggest a more comprehensible way of expressing the "I tried turning off all email options?" than "I ended up disabling them all"?
You have mailing list mode on, yes or no?
No. Because I couldn't stand what was being sent out.
Because it sounds like you are singing the praises of one-email-per-post in the top part, yet your screenshot at the bottom shows all email settings disabled.
Indeed I was, and indeed they are. Because DC wasn't sending me one email per post....
One moment there @PJH can you elaborate on the "intolerable and unreadable" thingy
This:
This is the email sent out for one post. Note all the crap at the end? over 75% of that screenshot is not the post I'm being notified about
Compared to (say):
where I can see more than one post on the page at the same time. And no - this isn't a HTML/plaintext email issue.
-
This is really frustrating:
Some people like this default, for some it is atrocious. I will add a user setting so you can override it.
Can you make a topic for this so I don't forget?
-
Stuck it in meta, since it's not a bug...
-
Ugh. Was it necessary to change my "-" list to a real list? That didn't really "add" value to the discussion does it? Now I feel compelled to make an off topic remark about my off topic reply. I made my point, or rather clarified @PJH 's point. Maybe I prefer adding "-" myself in front of every line instead of suffering with
- a list
that doesn't behave like I would expect it to
if I create a newline any sane editor makes me a new list item
it has been this way since WordPerfect - See? It is rather silly.
- So I did that shit once and since then I just use "-"
- Why? Because fiddling with style/markup/bbcode/html tags is a barrier to typing my reply. So I rather just type away and not lose my time or strain of thought by messing around.
Unless I'm missing something it is my right as author of the damn reply to do so anyway I want.
- a list
-
- test
- test 2
- worksforme ;)
-
- test- test 2- worksforme
Backslashes are cheating!
Also, quotes really love them it seems.
-
- a list
- is started when you start your post with "-"
First type some tekst
- then a "-"
- you don't get a list
- freaking thing is getting more and more silly
... to the bug-mobile!
Filled under: You're doing it WRONG! Don't start your post with a list! unless you want it to work
-
Well if you don't leave a blank line, then
- you
- won't
- get
- a
- list
But if you leave a blank line
- then
- you
- will.