So is the migration thing still happening?

If you can evade such a test in such a trivial fashion, that would suggest there is something amiss. Perhaps automatically condensing all entities back to characters before performing the test (and discarding the result to avoid chance of more XSS joy)

PJH

Or, perhaps, get rid of the test altogether. Or at least allow it to be configurable.

Onyx

@Arantor said:

If you can evade such a test in such a trivial fashion, that would suggest there is something amiss.

You really expect majority of (l)users to know how to insert a HTML entity of any kind?

dkf

@Onyx said:

You really expect majority of (l)users to know how to insert a HTML entity of any kind?

If they'd been using CS with Chrome, they'd have needed to!

Onyx

@Onyx said:

You really expect majority of (l)users who are not here on TDWTF to know how to insert a HTML entity of any kind?

FTFM

boomzilla

Only CS users with Chrome.

Arantor

@Onyx said:

You really expect majority of (l)users to know how to insert a HTML entity of any kind?

Discourse seems to expect users to...

ben_lubar

9855 / 263866 ( 3.7%) Error creating post 75834. Skipping.
Script Timed Out
9900 / 263866 ( 3.8%) Error creating post 75911. Skipping.
Script Timed Out
10007 / 263866 ( 3.8%) Error creating post 76294. Skipping.
Script Timed Out
10114 / 263866 ( 3.8%) Error creating post 76714. Skipping.
Script Timed Out
10192 / 263866 ( 3.9%) Error creating post 76936. Skipping.
Script Timed Out
10225 / 263866 ( 3.9%) Error creating post 77021. Skipping.
Script Timed Out
10231 / 263866 ( 3.9%) Error creating post 77044. Skipping.
Script Timed Out
10251 / 263866 ( 3.9%) Error creating post 77082. Skipping.
Connection timed out

@sam is there a way to check the number of sidekiq tasks that are waiting and (separately) to disable (or extend) the timeout on post creation? I'd like to do something like

while sidekiq_has_more_than_10_tasks_waiting
    sleep for a second
end

before giving the post data back to the base import script.

sam

Pretty sure you can do that with their API checkout the sidekiq wiki.

ben_lubar

sidekiq/lib/sidekiq.rb at main · sidekiq/sidekiq

Simple, efficient background processing for Ruby. Contribute to sidekiq/sidekiq development by creating an account on GitHub.

Ok, I think it's Sidekiq::Stats.new.enqueued.

@sam how about the timeout on whatever script is timing out? (The import base just prints the message, so I can't tell you the stack trace.)

ben_lubar

That number in parentheses is the number Sidekiq::Stats.new.enqueued returned. Oh dear.

Arantor

The real question that enquiring minds need to know: will those & entities be shown literally as-is or not?

ben_lubar

This user also failed to import (because the name suggester throws out everything after @)

@sam can you make the name suggester check to see if a name starts with @ and then throw out the @ instead of the whole thing?

creating users
      331 / 138275 (  0.2%)    Skipping user id 333 because email is blank
      604 / 138275 (  0.4%)    Skipping user id 1000 because email is blank
      605 / 138275 (  0.4%)    Skipping user id 1001 because email is blank
    28723 / 138275 ( 20.8%)    Failed to create user id  redacted@ymail.com: ["Username can't be blank", "Username must be present"]
    37256 / 138275 ( 26.9%)    Failed to create user id  redacted@hotmail.com: ["Username can't be blank", "Username must be present"]
   138275 / 138275 (100.0%)    
created: 138229 users
 failed: 5

uploading avatars
     1872 / 1872 (100.0%)    
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:320:in `raise_record_not_found_exception!': Couldn't find Category with 'id'=10 (ActiveRecord::RecordNotFound)
        from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:420:in `find_one'
        from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:404:in `find_with_ids'
        from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/relation/finder_methods.rb:68:in `find'
        from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.1.1/lib/active_record/querying.rb:3:in `find'
        from communityserver.rb:46:in `block in execute'
        from communityserver.rb:45:in `each'
        from communityserver.rb:45:in `execute'
        from /var/www/discourse/script/import_scripts/base.rb:43:in `perform'
        from communityserver.rb:129:in `<main>'

And that's when I realized that I forgot to restore from backup before I started the import over and I just tried to import TDWTF into an empty discourse forum. Wheeeee

ben_lubar

7662 / 263866 ( 2.9%) Error creating post 67823. Skipping.
Script Timed Out
7762 / 263866 ( 2.9%) Error creating post 68110. Skipping.
Script Timed Out
7814 / 263866 ( 3.0%) Error creating post 68298. Skipping.
Script Timed Out
7827 / 263866 ( 3.0%) Error creating post 68359. Skipping.
Connection timed out
7861 / 263866 ( 3.0%) Error creating post 68464. Skipping.
Script Timed Out
7948 / 263866 ( 3.0%) Error creating post 68883. Skipping.
Script Timed Out
Error creating post 68885. Skipping.
Connection timed out
7951 / 263866 ( 3.0%) Error creating post 68887. Skipping.
Script Timed Out
7969 / 263866 ( 3.0%) Error creating post 68949. Skipping.
Script Timed Out
8134 / 263866 ( 3.1%) Error creating post 69539. Skipping.
Script Timed Out
8239 / 263866 ( 3.1%) Error creating post 69804. Skipping.
Script Timed Out
8243 / 263866 ( 3.1%) Error creating post 69825. Skipping.
Script Timed Out
8312 / 263866 ( 3.2%) Error creating post 70146. Skipping.
Script Timed Out
8315 / 263866 ( 3.2%) Error creating post 70153. Skipping.
Script Timed Out
8322 / 263866 ( 3.2%) Error creating post 70171. Skipping.
Script Timed Out
8328 / 263866 ( 3.2%) Error creating post 70179. Skipping.
Script Timed Out
Error creating post 70183. Skipping.
Script Timed Out
8391 / 263866 ( 3.2%) Error creating post 70576. Skipping.
Script Timed Out
8414 / 263866 ( 3.2%) Error creating post 70645. Skipping.
Script Timed Out
8473 / 263866 ( 3.2%) Error creating post 70928. Skipping.
Script Timed Out
8513 / 263866 ( 3.2%) Error creating post 71174. Skipping.
Script Timed Out
8564 / 263866 ( 3.2%) Error creating post 71342. Skipping.
Script Timed Out
8634 / 263866 ( 3.3%) Error creating post 71627. Skipping.
Script Timed Out
8642 / 263866 ( 3.3%) Error creating post 71638. Skipping.
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:222:in rescue in io': Connection timed out (Redis::TimeoutError) from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:220:in io'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:228:in read' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:96:in block in call'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:201:in block (2 levels) in process' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:309:in ensure_connected'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:191:in block in process' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:270:in logging'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:190:in process' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis/client.rb:96:in call'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:1294:in block in smembers' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:37:in block in synchronize'
from /usr/local/lib/ruby/2.0.0/monitor.rb:211:in mon_synchronize' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:37:in synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-3.0.7/lib/redis.rb:1293:in smembers' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/redis-namespace-1.4.1/lib/redis/namespace.rb:352:in method_missing'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:30:in block in queues' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/connection_pool-2.0.0/lib/connection_pool.rb:58:in with'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq.rb:70:in redis' from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:29:in queues'
from /var/www/discourse/vendor/bundle/ruby/2.0.0/gems/sidekiq-3.1.3/lib/sidekiq/api.rb:50:in enqueued' from communityserver.rb:77:in block (2 levels) in execute'
from /var/www/discourse/script/import_scripts/base.rb:218:in block in create_posts' from /usr/local/lib/ruby/2.0.0/csv.rb:1716:in each'
from /var/www/discourse/script/import_scripts/base.rb:217:in create_posts' from communityserver.rb:76:in block in execute'
from /usr/local/lib/ruby/2.0.0/csv.rb:1266:in open' from communityserver.rb:75:in execute'
from /var/www/discourse/script/import_scripts/base.rb:43:in perform' from communityserver.rb:129:in <main>'

MathNerdCNU

So what happens when you try and import a post by a user on CS and DC but wasn't migrated across systems? As much as I would love to see all like, 6 posts of mine on CS, I'd rather not my account blowup because of 6 posts.

Maybe transform any 'account' on CS that wasn't in the migration batch to append _WTF_LE_OLD to de-conflict? Works until someone reads this post and does WTFery as to be expected.

ben_lubar

If you used the same email address, the posts will be attributed to you. Otherwise, it'll be a separate account with a name based on the name suggester. I'm only doing the migration locally to test my code. When this is finished, it'll be a fresh migration from the current databases.

ben_lubar

@sam it's going to be really important to disable email jobs during the import since there are 263866 posts from 138271 users. That's a total of thirty-six billion, four hundred eighty-five million, fifteen thousand, six hundred eighty-six emails or about one email for every 8 stars in the galaxy.

Edit: It seems that the flood of jobs is from this commit. It's funny how a commit named "FEATURE:" removed more code than it added.

Filed under: Me too!

sam

That is just for people who have mailing list mode on, are you enabling mailing list mode for everyone?

end

And if so I strongly recommend you do not do that @ben_lubar . Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.

Filed under: I'm going off the rails on a crazy train

ben_lubar

No, I'm not setting any preferences on the imported users, but each post is generating somewhere around 300 jobs.

sam

How many users in the db have mailing list mode on ?

That line is only enqueing one job, can you debug through to find the loop that is pumping jobs in?

ben_lubar

[1] pry(main)> User.where(mailing_list_mode: true).count
=> 2

Weird, I wonder what's been causing all the jobs then. I'll have to take another look when it finishes processing the users for my fixed avatar code.

Onyx

@codinghorror said:

Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.

You must be new her... we already did this, didn't we?

ben_lubar

Well, there are only two crazy people on this forum, which seems a bit low to me.

@Lorne_Kates @ender you might want to turn off crazy people mode when the import actually goes through.

Onyx

Bug: Discourse causes sanity. Sanityzation should be optional and defaulted to OFF

sam

@ben_lubar that is crazy town, just disable it for the users at the begining of the import and re-enable after done. Or perhaps a PR to temporarily disable various mail queuing that we can use during imports.

ben_lubar

FeatureTopicUsers runs for every post and then again for every topic at the end of the import.
ProcessPost runs for every post, but if I recall correctly, posts get re-cooked if the system notices they're out of date, so that could probably also be skipped.
The other jobs I could find were all notification/email jobs, which we can skip because this is an import.

Unless I'm mistaken, I can just turn off jobs during the import and everything will be fine.

@sam, am I mistaken?

sam

Its tricky, you can disable feature topic users, but you will need to walk through all topics and re-run it, after the import.

ProcessPost is kind of important though, it takes care of resizing images so you get light boxes and so on. See CookedPostProcessor.

ben_lubar

Won't this call CookedPostProcessor?

discourse/app/models/post.rb at main · discourse/discourse

A platform for community discussion. Free, open, simple. - discourse/discourse

PJH

@codinghorror said:

Mailing list "mode" aka "mail me every single thing that happens on the forum" is a crazy-people-only setting.

No. Not if you'd like to read the boards via email. Like CS lets you. One post per email. Without needless cruft.

Whatever DC is doing with emails is intolerable and unreadable, and I ended up disabling them all within the first couple of days.

Well almost all - I still get an occasional "Digest for MM-DD-YYY, Here's a brief summary of the discussion on What the Daily WTF? since we last saw you on <some time/date that has no bearing on when I last visited the website...>" which I can't figure out how to disable:

end

Er.. what? I can't understand this UK "English".

You have mailing list mode on, yes or no? Because it sounds like you are singing the praises of one-email-per-post in the top part, yet your screenshot at the bottom shows all email settings disabled.

And if that's the case, we definitely should not be sending you an email digest, and I suggest opening that as a new bug.

sam

@PJH said:

Whatever DC is doing with emails is intolerable and unreadable, and I ended up disabling them all within the first couple of days.

One moment there @PJH can you elaborate on the "intolerable and unreadable" thingy

What in particular are you referring to?

Keep in mind, we send stuff in the same email without html shananigens if that suites you better:

Luhmann

It is rather clear:

On CS it was possible to follow the forum by using e-mail only
On DC the setting exists but the resulting mails are "intolerable and unreadable"
Hence he disabled all mail settings as in the screenshot
But he occasionally still receives a digest mail.

Vous comprenez maintenant? Begrijp je het nu?

Filled under: English isn't even my second language

sam

@Luhmann said:

On DC the setting exists but the resulting mails are "intolerable and unreadable"

I am really not following, what about the emails above ^^^ is intolerable and unreadable?

Also, if digest is disabled and you are still getting it, its a bug and we will fix it.

end

Excellent translation.

@pjh I bet you have multiple accounts, and that digest is coming from the other account. Remember the days of the 3 char username and the alt account?

PJH

@codinghorror said:

Er.. what? I can't understand this UK "English".

Which part of

@PJH said:

I ended up disabling them all within the first couple of days.

did you have difficulty parsing Jeff?

Could you perhaps suggest a more comprehensible way of expressing the "I tried turning off all email options?" than "I ended up disabling them all"?

@codinghorror said:

You have mailing list mode on, yes or no?

No. Because I couldn't stand what was being sent out.

@codinghorror said:

Because it sounds like you are singing the praises of one-email-per-post in the top part, yet your screenshot at the bottom shows all email settings disabled.

Indeed I was, and indeed they are. Because DC wasn't sending me one email per post....

@sam said:

One moment there @PJH can you elaborate on the "intolerable and unreadable" thingy

This:

This is the email sent out for one post. Note all the crap at the end? over 75% of that screenshot is not the post I'm being notified about

Compared to (say):

where I can see more than one post on the page at the same time. And no - this isn't a HTML/plaintext email issue.

sam

This is really frustrating:

Some people like this default, for some it is atrocious. I will add a user setting so you can override it.

Can you make a topic for this so I don't forget?

PJH

Stuck it in meta, since it's not a bug...

Luhmann

Ugh. Was it necessary to change my "-" list to a real list? That didn't really "add" value to the discussion does it? Now I feel compelled to make an off topic remark about my off topic reply. I made my point, or rather clarified @PJH 's point. Maybe I prefer adding "-" myself in front of every line instead of suffering with

a list
that doesn't behave like I would expect it to
if I create a newline any sane editor makes me a new list item
it has been this way since WordPerfect
See? It is rather silly.
So I did that shit once and since then I just use "-"
Why? Because fiddling with style/markup/bbcode/html tags is a barrier to typing my reply. So I rather just type away and not lose my time or strain of thought by messing around.

Unless I'm missing something it is my right as author of the damn reply to do so anyway I want.

PJH

- test

test 2
worksforme ;)

Onyx

@PJH said:

- test- test 2- worksforme

Backslashes are cheating!

Also, quotes really love them it seems.

Luhmann

a list
is started when you start your post with "-"

First type some tekst

then a "-"
you don't get a list
freaking thing is getting more and more silly

... to the bug-mobile!

Filled under: You're doing it WRONG! Don't start your post with a list! unless you want it to work

PJH

Well if you don't leave a blank line, then

you
won't
get
a
list

But if you leave a blank line

then
you
will.