Imports (Was: {brace yourselves} the import is coming {Spoiler Alert: Not all of it} [i.e. blakeyrat was not utterly wrong for the first time ever] Filed under: append-only titles.)
-
So what's going on with the import, then?
The blue messages really put a damper on the whole reading experience - a far harsher abuse of mod power than the blakeygreen text ever was.And why is this thread's related thread is this thread?
-
How the hell does this topic have 702 posts in it?
-
The import has been happening for a long time now. The OP was a week ago...
-
Has it not imported the original "we're moving to Discourse" topic yet?
-
How would we tell?
-
-
I saw a big burst of errors half an hour ago, but all seems quiet on the western front atm.
memory and disk space are fine.
-
I think some posts were imported:
-
I think some posts were imported:
Indeed. Prior to the import last night, there were 331.7K posts on the About page. We're now at 361.1K. The new sidebar category is up to August 2008. Still a long ways to go.
-
So the success rate is as good as ever
-
It seems there's a flaw in that topic's time continuum. ;)
-
memory and disk space are fine.
So why aren't backups happening? I thought disk space was the problem.
Full-site backups are 2G+ - those ½G ones are @shadowmod taking the database only.[pjh@sofa ~]$ aws s3 ls s3://wtdwtf-back | tail -n10 2015-03-20 04:08:32 400130273 what-the-daily-wtf-2015-03-20-040000.tar.gz 2015-03-21 04:09:14 403987128 what-the-daily-wtf-2015-03-21-040000.tar.gz 2015-03-22 03:17:08 4097246586 what-the-daily-wtf-2015-03-22-030000.tar.gz 2015-03-22 04:10:07 405665827 what-the-daily-wtf-2015-03-22-040000.tar.gz 2015-03-24 04:09:41 459205585 what-the-daily-wtf-2015-03-24-040000.tar.gz 2015-03-25 04:08:46 462053637 what-the-daily-wtf-2015-03-25-040000.tar.gz 2015-03-26 09:19:45 468866848 what-the-daily-wtf-2015-03-26-090912.tar.gz 2015-03-26 09:33:18 468859871 what-the-daily-wtf-2015-03-26-092149.tar.gz 2015-03-27 04:09:36 470864251 what-the-daily-wtf-2015-03-27-040000.tar.gz 2015-03-28 04:15:12 481834274 what-the-daily-wtf-2015-03-28-040000.tar.gz [pjh@sofa ~]$
-
It makes the tar and then gzips it as a separate step, so you actually need space for the compressed backup + the uncompressed backup + the actual live data.
-
...... wat? there's these things called pipes that eliminate the need for that..... they're super easy to use too.
-
Correct me if I'm wrong, but using pipes doesn't totally erase the need for HDD space?
-
There's also
tar czf
which gzips, or if you want 15% better compression,tar cjf
for bzip2, or if you want 30% better compression,tar cJf
for xz. No need to type pipe characters anywhere.I can confirm that all three of these compressors are installed on the docker image.
-
Well, it would mean not having the uncompressed backup on the drive. You'd only need the compressed backup and the live data.
-
it does eliminate the nees to store the uncompressed step.
-
15 minute
-
Something about reading and high post counts… ;)
-
-
grep -A1 'Exception while creating' import2.log | sort | uniq -c
94 Couldn't find User without an ID 8736 EXECABORT Transaction discarded because of previous errors. 403 MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 2 Script Timed Out
grep 'Skipping' import2.log | sed -e 's/[ ]*[0-9]\+/#/g' | sort | uniq -c
9235 # /# (#.#%) Exception while creating post#. Skipping. 3913 # /# (#.#%) Skipping post#: category# not found 170361 # /# (#.#%) Skipping post#: parent post# not found
-
@PJH you can enable emailing when you get a chance. I'm not sure why the script isn't doing that on its own.
-
```
94 Couldn't find User without an ID 8736 EXECABORT Transaction discarded because of previous errors. 403 MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 2 Script Timed Out
So… the script will need running again then.
-
If we can figure out why redis is exploding, it'll only need to be run one more time.
-
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk.
Disk space issue maybe?
-
Maybe someone should lock the forum, take a backup, do the import on a local VM, take a backup, restore the VM backup on the original droplet, and then unlock the forum. But that would take ages if I did it.
-
Wouldn't exactly be a swift task if anyone else did it either. Plus you'd have the forum locked for hours, pissing off quite a lot of users.
-
Well, if the import had happened before the original switch to Discourse, maybe there wouldn't have been any problem. But that's the past and we can't change it.
-
But that's the past and we can't change it.
You're in luck! I invented a time machine next year. Should be here soon...
-
@PJH you can enable emailing when you get a chance. I'm not sure why the script isn't doing that on its own.
Done.
-
@ben_lubar http://what.thedailywtf.com/t/paypal-egg-cc-wtf/37910 Not sure if this is a known issue, but looks like a category was added...
-
It's a known thing; once the import is done, the topics will be Jeffed en-masse into the right categories
-
Cool
-
-
So ... instead of this data which has a verifiable checksum built in you propose instead using as a unique identifier $1$1$1$1$1$1$1 ?
Hm. What's with all the $1s in all the posts? Regex fail?
-
-
Hm. What's with all the $1s in all the posts? Regex fail?
because
s/\$1/[\*_ ]/
won't suffice... so it randomly puts
<p>
at the start of random posts to break the quoting.That said, I'm sure @ben_lubar said that was sorted, yet the newly imported posts are exhibiting both behaviours.
-
I'm sure @ben_lubar said that was sorted
But was it test- actually, almost certainly wasn't…
Not Ben's fault, I know
-
We can make Ben go through and tidy up, post by post, right?
-
Remember when this was a banner topic, and the background was blue? Turns out it was a bug:
-
So what's up with the import now?
-
Most of SSDS is imported (albeit with formating errors). What else would you need anyway?
-
-
Dear $deity, why?
Maybe I am not clued in, yet.
Filed Under: And, as many others, I probably never will be :D
-
Maybe I am not clued in, yet.
If you ever become clued in to Swampy, despair. It will surely involve severe psychosis and/or large quantities of psychotomimetic chemicals.
-
-
Needs some TL4 love, but it's a pivotal moment in wtdwtf history...
-
There's more than one SSDS thread... The one where spenk tried to actually use it was pretty epic. (Sorry spenk, SSDS doesn't know about one-letter file extensions. Please rename all your .c files to .txt)
Or maybe that was this thread? I dunno.
-
I guess I'll fix some markdown when I get around to reading it all.