WTDWTF Development
-
@fbmac said in WTDWTF Development:
Backup/restore is something Discourse did very well in the usability at least
until you got boned becuase you didn't have enough free space for two extra copies of your database and uploads, because they implemented the back up process so mindbendingly incompetently.
-
@accalia but this problem is unrelated to what I like in their backup system.
A single file and my yaml config is all I need to recreate an instance.
Ideally, it should be a single file, created and downloaded from its web interface. And a clean install should be able to return to that state with this file trough it's web interface too.
-
@ben_lubar said in WTDWTF Development:
Okay, to get a testing environment roughly equivalent to WTDWTF...
Now make that into a thread and sticky it.
Filed under: Phrasing, boom.
-
So I made that docker thing on hub.docker.com. What does that do for us? How would I use it? It claims to do builds or something but there's no obvious tangible outputs that I can see, not being familiar with that site.
For reference:
-
@boomzilla said in WTDWTF Development:
What does that do for us? How would I use it?
If you do
docker pull boomzillawtf/tdwtf
, then it'll install tdwtf on the machine, without your having to include Github stuff
-
@Yamikuronue setup.bash and update.bash work using the Docker Hub version (and so does the update_nodebb.bash script on the EC2 server this forum is hosted on).
If you want to get a testing environment but don't want to rebuild the entire docker image, you can just clone the repository without any of the submodules. You need the submodules if you're going to
docker build
it, though.
-
@Yamikuronue said in WTDWTF Development:
If you do docker pull boomzillawtf/tdwtf, then it'll install tdwtf on the machine, without your having to include Github stuff
For the next person who tries this.
I made a new VM and had docker installed, the hello world thing (mentioned in docker's install guide) worked, but I couldn't pull boomzillawtf/tdwtf until I did this:
You need to add your current user [suppose you're logged in as ubuntu] to docker group as follow
sudo usermod -aG docker ubuntu
then logout & login again into the system or restart the system. test by docker versionOf course, my username is
boomzilla
, notubuntu
.
-
So now I've managed to pull down the docker image.
docker pull boomzillawtf/tdwtf Using default tag: latest latest: Pulling from boomzillawtf/tdwtf 51f5c6a04d83: Already exists a3ed95caeb02: Already exists 7004cfc6e122: Already exists 5f37c8a7cfbd: Already exists 8ad7684cace4: Already exists 2cc6b7c79e1c: Already exists a57359d4ac5f: Already exists d4cd051aeffa: Already exists c1f0d9a512e3: Already exists 211938dac113: Already exists 9b9ba525b773: Already exists 48efcc7c3704: Already exists a83f00263256: Already exists a15eddae3462: Already exists 54b80ba9d270: Already exists 79a281115c84: Already exists 78e1cc4316fd: Already exists 6741abe628f5: Already exists c0be141281ee: Already exists 43f44d709d93: Already exists 4e6182694c88: Already exists bb65be3c3acf: Already exists Digest: sha256:8dbd00a0c0bcfc39585a665e1ea187dc197a93e67d0239cebc319b68abcb172e Status: Image is up to date for boomzillawtf/tdwtf:latest
@ben_lubar: what do I do next? The docs on
docker start
are particularly unhelpful. Apparently I've managed to download stuff but there's no obvious way to use it. Internet searches aren't being helpful either.
-
@boomzilla Oh...apparently
docker run
is the magic word.EDIT: Sort of. It started up and then seemed to crap out at the web install, which says it's listening, but doesn't actually seem to be.
And I guess that's still not a container that I can interact with.
$ docker logs --tail 100 -f boomzilla/tdwtf Error: No such container: boomzilla/tdwtf
-
@boomzilla said in WTDWTF Development:
boomzilla/tdwtf
That's the image name. Check
docker ps -a
for container names.Also, I made
setup.bash
for a reason. Run that.
-
@ben_lubar That's in github? I was going for a "bare" install, but I guess I could do that instead.
-
@ben_lubar said in WTDWTF Development:
Also, I made setup.bash for a reason. Run that.
$ git clone https://github.com/boomzillawtf/tdwtf.git Cloning into 'tdwtf'... remote: Counting objects: 834, done. remote: Compressing objects: 100% (717/717), done. remote: Total 834 (delta 84), reused 0 (delta 0), pack-reused 117 Receiving objects: 100% (834/834), 1.61 MiB | 0 bytes/s, done. Resolving deltas: 100% (129/129), done. Checking connectivity... done. boomzilla@bz-docker:~$ cd tdwtf/ boomzilla@bz-docker:~/tdwtf$ ./setup.bash [sudo] password for boomzilla: cp: cannot create regular file ‘/etc/nginx/’: Not a directory
:haha.exe:
-
@ben_lubar Can you hook this up to Travis to verify that it builds correctly on commit? You could then set something up to deploy automatically if you want (I've done it before) but it seems like "run tests, then build containers and verify the start successfully" should be happening on commit to Master
-
@Yamikuronue said in WTDWTF Development:
run tests
That's what Travis does.
@Yamikuronue said in WTDWTF Development:
build containers
The container builds automatically on Docker Hub.
@Yamikuronue said in WTDWTF Development:
verify the start successfully
It'll always start successfully unless NodeBB is completely broken. The actual stuff that needs to be tested about the container is done manually.
-
@ben_lubar said in WTDWTF Development:
It'll always start successfully
@boomzilla said in WTDWTF Development:
cp: cannot create regular file ‘/etc/nginx/’: Not a directory
-
@Yamikuronue said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
It'll always start successfully
@boomzilla said in WTDWTF Development:
cp: cannot create regular file ‘/etc/nginx/’: Not a directory
That's not the docker image, that's because @boomzilla doesn't have nginx installed on the host.
-
@ben_lubar Why isn't nginx inside the container?
-
@Yamikuronue said in WTDWTF Development:
@ben_lubar Why isn't nginx inside the container?
If nginx was inside the container, we couldn't do the 0 downtime reset.
-
@ben_lubar said in WTDWTF Development:
@Yamikuronue said in WTDWTF Development:
@ben_lubar Why isn't nginx inside the container?
0 downtime reset
You keep using those words. I do not think they mean what you think they mean.
-
@error said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
@Yamikuronue said in WTDWTF Development:
@ben_lubar Why isn't nginx inside the container?
0 downtime reset
You keep using those words. I do not think they mean what you think they mean.
Was there any downtime today? Because when the update happened, servercooties didn't notice any.
-
@ben_lubar Why doesn't it get its own container in the system, then? Like mongo and redis apparently do?
-
@Yamikuronue said in WTDWTF Development:
@ben_lubar Why doesn't it get its own container in the system, then?
It could have one, but it doesn't have anything that needs to be updated in one piece. It's just a reverse proxy that gets its upstream addresses switched by the update script.
-
@ben_lubar But it makes deploying a test system that much simpler.
-
@ben_lubar said in WTDWTF Development:
That's not the docker image, that's because @boomzilla doesn't have nginx installed on the host.
Yeah, I didn't realize that was required. I'm one step closer. It seems to be running. Which is to say, there are two node processes running app.js. But there doesn't seem to be a way to get there with a browser.
-
@ben_lubar I remember you saying something about using
docker exec wtdwtf-nodebb gdb ...
, but when I tried that:$ docker exec wtdwtf-nodebb gdb attach 5569 rpc error: code = 2 desc = "oci runtime error: exec failed: exec: \"gdb\": executable file not found in $PATH"
-
@ben_lubar said in WTDWTF Development:
Was there any downtime today?
Yes. I was getting 504 for over an hour earlier
-
@boomzilla said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
That's not the docker image, that's because @boomzilla doesn't have nginx installed on the host.
Yeah, I didn't realize that was required. I'm one step closer. It seems to be running. Which is to say, there are two node processes running app.js. But there doesn't seem to be a way to get there with a browser.
There's an example nginx config file in the repo.
-
@ben_lubar said in WTDWTF Development:
There's an example nginx config file in the repo.
I saw that in the output. I guess that's an obtuse way of saying that one needs to configure nginx?
And if I'm in a VM whose only purpose is to work with this, can I just copy it somewhere to make it work?
-
@boomzilla said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
There's an example nginx config file in the repo.
I saw that in the output. I guess that's an obtuse way of saying that one needs to configure nginx?
And if I'm in a VM whose only purpose is to work with this, can I just copy it somewhere to make it work?
Put it in
/etc/nginx/sites-enabled/
and add an /etc/hosts entry to the computer you work on so thatnodebb.local
points to the VM.
-
@ben_lubar said in WTDWTF Development:
Was there any downtime today?
Over 2 hours of it this morning. :p
-
@ben_lubar said in WTDWTF Development:
Put it in /etc/nginx/sites-enabled/ and add an /etc/hosts entry to the computer you work on so that nodebb.local points to the VM.
OK. That works. When I attach gdb to the process, I get basically the same result we've been seeing, which is to say, OOM errors, which I think is just an artifact of something else going on. It's not the real problem.
I'm going to try some of the other ideas, because unfortunately, it looks like docker and gdb don't play nicely together.
-
@boomzilla said in WTDWTF Development:
which I think is just an artifact of something else going on
could it be an artifact of how we're getting it to print the stack trace?
-
@Jaloopa said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
Was there any downtime today?
Yes. I was getting 504 for over an hour earlier
504? I was getting 502's. For quite a while too..
-
@swayde whatever it was it wasn't ok
-
@ben_lubar I can't say it isn't possible.
-
@ben_lubar said in WTDWTF Development:
@error said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
@Yamikuronue said in WTDWTF Development:
@ben_lubar Why isn't nginx inside the container?
0 downtime reset
You keep using those words. I do not think they mean what you think they mean.
Was there any downtime today? Because when the update happened, servercooties didn't notice any.
I don't know about that day, I was on my smartphone.
I see this waaay too often:
-
@error We're still trying to figure out why NodeBB is locking up, but I can assure you that when you see that screen, it's not because of the update script.
-
@error said in WTDWTF Development:
I don't know about that day, I was on my smartphone.
I see this waaay too often:Trust us, we do, too.
https://what.thedailywtf.com/topic/19990/the-official-server-cooties-tracking-thread/1
-
pjh@sofa:~/src/tdwtf$ ./setup.bash latest: Pulling from boomzillawtf/tdwtf a2295636c7aa: Pull complete 3035387b9e83: Pull complete c394bf6e2527: Pull complete bbdb7053f3db: Pull complete 29794a1c054d: Pull complete 6c7b30b083b7: Pull complete 10d6c13c8eca: Pull complete cd2fe83035eb: Pull complete 194f0bfa8577: Pull complete cbc38b00e7aa: Pull complete 91cb80b0e0d3: Pull complete 443cbce531fd: Pull complete 8453398fdbf6: Pull complete a0c0e16218ef: Pull complete c04268d0601d: Pull complete ceb63cae96d4: Pull complete 42f2913ead80: Pull complete 1db765860d59: Pull complete 50500f08371f: Pull complete 71a93603a2f4: Pull complete ecce3108a14a: Pull complete ef277ad174ee: Pull complete 718c4c6bb181: Pull complete a7762d71f3ab: Pull complete ffd5f19e33b1: Pull complete deaa8e8b1f98: Pull complete 29ee4c435cb3: Pull complete c115b728de5a: Pull complete 13aab5fca7bf: Pull complete 430646dcae2a: Pull complete 1d1899eea67c: Pull complete Digest: sha256:81f2d6f9ef004a4583a480efda3158c52acaf6e649454d88f8c9ec5c086bdb61 Status: Downloaded newer image for boomzillawtf/tdwtf:latest docker: 'network' is not a docker command. See 'docker --help'.
pjh@sofa:~/src/tdwtf$ docker --version Docker version 1.6.2, build 7c8fca2
May want to check docker version (and whether nginx is installed) near the start of the script.
-
@NedFodder said in PJH's <s>personnel department</s> thread:
@PJH It would work without the hard refresh if @ben_lubar would add this to the custom header:
$(window).on('action:ajaxify.contentLoaded', details_shim.init); $(window).on('action:posts.loaded', details_shim.init);
@ben_lubar said in PJH's <s>personnel department</s> thread:
Ok - what's the development process to getting this done? I basically hacked my way to getting my CSS change done, and feel sure I was .
Presume that
- the setup script has just been run,
- nodebb is up and running
- nginx on the host is up and running and http://nodebb.local takes you to the forum
- the forum has been set up to enable most/all of the plugsins that we have here (is this something else that could be done in Dockerfile, or more likely/reasonably in setup.bash)
How/where should custom.js be edited, and what's the procedure for getting it into the docker container to test?
Or is is my method of changing it directly in the container until tested, then move a copy out into source control the correct way?
-
And a related question, how do we go about (e.g.) hacking nodebb using this setup?
-
@PJH said in WTDWTF Development:
Or is is my method of changing it directly in the container until tested, then move a copy out into source control the correct way?
It's not correct, but it's not wrong either.
Basically, if you change https://github.com/boomzillawtf/tdwtf/blob/f4aecd92da35d98a97e837a162eee26a6da34bb0/update.bash#L4 to
docker build -t boomzillawtf/tdwtf .
, it'll build the image locally instead of downloading it from Docker Hub. You'll need togit submodule update --init --recursive
before you run that, though.
-
@PJH said in WTDWTF Development:
hacking nodebb
The best way for that is probably to change it directly in your local container and send a PR when that's done.
-
@ben_lubar This worked for me:
- Install docker and nginx
- Clone the repo,
npm install
- Configure nginx with
nginx-config-example
and add nodebb.local to /etc/hosts git submodule update --init --recursive
- Edit setup.bash and update.bash from
docker pull boomzillawtf/tdwtf
todocker build -t boomzillawtf/tdwtf .
./setup.bash
- Log into the forum, enable all the plugins we use (Aside: Is there no way for this to be automated?)
- Use the forum, notice the shim requires a hard refresh to work.
- Add my two-line fix to the end of
plugins/nodebb-plugin-tdwtf-customizations/custom.js
(inside thejshint ignore
block). npm run lint
andnpm test
./update.bash
- Use the forum, notice the shim works without a hard refresh.
Looks like everything is working. I have no idea how to submit a PR, though. I guess I was supposed to do something about forks or branches before I started changing code...
-
-
So now that I have the official test environment setup, I wanted to use it to test the plugins I've been playing with.
If you use the setup script as-is (with
docker pull boomzillawtf/tdwtf
), it seems like the only way to get the plugin loaded is tonpm publish
and download it from the admin page. Every change to the plugin code requires anothernpm publish
and update from the admin page. Not very friendly while you're still developing the plugin.So it seems better to use the local image (with
docker build -t boomzillawtf/tdwtf .
). I guessed that I'd have to get my source code in thetdwtf/plugins
folder in order to get it into the container. However, symlinks (npm link
andln -s
) don't work, but hard links (cp -al
) do. Now when I change code, all I do is rerunupdate.bash
.@ben_lubar am I doing it wrong?
-
@NedFodder
./update_local.bash
is probably what you want. Symlinks only work if they're relative and within the data that gets copied.
-
@ben_lubar said in WTDWTF Development:
./update_local.bash is probably what you want.
Cool, that wasn't there the last time I pulled the repo. Would it make sense to make a similar setup_local.bash that also has
git submodule update --init --recursive
?@ben_lubar said in WTDWTF Development:
Symlinks only work if they're relative and within the data that gets copied.
So
npm link
won't ever work, which is ok I guess. Since you said "within the data that gets copied", it finally dawned on my I should have looked atDockerfile
. Would it have been easier to edit that file instead of creating hardlinks to my code in the plugins folder?
-
@NedFodder said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
./update_local.bash is probably what you want.
Cool, that wasn't there the last time I pulled the repo. Would it make sense to make a similar setup_local.bash that also has
git submodule update --init --recursive
?@ben_lubar said in WTDWTF Development:
Symlinks only work if they're relative and within the data that gets copied.
So
npm link
won't ever work, which is ok I guess. Since you said "within the data that gets copied", it finally dawned on my I should have looked atDockerfile
. Would it have been easier to edit that file instead of creating hardlinks to my code in the plugins folder?I suggest
git submodule add https://github.com/NedFodder/nodebb-plugin-tdwtf-articles plugins/nodebb-plugin-tdwtf-articles && cd plugins/nodebb-plugin-tdwtf-articles && git checkout master && git remote set-url --push origin git@github.com:NedFodder/nodebb-plugin-tdwtf-articles.git && cd ../..
and then you'll have the plugin in your repository linked from the tdwtf repo and ready for a very simple pull request when you're ready with it.
-
@NedFodder said in WTDWTF Development:
Dockerfile
@ben_lubar, few questions about this that I can't seem to figure out from docker documentation....
What's the purpose of the following line:
COPY NodeBB/package.json /usr/src/app/
./package.json
exists, but./NodeBB/package.json
doesn't.RUN npm install ./plugins/*/ `cat ./plugins/other.txt`
Apart from the fact that the plugins in
other.txt
have version numbers and the directories in./plugins
don't, is there any particular reason for the two methods of specifying plugins?COPY emoji/emojione/assets/svg /usr/src/app/node_modules/nodebb-plugin-emoji-one/public/static/images COPY emoji/emojione/LICENSE.md /usr/src/app/node_modules/nodebb-plugin-emoji-one/public/static/images/
./emoji/emojione/
appears to be empty.COPY emoji/fontawesome/black/png/64 /usr/src/app/tdwtf-emoji/fontawesome
./emoji/fontawesome
appears to be empty.