WTDWTF Development
-
@TDWTF-NodeBB-Development
IIT we try to coordinate our development efforts, share knowledge, and propose ideas.We don't seem to have any kind of DevOps, and while it's not my specialty, I'd rather fill that role than work without any.
I'll start with a quick overview of the CI libraries and technologies I've been working with, what they are and how they fit in:
Travis CI
This tool automatically checks out each new commit on Github, loads a docker image, runs tasks like lint and unit tests, and emails out the result. There's a neat build status badge you can put on your Github.Grunt/Gulp
These are competing NodeJS task/build automation platforms. Both have wide adoption and many third-party modules available. Of the two, I prefer Gulp.Karma
This is a test framework that can load up a test harness (e.g. Mocha, Jasmine), a web browser (e.g. Firefox, Chrome, PhantomJS [a headless browser]), and then serve files to them through a built-in web server, and each file can be preprocessed. Endlessly configurable through plugins.Jasmine/Mocha
These are the test harnesses: the things that actually run your tests. Jasmine seems suited for client-side testing and comes with a built-in assertion library. Mocha seems better for server-side (NodeJS) testing, and supports several third-party assertion libraries (Chai being the most prominent one).As far as unit testing goes, I prefer a paradigm called Behavior Driven Development. This uses what are known as "spec tests." Each test is written as a specification. E.g.
describe( 'custom.scss', () => { it( 'prevents big tag abuse', () => { expect( $( 'big big big big big' ).css( 'font-size' ) ).toBe( '30px' ); } ); } );
See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.
I have a PR for TDWTF that implements all of these things in conjunction: https://github.com/boomzillawtf/tdwtf/pull/2/files
I'd love to know if I'm going down the right path with this or if I'm totally off-base. This should be an open, cooperative process. I don't want to just go off on my own here, or dictate what our processes should be. Feedback is very welcome - both positive and negative - and I hope we can collaborate more closely.
-
@error said in WTDWTF Development:
See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.
Reasonable, though a bit reliant on the particulars of the behaviour rather than the general principle. When is a test testing the specification, and not the implementation, and when is a test really a test and not just an experiment with the code? It's a deceptively difficult thing…
Which isn't that I disapprove (and Travis/etc. is a great move!)
-
@dkf said in WTDWTF Development:
@error said in WTDWTF Development:
See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.
Reasonable, though a bit reliant on the particulars of the behaviour rather than the general principle. When is a test testing the specification, and not the implementation, and when is a test really a test and not just an experiment with the code? It's a deceptively difficult thing…
The
describe
andit
methods both take a string followed by a function. I believe the string documents the intended behavior, while the function tests that the implementation follows it.
I like that you can read through just the string parameters to get an idea of what it does, and that when a test goes red it calls out to you exactly which specified behavior you broke. If you're deliberately changing the behavior, then it also forces that to be a conscious decision, and you should rewrite the spec to match the new reality.I like to specify expected behavior and also what happens when a contract is violated. E.g.
it( 'throws an Error when given a non-number', () => { expect( () => foo( 'bar' ) ).toThrow(); } ); it( 'returns null when given a non-finite number', () => { expect( foo( +Infinity ) ).toBe( null ); expect( foo( -Infinity ) ).toBe( null ); } );
Many times now I've been surprised at the behavior of my functions when given unexpected inputs, and the spec tests have shown this to me before it could arise as a hard-to-pinpoint bug in a program. (E.g., the first test above probably returned
NaN
before I wrote it and saw it fail.)
-
@error I vastly prefer to test client-side code server-side with Mocha, if we can. It's much easier to do in a CI pipeline.
Don't forget Istanbul!
-
@Yamikuronue Travis CI runs our Karma tests just fine.
I did forget
PolandIstanbul.
-
@error Sure, but now you've introduced another point of failure into the build system. And heaven forbid you test any classes that have any dependencies or rely on anything that's loaded globally into the ecosystem. I've seen it turn into a mess often enough to be gunshy.
-
@Yamikuronue I really really want to get us to over to using AMD or SystemJS modules, and eliminate global dependencies wherever possible. Unfortunately, I consider an integration testing environment a prerequisite for that, and I haven't succeeded in setting up a reasonable facsimile environment where I can run such integration tests. Containers I think would be the key, but I don't have the experience in configuring containers.
-
@error Two things:
One, you can (and should) do integration testing without booting up a full system, by mocking out the troublesome bits of the dependencies.
Two, you don't need a container infrastructure to deploy to a staging server, I have that running from Travis for one of my websites. However, I believe @ben_lubar is running containers for nodebb at the moment, so you'll want to talk to him about perhaps deploying another container for staging we can test plugins in.
-
@Yamikuronue said in WTDWTF Development:
However, I believe @ben_lubar is running containers for nodebb at the moment
They're using the containers-as-VMs antipattern, but if we had a NodeBB fork that came with the plugins we use pre-installed, I could make it use Docker the way Docker should be used.
-
@ben_lubar Why do the plugins need to be pre-installed? Can't we put together a build script that installs them and bakes a container? That's how Docker's meant to be done with CI.
-
@Yamikuronue pre-installed in the container is what I meant.
-
-
@ben_lubar, error, y'all may want to think twice before dusting up with mistress QA over testing practices.....
just a thought.
-
@ben_lubar Like I said. From what I understand, the idea is that with Docker and CI, you pull down a base image, configure the software, install plugins, and bake that into a container, which is then your deployable artifact. Isn't it? Granted, my knowledge comes from watching a lot of talks about Netflix, so maybe they have some magic fairies we don't have, but that seems to be the general idea.
-
@Yamikuronue oh wow, I'm being silly. We can just have a dockerfile that depends on another dockerfile.
-
@accalia No, I definitely welcome @Yamikuronue's expertise. I'm glad she finally took notice of this topic.
I'm not in any way trying to be the director of this project. I just want to actually push it forward, because right now I just see a couple of devs working in silos, and I think we're capable of much more if we cooperate.
-
@ben_lubar I believe they call them "layers."
-
@error no, each command is a layer. So if you do RUN twice, you get two layers.
-
-
@Yamikuronue Maybe I was just looking in the wrong places.
I, too, have been contributing to upstream projects. (That particular one was just because I couldn't proxy to TDWTF because of our corporate proxy at work.)
-
@error Yeah, I figured i'd try to help with nodeBB's test coverage to help prevent some of the worst of Discourse, but I've been busy with my own projects since I issued that PR
-
@ben_lubar said in WTDWTF Development:
@Yamikuronue pre-installed in the container is what I meant.
you can npm install plugins in a Dockerfile, I did that when I was messing with these things
-
-
@ben_lubar, you can do LESS with:
npm install -g less lessc custom.css wherever/nodebb/wants/the/file.css
-
@Yamikuronue yeah, the problem is that it's stored in the database currently. What I think we should do is make a NodeBB plugin in that plugins folder that just adds the header scripts and the LESS and then get rid of the admin panel LESS/HTML.
-
-
@Yamikuronue said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
it's stored in the database currently.
@julianlam RU ?
Where else would you store admin panel settings? The uploads folder?
-
@ben_lubar said in WTDWTF Development:
Where else would you store admin panel settings?
Why the fuck is custom CSS an admin panel setting and not an upload of some kind?
-
@Yamikuronue said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
Where else would you store admin panel settings?
Why the fuck is custom CSS an admin panel setting and not an upload of some kind?
Best guess: Discourse did it that way
-
@ben_lubar My minecraft server's control panel does it better: there's a pretty basic text editor that serializes to the config files that are required on disk for the server to pick up on the settings. Something like that would be appropriate here.
-
@Yamikuronue why do that when you can keep it in memory with the rest of the settings that are cached and synchronized?
-
@ben_lubar Why do all that work when you can have nginx serve the static file directly?
-
Storing custom CSS in a database field isn't too wtf to be fair. You don't have to deal with file system permissions issues.
-
@Yamikuronue the stuff that I'm talking about is included in the HTML
<head>
tag on every page.
-
-
-
@ben_lubar Why does custom CSS belong in the header of every page tho?
-
@Yamikuronue it's not like it's being sent on every page view, just the cold loads.
-
@ben_lubar And is never cacheable.
-
@ben_lubar said in WTDWTF Development:
put a google-site-verification tag?
In a static file somewhere?
-
@Tsaukpaetra said in WTDWTF Development:
@ben_lubar said in WTDWTF Development:
put a google-site-verification tag?
In a static file somewhere?
this, exactly this.
-
@accalia Really, in a SPA like this, everything but the API can be flat, cacheable files. Then node only has to process the incoming AJAX calls and return data, serialize data, et cetera.
-
@Yamikuronue said in WTDWTF Development:
Istanbul
No, I have nothing else to contribute, except being happy I now know what a bunch of stuff in sockmafia is.
-
$ docker build -t boomzillawtf/tdwtf . Sending build context to Docker daemon 12.46 MB Step 1 : FROM node:4 ---> 1f9967dbfc82 Step 2 : RUN mkdir -p /usr/src/app ---> Using cache ---> 39feb1af1126 Step 3 : WORKDIR /usr/src/app ---> Using cache ---> f2a28d576ea8 Step 4 : ENV NODE_ENV production daemon false silent false ---> Using cache ---> 2ae2f9403bb2 Step 5 : COPY NodeBB/package.json /usr/src/app/ ---> Using cache ---> dff1660d8305 Step 6 : RUN npm install ---> Using cache ---> 1261b0b380f1 Step 7 : COPY NodeBB /usr/src/app ---> Using cache ---> b14d4851d6af Step 8 : COPY plugins /usr/src/app/plugins ---> Using cache ---> 45bd2c32379e Step 9 : RUN npm install ./plugins/*/ `cat ./plugins/other.txt` ---> Using cache ---> f9621c8c23e0 Step 10 : RUN echo public/uploads/*/ > .make-uploads-folders ---> Using cache ---> e3c453de3247 Step 11 : EXPOSE 4567 ---> Using cache ---> 18a2c091de10 Step 12 : VOLUME /usr/src/app/docker ---> Using cache ---> d6a48f2e4162 Step 13 : VOLUME /usr/src/app/public/uploads ---> Using cache ---> 31410ea5d356 Step 14 : RUN ln -s /usr/src/app/docker/config.json /usr/src/app/config.json ---> Using cache ---> 2a0db905fa9f Step 15 : CMD cat .make-uploads-folders | xargs mkdir -p && ./nodebb upgrade && echo 1 > pidfile && exec node loader.js ---> Using cache ---> 95906ee730f2 Successfully built 95906ee730f2 $ docker run -d --name wtdwtf2-nodebb --net wtdwtf2 --ip 172.19.0.253 --restart unless-stopped -v /usr/share/nginx/wtdwtf-nodebb.config:/usr/src/app/docker:ro -v /usr/share/nginx/wtdwtf-nodebb.uploads:/usr/src/app/public/uploads boomzillawtf/tdwtf f6492ac936f70d0aa0ebc3c1ba0ca343326d456c85fcd839d3817ecd77ee0d40
It works!
I'll have to ask @julianlam why version 2.0.0 isn't on NPM or GitHub but it's somehow asking me to install it.
Edit:
-
Okay, to get a testing environment roughly equivalent to WTDWTF:
- Clone that repository
git submodule update --init --recursive
edit: this step is no longer required if you don't want to recompile the testing environment./setup.bash
- Set up nginx as a reverse proxy using the example config file. The most important part is the upstream block.
To update:
git pull
git submodule update --init --recursive
edit: this step is no longer required if you don't want to recompile the testing environment./update.bash
The forum shouldn't go down for any noticeable amount of time.
If your forum instance is accessible from the internet, I suggest changing the
not-secret wtdwtf test
to random gibberish, such as the result ofuuidgen
. The config file is in/usr/share/nginx/wtdwtf-nodebb.config/config.json
on the host.Once @boomzilla handles #3, there'll be a prebuilt version accessible via
docker pull
. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.
-
@ben_lubar said in WTDWTF Development:
Okay, to get a testing environment roughly equivalent to WTDWTF:
- Clone that repository
git submodule update --init --recursive
./setup.bash
- Set up nginx as a reverse proxy using the example config file. The most important part is the upstream block.
To update:
git pull
git submodule update --init --recursive
./update.bash
The forum shouldn't go down for any noticeable amount of time.
If your forum instance is accessible from the internet, I suggest changing the
not-secret wtdwtf test
to random gibberish, such as the result ofuuidgen
. The config file is in/usr/share/nginx/wtdwtf-nodebb.config/config.json
on the host.Once @boomzilla handles #3, there'll be a prebuilt version accessible via
docker pull
. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.
-
@Yamikuronue said in WTDWTF Development:
Why the fuck is custom CSS an admin panel setting and not an upload of some kind?
I suspect that most people do far far less in there than we do. And it's handy and easy to edit right there.
-
@ben_lubar said in WTDWTF Development:
Once @boomzilla handles #3, there'll be a prebuilt version accessible via docker pull. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.
I did see that stuff in my inbox. I have out of town guests, so definitely won't happen until at least tomorrow.
-
@boomzilla said in WTDWTF Development:
it's handy and easy to edit right there.
Do that instead. Those are all flatfiles the server parses, as the control panel is a third-party add-on.
-
@lucas1 storing in the database simplify backups a little.
Backup/restore is something Discourse did very well in the usability at least