WTDWTF Development


  • Winner of the 2016 Presidential Election

    @TDWTF-NodeBB-Development
    IIT we try to coordinate our development efforts, share knowledge, and propose ideas.

    We don't seem to have any kind of DevOps, and while it's not my specialty, I'd rather fill that role than work without any.

    I'll start with a quick overview of the CI libraries and technologies I've been working with, what they are and how they fit in:

    Travis CI
    This tool automatically checks out each new commit on Github, loads a docker image, runs tasks like lint and unit tests, and emails out the result. There's a neat build status badge you can put on your Github.

    Grunt/Gulp
    These are competing NodeJS task/build automation platforms. Both have wide adoption and many third-party modules available. Of the two, I prefer Gulp.

    Karma
    This is a test framework that can load up a test harness (e.g. Mocha, Jasmine), a web browser (e.g. Firefox, Chrome, PhantomJS [a headless browser]), and then serve files to them through a built-in web server, and each file can be preprocessed. Endlessly configurable through plugins.

    Jasmine/Mocha
    These are the test harnesses: the things that actually run your tests. Jasmine seems suited for client-side testing and comes with a built-in assertion library. Mocha seems better for server-side (NodeJS) testing, and supports several third-party assertion libraries (Chai being the most prominent one).

    As far as unit testing goes, I prefer a paradigm called Behavior Driven Development. This uses what are known as "spec tests." Each test is written as a specification. E.g.

    describe( 'custom.scss', () => {
        it( 'prevents big tag abuse', () => {
            expect( $( 'big big big big big' ).css( 'font-size' ) ).toBe( '30px' );
        } );
    } );
    

    See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.

    I have a PR for TDWTF that implements all of these things in conjunction: https://github.com/boomzillawtf/tdwtf/pull/2/files

    I'd love to know if I'm going down the right path with this or if I'm totally off-base. This should be an open, cooperative process. I don't want to just go off on my own here, or dictate what our processes should be. Feedback is very welcome - both positive and negative - and I hope we can collaborate more closely.


  • Discourse touched me in a no-no place

    @error said in WTDWTF Development:

    See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.

    Reasonable, though a bit reliant on the particulars of the behaviour rather than the general principle. When is a test testing the specification, and not the implementation, and when is a test really a test and not just an experiment with the code? It's a deceptively difficult thing…

    Which isn't that I disapprove (and Travis/etc. is a great move!)


  • Winner of the 2016 Presidential Election

    @dkf said in WTDWTF Development:

    @error said in WTDWTF Development:

    See how it kind of reads like English, each test documents how a component should behave while simultaneously verifying that behavior.

    Reasonable, though a bit reliant on the particulars of the behaviour rather than the general principle. When is a test testing the specification, and not the implementation, and when is a test really a test and not just an experiment with the code? It's a deceptively difficult thing…

    The describe and it methods both take a string followed by a function. I believe the string documents the intended behavior, while the function tests that the implementation follows it.
    I like that you can read through just the string parameters to get an idea of what it does, and that when a test goes red it calls out to you exactly which specified behavior you broke. If you're deliberately changing the behavior, then it also forces that to be a conscious decision, and you should rewrite the spec to match the new reality.

    I like to specify expected behavior and also what happens when a contract is violated. E.g.

    it( 'throws an Error when given a non-number', () => {
        expect( () => foo( 'bar' ) ).toThrow();
    } );
    it( 'returns null when given a non-finite number', () => {
        expect( foo( +Infinity ) ).toBe( null );
        expect( foo( -Infinity ) ).toBe( null );
    } );
    

    Many times now I've been surprised at the behavior of my functions when given unexpected inputs, and the spec tests have shown this to me before it could arise as a hard-to-pinpoint bug in a program. (E.g., the first test above probably returned NaN before I wrote it and saw it fail.)


  • mod

    @error I vastly prefer to test client-side code server-side with Mocha, if we can. It's much easier to do in a CI pipeline.

    Don't forget Istanbul!


  • Winner of the 2016 Presidential Election

    @Yamikuronue Travis CI runs our Karma tests just fine.

    I did forget PolandIstanbul.


  • mod

    @error Sure, but now you've introduced another point of failure into the build system. And heaven forbid you test any classes that have any dependencies or rely on anything that's loaded globally into the ecosystem. I've seen it turn into a mess often enough to be gunshy.


  • Winner of the 2016 Presidential Election

    @Yamikuronue I really really want to get us to over to using AMD or SystemJS modules, and eliminate global dependencies wherever possible. Unfortunately, I consider an integration testing environment a prerequisite for that, and I haven't succeeded in setting up a reasonable facsimile environment where I can run such integration tests. Containers I think would be the key, but I don't have the experience in configuring containers.


  • mod

    @error Two things:

    One, you can (and should) do integration testing without booting up a full system, by mocking out the troublesome bits of the dependencies.

    Two, you don't need a container infrastructure to deploy to a staging server, I have that running from Travis for one of my websites. However, I believe @ben_lubar is running containers for nodebb at the moment, so you'll want to talk to him about perhaps deploying another container for staging we can test plugins in.



  • @Yamikuronue said in WTDWTF Development:

    However, I believe @ben_lubar is running containers for nodebb at the moment

    They're using the containers-as-VMs antipattern, but if we had a NodeBB fork that came with the plugins we use pre-installed, I could make it use Docker the way Docker should be used.


  • mod

    @ben_lubar Why do the plugins need to be pre-installed? Can't we put together a build script that installs them and bakes a container? That's how Docker's meant to be done with CI.



  • @Yamikuronue pre-installed in the container is what I meant.


  • Winner of the 2016 Presidential Election


  • sockdevs

    @ben_lubar, error, y'all may want to think twice before dusting up with mistress QA over testing practices.....

    just a thought.


  • mod

    @ben_lubar Like I said. From what I understand, the idea is that with Docker and CI, you pull down a base image, configure the software, install plugins, and bake that into a container, which is then your deployable artifact. Isn't it? Granted, my knowledge comes from watching a lot of talks about Netflix, so maybe they have some magic fairies we don't have, but that seems to be the general idea.



  • @Yamikuronue oh wow, I'm being silly. We can just have a dockerfile that depends on another dockerfile.


  • Winner of the 2016 Presidential Election

    @accalia No, I definitely welcome @Yamikuronue's expertise. I'm glad she finally took notice of this topic.
    I'm not in any way trying to be the director of this project. I just want to actually push it forward, because right now I just see a couple of devs working in silos, and I think we're capable of much more if we cooperate.


  • Winner of the 2016 Presidential Election

    @ben_lubar I believe they call them "layers."



  • @error no, each command is a layer. So if you do RUN twice, you get two layers.


  • mod

    @error said in WTDWTF Development:

    I just see a couple of devs working in silos

    Yeah...

    :D


  • Winner of the 2016 Presidential Election

    @Yamikuronue Maybe I was just looking in the wrong places.
    I, too, have been contributing to upstream projects. (That particular one was just because I couldn't proxy to TDWTF because of our corporate proxy at work.)


  • mod

    @error Yeah, I figured i'd try to help with nodeBB's test coverage to help prevent some of the worst of Discourse, but I've been busy with my own projects since I issued that PR



  • @ben_lubar said in WTDWTF Development:

    @Yamikuronue pre-installed in the container is what I meant.

    you can npm install plugins in a Dockerfile, I did that when I was messing with these things




  • mod

    @ben_lubar, you can do LESS with:

    npm install -g less
    
    lessc custom.css wherever/nodebb/wants/the/file.css
    


  • @Yamikuronue yeah, the problem is that it's stored in the database currently. What I think we should do is make a NodeBB plugin in that plugins folder that just adds the header scripts and the LESS and then get rid of the admin panel LESS/HTML.


  • mod

    @ben_lubar said in WTDWTF Development:

    it's stored in the database currently.

    0_1464482093803_upload-f5e4701b-7a8c-4860-b3ae-65f49f5b4961

    @julianlam :wtf: :flag_ru: :smoking: ?



  • @Yamikuronue said in WTDWTF Development:

    @ben_lubar said in WTDWTF Development:

    it's stored in the database currently.

    0_1464482093803_upload-f5e4701b-7a8c-4860-b3ae-65f49f5b4961

    @julianlam :wtf: RU :smoking: ?

    Where else would you store admin panel settings? The uploads folder?


  • mod

    @ben_lubar said in WTDWTF Development:

    Where else would you store admin panel settings?

    Why the fuck is custom CSS an admin panel setting and not an upload of some kind?



  • @Yamikuronue said in WTDWTF Development:

    @ben_lubar said in WTDWTF Development:

    Where else would you store admin panel settings?

    Why the fuck is custom CSS an admin panel setting and not an upload of some kind?

    Best guess: Discourse did it that way


  • mod

    @ben_lubar My minecraft server's control panel does it better: there's a pretty basic text editor that serializes to the config files that are required on disk for the server to pick up on the settings. Something like that would be appropriate here.



  • @Yamikuronue why do that when you can keep it in memory with the rest of the settings that are cached and synchronized?


  • mod

    @ben_lubar Why do all that work when you can have nginx serve the static file directly?



  • Storing custom CSS in a database field isn't too wtf to be fair. You don't have to deal with file system permissions issues.



  • @Yamikuronue the stuff that I'm talking about is included in the HTML <head> tag on every page.


  • mod



  • @Yamikuronue said in WTDWTF Development:

    @ben_lubar :facepalm:

    Where else would you put a google-site-verification tag?


  • mod

    @ben_lubar Why does custom CSS belong in the header of every page tho?



  • @Yamikuronue it's not like it's being sent on every page view, just the cold loads.


  • mod

    @ben_lubar And is never cacheable.


  • Impossible Mission Players - A

    @ben_lubar said in WTDWTF Development:

    put a google-site-verification tag?

    In a static file somewhere?


  • sockdevs

    @Tsaukpaetra said in WTDWTF Development:

    @ben_lubar said in WTDWTF Development:

    put a google-site-verification tag?

    In a static file somewhere?

    this, exactly this.


  • mod

    @accalia Really, in a SPA like this, everything but the API can be flat, cacheable files. Then node only has to process the incoming AJAX calls and return data, serialize data, et cetera.


  • Winner of the 2016 Presidential Election

    @Yamikuronue said in WTDWTF Development:

    Istanbul

    Not Constantinople?

    No, I have nothing else to contribute, except being happy I now know what a bunch of stuff in sockmafia is.



  • $ docker build -t boomzillawtf/tdwtf .
    Sending build context to Docker daemon 12.46 MB
    Step 1 : FROM node:4
     ---> 1f9967dbfc82
    Step 2 : RUN mkdir -p /usr/src/app
     ---> Using cache
     ---> 39feb1af1126
    Step 3 : WORKDIR /usr/src/app
     ---> Using cache
     ---> f2a28d576ea8
    Step 4 : ENV NODE_ENV production daemon false silent false
     ---> Using cache
     ---> 2ae2f9403bb2
    Step 5 : COPY NodeBB/package.json /usr/src/app/
     ---> Using cache
     ---> dff1660d8305
    Step 6 : RUN npm install
     ---> Using cache
     ---> 1261b0b380f1
    Step 7 : COPY NodeBB /usr/src/app
     ---> Using cache
     ---> b14d4851d6af
    Step 8 : COPY plugins /usr/src/app/plugins
     ---> Using cache
     ---> 45bd2c32379e
    Step 9 : RUN npm install ./plugins/*/ `cat ./plugins/other.txt`
     ---> Using cache
     ---> f9621c8c23e0
    Step 10 : RUN echo public/uploads/*/ > .make-uploads-folders
     ---> Using cache
     ---> e3c453de3247
    Step 11 : EXPOSE 4567
     ---> Using cache
     ---> 18a2c091de10
    Step 12 : VOLUME /usr/src/app/docker
     ---> Using cache
     ---> d6a48f2e4162
    Step 13 : VOLUME /usr/src/app/public/uploads
     ---> Using cache
     ---> 31410ea5d356
    Step 14 : RUN ln -s /usr/src/app/docker/config.json /usr/src/app/config.json
     ---> Using cache
     ---> 2a0db905fa9f
    Step 15 : CMD cat .make-uploads-folders | xargs mkdir -p && ./nodebb upgrade && echo 1 > pidfile && exec node loader.js
     ---> Using cache
     ---> 95906ee730f2
    Successfully built 95906ee730f2
    $ docker run -d --name wtdwtf2-nodebb --net wtdwtf2 --ip 172.19.0.253 --restart unless-stopped -v /usr/share/nginx/wtdwtf-nodebb.config:/usr/src/app/docker:ro -v /usr/share/nginx/wtdwtf-nodebb.uploads:/usr/src/app/public/uploads boomzillawtf/tdwtf
    f6492ac936f70d0aa0ebc3c1ba0ca343326d456c85fcd839d3817ecd77ee0d40
    

    It works!

    0_1464498952269_upload-9c9218a7-556b-4725-8291-1b6e3cda98e3

    I'll have to ask @julianlam why version 2.0.0 isn't on NPM or GitHub but it's somehow asking me to install it.

    Edit:



  • Okay, to get a testing environment roughly equivalent to WTDWTF:

    • Clone that repository
    • git submodule update --init --recursive edit: this step is no longer required if you don't want to recompile the testing environment
    • ./setup.bash
    • Set up nginx as a reverse proxy using the example config file. The most important part is the upstream block.

    To update:

    • git pull
    • git submodule update --init --recursive edit: this step is no longer required if you don't want to recompile the testing environment
    • ./update.bash

    The forum shouldn't go down for any noticeable amount of time.

    If your forum instance is accessible from the internet, I suggest changing the not-secret wtdwtf test to random gibberish, such as the result of uuidgen. The config file is in /usr/share/nginx/wtdwtf-nodebb.config/config.json on the host.

    Once @boomzilla handles #3, there'll be a prebuilt version accessible via docker pull. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.



  • @ben_lubar said in WTDWTF Development:

    Okay, to get a testing environment roughly equivalent to WTDWTF:

    • Clone that repository
    • git submodule update --init --recursive
    • ./setup.bash
    • Set up nginx as a reverse proxy using the example config file. The most important part is the upstream block.

    To update:

    • git pull
    • git submodule update --init --recursive
    • ./update.bash

    The forum shouldn't go down for any noticeable amount of time.

    If your forum instance is accessible from the internet, I suggest changing the not-secret wtdwtf test to random gibberish, such as the result of uuidgen. The config file is in /usr/share/nginx/wtdwtf-nodebb.config/config.json on the host.

    Once @boomzilla handles #3, there'll be a prebuilt version accessible via docker pull. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.

    @TDWTF-NodeBB-Development



  • @Yamikuronue said in WTDWTF Development:

    Why the fuck is custom CSS an admin panel setting and not an upload of some kind?

    I suspect that most people do far far less in there than we do. And it's handy and easy to edit right there.



  • @ben_lubar said in WTDWTF Development:

    Once @boomzilla handles #3, there'll be a prebuilt version accessible via docker pull. I'm not going to set this thing up on the live forum until that's done and we're sure it's ready.

    I did see that stuff in my inbox. I have out of town guests, so definitely won't happen until at least tomorrow.


  • mod

    @boomzilla said in WTDWTF Development:

    it's handy and easy to edit right there.

    0_1464614865295_upload-9eedadce-b5cc-460f-8a09-ab2be109102c

    0_1464614919290_upload-9af2f0f8-6a6f-4c83-a2dc-78f092defde6

    Do that instead. Those are all flatfiles the server parses, as the control panel is a third-party add-on.



  • @lucas1 storing in the database simplify backups a little.

    Backup/restore is something Discourse did very well in the usability at least


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.