Project layout - layered vs compartmentalized



  • For the longest time, accepted wisdom when organizing web projects has been to group likes with likes - controllers with controllers, javascripts with javascripts, styles with styles etc. If you create a new ASP.NET MVC, Rails or Flask project using standard templates, you'll get something like that. I'll call this organization "layered" because a guy on SO did so.

    Let's examine it on a generic web store project. There's the admin section and the site section (here imagined as two entirely separate apps). Public facing site has two pages - Home and Catalogue. There's also the Basket screen inside the catalogue. Products on both screens are displayed using the generic product widget. I avoided any known frameworks or MVC trappings. Let's just treat backend code as simple py files and concentrate on the big picture.

    ├── public
    │   ├── fonts
    │   ├── images
    │   │   ├── home
    │   │   │   ├── logo_large.jpg
    │   │   │   └── stock_handshake.jpg
    │   │   └── logo.png
    │   ├── js
    │   │   ├── admin.min.js (generated)
    │   │   ├── pages
    │   │   │   └── home
    │   │   │       └── home.min.js (generated)
    │   │   └── site.min.js (generated)
    │   └── style
    │       └── app.hr344hg3.css (generated)
    └── src
        ├── admin
        ├── admin.py
        ├── common
        ├── site
        │   ├── api
        │   │   ├── api.py
        │   │   ├── catalogue
        │   │   │   ├── basket.py
        │   │   │   └── catalogue.py
        │   │   └── home
        │   │       └── home.py
        │   ├── dal
        │   │   └── data_driver.py
        │   ├── js
        │   │   ├── admin.js
        │   │   ├── pages
        │   │   │   └── home
        │   │   │       └── home.js
        │   │   ├── site.js
        │   │   ├── vendor
        │   │   │   └── jquery.js
        │   │   └── widgets
        │   │       └── product.js
        │   ├── lib
        │   │   └── tools.py
        │   ├── style
        │   │   ├── app.scss
        │   │   ├── common
        │   │   │   └── colors.scss
        │   │   ├── pages
        │   │   │   ├── catalogue
        │   │   │   │   ├── basket.scss
        │   │   │   │   └── product.scss
        │   │   │   └── home
        │   │   │       └── home.scss
        │   │   └── vendor
        │   │       └── bootstrap.scss
        │   └── templates
        │       ├── catalogue
        │       │   ├── basket.html
        │       │   └── catalogue.html
        │       ├── components
        │       │   └── product.html
        │       ├── home
        │       │   └── home.html
        │       └── layout.html
        └── site.py
    

    As you can see, each asset group is inside its own little section, with subfolder structures roughly matching between them. We use some scripting library like grunt or gulp to generate minimized javascripts and styles and copy them to the static folder. Images are kept under static and added to vc there.

    So that's "layered" structure.

    However, lately there's been some grumbling about this accepted wisdom. With everything on the web moving towards components, why not do so with the project structure too? Instead of code for your Users page being strewn all over the place, group it all together under a single folder. I'll call this compartmentalized structure, because I like to challenge my spelling skills.

    Here's the same project using this kind of layout:

    ├── public
    │   ├── fonts
    │   ├── images
    │   │   ├── common
    │   │   │   └── logo.png (generated)
    │   │   └── home
    │   │       ├── logo_large.jpg (generated)
    │   │       └── stock_handshake.jpg (generated)
    │   ├── js
    │   │   ├── admin.min.js (generated)
    │   │   ├── home
    │   │   │   └── home.min.js (generated)
    │   │   └── site.min.js (generated)
    │   └── style
    │       └── app.hr344hg3.css (generated)
    └── src
        ├── admin
        ├── admin.py
        ├── common
        ├── site
        │   ├── api
        │   │   ├── api.py
        │   │   ├── catalogue
        │   │   │   ├── basket.html
        │   │   │   ├── basket.py
        │   │   │   ├── basket.scss
        │   │   │   ├── catalogue.html
        │   │   │   ├── catalogue.py
        │   │   │   └── product.scss
        │   │   ├── common
        │   │   │   ├── colors.scss
        │   │   │   └── logo.png
        │   │   ├── components
        │   │   │   ├── product.html
        │   │   │   └── product.js
        │   │   ├── home
        │   │   │   ├── home.html
        │   │   │   ├── home.js
        │   │   │   ├── home.py
        │   │   │   ├── home.scss
        │   │   │   └── images
        │   │   │       ├── logo_large.jpg
        │   │   │       └── stock_handshake.jpg
        │   │   ├── site.js
        │   │   └── vendor
        │   │       ├── bootstrap.scss
        │   │       └── jquery.js
        │   ├── dal
        │   │   └── data_driver.py
        │   └── lib
        │       └── tools.py
        └── site.py
    

    As you can see, everything related to home is next to each other - backend code, template, javascript, styles and images. Same goes for catalogues. Common assets are also grouped together in this example, although I could conceive keeping these under a mini layered structure instead. Note that images are kept inside src too and must be collected under static using grunt/gulp.

    So which is better? Here's my breakdown:

    • Layered structure is easier to set up. Most templates, frameworks and tools expect a layout like this. Compartmentalized structure requires a lot more scripting inside grunt/gulp to group everything together for deployment. Also, the way "shared" stuff (common, components...) is strewn all over the place feels clumsy.

    • Compartmentalized layout is easier to edit once you get going. In reality, you usually just work on one page at a time. So I imagine it'd be pretty cool to have everything in one place, instead of chasing it all over the project. Also, it follows the "web components" idea, which is the future of web (for now :-)). So for my new project, I'd like to take a shot at this.

    So what do you think? What are your experiences with each, if any? Any glaring problems with "compartmentalization" that I missed?

    If you want to play around with these faux projects, code is here: https://github.com/thecartman82/project_structure



  • I prefer compartmentalized because it makes more sense from the standpoint of having multiple compartments each being their own module/app/program/whatever.

    The layered style is exactly why I hate Linux's directory layout and prefer Windows' layout.



  • @LB_ said:

    I prefer compartmentalized

    +1.
    But the projects I work on are small



  • @swayde said:

    But the projects I work on are small

    IMO the smaller the better. I'm not a fan of huge libraries/programs/apps that try to do everything - I like small bite-sized things that I can consume on an as-needed basis. Also, by having them be smaller, they are much more approachable/maintainable and they can generally be really good at the one thing they do, rather than mediocre at a whole slew of various things.



  • I'd like to point out, for reasons of being pedantic, LAYERED is an architectural style, and doesn't really apply to the layout of code files in a project.

    They (the project's code files) can be laid out anyway that the programmer or team sees fit (as in your case, a compartmentalized view), but the overall architecture is still layered as each considered layered. Generally, each layer only communicates with layers directly next (above or below) to it.



  • Yeah I know. I need a better name for that style.



  • @cartman82 said:

    Yeah I know. I need a better name for that style.

    No worries! I figured as much, cause I really don't know how to describe that project layout as well. It's almost as if the project is structured by functionality.


  • Trolleybus Mechanic

    Allow me to present "Address.ascx". It's a user control that allows one to input an address (it has address 1, state, postal code, etc).

    Does this get compartmentalized under User (because user's have addresses), or Shipping (because you ship to addresses) or Marketing (because of mailing lists)?

    BUT WAIT THERE'S MORE!

    The address control contains a custom Regex validator. It is given an enum-- phone, postal code, email, etc. You put the regex validator next to a textbox, give it an enum, and it loads the correct regex. Where does this get compartmentalized? Validators? Address? Phone controls?

    The moment you have any sort of reuse, your compartmentalization goes haywire.



  • +1 for compartments. Each endpoint handler is a server in its own right. Make the endpoints the "compartments" for great sanity.

    Keep reusable code in libraries.

    Especially useful when you can just transparently do stuff like run a returnJSON method on arbitrary code. You can transparently lift library code to being an endpoint if you have generic "serve this object" methods. Or you can stack/decorate constraints at the endpoint without messing around with the reusable core.

    If you're seriously thinking about this kind of thing (using organization to make your job easier), you're one step closer to the Haskell dark side. "Components" are what objects were supposed to be, except for the fact that you use a terrible metaphor for organizing your code (classes), which is why you end up with "layers" in the first place.



  • You put it in the folder that contains all the other components you mentioned. It's clearly a generic thing, so it should go in a generic place. Usually the root folder is sufficient.



  • Except you don't want your API mixed-in with your static files, it makes it hard to scale. (You can put the static files your content delivery network, for example. Then you only have to scale the API.)



  • @Lorne_Kates said:

    Allow me to present "Address.ascx". It's a user control that allows one to input an address (it has address 1, state, postal code, etc).

    Does this get compartmentalized under User (because user's have addresses), or Shipping (because you ship to addresses) or Marketing (because of mailing lists)?

    In the above example, it would go under src/site/api/components, next to the "product" component.

    @Lorne_Kates said:

    The address control contains a custom Regex validator. It is given an enum-- phone, postal code, email, etc. You put the regex validator next to a textbox, give it an enum, and it loads the correct regex. Where does this get compartmentalized? Validators? Address? Phone controls?

    I didn't quite understand this one, but if the regex validator is also a component, the same as above. The regexes themselves would go somewhere under src/common, presuming you'd want to reuse them elsewhere.

    @blakeyrat said:

    Except you don't want your API mixed-in with your static files, it makes it hard to scale. (You can put the static files your content delivery network, for example. Then you only have to scale the API.)

    Can't grunt/gulp handle that?

    Script collects my dev images from all over the source tree and plops them under static/images. Then the code files and templates go to my app server, static stuff gets sent to the CDN?

    The idea is to have locality during development. To deploy this stuff, you will either way need to group them based on asset type.



  • Yeah, I would just use file extension filters.



  • @cartman82 said:

    Can't grunt/gulp handle that?

    I don't know how your shitty-ass open source build shit works. Presumably?

    But the real point is: I see laying out the API organization as COMPLETELY SEPARATE from laying out the static file organization. In most companies, they're worked on by entirely different teams.



  • This is essentially the same problem that Gmail addressed when it introduced tags for emails instead of folders. There's always a bunch of different ways to organize stuff in a hierarchy; none is ever objectively "right".

    Fortunately, filesystems can also do tags: they're called "links".

    Make a compartmentalized tree that keeps you happy; automate the creation of a forest of links to create the layered view that keeps your build system (and your more traditionalist coworkers) happy. Or vice versa (if you use hard links it doesn't matter which tree is derived from the other).



  • Isn't that what GoboLinux does?



  • Pretty much. It uses a completely non-LFSH filesystem layout, where each package gets wrapped in its own dedicated directory under /Programs (the slogan is "the filesystem is the database") and they have a script that walks the installed package directories to maintain a mostly-LFSH-compatible forest of symlinks for backward compatibility with broken apps and luddite sysadmins.

    I've not used it myself but I'm quite prepared to believe that it's not completely insane.



  • @blakeyrat said:

    In most companies, they're worked on by entirely different teams.

    I see.

    That's a good point. In larger companies, each page is made by teams of UX-ers, designers, frontend and backend devs. So they might benefit from separating all the various technologies they use into their own fiefdoms.

    I'm coming to this from the perspective of a full-stack developer, so I want to work on all the related files together.

    @flabdablet said:

    Make a compartmentalized tree that keeps you happy; automate the creation of a forest of links to create the layered view that keeps your build system (and your more traditionalist coworkers) happy. Or vice versa (if you use hard links it doesn't matter which tree is derived from the other).

    Interesting idea. But I'm not sure I get much with maintaining this forest of symlinks as opposed to just copying everything where it needs to go during the build.



  • @LB_ said:

    IMO the smaller the better. I'm not a fan of huge libraries/programs/apps that try to do everything - I like small bite-sized things that I can consume on an as-needed basis. Also, by having them be smaller, they are much more approachable/maintainable and they can generally be really good at the one thing they do, rather than mediocre at a whole slew of various things.

    That is a good résumé of the Unix philosophy !


Log in to reply