Existential architecture fears — a case for DDD & framework agnosticism
I have 3 fears about the future of Agorakit, from a technical standpoint:
First, we desperately need static analysis, and it needs to be enforced in the CI pipeline. In a monolith, it's just too easy to accidentally break things without it. Not only will it prevent myriad bugs, but it will make it easier to follow code in a robust IDE. It's going to enforce type safety and a number of other issues as we increase the level on PHPStan from 0. Thankfully, this is already a work in progress and I'm confident we'll have it by the end of the year, hopefully to at least level 4.
Second, I believe Laravel is a governance ticking time bomb. Otwell is BDFL, has no interest in collective governance, adopted the classically untenable "I don't talk politics anymore" stance on social media, and took VC money over a year ago. The signs are all there for a Rails-level community fiasco. Now, I don't think the solution is to abandon Laravel, as I've stated previously, but rather to be careful to not implement Laravel "magic" that would be onerous to rebuild on a different framework, and to consciously choose architecture patterns that keep us decoupled from the framework as much as possible. The "Clean Architecture" pattern has more to say about this: https://softwarepatternslexicon.com/patterns-php/8/8/
Third, I expect the project to grow considerably over the next 5 years, and I am concerned MVC is going to become a bit of a nightmare as the models & controllers continue to grow. At a certain point, the project starts to become unapproachable because of the sheer size of the classes and their many interactions. This was my experience working on Vanilla Forums for a decade — the object bloat is very real. Static analysis will help manage this, in the sense it will find the errors we inevitably make, but I'd like us to consider another tool that improves communication in a project by making sure the product language and code language stay unified: The "Domain Driven Design" (DDD) pattern: https://softwarepatternslexicon.com/patterns-php/8/9/
I think there's a lot of overlap between Clean Architecture & DDD, so I don't think it's an either-or choice, it's just overlapping concepts I'm linking here for discussion. There are an increasing number of "Laravel DDD" articles & examples over the last few years:
- A Simple Guide to Domain-Driven Design (DDD) in Laravel
- Implementing Domain-Driven Architecture in Laravel: Setup, Advantages, and Practical Use Cases
- Example implementation of DDD in Laravel
The second & third fears are not "stop everything and change this" issues. However, if we plan to add more complex features like Badges & Collaborative Moderation, I think this is a good moment to consider whether we want to use those to start a shift in that direction.
As the first article in the list notes, for a simple project it is overkill to implement DDD. The benefit of it doesn't really kick in until the complexity has risen to the point that the extra overhead pays off. So what I'm trying to do is look into the future to see if that trade is worth it, and my sense is that it will be. If Agorakit wants to be the "all-in-one" community self-governance tool without adopting a plugin-driven framework, it feels inevitable that the overall size will grow to the point that the benefits will be worth it. And, if we wait until that future arrives, the refactoring will only be harder.
Lastly, I'm no DDD / Clean Architecture expert! 😅 I've certainly studied the concepts for many years & done the workshops, but the code I've tended to end up in wasn't a good candidate for either. So, I'm thinking about this as a collaborative learning experience, not a "Here is Lincoln's interpretation of DDD we must follow" directive. I'm sure it will involve a few experiments and wrong turns along the way.
Some things to mull over & discuss as the weather turns to winter (in the north). Let me know what you think about all this.
Great thoughts, thank you for taking care of the future of the project.
I took the time to read a bit about all the concepts you introduced, some I knew some were new.
Quick thoughts :
- I'm all for using more automation for code quality, and will be happy to help reach level 4 (sounds like a nice secret accomplishment)
- If I were to start today, maybe I'd be mature enough to start directly with symphony, but with what we already have, a complete exit from laravel would be costly in time (complete rewrite) and might kill the project (if not, our motivation :). I'm 100% for not using more laravel magic from now on, upgrade only when necessary since frankly we don't get that much with newer releases (except security when it's not supported anymore), and try to use generic packages when a new need arrives.
- 20 years ago I was happy with my spaghetti code and was considering mvc as overkill complexity. 10 years later I understood the interest and fully delved in. Maybe now is the time to step up :)
- This said I'd like to keep the project accessible for non experts coders (I'm not an expert either, but am happy to learn), simply to increase to chance to get more help. Maybe a non concern, a well organized codebase might be easier to approach.
- Some of the goals of DDD/clean design might be non goals for us and as I understand it we should not aim for them to avoid increasing complexity.
- For example imho, we don't want to be database type / datastore independent (something like "we can store discussions in text files and queries still works" kind of abstraction). Even sqlite support is maybe a distraction (maybe). Otoh, eloquent could be considered as something we want to abstract from, as a future laravel exit strategy. But at that point someone will come up with a package that does it for you I think.
- Same for the fact that Agorakit will probably always be served as html and not as an spa.
- I'd love to reduce fat in controllers and models so something is definitely needed. Also some parts are duplicated multiple times. I tried traits, some services, but it's not enough at all.
- I don't really see the interest of unit testing, but I can be convinced (probably easily) otherwise. I just love end to end testing's ease of creation and usually ease of interpretation. We should aim 100% coverage of "usual user activities". Mocking stuff to test smaller parts of the code seems a waste as long as we don't have better end to end testing. We are not doing banking stuff with complex calculations. (I might be very wrong)
- It would be nice to have clear directory structure someone immediately understands by looking at the repository. Not too many levels if possible. Let's say we plan a 10 times increase of code / features. By simply adding one or two level of directories we can keep it humanly manageable when looking at a list of files. Something like : create a subdirectory if more than 10 files will be there, each class should not contain more that let's say 10 functions with each having not more than 3O lines of code.
- On the contrary I'd like also to avoid having a lot of very small classes that do nothing but glue parts together and slow down understanding of the whole (that is, only if this "glue" is probably the only type we'll use, like the case of datastore adapter I mentioned). For example, controllers with only one method seems counterintuitive.
- just to be sure : it's a bad idea to introduce a plugin architecture, right?
- I guess some of those ideas are short sighted, as I said I hope to grasp more of all this. I shall not slow down the project, and want to learn.
Agreed. I am firmly on "team hypermedia" and want to stick to "native Web" methodologies to the extent possible. I'd honestly prefer to avoid JSON APIs entirely, but I won't die on that hill if we need them. I think there is tremendous overlooked value in hypermedia, but I'm not a purist.
I think these are overlapping concerns. While DDD is a *little* esoteric at first, it could dramatically improve the ability of a new contributor to onboard in exchange for a slightly steeper "front edge" of the learning curve. In general, I've found that sort of trade off worth it. Folks have the most energy for learning at the start because they have clear & specific motivation. And I do think there is value in contributors learning a bit along the way — I hope there is always educational value in participating in open source.
Yes, this is (often) "boilerplate" stuff that PHP is fairly good at avoiding and I certainly don't want to add a bunch of layers of OOO abstractions just to make it "technically correct". This is functional software, not an art project to prove abstraction skills. 🙃
It's not a "bad" idea, but leaning on it heavily produces complexity and makes updates a bit of a nightmare. I'm currently watching Flarum struggle to reach 2.0 and it appears bogged down, in part, by the size of the plugin ecosystem with independent authors who haven't updated their plugins yet. This is the blessing & curse of plugins: it adds WIDE amounts of functionality you couldn't produce yourself, but also a tremendous loss of focus waiting on features to update that may not truly serve your goals and lots of duplicate features that now you need to support multiple versions of. My preference, today, is to stay away from plugins and revisit the idea if there is a clear & pressing need. Once you allow plugins, there's no real "undo" without a lot of bad feelings.
I'd also rather spend effort thinking about how to allow for theming options if we're going to look at customizations.
I will admit I am lightly in favor of considering PostgreSQL support / compatibility because they have a better governance model than MySQL, MariaDB, or Percona which are all tightly coupled to corporations. Looking into my crystal ball, I think PostgreSQL has a brighter future. I do not have a sense of how challenging that would be yet, so I don't have a strong opinion either way today. In theory, they are ~95% cross-compatible anyway in a simpler app like Agorakit today. My naive hope is it's pretty painless.
Beyond that, I'd agree we're not trying to support a NoSQL engine or anything like that, at least not for the core data you're referencing.
I think there is value in unit tests, not least because it forces you to write better code by changing your perspective, but agree we'll get faster value from E2E tests (& integration tests) as a higher priority. One challenge is that E2E tests naturally consume far more resources, so you start to hit a point of diminishing returns much faster — by the 100th E2E test you may find the CI pipeline is taking so long it's slowing down other work.
I also think that static analysis serves a very complimentary purpose as unit tests, so that's a good focus for now.