[Feature] Add support for different Clojure platforms.

[conjurernix]

I've been thinking for a while on how to add support for different platforms (Clojurescript, cljc or any other platform that Clojure can run now or in the future) to the poly tool, and I (hopefully) came up with a suggestion that will not end up as a breaking change or an infinite amount of maintenance.

The idea is to add an additional :platforms configuration in workspace.edn and a new platform:<insert_platform> named argument.

So a default workspace generated by the poly tool would look like this:

{:top-namespace "test-ws"
  :interface-ns "interface"
  :default-profile-name "default"
  :compact-views #{}
  :vcs {:name "git"
        :auto-add false}
  :tag-patterns {:stable "stable-*"
                 :release "v[0-9]*"}
  :projects {"development" {:alias "dev"}}
  :platforms {:default :clj
                 :clj {:bases-path "bases"
                         :components-path "components"
                         :projects-path "projects"
                         :file-extension "clj"}}}

:default key indicates which "platform" will be used by default when the platform argument is not given in poly tool's commands

:bases-path indicates where base bricks will be created for the given platform. (In the above case there's only one platform ("clj") which will create the brick under the ./bases/ directory.

:components-path indicates where component bricks will be created for the given platform. (In the above case there's only one platform ("clj") which will create the brick under the ./components/ directory.

:projects-path indicates where projects will be created for the given platform. (In the above case there's only one platform ("clj") which will create the brick under the ./projects/ directory.

:file-extension designates the extension of the Clojure source files that are generated within the brick. For the above example it will generate interface.clj and core.clj source files (same for tests).

Thus with the addition of the above :platforms key, compatibilty would not break for the current users of the poly tool.

If one would want to have a full-stack Clojure(Script) polylith workspace that would entail having a mix of .clj, .cljs and .cljc files. With the new feature he would be able to have a workspace.edn that would look like this:

{:top-namespace "test-ws"
  :interface-ns "interface"
  :default-profile-name "default"
  :compact-views #{}
  :vcs {:name "git"
        :auto-add false}
  :tag-patterns {:stable "stable-*"
                 :release "v[0-9]*"}
  :projects {"development" {:alias "dev"}}
  :platforms {:default :clj
             :clj {:bases-path "bases/clj"
                   :components-path "components/clj"
                   :projects-path "projects/clj"
                   :file-extension "clj"}
             :cljs {:bases-path "bases/cljs"
                    :components-path "components/cljs"
                    :projects-path "projects/cljs"
                    :file-extension "cljs"}
             :cljc {:bases-path "bases/cljc"
                    :components-path "components/cljc"
                    :file-extension "cljc"}}}

which would result in a directory tree looking like the following:

├── components
│   ├── clj
│   ├── cljc
│   └── cljs
├── bases
│   ├── clj
│   ├── cljc
│   └── cljs
├── development
│   ├── clj
│   └── cljs
└── projects
    ├── clj
    └── cljs

When using reporting commands like poly info the output would be separated for each "platform" like so:

  Platform: clj

  projects: 2   interfaces: 4
  bases:    0   components: 5

  project      alias  status   dev
  --------------------------   ---
  service      ?1      ---     ---
  development  dev     s--     s--

  interface    brick          ?1    dev
  -------------------------   ---   ---
  database     database1      ---   ---
  test-helper  test-helper1   ---   ---
  user         admin          ---   ---
  user         user1          ---   ---
  util         util1          ---   ---

  Platform: cljs

  projects: 1   interfaces: 2
  bases:    0   components: 2

.
.
.

Additionally the platform named argument could be introduced in reporting/testing commands as well.
For example running poly info platform:clj would only output information regarding bricks of the clj platform, or poly test platform:clj would only run tests of clj bricks.

Furthermore, this could open up to other niceties as well, for example having a :default-project-deps key that would be used like this:

{:top-namespace "test-ws"
  :interface-ns "interface"
  :default-profile-name "default"
  :compact-views #{}
  :vcs {:name "git"
        :auto-add false}
  :tag-patterns {:stable "stable-*"
                 :release "v[0-9]*"}
  :projects {"development" {:alias "dev"}}
  :platforms {:default :clj
             :clj {:bases-path "bases/clj"
                   :components-path "components/clj"
                   :projects-path "projects/clj"
                   :default-project-deps {org.clojure/clojure          {:mvn/version "1.10.1"}
                                          org.clojure/tools.deps.alpha {:mvn/version "0.12.985"}}
                   :file-extension "clj"}
             :cljs {:bases-path "bases/cljs"
                    :components-path "components/cljs"
                    :projects-path "projects/cljs"
                    :file-extension "cljs"}
             :cljc {:bases-path "bases/cljc"
                    :components-path "components/cljc"
                    :file-extension "cljc"}}}

so that when a new project is created for that specific platform, the generated deps.edn would use the :default-project-deps map instead of having it hard-coded within the poly tool's implementation. This is only one of different kind of utilities that can be built upon this suggested feature.

A (partially at the moment) implementation of this can be found in this fork: https://github.com/DrBuchkov/polylith/tree/feature/platforms
Current status:

• Create commands have been adjusted, and passing tests.
• Report commands have yet to be adjusted.
• Some workspace related tests fail.

[tengstrand]

Hi Nikolas!

Thank you for working on this difficult problem. With that said, we lack a clear problem formulation here and it would be nice if you could elaborate a bit more around what problem you are trying to solve before jumping into the implementation.

This is what we came up with at today's meeting:

• Leiningen and shadow-cljs are two popular choices for frontend development, and Polylith is built on top of tools.deps. We can see a mismatch here which makes it hard to reuse functionality in the poly tool, because frontend and backend work so differently.
• We think Clojure backend systems are mostly just code. You can use any editor and run a JVM REPL to work with it. Clojure frontend is a different animal in our opinion and is more complex than the backend code.

Polylith tries to solve a number of problems, like code sharing and to speed up the feedback loop. A lot of that is already addressed by the dynamic nature of frontend development.

For example, Polylith solves sharing by introducing components and interfaces that are used from projects, by combining a set of bricks. The equivalent to Polylith projects in e.g. shadow-cljs are builds.
The benefits of components and interfaces are very small in frontend projects. This is due to the fact that most Clojure frontend projects make use of re-frame and register event handlers, which are not functional interfaces. The same goes for UI components where each UI component usually exposes one single render function.

We looked at the production frontend codebase at Furkan's company and saw:

• 30% of the code is dummy UI components, each has their own namespace with only one function named render exposed publicly. The render function is a reagent component.
• 50% of the code is re-frame event handlers and subscriptions. This code is not used anywhere else apart from the namespace they live in. They are exposed as re-frame events and subscriptions and used indirectly within the re-frame’s subscribe/dispatch mechanism.
• 15% is utility namespaces, usually creating a bridge between external Clojure and JavaScript libraries. This is the only place that we can see the value of having Polylith components since these namespaces usually expose a set of functions that are used from other places.
• The final 5% is plain JavaScript code that is needed and used from the utility namespaces.

As you can see, only 15% of the codebase could partially benefit from Polylith. We also don't have a clear idea about how to fit in the base concept. Shadow-cljs already handles the project concept with it’s builds and modules.

We think that Polylith is not a good fit for a frontend codebase and we believe that solutions should come from a need or a problem that people currently have in their workflow. We are always open for suggestions and willing to discuss them. It would be helpful if you could elaborate more on which problems you are trying to solve.

We think the main problem you are trying to solve is about sharing code between backend and frontend via cljc code.
One solution, that is already in place, is to let backend clj and cljc code live together in a Polylith workspace and to let the frontend code live under another directory (in the same repository) where it can make use of the cljc components by directly including them as code.

The Polylith Team

[conjurernix]

Hello Joakim,

Thanks for putting the effort to read up on and consider my proposal and taking the time to respond.

I have a few notes however. My suggestion is a proposal for allowing the user of the polylith tool to integrate any current or future Clojure implementation in the future, not necessarily ClojureScript although that would be the usual suspect. Nowhere did i ask for the maintainers of the tool to support every single curent or future Clojure implementation, rather to abstract the logic that is specific to Clojure JVM into a configuration that can be hacked by a user if they wanted.

Adding any support for any platform is outside the above proposal, rather suggesting a solution that may allow users to do so if they pleased. What i showed above with the clj/cljs/cljc was merely an example of a possible configuration a user could add to his workspace.edn in order to make his polylith workspace 'polyglot'. In my original post I intentionally didn't mention the word 'frontend' since whether this feature is used to integrate your frontend project, is not this feature's concern, but probably would be the most common usecase, which is covered.

As far as building Clojurescript goes, you can still use tools.deps to do so. In fact in all the Clojurescript projects I've worked on, even ones that have been started by other people and I have joined later, have been using deps.edn to pull cljs dependencies, create the "classpath" and use deps.edn aliases to setup the development environment, and only using shadow-cljs for building and for pulling npm dependencies. Whether someone wants to use shadow-cljs is up to him and can easily do so manually. I never insinuated that the poly tool should handle any tools other than tools.deps.

On the dissection of a Clojurescript project, I feel that this is completely anecdotal. reagent and re-frame are not the only Clojurescript frameworks, and in fact even in the aforementioned (barring re-frame event handlers), reagent components are simply functions that can be put into components, which might have a dependency to another component.
A simple example is having two components of the ui-form interface, which might have a dependency to the ui-input-field interface. Depending which ones you add as extra-deps in your aliases your form will look different. Sharing code between CLJ/CLJS is not even the main concern here, but it is also enabled.
I just don't see why polylth's tooling can't be of use here. You'd still want to check your bricks' dependencies, check your project's library dependencies or run your tests.

In essence, I see a future where you might have a workspace where some services are written in Clojure JVM, some other services are written in Clojerl (for example), a couple of web applications are written in Cljs (for example admin/client applications), and mobile application is written in ClojureDart, where they can all be developed in a single environment and possibly have code shared between them, while still leveraging each of polylith's features for the variety of your projects.

I'm willing to further develop the fork and can maybe discuss further my particular use-case if the Polylith Team sees any value in what I'm saying.

[crimeminister]

I'm interested in using polylith for a shadow-cljs based Node.js project, just as a point of anecdotal interest. I imagine that many of the same advantages accrued from using it with a Java-based project will apply for a project like this as well.

[tengstrand]

The poly tool is already prepared to support multiple languages. If you go to the core namespace of poly-cli and read the comment there, you will see the steps that is performed to read a workspace:

(-> user-input
    workspace-clj/workspace-from-disk
    workspace/enrich-workspace
    change/with-changes)

The user-input is a data structure that holds the user input that is passed in when the poly command is executed.
That data structure is passed into workspace-clj/workspace-from-disk that reads clj/cljc source code from disk
and puts the result in a data structure in memory. The following steps operate on this data structure, including all available commands.

The idea here is that a new language, e.g. Kotlin, can be supported by adding a workspace-kotlin component containing a function workspace-from-disk or similar, which means that all other code in the workspace could be reused, except for the test command which needs to be custom made.
We can't share Kotlin and Clojure code which is the main reason we want to keep them separated in different workspaces (but they can coexist in the same repository if we want).

As I see it, the path forward for you would be to implement a workspace-cljs component and probably a new cli tool for the ClojureScript code (and a separate base and project).

Why I think it’s a good idea to keep each language in its own workspace:

• Your branch can be compatible with the main branch.
• Most of the code in the Polylith workspace can be reused when you create your own cli tool.
• Keep it simpler by having one workspace per language, instead of combining everything into one single workspace.

[alidcast]

The front-end code re-use story is more clear when an app targets different platforms.

E.g. using react-native, react-native-web, and electron to target mobile, web, and desktop respectively, there's more of a need to have separate packages where components are shared.

So being able to using the same deps tool for a monorepo setup in front-end would be useful. Building for these platforms is indeed more complicated, but that can be left to user space.

[alidcast]

For an example codebase take a look at DevHub: https://github.com/devhubapp/devhub

It's a JS app and they use typescript to handle the builds, but you can see how much more logic is split into separate packages to accommodate re-use across platforms.

[colliderwriter]

I haven't seen my use cases mentioned, so here goes:

I've been using Malli models as a single source of truth for both front and back ends in a monorepo. I template component code for both the front and back ends from the models.

• validators for the front end
• widget layout for the front end
• websocket message serdes shared by both the front and back end
• database schema
• database queries

I stole my monorepo layout ideas from metosin's reitit repo: I started out using lein-parent for common dependency management, but switched to tools.deps as soon as I could. I use cljc files as much as possible to make pure functions testable on the JVM for simplicity's sake.

My back end code already has interfaces because I use Duct. I wish Poly would embrace Duct because it not only defines components, it allows you to reason about them ™️ . I recognize that there are philosophical differences between the two, but only two tools I use have ever demanded interfaces around components for abstraction purposes.I wish one could provide tooling over the other.

Would I like tooling to help me keep my garden of EDN clean? I certainly would. There's a lot of cognitive load in the config stack which i would love to outsource to a tool.

What would I like most from Poly? A shadow-cljs project type and clj / cljc / cljs agnosticism.

Thanks for making Poly even though I can't currently use it. It has given much food for thought.

[dmg46664]

I second @colliderwriter's sentiments.
I think the core concepts of interface based components, and then bases & project with dependencies are great! 🙌

However I don't think the poly tool should dictate directory structure beyond that (or maybe not even that), and should essentially be able to work with any subsequent layout. You can't guarantee that bricks can't be fully agnostic or do something strange.

Moving to deps.edn felt freeing from Apache Maven because the latter imposed a ton of convention, just in case, to play nice. One shouldn't have to create src/main/java unless it's needed otherwise the code base just feels bloated. Likewise, poly feels like it is going down the Maven path.

At the moment I can't use poly test because it fails poly check.
In any event I'm using kaocha directly because of the --watch --failfast options which I don't believe poly caters for anyway.
So I only use poly info (with errors) and poly create... and then delete a bunch stuff.

I would much prefer poly to be a set of tools/libraries that can be mixed and matched rather than a monolith approach.

[tengstrand]

What errors do you get when you run poly check?
Can you elaborate on why the current workspace structure isn't working well for you, and what problems it's causing you?

[furkan3ayraktar]

Polylith comes with a default test runner which relies on default Clojure test runner. However, it allows adding custom test runners. polylith-coacha is one of the custom test runners developed by @imrekoszo.

If I remember correctly, @imrekoszo mentioned (probably on Slack) that the mechanisms in koacha that watch changes and run test do not work with this custom test-runner yet.

Polylith is an opinionated software architecture. We chose to use a pre-defined yet simple directory structure in order to create a simple architecture. We claim that it solves many issues that software industry faces today. We are doing that by utilizing a simple concept, a pre-defined directory structure. On top of that, we provide a CLI which can understand that structure and help the developer. You can refer to the documentation to learn more about what Polylith is and more. You can also have a look at Sean's monorepo and Polylith blog series to see some real life benefits.

As Joakim already mentioned, it would be great if you can elaborate more on the issues that you are facing with Polylith. We can discuss more and help you get up to speed with Polylith.

[dmg46664]

@tengstrand Thanks. I already filed a possible bug earlier today #259, before posting the above comment.

@furkan3ayraktar Thanks.

• Kaocha is an opinionated test runner, i.e. it cares a lot about the order of running tests with modes of prioritising the previously failed tests... and doing this at incredibly high speed. As I believe poly is also an opinionated test runner trying to optimise on dependencies and classloading, I doubt these will ever be reconciled (spawning one from the other).
• I've already read all Sean's blog posts, and both of the official books. Polylith makes assumptions about the backend services sharing code, as opposed to different clients & the backend sharing code. The difference in opinion is me pushing to support the latter. I think the flexibility would extend beyond my situation to Colliderwriter's as well (and others perhaps not listed).

[imrekoszo]

A polylith workspace contains many "top-level" deps.edn-s (i.e. polylith projects). poly test finds out which of these are affected, and for each of these: 1/ it assembles a classpath 2/ instructs the configured test runner to run all tests found on that classpath. polylith-kaocha currently slots into point 2 there, a kaocha "process" is started by poly for each of the affected projects in sequence.

In the above workflow, the value added by poly is the set of projects affected (out of all the projects) and their respective test classpaths (which include tests from bricks referenced by the project as well). The main value added by Kaocha (at least in my opinion) is its plugin infrastructure, providing extensibility wrt test types, output etc.

As far as I am aware, a normal kaocha --watch -based workflow assumes a single deps.edn (classpath) for which a process is started (and kept in the foreground indefinitely), which then listens for changes in sources and reruns tests (out of all tests on the classpath) as needed (I'm not sure how it figures out which tests need to be rerun).

poly test's iteration time is longer than kaocha --watch's, given that the former launches (potentially multiple) classloaders while the latter works in a single one (as far as I'm aware) throughout the entire process. Therefore I consider the latter closer in purpose to a development-time REPL where you re-execute tests after making a change.

As far as I'm aware, polylith's dev project includes all sources and tests from the workspace and so it should be possible to create a poly test command that runs all the tests in the workspace in the context of the dev project (confirmation needed @furkan3ayraktar or @tengstrand). It should be possible to craft a command line that executes (most if not) all of the tests along the lines of clojure -Srepro -M:poly test :project project:dev. If your workspace uses polylith-kaocha then this should create a single classloader and hand it to kaocha.

The only thing that's left is telling kaocha to stay alive and run in watch mode. For that one would want to pass pass-through cli args to poly but I don't currently know how to do that.

[tengstrand]

FYI. We have made another proposal where we discuss how to best support ClojureScript, and it should be a good base for adding support for other Clojure dialects in the future.

[furkan3ayraktar]

I'm closing this discussion in favor of #525 where we're continuing the conversation.