add nodes

[kwladyka]

It will be so great to have nodes like in jekins here. So for example i can run some commands on another machine on different system as a part of CD.

[flosell]

This has been mentioned offline multiple times and @sroidl and me actually brainstormed multiple different approaches to this. Unfortunately, none of the were simple extensions but required deeper design changes so we haven't started actively going in this direction yet.

Also, it gets trickier no to violate one of the basic principles LambdaCD is designed with: do one thing and do it well. LambdaCD is a tool to orchestrate parts of a delivery pipeline, not a clustered job scheduler (there are other tools out there that specialise in this).

So if you have a method to execute commands on another machine (this can be as complex as a Mesos or Kubernetes cluster and as trivial as an SSH daemon on some agent hosts), he most straightforward extension I could think of would be to just wrap this into a a step that feels similar to shell/bash:

(ns lambdacd-remote-ssh.core)

(defn bash [ctx cwd & cmds]
  (let [host (select-host ctx)]
    (scp-working-directory ctx cwd host)
    (execute-via-ssh ctx cmds)))

(defn step-that-does-stuff-remotely [args ctx]
  (remote-ssh/bash ctx (:cwd args) "./do-stuff.sh"))

[kwladyka]

I don't think it is against principle. For example you want to have tests on OS X and windows. Build the same app but on different system, because it is needed. But you have to have release new version in the same time. So then nodes are important. Not to do things parallel, but to do things in the right way. Also if you manage many services etc. you want work with them on different environment. You can build multiple lambdacd project, but then it lose simplicity. But maybe i am wrong.

Another example: you have complex backend, IOS and OS X application. After each change any of this you want to run tests all of them to be sure nothing break between them like in API. Like versions are compatible. But each system works on different environment. Lambdacd is not designed to do such things now.

So on the end in our company we don't use lambdacd. It is powerful and i love idea doing this in Clojure code, but still not enough futures to use it with real complex issues like i described before. Anyway not my choice :)

I would just wish myself to use CD in Clojure code without GUI for settings instead of things like jenkins etc.

[flosell]

I don't disagree with you that this is an important feature to have. What I meant to say was, I'm not quite sure yet which parts of this feature should be implemented in the core of LambdaCD and where it makes sense to delegate to a different (pluggable) tool that's purpose-built for this (e.g. a cluster manager).

Apart from this, what about the approach I outlined, support to wrap scp and ssh (or other remoting approaches e.g. for Windows) to run commands on a remote machine? Is this something that makes sense?

[kwladyka]

Like lambdacd project connect with nodes which are only remote ssh/scp ends?

It will be ok, but much better will be to use it with lambdacd modules for git etc. So i can run git clone repo with some options to choose branch or tag directly in SSH, but if we already have this future in lambdacd to not do it in this way :)

Maybe code should looks like now, but with wrap like:

(node :osx ...)

Where :osx is label and can mean one of node marked as :osx.

[flosell]

Thanks for the feedback! I tried combining it with my thoughts to write up motivations and different approaches in #168. Don't rely on this being implemented right away but at least now there's a central point to refer to and discuss different approaches.

Writing it, I came to think that maybe the main challenge isn't node management or connectivity but distributing the code: Since LambdaCD pipelines can be arbitrary code with arbitrary dependencies, we'll have to have some way push all this to the remote agent.

Anyway, I'm closing this issue in favour of #168.