# Overview [Radicle]: https://radicle.xyz/ [Git]: https://git-scm.com/ [CI]: https://en.wikipedia.org/wiki/Continuous_integration [Radicle] is a peer-to-peer collaboration system built on top of the [Git] version control system. Radicle has support for integrating with continuous integration ([CI]) systems, using an architecture where a "broker" listens to events about changes to repositories stored in a node, and launching the appropriate "adapter" for each change, according to its configuration. This means each node can opt into running CI for what projects and changes according to the interests of the person whose node it is. * The delegates for a repository might run CI on all patches to make merge decisions with more confidences. * Someone else, who is contributing to a project, might only care about patches they themselves created, and only run CI for those. * A third party might run CI for projects they use, to know if it's OK to deploy to their production systems. Radicle provides its own, very simple "native CI" solution. It's just good enough for the Radicle project to use itself. In addition, there are adapters that integrate with external CI systems. ## Goal of Radicle CI Context: The user is a software developer working on a project that uses Radicle for version control. The project has an automated test suite, and in-repository configuration for how to build the project and run the test suite, in a format suitable for the CI engine being used. In the long run, the goal for CI in Radicle is "anything that makes it easier, more fun, and faster to produce working software", but that's not a concrete goal. At this stage in the development of Radicle CI has two concrete goals: * When I create a patch to propose a change, I am automatically told if the project branch with my committed changes fails to build or pass its test suite. I can also manually check what the status of that process ("CI run") is, and find out what the build log is, to investigate any problems. - This is "build and test the patch branch". * When a project delegate merges my patch, both they and I are automatically told if the merge fails due to a merge conflict, or if, after the merge the project no longer builds or its test suite fails. - This is "build and test the master branch after the merge". This is useful, because sometimes a merged change breaks the build or the test suite, even when there are no merge conflicts. It is not yet clear how notifications will work. ## Components when integrating an external CI system For external CI, the components are: * the Radicle node * the CI broker * an adapter executable for each supported external CI instance * the external CI instance The first three of these run on the same host, but the external CI instance can run anywhere. The adapter talks to the CI instance using whatever protocol the CI instance supports, such as HTTP. ~~~dot digraph "" { radicle_node [label="Radicle node"]; broker [label="CI broker"]; adapter [label="Adapter"]; engine [label="External CI system"]; radicle_node -> broker [label="change event"]; broker -> adapter [label="invoke"]; adapter -> engine [label="run"]; engine -> adapter [label="web hook?"]; adapter -> broker [label="result"]; } ~~~ External CI integration works like this: * a repository known to the node changes - a Git ref is updated - the ref can be a branch, tag, or something else, such as a Radicle COB - the node emits an event describing the change * the CI broker listens to node events - the broker subscribes to node events via the node control socket, which is a Unix domain socket * the CI broker filters events, based on its configuration, and the configuration for the repository involved * for an event that passes its filters, the CI broker spawns the appropriate adapter process - there is a different adapter for each different CI implementation * the broker sends a request object to the adapter as a child process, via the child's stdin, and reads any responses from the child's stdout - the request is JSON - the responses are in the form of JSON Lines: a JSON object per line serialized to not contain newline characters * the adapter communicates with the external CI instance in whatever way is suitable for that instance - this is usually over HTTP - it may involve the CI instance making a web hook request back to the adapter ~~~plantuml @startuml node -> broker : RefsFetched event broker -> adapter : spawn broker -> adapter : send request adapter-> engine : trigger run engine -> worker : start run note over worker : perform the run engine -> adapter : response with run id adapter -> broker : response: run id worker -> engine : run finished engine -> adapter : web hook? adapter -> broker : response: result @enduml ~~~ ## Components in native CI CI support in Radicle consists of several components. For native CI they are: * the Radicle node * the CI broker * the native CI executable These all run on the same host. ~~~dot digraph "" { radicle_node [label="Radicle node"]; broker [label="CI broker"]; native [label="Native CI"]; commands [label="Shell commands \n from .radicle/native.yaml"]; radicle_node -> broker [label="change event"]; broker -> native [label="invoke"]; native -> commands [label="run"]; native -> broker [label="result"]; } ~~~ ~~~plantuml @startuml node -> broker : RefsFetched event broker -> adapter : spawn broker -> adapter : send request adapter -> broker : response: run id note over adapter : perform the run adapter -> broker : response: result @enduml ~~~ # The adapter The adapter process reads the request to perform a CI run on a specific commit in a repository, and responds with the id of the run, then later with the status of the finished run. For native CI, the adapter actually is the CI engine and performs the CI run itself. For external CI, the adapter process does whatever it needs to do to get the external CI engine instance to perform the CI run. If the CI engine calls back via web hook to notify of the run finishing, the adapter process needs to receive the call and process it. External CI engines allow complex pipelines to be written and support a variety of workflows. Different jobs or tasks can be triggered based on different events (e.g. `push`, `patch created`, `patch updated`, etc.), to satisfy different workflow needs. (Only a few of these are actually implemented yet, but the scaffolding to support more is there.) Some examples of real-world use-cases: - trigger "fast" tests on every push to any branch - trigger "relatively fast" tests only when a Patch is created / updated - trigger "full test suite" on every push to the default branch (e.g. `main`) In order to allow developers using Radicle the same flexibility that they are used to on other forges, we want the broker to pass on whatever information it already has from the node events to the adapters, so they can pass it on to external CI systems, as appropriate. # Report generation The CI broker has an SQLite database file for persistent storage of information of the CI runs it triggers. This is used to generate report pages, among other things. The report page generation is done in its own thread, separate from the main thread of the CI broker. This allows the reporting to happens independently of what the main thread is doing. In particular, it means the report generation happens even while the main thread is busy running an adapter. When it comes to per-run logs, the adapter can include a URL to one in the first response message. The URL will be included as a link in the report HTML.