     [file version.texi not included]

Cuirass
*******

This manual is for Cuirass version VERSION, a build automation server.

Copyright © 2016, 2017 Mathieu Lirzin

Copyright © 2017 Mathieu Othacehe

Copyright © 2018 Ludovic Courtès

Copyright © 2018 Clément Lassieur

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

Introduction
************

Cuirass is a general-purpose build automation server that checks out source files from VCS (Version Control System) repositories, executes a build job, and notifies the results of that job.  A build job consists of a combination of tasks such as compiling source code to binary code and running automated tests. Notification is achieved by using a database that stores the build results associated with an HTTP server that provides a convenient way to access them.

Cuirass is inspired by the Hydra (https://nixos.org/hydra/) continuous build system. Unlike Hydra, it is built on top of the GNU Guix (https://www.gnu.org/software/guix/) functional package manager.

The goal of Cuirass is to provide both on-demand, scheduled, and triggered builds.  A Build server is an important tool in the software development process, because it allows modifying the source code while ensuring the portability and robustness of those changes.  It is the basis of the Continuous integration practice.

Continuous Integration
======================

Overview
********

cuirass acts as a daemon polling VCS (version control system) repositories for changes, and evaluating a derivation when something has changed (see Derivations (Guix)). As a final step the derivation is realized and the result of that build allows you to know if the job succeeded or not.

What is actually done by cuirass is specified in a job specification which is represented as an association list which is a basic and traditional Scheme data structure.  Here is an example of what a specification might look like:

      `((#:name . "hello")
        (#:url . "git://git.savannah.gnu.org/guix.git")
        (#:branch . "master")
        (#:no-compile? . #t)
        (#:load-path . ".")
        (#:proc . cuirass-jobs)
        (#:file . "/tmp/drv-file.scm")
        (#:arguments (subset . "hello")))

In this specification the keys are Scheme keywords which have the nice property of being self evaluating.  This means that they can't refer to another value like symbols do.

     Currently Cuirass only supports Git repositories, and only over the git and ``smart'' HTTP(S) transports (Git's so-called ``dumb HTTP'' transport, where the HTTP server does not know about Git, is not supported.)

Currently the only way to add those specifications to cuirass is to put a list of them in a file and set the --specifications command line option argument with the file name when launching the daemon (see Invocation).  The specifications are persistent (they are kept in a SQLite database) so the next time cuirass is run the previously added specifications will remain active even if you don't keep the --specifications option.

Invoking cuirass
****************

The usual way to invoke cuirass is as follows:

     cuirass --specifications SPECS

Additionally the following options can be used.

--one-shot
     Instead of executing cuirass as a daemon looping over the jobs. Only evaluate and build the specifications once.

--cache-directory=DIRECTORY
     DIRECTORY is the place where the VCS repositories used by the jobs are stored.

--specifications=SPECIFICATIONS-FILE
-S SPECIFICATIONS-FILE
     Add the specifications defined in SPECIFICATIONS-FILE in the job database before launching the evaluation and build processes.

--database=DATABASE
-D DATABASE
     Use DATABASE as the database containing the jobs and the past build results. Since cuirass uses SQLite as a database engine, DATABASE must be a file name.  If the file doesn't exist, it will be created.

--port=NUM
-p NUM
     Make the HTTP interface listen on port NUM.  Use port 8080 by default.

--listen=HOST
     Make the HTTP interface listen on network interface for HOST.  Use localhost by default.

--interval=N
-I N
     Wait N seconds between each poll.

--use-substitutes
     This can be useful when you are not interested in building the dependencies of a particular job.

--threads=N
     Use up to N kernel threads.

     N should be lower than or equal to the number of CPU cores on the machine.  In general though, having a large N is not very useful since the work of Cuirass is primarily I/O-bound---on the contrary, large values of N may increase overhead.  The default value should be appropriate for most cases.

--version
-V
     Display the actual version of cuirass.

--help
-h
     Display an help message that summarize all the options provided.

Database schema
***************

Cuirass uses a SQLite database to store information about jobs and past build results, but also to coordinate the execution of jobs.

The database contains the following tables: Specifications, Stamps, Evaluations, Derivations, and Builds.  The purpose of each of these tables is explained below.

Specifications
==============

This table stores specifications describing the repository from whence Cuirass fetches code and the environment in which it will be processed. Entries in this table must have values for the following text fields:

repo_name
     This field holds the name of the repository.  This field is also the primary key of this table.  Although this field is called repo_name in the database, it's called name in the specification itself.

url
     The URL of the repository.

load_path
     This field holds a colon-separated list of directories that are prepended to the Guile load path when evaluating file (see below.)

     Each entry that is not an absolute file name is interpreted relative to the source code checkout.  Often, load_path has just one entry, ".".

     When load_path is empty, the load path is left unchanged.

file
     The absolute name of the Scheme file containing PROC.

proc
     This text field holds the name of the procedure in the Scheme file FILE that produces a list of jobs.

arguments
     A list of arguments to be passed to PROC.  This can be used to produce a different set of jobs using the same PROC.

The following columns are optional:

branch
     This text field determines which branch of the repository Cuirass should check out.

tag
     This text field is an alternative to using BRANCH or REVISION.  It tells Cuirass to check out the repository at the specified tag.

revision
     This text field is an alternative to using BRANCH or TAG.  It tells Cuirass to check out the repository at a particular revision.  In the case of a git repository this would be a commit hash.

no_compile_p
     When this integer field holds the value 1 Cuirass will skip compilation for the specified repository.

Stamps
======

When a specification is processed, the repository must be downloaded at a certain revision as specified.  The Stamps table stores the current revision for every specification when it is being processed.

The table only has two text columns: specification, which references a specification from the Specifications table via the field repo_name, and stamp, which holds the revision (e.g. a commit hash).

Evaluations
===========

An evaluation relates a specification with the revision of the repository specified therein.  Derivations and builds (see below) each belong to a specific evaluation.

The Evaluations table has the following columns:

id
     This is an automatically incrementing numeric identifier.

specification
     This field holds the repo_name of a specification from the Specifications table.

revision
     This text field holds the revision string (e.g. a git commit) of the repository specified in the related specification.

Derivations
===========

This table associates a tuple of the absolute derivation file name and evaluation identifier with a job name.

derivation
     This column holds the absolute file name of the Guix derivation that is supposed to be evaluated for this job.

evaluation
     This field holds the id of an evaluation from the Evaluations table.

job_name
     This text field holds the name of the job.

system
     This text field holds the system name of the derivation.

nix_name
     This text field holds the name of the derivation ---e.g., coreutils-8.24.

Builds
======

This table holds records of completed or failed package builds.  Note that builds are not in a one to one relationship with derivations in order to keep track of non-deterministic compilations.

id
     This is an automatically incrementing numeric identifier.

derivation
     This text field holds the absolute name of the derivation file that resulted in this build.

evaluation
     This integer field references the evaluation identifier from the Evaluations table, indicating to which evaluation this build belongs.

log
     This text field holds the absolute file name of the build log file.

status
     This integer field holds the build status of the derivation.

timestamp
     This integer field holds a timestamp taken at build creation time.

starttime
     This integer field holds a timestamp taken at build start time. Currently, it has the same value as the timestamp above.

stoptime
     This integer field holds a timestamp taken at build stop time. Currently, it has the same value as the timestamp above.

Outputs
=======

This table keep tracks for every eventual build outputs. Each build stored in Builds table may have zero (if it has failed), one or multiple outputs.

build
     This field holds the id of a build from the Builds table.

name
     This text field holds the name of the output.

path
     This text field holds the path of the output.

Web API
*******

Cuirass web API is derived from Hydra one, see Hydra API description (https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml).

For now only a subset of this API is implemented.

API description
===============

Build information
-----------------

It is possible to query Cuirass web server for build informations. The dedicated API is "/build/BUILD-ID" where BUILD-ID is the unique id associated to the build in database.

For instance, querying a local Cuirass web server can be done with curl and jq to format the JSON response :

     $ curl -s "http://localhost:8080/build/2" | jq

     {
       "id": 2,
       "jobset": "guix",
       "job": "acpica-20150410-job",
       "timestamp": 1501347493,
       "starttime": 1501347493,
       "stoptime": 1501347493,
       "buildoutputs": {
         "out": {
           "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410"
         }
       },
       "system": "x86_64-linux",
       "nixname": "acpica-20150410",
       "buildstatus": 0,
       "busy": 0,
       "priority": 0,
       "finished": 1,
       "buildproducts": null,
       "releasename": null,
       "buildinputs_builds": null
     }

If requested BUILD-ID is not known, the HTTP code 404 is answered with a JSON error message. For example:

     $ curl -s "http://localhost:8080/build/fff"

     {"error" : "Build with ID fff doesn't exist."}

The nominal output is a JSON object whose fields are described hereafter.

id
     The unique build id.

jobset
     The associated specification name, as a string.

job
     The associated job-name, as a string.

timestamp
     Timestamp taken at build creation time.

starttime
     Timestamp taken at build start time.

stoptime
     Timestamp taken at build stop time.

buildoutputs
     Build outputs as a JSON object. The keys names are referring to the eventual output names. The associated value is another JSON object which only key is path. path value is the output directory in store as a string.

system
     System name of the build, as a string.

nixname
     Derivation name, as a string.

buildstatus
     Build status, as an integer. Possible values are :

          0 -> succeeded
          1 -> failed
          2 -> failed dependency
          3 -> failed other
          4 -> cancelled

busy
     Whether the build is pending, as an integer (not implemented yet).

priority
     Build priority, as an integer (not implemented yet).

finished
     Build finished, as an integer (not implemented yet : always 1).

buildproducts
     Build products in store as a JSON object (not implemented yet).

releasename
     Unknown, not implemented yet.

buildinputs_builds
     Inputs used for the build, as a JSON object (not implemented yet).

Build raw log output
--------------------

It is possible to ask Cuirass for the raw build output log with the API "/build/BUILD-ID/log/raw" where BUILD-ID is the unique id associated to the build in database.

The output is a raw text, for example:

     $ curl http://localhost:8080/build/2/log/raw

     starting phase `set-SOURCE-DATE-EPOCH'
     phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
     starting phase `set-paths'
     ...

If requested BUILD-ID is not known, the HTTP code 404 is answered with a JSON error message. For example:

     $ curl -s "http://localhost:8080/build/fff/log/raw"

     {"error" : "Build with ID fff doesn't exist."}

Latest builds
-------------

The list of latest builds can be obtained with the API "/api/latestbuilds".  The output is a JSON array of builds. Builds are represented as in "/build/BUILD-ID API.

This request accepts a mandatory parameter and multiple optional ones.

nr
     Limit query result to nr elements. This parameter is mandatory.

jobset
     Filter query result to builds with the given jobset.

job
     Filter query result to builds with the given job name.

system
     Filter query result to builds with the given system.

For example, to ask for the ten last builds:

     $ curl "http://localhost:8080/api/latestbuilds?nr=10"

or the five last builds where jobset ``guix'':

     $ curl "http://localhost:8080/api/latestbuilds?nr=5&jobset=guix"

If no builds matching given parameters are found, an empty JSON array is returned.

Contributing
************

Everyone is welcome to contribute to Cuirass.  You can report bugs, send patches and share your ideas with others by sending emails the mailing list.

Development is done using the Git distributed version control system. Thus, access to the repository is not strictly necessary.  We welcome contributions in the form of patches as produced by git format-patch.  Please write commit logs in the ChangeLog format (see Change Logs (GNU Coding Standards)); you can check the commit history for examples.

When posting a patch to the mailing list, use [PATCH] ... as a subject.  You may use your email client or the git send-email command.  We prefer to get patches in plain text messages, either inline or as MIME attachments.  You are advised to pay attention if your email client changes anything like line breaks or indentation which could potentially break the patches.

GNU Free Documentation License
******************************

     [file fdl-1.3.texi not included]