path: root/doc/cuirass.texi

\input texinfo
@setfilename cuirass.info
@documentencoding UTF-8
@include version.texi
@settitle Cuirass Reference Manual
@setchapternewpage odd

@copying

This manual is for Cuirass version @value{VERSION}, a build automation
server.

Copyright @copyright{} 2016, 2017 Mathieu Lirzin@*
Copyright @copyright{} 2017, 2020 Mathieu Othacehe@*
Copyright @copyright{} 2018 Ludovic Courtès@*
Copyright @copyright{} 2018 Clément Lassieur

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@end quotation
@end copying

@dircategory Software development
@direntry
* Cuirass: (cuirass).       Build automation server.
@end direntry

@titlepage
@title Cuirass Reference Manual
@subtitle Build automation server
@subtitle for version @value{VERSION}, @value{UPDATED}
@author by Mathieu Lirzin

@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top Cuirass
@insertcopying
@end ifnottex

@c *********************************************************************
@menu
* Introduction::                What is Cuirass about?

Tutorial sections:
* Overview::                    A quick tour of Cuirass

Reference sections:
* Invocation::                  How to run Cuirass.
* Database::                    About the database schema.
* Web API::                     Description of the Web API.

* Contributing::                Your help needed!
* GNU Free Documentation License::  The license of this manual.
* Concept Index::               Concepts.

@end menu

@c *********************************************************************
@node Introduction
@unnumbered Introduction
@cindex introduction

@dfn{Cuirass} is a general-purpose build automation server that checks
out source files from @acronym{VCS, Version Control System}
repositories, executes a @dfn{build job}, and notifies the results of
that job.  A build job consists of a combination of tasks such as
compiling source code to binary code and running automated tests.
Notification is achieved by using a database that stores the build
results associated with an HTTP server that provides a convenient way to
access them.

Cuirass is inspired by the @url{https://nixos.org/hydra/, Hydra}
continuous build system. Unlike Hydra, it is built on top of the
@url{https://www.gnu.org/software/guix/, GNU Guix} functional package
manager.

The goal of Cuirass is to provide both on-demand, scheduled, and
triggered builds.  A Build server is an important tool in the software
development process, because it allows modifying the source code while
ensuring the portability and robustness of those changes.  It is the
basis of the @dfn{Continuous integration} practice.

@menu
* Continuous Integration::      A Software development practice
@end menu

@c *********************************************************************
@node Continuous Integration
@unnumberedsec Continuous Integration

@c *********************************************************************
@node Overview
@chapter Overview

@command{cuirass} acts as a daemon polling @acronym{VCS, version control
system} repositories (called @dfn{inputs}) for changes, and evaluating a
derivation when an input has changed (@pxref{Derivations, Derivations,, guix,
Guix}).  As a final step the derivation is realized and the result of that
build allows you to know if the job succeeded or not.

What is actually done by @command{cuirass} is specified in a @dfn{job
specification} which is represented as an association list which is a
basic and traditional Scheme data structure.  Here is an example of what
a specification might look like:

@lisp
 '((#:name . "foo-master")
   (#:load-path-inputs . ("guix"))
   (#:package-path-inputs . ("packages"))
   (#:proc-input . "conf")
   (#:proc-file . "drv-list.scm")
   (#:proc . cuirass-jobs)
   (#:proc-args (subset . "foo"))
   (#:inputs . (((#:name . "guix")
                 (#:url . "git://git.savannah.gnu.org/guix.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "conf")
                 (#:url . "git://my-personal-conf.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))
                ((#:name . "packages")
                 (#:url . "git://my-custom-packages.git")
                 (#:load-path . ".")
                 (#:branch . "master")
                 (#:no-compile? . #t))))
   (#:build-outputs .
    (((#:job . "hello*")
      (#:type . "license")
      (#:output . "out")
      (#:path . "share/doc/hello-2.10/COPYING")))))
@end lisp

In this specification the keys are Scheme keywords which have the nice
property of being self evaluating.  This means that they can't refer to
another value like symbols do.

There are three inputs: one tracking the Guix repository, one tracking the
repository containing the @code{proc}, and one tracking the repository
containing the custom packages (see @code{GUIX_PACKAGE_PATH}).
@code{#:load-path-inputs}, @code{#:package-path-inputs} and
@code{#:proc-input} refer to these inputs by their name.

The @code{#:build-outputs} list specifies the files that will be made
available for download, through the Web interface. Here, the
@code{COPYING} file, in the @code{"out"} output, for all jobs whose name
matches @code{"hello*"} regex.

@quotation Note
@c This refers to
@c <https://github.com/libgit2/libgit2sharp/issues/1094#issuecomment-112306072>.
Currently Cuirass only supports Git repositories, and only over the
@code{git} and ``smart'' HTTP(S) transports (Git's so-called ``dumb
HTTP'' transport, where the HTTP server does not know about Git, is not
supported.)
@end quotation

Currently the only way to add those specifications to cuirass is to put
a list of them in a file and set the @code{--specifications} command
line option argument with the file name when launching the daemon
(@pxref{Invocation}).  The specifications are persistent (they are kept
in a SQLite database) so the next time @command{cuirass} is run the
previously added specifications will remain active even if you don't
keep the @code{--specifications} option.

@c *********************************************************************
@node Invocation
@chapter Invoking cuirass
@cindex invoking cuirass
@cindex cuirass invocation
@cindex options for invoking cuirass

The usual way to invoke @code{cuirass} is as follows:

@example
cuirass --specifications @var{specs}
@end example

Additionally the following options can be used.

@table @code
@item --one-shot
Instead of executing @code{cuirass} as a daemon looping over the jobs.
Only evaluate and build the specifications once.

@item --cache-directory=@var{directory}
@var{directory} is the place where the VCS repositories used by the jobs
are stored.

@item --specifications=@var{specifications-file}
@itemx -S @var{specifications-file}
Add the specifications defined in @var{specifications-file} in the job
database before launching the evaluation and build processes.

@item --database=@var{database}
@itemx -D @var{database}
Use @var{database} as the database containing the jobs and the past
build results. Since @code{cuirass} uses SQLite as a database engine,
@var{database} must be a file name.  If the file doesn't exist, it will
be created.

@item --ttl=@var{duration}
Cuirass registers build results as garbage collector (GC) roots, thereby
preventing them from being deleted by the GC.  The @option{--ttl} option
instructs it to keep those GC roots live for at least @var{duration}---e.g.,
@code{1m} for one month, @code{2w} for two weeks, and so on.  The default is
30 days.

Those GC roots are typically stored in
@file{/var/guix/gcroots/profiles/per-user/@var{user}/cuirass}, where @var{user} is the
user under which Cuirass is running.

@item --port=@var{num}
@itemx -p @var{num}
Make the HTTP interface listen on port @var{num}.  Use port 8080 by
default.

@item --listen=@var{host}
Make the HTTP interface listen on network interface for @var{host}.  Use
localhost by default.

@item --interval=@var{n}
@itemx -I @var{n}
Wait @var{n} seconds between each poll.

@item --use-substitutes
This can be useful when you are not interested in building the
dependencies of a particular job.

@item --threads=@var{n}
Use up to @var{n} kernel threads.

@var{n} should be lower than or equal to the number of CPU cores on the
machine.  In general though, having a large @var{n} is not very useful
since the work of Cuirass is primarily I/O-bound---on the contrary,
large values of @var{n} may increase overhead.  The default value should
be appropriate for most cases.

@item --version
@itemx -V
Display the actual version of @code{cuirass}.

@item --help
@itemx -h
Display an help message that summarize all the options provided.
@end table

@c *********************************************************************
@node Database
@chapter Database schema
@cindex cuirass database
@cindex sqlite database
@cindex persistent configuration

Cuirass uses a SQLite database to store information about jobs and past
build results, but also to coordinate the execution of jobs.

The database contains the following tables: @code{Specifications},
@code{Inputs}, @code{Checkouts}, @code{Evaluations}, @code{Builds} and
@code{Outputs}.  The purpose of each of these tables is explained below.

@section Specifications
@cindex specifications, database

This table stores specifications describing the repositories from whence
Cuirass fetches code and the environment in which it will be processed.
Entries in this table must have values for the following text fields:

@table @code
@item name
This field holds the name of the specification.  This field is also the
primary key of this table.

@item load_path_inputs
This field holds a list of input names whose load path is prepended to Guile's
@code{%load-path} when evaluating @code{proc_file}.

@item package_path_inputs
This field holds a list of input names whose load path is prepended to
@code{GUIX_PACKAGE_PATH} when evaluating @code{proc_file}.

@item proc_input
The name of the input containing @code{proc}.

@item proc_file
The path of the Scheme file containing @code{proc}, relative to
@code{proc_input}.

@item proc
This text field holds the name of the procedure in the Scheme file
@code{proc_file} that produces a list of jobs.

@item proc_args
A list of arguments to be passed to @code{proc}.  This can be used to produce
a different set of jobs using the same @code{proc}.
@end table

@section Inputs
@cindex inputs, database

This table stores the data related to the repositories that are periodically
fetched by Cuirass.  Entries in this table must have values for the following
text fields:

@table @code
@item specification
This field holds the name of the specification from the @code{Specifications}
table associated with the input.  Every input belongs to a specification, and
that specification can refer to its inputs.

@item name
This field holds the name of the input.  That name can be used as a key by the
@code{proc} if it needs access to its resulting checkout.

@item url
The URL of the repository.

@item load_path
Used by a specification when it refers to an input's load path.  See
@code{load_path_inputs} and @code{package_path_inputs}.

@end table

The following columns are optional:

@table @code
@item branch
This text field determines which branch of the repository Cuirass should
check out.

@item tag
This text field is an alternative to using @code{branch} or @code{revision}.
It tells Cuirass to check out the repository at the specified tag.

@item revision
This text field is an alternative to using @code{branch} or @code{tag}.  It
tells Cuirass to check out the repository at a particular commit.

@item no_compile_p
When this integer field holds the value @code{1} Cuirass will skip
compilation for the specified repository.
@end table

@section Checkouts
@cindex checkouts, database

When a specification is processed, the repositories must be downloaded at a
certain revision as specified.  The download is called a checkout.  The
@code{Checkouts} table stores the new checkouts for every specification when
it is being processed.

The @code{Checkouts} table has the following columns:

@table @code
@item specification
The specification associated with the checkout.

@item revision
The revision of the checkout. Within the same specification, two checkouts
can't be identical: they can't have the same revision.

@item evaluation
The evaluation that was triggered by the addition of that new checkout.

@item input
The input associated with the checkout.

@item directory
The directory into which the checkout was extracted.
@end table

@section Evaluations
@cindex evaluations, database

An evaluation relates a specification with the revision of the repository
specified therein.  Builds (see below) belong to a specific evaluation.

The @code{Evaluations} table has the following columns:

@table @code
@item id
This is an automatically incrementing numeric identifier.

@item specification
This field holds the @code{name} of a specification from the
@code{Specifications} table.

@item commits
This text field holds the revisions (space separated commit hashes) of the
repositories specified as inputs of the related specification.
@end table

@section Builds
@cindex builds, database

This table holds records of the derivations and their build status.  Note that
a job will be registered here only if its derivation doesn't already exist.

@table @code
@item derivation
This text field holds the absolute name of the derivation file that
resulted in this build.

@item evaluation
This integer field references the evaluation identifier from the
@code{Evaluations} table, indicating to which evaluation this build
belongs.

@item job_name
This text field holds the name of the job.

@item system
This text field holds the system name of the derivation.

@item nix_name
This text field holds the name of the derivation ---e.g.,
@code{coreutils-8.24}.

@item log
This text field holds the absolute file name of the build log file.

@item status
This integer field holds the build status of the derivation.

@item timestamp
This integer field holds a timestamp taken at build creation time.

@item starttime
This integer field holds a timestamp taken at build start time.
Currently, it has the same value as the @code{timestamp} above.

@item stoptime
This integer field holds a timestamp taken at build stop time.
Currently, it has the same value as the @code{timestamp} above.

@end table

@section Outputs
@cindex outputs, database

This table keep tracks for every eventual build outputs. Each build
stored in @code{Builds} table may have zero (if it has failed), one or
multiple outputs.

@table @code
@item derivation
This field holds the @code{derivation} of a build from the @code{Builds}
table.

@item name
This text field holds the name of the output.

@item path
This text field holds the path of the output.

@end table

@c *********************************************************************
@node Web API
@chapter Web API
@cindex web api

Cuirass web API is derived from Hydra one, see @url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API description}.

For now only a subset of this API is implemented.

@section API description
@cindex description, json

@subsection Build information

It is possible to query Cuirass web server for build informations. The
dedicated API is "/build/@var{build-id}" where @var{build-id} is the
unique id associated to the build in database.

The build information can also be queried by output. For example,
@samp{/output/kg9mirg6xbvzcp0a98v7326n1nvvwgsj-hello-2.10} will return
the details of the output, along with the build if available.

For instance, querying a local Cuirass web server can be done with
@code{curl} and @code{jq} to format the JSON response :

@example
$ curl -s "http://localhost:8080/build/2" | jq

@{
  "id": 2,
  "jobset": "guix",
  "job": "acpica-20150410-job",
  "timestamp": 1501347493,
  "starttime": 1501347493,
  "stoptime": 1501347493,
  "buildoutputs": @{
    "out": @{
      "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410"
    @}
  @},
  "system": "x86_64-linux",
  "nixname": "acpica-20150410",
  "buildstatus": 0,
  "busy": 0,
  "priority": 0,
  "finished": 1,
  "buildproducts": null,
  "releasename": null,
  "buildinputs_builds": null
@}
@end example

If requested @var{build-id} is not known, the HTTP code 404 is
answered with a JSON error message. For example:

@example
$ curl -s "http://localhost:8080/build/fff"

@{"error" : "Build with ID fff doesn't exist."@}
@end example

The nominal output is a JSON object whose fields are described
hereafter.

@table @code
@item id
The unique build id.

@item jobset
The associated specification name, as a string.

@item job
The associated job-name, as a string.

@item timestamp
Timestamp taken at build creation time.

@item starttime
Timestamp taken at build start time.

@item stoptime
Timestamp taken at build stop time.

@item buildoutputs
Build outputs as a JSON object. The keys names are referring to the
eventual output names. The associated value is another JSON object which
only key is @code{path}. @code{path} value is the output directory in
store as a string.

@item system
System name of the build, as a string.

@item nixname
Derivation name, as a string.

@item buildstatus
Build status, as an integer. Possible values are :

@example
0 -> succeeded
1 -> failed
2 -> failed dependency
3 -> failed other
4 -> cancelled
@end example

@item busy
Whether the build is pending, as an integer (not implemented yet).

@item priority
Build priority, as an integer (not implemented yet).

@item finished
Build finished, as an integer (not implemented yet : always 1).

@item buildproducts
Build products in store as a JSON object (not implemented yet).

@item releasename
Unknown, not implemented yet.

@item buildinputs_builds
Inputs used for the build, as a JSON object (not implemented yet).

@end table

@subsection Build raw log output

It is possible to ask Cuirass for the raw build output log with the API
"/build/@var{build-id}/log/raw" where @var{build-id} is the
unique id associated to the build in database.

The output is a raw text, for example:

@example
$ curl http://localhost:8080/build/2/log/raw

starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
...
@end example

If requested @var{build-id} is not known, the HTTP code 404 is
answered with a JSON error message. For example:

@example
$ curl -s "http://localhost:8080/build/fff/log/raw"

@{"error" : "Build with ID fff doesn't exist."@}
@end example

@subsection Latest builds

The list of latest builds can be obtained with the API
"/api/latestbuilds".  The output is a JSON array of
builds. Builds are represented as in "/build/@var{build-id} API.

This request accepts a mandatory parameter and multiple optional ones.

@table @code
@item nr
Limit query result to nr elements. This parameter is @emph{mandatory}.

@item jobset
Filter query result to builds with the given @code{jobset}.

@item job
Filter query result to builds with the given @code{job} name.

@item system
Filter query result to builds with the given @code{system}.

@end table

For example, to ask for the ten last builds:

@example
$ curl "http://localhost:8080/api/latestbuilds?nr=10"
@end example

or the five last builds where jobset ``guix'':

@example
$ curl "http://localhost:8080/api/latestbuilds?nr=5&jobset=guix"
@end example

If no builds matching given parameters are found, an empty JSON array is
returned.

@c *********************************************************************
@node Contributing
@chapter Contributing

Everyone is welcome to contribute to Cuirass.  You can report bugs, send
patches and share your ideas with others by sending emails the
@email{guix-devel@@gnu.org, mailing list}.

Development is done using the Git distributed version control system.
Thus, access to the repository is not strictly necessary.  We welcome
contributions in the form of patches as produced by @code{git
format-patch}.  Please write commit logs in the ChangeLog format
(@pxref{Change Logs,,, standards, GNU Coding Standards}); you can check
the commit history for examples.

When posting a patch to the mailing list, use @samp{[PATCH] @dots{}} as
a subject.  You may use your email client or the @command{git
send-email} command.  We prefer to get patches in plain text messages,
either inline or as MIME attachments.  You are advised to pay attention
if your email client changes anything like line breaks or indentation
which could potentially break the patches.

@c *********************************************************************
@node GNU Free Documentation License
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License
@include fdl-1.3.texi

@c *********************************************************************
@node Concept Index
@unnumbered Concept Index
@printindex cp

@bye