\input texinfo
@setfilename cuirass.info
@documentencoding UTF-8
@include version.texi
@settitle Cuirass Reference Manual
@setchapternewpage odd

@copying

This manual is for Cuirass version @value{VERSION}, a build automation
server.

Copyright @copyright{} 2016, 2017 Mathieu Lirzin@*
Copyright @copyright{} 2017 Mathieu Othacehe

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@end quotation
@end copying

@dircategory Software development
@direntry
* Cuirass: (cuirass).       Build automation server.
@end direntry

@titlepage
@title Cuirass Reference Manual
@subtitle Build automation server
@subtitle for version @value{VERSION}, @value{UPDATED}
@author by Mathieu Lirzin

@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top Cuirass
@insertcopying
@end ifnottex

@c *********************************************************************
@menu
* Introduction::                What is Cuirass about?

Tutorial sections:
* Overview::                    A quick tour of Cuirass

Reference sections:
* Invocation::                  How to run Cuirass.
* Database::                    About the database schema.
* Web API::                     Description of the Web API.

* Contributing::                Your help needed!
* GNU Free Documentation License::  The license of this manual.
* Concept Index::               Concepts.

@end menu

@c *********************************************************************
@node Introduction
@unnumbered Introduction
@cindex introduction

@dfn{Cuirass} is a general-purpose build automation server that checks
out source files from @acronym{VCS, Version Control System}
repositories, executes a @dfn{build job}, and notifies the results of
that job.  A build job consists of a combination of tasks such as
compiling source code to binary code and running automated tests.
Notification is achieved by using a database that stores the build
results associated with an HTTP server that provides a convenient way to
access them.

Cuirass is inspired by the @url{https://nixos.org/hydra/, Hydra}
continuous build system. Unlike Hydra, it is built on top of the
@url{https://www.gnu.org/software/guix/, GNU Guix} functional package
manager.

The goal of Cuirass is to provide both on-demand, scheduled, and
triggered builds.  A Build server is an important tool in the software
development process, because it allows modifying the source code while
ensuring the portability and robustness of those changes.  It is the
basis of the @dfn{Continuous integration} practice.

@menu
* Continuous Integration::      A Software development practice
@end menu

@c *********************************************************************
@node Continuous Integration
@unnumberedsec Continuous Integration

@c *********************************************************************
@node Overview
@chapter Overview

@command{cuirass} acts as a daemon polling @acronym{VCS, version control
system} repositories for changes, and evaluating a derivation when
something has changed (@pxref{Derivations, Derivations,, guix, Guix}).
As a final step the derivation is realized and the result of that build
allows you to know if the job succeeded or not.

What is actually done by @command{cuirass} is specified in a @dfn{job
specification} which is represented as an association list which is a
basic and traditional Scheme data structure.  Here is an example of what
a specification might look like:

@lisp
 `((#:name . "hello")
   (#:url . "git://git.savannah.gnu.org/guix.git")
   (#:branch . "master")
   (#:no-compile? . #t)
   (#:load-path . ".")
   (#:proc . cuirass-jobs)
   (#:file . "/tmp/drv-file.scm")
   (#:arguments (subset . "hello")))
@end lisp

In this specification the keys are Scheme keywords which have the nice
property of being self evaluating.  This means that they can't refer to
another value like symbols do.

@quotation Note
@c This refers to
@c <https://github.com/libgit2/libgit2sharp/issues/1094#issuecomment-112306072>.
Currently Cuirass only supports Git repositories, and only over the
@code{git} and ``smart'' HTTP(S) transports (Git's so-called ``dumb
HTTP'' transport, where the HTTP server does not know about Git, is not
supported.)
@end quotation

Currently the only way to add those specifications to cuirass is to put
a list of them in a file and set the @code{--specifications} command
line option argument with the file name when launching the daemon
(@pxref{Invocation}).  The specifications are persistent (they are kept
in a SQLite database) so the next time @command{cuirass} is run the
previously added specifications will remain active even if you don't
keep the @code{--specifications} option.

@c *********************************************************************
@node Invocation
@chapter Invoking cuirass
@cindex invoking cuirass
@cindex cuirass invocation
@cindex options for invoking cuirass

The usual way to invoke @code{cuirass} is as follows:

@example
cuirass --specifications @var{specs}
@end example

Additionally the following options can be used.

@table @code
@item --one-shot
Instead of executing @code{cuirass} as a daemon looping over the jobs.
Only evaluate and build the specifications once.

@item --cache-directory=@var{directory}
@var{directory} is the place where the VCS repositories used by the jobs
are stored.

@item --specifications=@var{specifications-file}
@itemx -S @var{specifications-file}
Add the specifications defined in @var{specifications-file} in the job
database before launching the evaluation and build processes.

@item --database=@var{database}
@itemx -D @var{database}
Use @var{database} as the database containing the jobs and the past
build results. Since @code{cuirass} uses SQLite as a database engine,
@var{database} must be a file name.  If the file doesn't exist, it will
be created.

@item --port=@var{num}
@itemx -p @var{num}
Make the HTTP interface listen on port @var{num}.  Use port 8080 by
default.

@item --interval=@var{n}
@itemx -I @var{n}
Wait @var{n} seconds between each poll.

@item --use-substitutes
This can be useful when you are not interested in building the
dependencies of a particular job.

@item --version
@itemx -V
Display the actual version of @code{cuirass}.

@item --help
@itemx -h
Display an help message that summarize all the options provided.
@end table

@c *********************************************************************
@node Database
@chapter Database schema
@cindex cuirass database
@cindex sqlite database
@cindex persistent configuration

Cuirass uses a SQLite database to store information about jobs and past
build results, but also to coordinate the execution of jobs.

The database contains the following tables: @code{Specifications},
@code{Stamps}, @code{Evaluations}, @code{Derivations}, and
@code{Builds}.  The purpose of each of these tables is explained below.

@section Specifications
@cindex specifications, database

This table stores specifications describing the repository from whence
Cuirass fetches code and the environment in which it will be processed.
Entries in this table must have values for the following text fields:

@table @code
@item repo_name
This field holds the name of the repository.  This field is also the
primary key of this table.

@item url
The URL of the repository.

@item load_path
This text field holds the name of the subdirectory in the checked out
repository that is passed to the @code{evaluate} tool as the Guile load
path.  This directory is interpreted relative to the repository in the
Cuirass cache directory.  This will usually be the current directory
@code{"."}.

@item file
The absolute name of the Scheme file containing PROC.

@item proc
This text field holds the name of the procedure in the Scheme file FILE
that produces a list of jobs.

@item arguments
A list of arguments to be passed to PROC.  This can be used to produce a
different set of jobs using the same PROC.
@end table

The following columns are optional:

@table @code
@item branch
This text field determines which branch of the repository Cuirass should
check out.

@item tag
This text field is an alternative to using BRANCH or REVISION.  It tells
Cuirass to check out the repository at the specified tag.

@item revision
This text field is an alternative to using BRANCH or TAG.  It tells
Cuirass to check out the repository at a particular revision.  In the
case of a git repository this would be a commit hash.

@item no_compile_p
When this integer field holds the value @code{1} Cuirass will skip
compilation for the specified repository.
@end table

@section Stamps
@cindex stamps, database

When a specification is processed, the repository must be downloaded at
a certain revision as specified.  The @code{Stamps} table stores the
current revision for every specification when it is being processed.

The table only has two text columns: @code{specification}, which
references a specification from the @code{Specifications} table via the
field @code{repo_name}, and @code{stamp}, which holds the revision
(e.g. a commit hash).

@section Evaluations
@cindex evaluations, database

An evaluation relates a specification with the revision of the
repository specified therein.  Derivations and builds (see below) each
belong to a specific evaluation.

The @code{Evaluations} table has the following columns:

@table @code
@item id
This is an automatically incrementing numeric identifier.

@item specification
This field holds the @code{repo_name} of a specification from the
@code{Specifications} table.

@item revision
This text field holds the revision string (e.g. a git commit) of the
repository specified in the related specification.
@end table

@section Derivations
@cindex derivations, database

This table associates a tuple of the absolute derivation file name and
evaluation identifier with a job name.

@table @code
@item derivation
This column holds the absolute file name of the Guix derivation that is
supposed to be evaluated for this job.

@item evaluation
This field holds the @code{id} of an evaluation from the
@code{Evaluations} table.

@item job_name
This text field holds the name of the job.

@item system
This text field holds the system name of the derivation.

@item nix_name
This text field holds the name of the derivation ---e.g.,
@code{coreutils-8.24}.

@end table

@section Builds
@cindex builds, database

This table holds records of completed or failed package builds.  Note
that builds are not in a one to one relationship with derivations in
order to keep track of non-deterministic compilations.

@table @code
@item id
This is an automatically incrementing numeric identifier.

@item derivation
This text field holds the absolute name of the derivation file that
resulted in this build.

@item evaluation
This integer field references the evaluation identifier from the
@code{Evaluations} table, indicating to which evaluation this build
belongs.

@item log
This text field holds the absolute file name of the build log file.

@item status
This integer field holds the build status of the derivation.

@item timestamp
This integer field holds a timestamp taken at build creation time.

@item starttime
This integer field holds a timestamp taken at build start time.
Currently, it has the same value as the @code{timestamp} above.

@item stoptime
This integer field holds a timestamp taken at build stop time.
Currently, it has the same value as the @code{timestamp} above.

@end table

@section Outputs
@cindex outputs, database

This table keep tracks for every eventual build outputs. Each build
stored in @code{Builds} table may have zero (if it has failed), one or
multiple outputs.

@table @code
@item build
This field holds the @code{id} of a build from the
@code{Builds} table.

@item name
This text field holds the name of the output.

@item path
This text field holds the path of the output.

@end table

@c *********************************************************************
@node Web API
@chapter Web API
@cindex web api

Cuirass web API is derived from Hydra one, see @url{https://github.com/NixOS/hydra/blob/master/doc/manual/api.xml, Hydra API description}.

For now only a subset of this API is implemented.

@section API description
@cindex description, json

@subsection Build information

It is possible to query Cuirass web server for build informations. The
dedicated API is "/build/@var{build-id}" where @var{build-id} is the
unique id associated to the build in database.

For instance, querying a local Cuirass web server can be done with
@code{curl} and @code{jq} to format the JSON response :

@example
$ curl -s "http://localhost:8080/build/2" | jq

@{
  "id": 2,
  "project": "guix",
  "jobset": "master",
  "job": "acpica-20150410-job",
  "timestamp": 1501347493,
  "starttime": 1501347493,
  "stoptime": 1501347493,
  "buildoutputs": @{
    "out": @{
      "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410"
    @}
  @},
  "system": "x86_64-linux",
  "nixname": "acpica-20150410",
  "buildstatus": 0,
  "busy": 0,
  "priority": 0,
  "finished": 1,
  "buildproducts": null,
  "releasename": null,
  "buildinputs_builds": null
@}
@end example

If requested @var{build-id} is not known, the HTTP code 404 is
answered with a JSON error message. For example :

@example
$ curl -s "http://localhost:8080/build/fff"

@{"error" : "Build with ID fff doesn't exist."@}
@end example

The nominal output is a JSON object whose fields are described
hereafter.

@table @code
@item id
The unique build id.

@item project
The associated specification name, as a string.

@item jobset
The associated specification branch, as a string.

@item job
The associated job-name, as a string.

@item timestamp
Timestamp taken at build creation time.

@item starttime
Timestamp taken at build start time.

@item stoptime
Timestamp taken at build stop time.

@item buildoutputs
Build outputs as a JSON object. The keys names are referring to the
eventual output names. The associated value is another JSON object which
only key is @code{path}. @code{path} value is the output directory in
store as a string.

@item system
System name of the build, as a string.

@item nixname
Derivation name, as a string.

@item buildstatus
Build status, as an integer. Possible values are :

@example
0 -> succeeded
1 -> failed
2 -> failed dependency
3 -> failed other
4 -> cancelled
@end example

@item busy
Whether the build is pending, as an integer (not implemented yet).

@item priority
Build priority, as an integer (not implemented yet).

@item finished
Build finished, as an integer (not implemented yet : always 1).

@item buildproducts
Build products in store as a JSON object (not implemented yet).

@item releasename
Unknown, not implemented yet.

@item buildinputs_builds
Inputs used for the build, as a JSON object (not implemented yet).

@end table

@subsection Build raw log output

It is possible to ask Cuirass for the raw build output log with the API
"/build/@var{build-id}/log/raw" where @var{build-id} is the
unique id associated to the build in database.

The output is a raw text, for example :

@example
$ curl http://localhost:8080/build/2/log/raw

starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
...
@end example

If requested @var{build-id} is not known, the HTTP code 404 is
answered with a JSON error message. For example :

@example
$ curl -s "http://localhost:8080/build/fff/log/raw"

@{"error" : "Build with ID fff doesn't exist."@}
@end example

@subsection Latest builds

The list of latest builds can be obtained with the API
"/api/latestbuilds".  The output is a JSON array of
builds. Builds are represented as in "/build/@var{build-id} API.

This request accepts a mandatory parameter and multiple optional ones.

@table @code
@item nr
Limit query result to nr elements. This parameter is @emph{mandatory}.

@item project
Filter query result to builds with the given @code{project}.

@item jobset
Filter query result to builds with the given @code{jobset}.

@item job
Filter query result to builds with the given @code{job} name.

@item system
Filter query result to builds with the given @code{system}.

@end table

For example, to ask for the ten last builds :

@example
$ curl "http://localhost:8080/api/latestbuilds?nr=10"
@end example

or the five last builds which project is ``guix'' and jobset ``master' :

@example
$ curl "http://localhost:8080/api/latestbuilds?nr=5&project=guix&jobset=master"
@end example

If no builds matching given parameters are found and empty JSON array is returned.

@c *********************************************************************
@node Contributing
@chapter Contributing

Everyone is welcome to contribute to Cuirass.  You can report bugs, send
patches and share your ideas with others by sending emails the
@email{guix-devel@@gnu.org, mailing list}.

Development is done using the Git distributed version control system.
Thus, access to the repository is not strictly necessary.  We welcome
contributions in the form of patches as produced by @code{git
format-patch}.  Please write commit logs in the ChangeLog format
(@pxref{Change Logs,,, standards, GNU Coding Standards}); you can check
the commit history for examples.

When posting a patch to the mailing list, use @samp{[PATCH] @dots{}} as
a subject.  You may use your email client or the @command{git
send-email} command.  We prefer to get patches in plain text messages,
either inline or as MIME attachments.  You are advised to pay attention
if your email client changes anything like line breaks or indentation
which could potentially break the patches.

@c *********************************************************************
@node GNU Free Documentation License
@appendix GNU Free Documentation License
@cindex license, GNU Free Documentation License
@include fdl-1.3.texi

@c *********************************************************************
@node Concept Index
@unnumbered Concept Index
@printindex cp

@bye