Add support for multiple inputs.

* Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql.
* bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that
was used afterwards as %GUIX-PACKAGE-PATH.
* bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path,
spec-package-path, format-checkouts): New procedures.
(%not-colon): Remove variable.
(main): Take the load path, package path and PROC from the checkouts that
result from the inputs.  Format the checkouts before sending them to the
procedure.  Remove the LOAD-PATH argument.
* doc/cuirass.texi (Overview, Database schema): Document the changes.
* examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm,
hello-subset.scm, random.scm}: Adapt to the new specification format.
* examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT.
(package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle
the new format of its return value.
* examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT.
Rename the checkout from 'random (which is a specification) to 'cuirass (which
is a checkout resulting from an input).
* src/cuirass/base.scm (fetch-repository): Rename to fetch-input.  Rename SPEC
to INPUT.  Return a checkout object instead of returning two values.
(evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of
SOURCE.  Remove TOKENIZE and LOAD-PATH.  Pass the CHECKOUTS instead of the
SOURCE to "evaluate".  Remove %GUIX-PACKAGE-PATH.  Build the EVAL object
instead of getting it from "evaluate".
(compile?, fetch-inputs, compile-checkouts): New procedures.
(process-specs): Fetch all inputs instead of only fetching one repository.
The result of that fetching operation is a list of CHECKOUTS whose COMMITS are
used as a STAMP.
(%guix-package-path, set-guix-package-path): Remove them.
* src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures.
(db-add-specification, db-get-specifications): Adapt to the new specification
format.  Add/get all inputs as well.
(db-add-evaluation): Rename REVISION to COMMITS.  Store COMMITS as space
separated commit hashes.
(db-get-builds): Rename REPO_NAME to NAME.
(db-get-stamp): Rename COMMIT to STAMP.  Return #f when there is no STAMP.
(db-add-stamp): Rename COMMIT to STAMP.  Deal with DB-GET-STAMP's new return
value.
(db-get-evaluations): Rename REVISION to COMMITS.  Tokenize COMMITS.
* src/cuirass/utils.scm (%non-blocking): Export it.
* src/schema.sql (Inputs): New table that refers to the Specifications table.
(Specifications): Move input related fields to the Inputs table.  Rename
REPO_NAME to NAME.  Rename ARGUMENTS to PROC_ARGS.  Rename FILE to PROC_FILE.
Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to
the Inputs table.
(Stamps): Rename REPO_NAME to NAME.
(Evaluations): Rename REPO_NAME to NAME.  Rename REVISION to COMMITS.
(Specifications_index): Replace with Inputs_index.
* src/sql/upgrade-1.sql: New file.
* tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to
the new specifications format.  Rename REVISION to COMMITS.
* tests/http.scm (evaluations-query-result, fill-db): Idem.

1 files changed, 92 insertions, 52 deletions

diff --git a/doc/cuirass.texi b/doc/cuirass.texi
index 4dbb723..6cf82ba 100644
--- a/doc/cuirass.texi
+++ b/doc/cuirass.texi
@@ -105,10 +105,10 @@ basis of the @dfn{Continuous integration} practice.
 @chapter Overview
 
 @command{cuirass} acts as a daemon polling @acronym{VCS, version control
-system} repositories for changes, and evaluating a derivation when
-something has changed (@pxref{Derivations, Derivations,, guix, Guix}).
-As a final step the derivation is realized and the result of that build
-allows you to know if the job succeeded or not.
+system} repositories (called @dfn{inputs}) for changes, and evaluating a
+derivation when an input has changed (@pxref{Derivations, Derivations,, guix,
+Guix}).  As a final step the derivation is realized and the result of that
+build allows you to know if the job succeeded or not.
 
 What is actually done by @command{cuirass} is specified in a @dfn{job
 specification} which is represented as an association list which is a
@@ -116,20 +116,40 @@ basic and traditional Scheme data structure.  Here is an example of what
 a specification might look like:
 
 @lisp
- `((#:name . "hello")
-   (#:url . "git://git.savannah.gnu.org/guix.git")
-   (#:branch . "master")
-   (#:no-compile? . #t)
-   (#:load-path . ".")
+ '((#:name . "foo-master")
+   (#:load-path-inputs . ("guix"))
+   (#:package-path-inputs . ("packages"))
+   (#:proc-input . "conf")
+   (#:proc-file . "drv-list.scm")
    (#:proc . cuirass-jobs)
-   (#:file . "/tmp/drv-file.scm")
-   (#:arguments (subset . "hello")))
+   (#:proc-args (subset . "foo"))
+   (#:inputs . (((#:name . "guix")
+                 (#:url . "git://git.savannah.gnu.org/guix.git")
+                 (#:load-path . ".")
+                 (#:branch . "master")
+                 (#:no-compile? . #t))
+                ((#:name . "conf")
+                 (#:url . "git://my-personal-conf.git")
+                 (#:load-path . ".")
+                 (#:branch . "master")
+                 (#:no-compile? . #t))
+                ((#:name . "packages")
+                 (#:url . "git://my-custom-packages.git")
+                 (#:load-path . ".")
+                 (#:branch . "master")
+                 (#:no-compile? . #t)))))
 @end lisp
 
 In this specification the keys are Scheme keywords which have the nice
 property of being self evaluating.  This means that they can't refer to
 another value like symbols do.
 
+There are three inputs: one tracking the Guix repository, one tracking the
+repository containing the @code{proc}, and one tracking the repository
+containing the custom packages (see @code{GUIX_PACKAGE_PATH}).
+@code{#:load-path-inputs}, @code{#:package-path-inputs} and
+@code{#:proc-input} refer to these inputs by their name.
+
 @quotation Note
 @c This refers to
 @c <https://github.com/libgit2/libgit2sharp/issues/1094#issuecomment-112306072>.
@@ -229,47 +249,69 @@ Cuirass uses a SQLite database to store information about jobs and past
 build results, but also to coordinate the execution of jobs.
 
 The database contains the following tables: @code{Specifications},
-@code{Stamps}, @code{Evaluations}, @code{Derivations}, and
+@code{Inputs}, @code{Stamps}, @code{Evaluations}, @code{Derivations} and
 @code{Builds}.  The purpose of each of these tables is explained below.
 
 @section Specifications
 @cindex specifications, database
 
-This table stores specifications describing the repository from whence
+This table stores specifications describing the repositories from whence
 Cuirass fetches code and the environment in which it will be processed.
 Entries in this table must have values for the following text fields:
 
 @table @code
-@item repo_name
-This field holds the name of the repository.  This field is also the
-primary key of this table.  Although this field is called
-@code{repo_name} in the database, it's called @code{name} in the
-specification itself.
-
-@item url
-The URL of the repository.
+@item name
+This field holds the name of the specification.  This field is also the
+primary key of this table.
 
-@item load_path
-This field holds a colon-separated list of directories that are
-prepended to the Guile load path when evaluating @code{file} (see
-below.)
+@item load_path_inputs
+This field holds a list of input names whose load path is prepended to Guile's
+@code{%load-path} when evaluating @code{proc_file}.
 
-Each entry that is not an absolute file name is interpreted relative to
-the source code checkout.  Often, @code{load_path} has just one entry,
-@code{"."}.
+@item package_path_inputs
+This field holds a list of input names whose load path is prepended to
+@code{GUIX_PACKAGE_PATH} when evaluating @code{proc_file}.
 
-When @code{load_path} is empty, the load path is left unchanged.
+@item proc_input
+The name of the input containing @code{proc}.
 
-@item file
-The absolute name of the Scheme file containing PROC.
+@item proc_file
+The path of the Scheme file containing @code{proc}, relative to
+@code{proc_input}.
 
 @item proc
-This text field holds the name of the procedure in the Scheme file FILE
-that produces a list of jobs.
+This text field holds the name of the procedure in the Scheme file
+@code{proc_file} that produces a list of jobs.
+
+@item proc_args
+A list of arguments to be passed to @code{proc}.  This can be used to produce
+a different set of jobs using the same @code{proc}.
+@end table
+
+@section Inputs
+@cindex inputs, database
+
+This table stores the data related to the repositories that are periodically
+fetched by Cuirass.  Entries in this table must have values for the following
+text fields:
+
+@table @code
+@item specification
+This field holds the name of the specification from the @code{Specifications}
+table associated with the input.  Every input belongs to a specification, and
+that specification can refer to its inputs.
+
+@item name
+This field holds the name of the input.  That name can be used as a key by the
+@code{proc} if it needs access to its resulting checkout.
+
+@item url
+The URL of the repository.
+
+@item load_path
+Used by a specification when it refers to an input's load path.  See
+@code{load_path_inputs} and @code{package_path_inputs}.
 
-@item arguments
-A list of arguments to be passed to PROC.  This can be used to produce a
-different set of jobs using the same PROC.
 @end table
 
 The following columns are optional:
@@ -280,13 +322,12 @@ This text field determines which branch of the repository Cuirass should
 check out.
 
 @item tag
-This text field is an alternative to using BRANCH or REVISION.  It tells
-Cuirass to check out the repository at the specified tag.
+This text field is an alternative to using @code{branch} or @code{revision}.
+It tells Cuirass to check out the repository at the specified tag.
 
 @item revision
-This text field is an alternative to using BRANCH or TAG.  It tells
-Cuirass to check out the repository at a particular revision.  In the
-case of a git repository this would be a commit hash.
+This text field is an alternative to using @code{branch} or @code{tag}.  It
+tells Cuirass to check out the repository at a particular commit.
 
 @item no_compile_p
 When this integer field holds the value @code{1} Cuirass will skip
@@ -296,14 +337,13 @@ compilation for the specified repository.
 @section Stamps
 @cindex stamps, database
 
-When a specification is processed, the repository must be downloaded at
-a certain revision as specified.  The @code{Stamps} table stores the
-current revision for every specification when it is being processed.
+When a specification is processed, the repositories must be downloaded at a
+certain revision as specified.  The @code{Stamps} table stores the current
+revisions for every specification when it is being processed.
 
-The table only has two text columns: @code{specification}, which
-references a specification from the @code{Specifications} table via the
-field @code{repo_name}, and @code{stamp}, which holds the revision
-(e.g. a commit hash).
+The table only has two text columns: @code{specification}, which references a
+specification from the @code{Specifications} table via the field @code{name},
+and @code{stamp}, which holds the revisions (space separated commit hashes).
 
 @section Evaluations
 @cindex evaluations, database
@@ -319,12 +359,12 @@ The @code{Evaluations} table has the following columns:
 This is an automatically incrementing numeric identifier.
 
 @item specification
-This field holds the @code{repo_name} of a specification from the
+This field holds the @code{name} of a specification from the
 @code{Specifications} table.
 
-@item revision
-This text field holds the revision string (e.g. a git commit) of the
-repository specified in the related specification.
+@item commits
+This text field holds the revisions (space separated commit hashes) of the
+repositories specified as inputs of the related specification.
 @end table
 
 @section Derivations