diff options
author | Clément Lassieur <clement@lassieur.org> | 2018-08-11 20:30:11 +0200 |
---|---|---|
committer | Clément Lassieur <clement@lassieur.org> | 2018-08-27 15:38:44 +0200 |
commit | 8d40c49170971ad7bbf8b97336934dbb3d949fc1 (patch) | |
tree | fe272b71fe83409579418ed02564d4805e92f9ed /src/cuirass/base.scm | |
parent | 4612a3a70f1e70afa4e0ce00e8cb1a7848dddf58 (diff) | |
download | cuirass-8d40c49170971ad7bbf8b97336934dbb3d949fc1.tar cuirass-8d40c49170971ad7bbf8b97336934dbb3d949fc1.tar.gz |
database: Add a Checkouts table.
It is used to know when a new evaluation must be triggered and to display
input changes.
* Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-3.sql'.
* bin/cuirass.in (main): Call DB-SET-EVALUATION-DONE at startup to clear
'in-progress' evaluations.
* bin/evaluate.in (input-checkout, format-checkouts): Rename '#:name' to
'#:input'.
* doc/cuirass.texi (Stamps): Remove section.
(Checkouts): New section.
* src/cuirass/base.scm (fetch-input, fetch-inputs, compile-checkouts): Rename
'#:name' to '#:input'.
(evaluate): Remove the COMMITS argument. Add an EVAL-ID argument. Don't call
DB-ADD-EVALUATION because it was called sooner. Remove the EVAL-ID argument
to AUGMENT-JOB because it's a closure.
(build-packages): Add an EVAL-ID argument. Call DB-SET-EVALUATION-DONE once
all the derivations are registered.
(process-specs): Replace the stamping mechanism by the primary key constraint
of the Checkouts table: call "evaluate" only when DB-ADD-EVALUATION is true,
which means that at least one checkout was added. Change the EVALUATE and
BUILD-PACKAGES arguments accordingly.
* src/cuirass/database.scm (db-add-stamp, db-get-stamp): Remove procedures.
(db-set-evaluations-done, db-set-evaluation-done): New exported procedure.
(db-add-checkout): New procedure that returns #f if a checkout with the same
revision already exists.
(db-add-evaluation): Replace the EVAL argument with a SPEC-NAME and a
CHECKOUTS arguments. Insert the evaluation only if at least one checkout was
inserted. Return #f otherwise.
(db-get-checkouts): New procedure.
(db-get-evaluations, db-get-evaluations-build-summary): Handle the
'in_progress' column, remove the 'commits' column. Return the result of
DB-GET-CHECKOUTS as part of the evaluation.
* src/cuirass/templates.scm (input-changes, evaluation-badges): New
procedures.
(evaluation-info-table): Rename "Commits" to "Input changes". Use
INPUT-CHANGES to display the input changes that triggered the evaluation. Use
EVALUATION-BADGES to display a message indicating that the evaluation is in
progress.
* src/schema.sql (Stamps): Remove table.
(Checkouts): New table.
(Evaluations): Remove the 'commits' column. Add an 'in_progress' column.
* src/sql/upgrade-3.sql: New file with SQL queries to upgrade the database.
* tests/database.scm (make-dummy-eval): Remove procedure.
(make-dummy-checkouts): New procedure.
("sqlite-exec"): Remove the 'commits' column. Add the 'in_progress' column.
("db-update-build-status!", "db-get-builds", "db-get-pending-derivations"):
Update the arguments of DB-ADD-EVALUATION accordingly.
* tests/http.scm (hash-table=?): Add support for lists of hash tables.
(evaluations-query-result): Replace '#:commits' with '#:checkouts'. Return a
list instead of returning one element, for symmetry.
("fill-db"): Add a new input so that the second checkout can refer to it.
Replace EVALUATION1 and EVALUATION2 with CHECKOUTS1 and CHECKOUTS2. Update
the arguments of DB-ADD-EVALUATION accordingly.
("/api/queue?nr=100"): Take the CAR of the EVALUATIONS-QUERY-RESULT list to
make it symmetrical with the other argument of HASH-TABLE=?.
Diffstat (limited to 'src/cuirass/base.scm')
-rw-r--r-- | src/cuirass/base.scm | 49 |
1 files changed, 21 insertions, 28 deletions
diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm index 1ec122c..deee05b 100644 --- a/src/cuirass/base.scm +++ b/src/cuirass/base.scm @@ -178,7 +178,7 @@ read-only directory." (string-append (%package-cachedir) "/" name)) directory))) - `((#:name . ,name) + `((#:input . ,name) (#:directory . ,directory) (#:commit . ,commit) (#:load-path . ,(assq-ref input #:load-path)) @@ -248,10 +248,10 @@ fibers." (logior (@ (fibers epoll) EPOLLERR) (@ (fibers epoll) EPOLLHUP))))) -(define (evaluate store spec checkouts commits) +(define (evaluate store spec eval-id checkouts) "Evaluate and build package derivations defined in SPEC, using CHECKOUTS. -Return a list of jobs." - (define (augment-job job eval-id) +Return a list of jobs that are associated to EVAL-ID." + (define (augment-job job) (let ((drv (read-derivation-from-file (assq-ref job #:derivation)))) `((#:eval-id . ,eval-id) @@ -275,14 +275,9 @@ Return a list of jobs." (close-pipe port) (match result (('evaluation jobs) - (let* ((spec-name (assq-ref spec #:name)) - (eval-id (db-add-evaluation - `((#:specification . ,spec-name) - (#:commits . ,commits))))) - (log-message "created evaluation ~a for '~a'" eval-id spec-name) - (map (lambda (job) - (augment-job job eval-id)) - jobs)))))) + (let* ((spec-name (assq-ref spec #:name))) + (log-message "evaluation ~a for '~a' completed" eval-id spec-name) + (map augment-job jobs)))))) ;;; @@ -539,7 +534,7 @@ started)." (spawn-builds store valid) (log-message "done with restarted builds")))) -(define (build-packages store jobs) +(define (build-packages store jobs eval-id) "Build JOBS and return a list of Build results." (define (register job) (let* ((name (assq-ref job #:job-name)) @@ -576,6 +571,10 @@ started)." (define derivations (filter-map register jobs)) + (log-message "evaluation ~a registered ~a new derivations" + eval-id (length derivations)) + (db-set-evaluation-done eval-id) + (spawn-builds store derivations) (let* ((results (filter-map (cut db-get-build <>) derivations)) @@ -625,7 +624,7 @@ started)." (results (par-map %non-blocking thunks))) (map (lambda (checkout) (log-message "fetched input '~a' of spec '~a' (commit ~s)" - (assq-ref checkout #:name) + (assq-ref checkout #:input) (assq-ref spec #:name) (assq-ref checkout #:commit)) checkout) @@ -638,7 +637,7 @@ started)." (lambda (checkout) (lambda () (log-message "compiling input '~a' of spec '~a' (commit ~s)" - (assq-ref checkout #:name) + (assq-ref checkout #:input) (assq-ref spec #:name) (assq-ref checkout #:commit)) (compile checkout))) @@ -646,7 +645,7 @@ started)." (results (par-map %non-blocking thunks))) (map (lambda (checkout) (log-message "compiled input '~a' of spec '~a' (commit ~s)" - (assq-ref checkout #:name) + (assq-ref checkout #:input) (assq-ref spec #:name) (assq-ref checkout #:commit)) checkout) @@ -656,15 +655,10 @@ started)." "Evaluate and build JOBSPECS and store results in the database." (define (process spec) (with-store store - (let* ((stamp (db-get-stamp spec)) - (name (assoc-ref spec #:name)) + (let* ((name (assoc-ref spec #:name)) (checkouts (fetch-inputs spec)) - (commits (map (cut assq-ref <> #:commit) checkouts)) - (commits-str (string-join commits))) - (unless (equal? commits-str stamp) - ;; Immediately mark SPEC's INPUTS as being processed so we don't - ;; spawn a concurrent evaluation of that same commit. - (db-add-stamp spec commits-str) + (eval-id (db-add-evaluation name checkouts))) + (when eval-id (compile-checkouts spec (filter compile? checkouts)) (spawn-fiber (lambda () @@ -672,13 +666,12 @@ started)." (log-message "failed to evaluate spec '~a'" (evaluation-error-spec-name c)) #f)) - (log-message "evaluating spec '~a': stamp ~s different from ~s" - name commits-str stamp) + (log-message "evaluating spec '~a'" name) (with-store store - (let ((jobs (evaluate store spec checkouts commits))) + (let ((jobs (evaluate store spec eval-id checkouts))) (log-message "building ~a jobs for '~a'" (length jobs) name) - (build-packages store jobs)))))) + (build-packages store jobs eval-id)))))) ;; 'spawn-fiber' returns zero values but we need one. *unspecified*)))) |