base: Let sqlite handle deduplication of the list of pending derivations.

Previously we would make a SQL query that would return many build jobs,
and then call 'delete-duplicates' on that.  This was extremely wasteful
because the list of returned by the query was huge leading to a heap of
several tens of GiB on a big database, and 'delete-duplicates' would
lead to more GC and it would take ages.

Furthermore, since 'delete-duplicates' is written in C as of Guile
2.2.3, it is uninterruptible from Fiber's viewpoint.  Consequently, the
kernel thread running the 'restart-builds' fiber would never schedule
other fibers, which could lead to deadlocks--e.g., since fibers are
scheduled on a circular shuffled list of kernel threads, once every N
times, a web server fiber would be sent to that kernel thread and not be
serviced.

* src/cuirass/base.scm (shuffle-jobs): Remove.
(shuffle-derivations): New procedure.
(spawn-builds): Take a list of derivations instead of a list of jobs.
(restart-builds): Remove 'builds' parameter.  Remove 'delete-duplicates'
call.  Remove done/remaining partitioning.
(build-packages): Adjust to pass 'spawn-builds' a list of derivations.
* bin/cuirass.in (main): Remove computation of PENDING.  Remove second
parameter in call to 'restart-builds'.

1 files changed, 4 insertions, 7 deletions

diff --git a/bin/cuirass.in b/bin/cuirass.in
index fa0d6af..d27167c 100644
--- a/bin/cuirass.in
+++ b/bin/cuirass.in
@@ -128,12 +128,9 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@"
                                 new-specs)))
                (if one-shot?
                    (process-specs db (db-get-specifications db))
-                   (let ((exit-channel (make-channel))
-                         (pending
-                          (begin
-                            (clear-build-queue db)
-                            (log-message "retrieving list of pending builds...")
-                            (db-get-builds db '((status pending))))))
+                   (let ((exit-channel (make-channel)))
+
+                     (clear-build-queue db)
 
                      ;; First off, restart builds that had not completed or
                      ;; were not even started on a previous run.
@@ -142,7 +139,7 @@ exec ${GUILE:-@GUILE@} --no-auto-compile -e main -s "$0" "$@"
                        'restart-builds exit-channel
                        (lambda ()
                          (with-database db
-                           (restart-builds db pending)))))
+                           (restart-builds db)))))
 
                      (spawn-fiber
                       (essential-task