aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAge
* base: Let sqlite handle deduplication of the list of pending derivations.Ludovic Courtès2018-04-05
| | | | | | | | | | | | | | | | | | | | | | | | | Previously we would make a SQL query that would return many build jobs, and then call 'delete-duplicates' on that. This was extremely wasteful because the list of returned by the query was huge leading to a heap of several tens of GiB on a big database, and 'delete-duplicates' would lead to more GC and it would take ages. Furthermore, since 'delete-duplicates' is written in C as of Guile 2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the kernel thread running the 'restart-builds' fiber would never schedule other fibers, which could lead to deadlocks--e.g., since fibers are scheduled on a circular shuffled list of kernel threads, once every N times, a web server fiber would be sent to that kernel thread and not be serviced. * src/cuirass/base.scm (shuffle-jobs): Remove. (shuffle-derivations): New procedure. (spawn-builds): Take a list of derivations instead of a list of jobs. (restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates' call. Remove done/remaining partitioning. (build-packages): Adjust to pass 'spawn-builds' a list of derivations. * bin/cuirass.in (main): Remove computation of PENDING. Remove second parameter in call to 'restart-builds'.
* database: Add 'db-get-pending-derivations'.Ludovic Courtès2018-04-05
| | | | | * src/cuirass/database.scm (db-get-pending-derivations): New procedure. * tests/database.scm ("database")["db-get-pending-derivations"]: New test.
* utils: Add critical sections.Ludovic Courtès2018-04-02
| | | | | | | | | | | * src/cuirass/utils.scm (make-critical-section) (call-with-critical-section): New procedures. (with-critical-section): New macro. * src/cuirass/http.scm (with-database-access): Remove. (handle-build-request, handle-builds-request, url-handler): Use 'with-critical-section' instead of 'with-database-access'. (run-cuirass-server): Remove 'spawn-fiber' call. Use 'make-critical-section' instead.
* base: Do not resort to Coreutils' "chmod".Ludovic Courtès2018-04-01
| | | | | * src/cuirass/base.scm (make-writable-copy)[chmod+w]: New procedure. Replace 'system*' call with 'file-system-fold' call.
* base: Make a writable copy of the checkout only when #:no-compile? is false.Ludovic Courtès2018-04-01
| | | | | | | | | | | | | | | This avoids copying things back and forth. * src/cuirass/base.scm (fetch-repository): Add #:writable-copy? parameter. Call 'make-writable-copy' when it's true. (copy-repository-cache): Remove. (make-writable-copy): New procedure. (evaluate): Add 'source' parameter and pass it to the 'evaluate' program. (process-specs): Define 'compile?'. Pass #:writable-copy? to 'fetch-repository'. Remove call to 'copy-repository-cache'. Remove computation of the checkout directory name. Pass CHECKOUT to 'evaluate'. * bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove computation of the checkout directory name.
* database: 'db-get-builds' honors 'status+submission-time' ordering again.Ludovic Courtès2018-03-29
| | | | | | | | | | | | | | | Fixes a regression introduced in 1bab5c4e56eb1849edc2cf0b23d433aeb2cac421 whereby the 'status+submission-time' order would no longer be honored. As a result, /api/queue would return the queue ordered by build IDs, making it largely useless. * src/cuirass/database.scm (db-get-builds): Remove 'order' and rename 'order-column-name' to 'order'. Add case for 'status+submission-time'. * tests/database.scm ("database")["db-get-builds"]: Move below "db-update-build-status!" test. Add case for the 'status+submission-time' order.
* http: Process client connections really concurrently, again.Ludovic Courtès2018-03-28
| | | | | | | | | | | | | | | | This reinstates c47dfdf82b4be62501a7932eaec4c124566a1829 and fixes the issues that led to the revert in b71f0cdca5aeb82e5eb24f54b32e3f09fee22bad. Before that, 'run-server' would force sequential processing of client requests one after another. * src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop instead of 'run-server'. Spawn a database fiber. (with-database-access): New macro. (handle-build-request): Expect 'db-channel' and use 'with-database-access'. (handle-builds-request): Likewise. (url-handler): Likewise.
* base: Add 'cancel-old-builds'.Ludovic Courtès2018-03-28
| | | | * src/cuirass/base.scm (cancel-old-builds): New procedure.
* database: Set a 'busy_timeout' to handle concurrent accesses.Ludovic Courtès2018-03-25
| | | | | | | | | | | | Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while accessing the database: see <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>. Suggested by Danny Milosavljevic <dannym@scratchpost.org>. * src/cuirass/database.scm (wal-mode): Rename to... (set-db-options): ... this. Add call to 'sqlite-exec' for 'busy_timeout'.
* 'with-store' and 'with-database' and written in terms of 'unwind-protect'.Ludovic Courtès2018-03-19
| | | | | * src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'. * src/cuirass/database.scm (with-database): Likewise.
* utils: Add 'unwind-protect'.Ludovic Courtès2018-03-19
| | | | * src/cuirass/utils.scm (unwind-protect): New macro.
* http: Correctly set #:timestamp for non-finished builds.Ludovic Courtès2018-03-18
| | | | | | | | | Fixes a regression introduced in f083282fd3bf813fda0b54ed33278d2d5325dfa1, whereby we'd return 0 as the timestamp for everything in /api/queue. * src/cuirass/http.scm (build->hydra-build): Make 'finished?' a Boolean. Move 'bool->int' call in #:finished definition.
* base: Catch errors in the 'process-build-log' handler.Ludovic Courtès2018-03-18
| | | | | | | | | | | Previously, when an exception was raised from 'handle-build-event' (e.g., a "database is locked" error), we'd throw, thereby leaving PORT open and we'd never read from it again. Thus, the corresponding 'guix-daemon' process would eventually get stuck in a 'write' call to that socket, and its build processes would stall. * src/cuirass/base.scm (exception-reporter): New procedure. (spawn-builds): Use it.
* base: Move database update from 'evaluate' process to the main process.Ludovic Courtès2018-03-01
| | | | | | | | | | | | Fixes <https://bugs.gnu.org/30618>. Reported by Andreas Enge <andreas@enge.fr>. * bin/evaluate.in (fill-job): Remove. (main): Remove 'database' command-line argument. Remove DB and its uses. Write an (evaluation EVAL JOBS) sexp. * src/cuirass/base.scm (evaluate)[augment-job]: New procedure. Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call 'db-add-evaluation' and 'db-add-derivation'.
* base: 'spawn-builds' correctly keeps track of remaining builds.Ludovic Courtès2018-03-01
| | | | | | | Fixes <https://bugs.gnu.org/30645>. Reported by Andreas Enge <andreas@enge.fr>. * src/cuirass/base.scm (spawn-builds): Fix TOTAL vs. COUNT mismatches.
* http: Return build completion time as #:timestamp when completed.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/http.scm (build->hydra-build): Set #:timestamp to #:stoptime when BUILD is finished.
* database: Indent 'db-get-builds'.Ludovic Courtès2018-03-01
| | | | * src/cuirass/database.scm (db-get-builds): Reindent.
* database: Adjust style of 'assqx-ref'.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/database.scm (db-get-builds)[assqx-ref]: Rewrite with 'match'.
* http: 'request-parameters' always returns a list.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/http.scm (request-parameters): Return the empty list when QUERY is #f.
* http: Fix interpretation of non-numerical parameters.Ludovic Courtès2018-03-01
| | | | | | | | | | Fixes a regression introduced in 593cb7be108ed97bca5371aad2e53fa8ce4817ba. * src/cuirass/http.scm (request-parameters): Fix fallback case in 'match' form. Previously it would return a procedure in this case, as returned by (const param), leading to a failure down the road in 'sqlite-bind-arguments' as could be seen by running tests/http.scm.
* logging: "Defensive programming" for 'log-monitoring-stats'.Ludovic Courtès2018-02-27
| | | | | | | | I've seen 'scandir' report #f once, even though that's theoretically impossible. * src/cuirass/logging.scm (log-monitoring-stats): Return '() if 'scandir' returns #f.
* database: Fix grouping in db-get-builds.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/database.scm (db-get-builds): Fix grouping.
* http: Interpret id and nr request-parameters as numbers.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/http.scm (request-parameters): Interpret id and nr parameters as numbers.
* database: db-get-builds: Remove debugging output.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/database.scm (db-get-builds): Remove debugging output. (db-get-build): Remove debugging output.
* http: Convert build-id URL part to number.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/http.scm (url-handler): Convert build-id URL part to number.
* database: db-get-builds: Inline output selection.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/database.scm (db-get-builds): Inline output selection.
* database: Simplify 'db-get-builds'.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/database.scm (db-get-builds): Modify. (db-get-build): Modify.
* cuirass: Clear the build queue when starting.Ludovic Courtès2018-02-14
| | | | | * src/cuirass/base.scm (clear-build-queue): New procedure. * bin/cuirass.in (main): Call it.
* sql: Add indices to speed up common queries.Ludovic Courtès2018-02-14
| | | | * src/schema.sql: Add indices.
* Revert "http: Process client connections really concurrently."Ludovic Courtès2018-02-14
| | | | | | | | This reverts commit c47dfdf82b4be62501a7932eaec4c124566a1829. Processing connections concurrently would require having separate database handles. See <https://lists.gnu.org/archive/html/guix-devel/2018-02/msg00206.html>.
* database: Make 'db-add-derivation' idempotent.Ludovic Courtès2018-02-14
| | | | | | | | * src/cuirass/database.scm (db-add-derivation): Catch 'sqlite-error and handle SQLITE_CONSTRAINT_PRIMARYKEY. (SQLITE_CONSTRAINT_UNIQUE): New variable. * tests/database.scm ("database")["db-add-derivation"]: Add extra call to 'db-add-derivation'.
* http: Process client connections really concurrently.Ludovic Courtès2018-02-10
| | | | | | | | Before that, 'run-server' would force sequential processing of client requests one after another. * src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop instead of 'run-server'.
* utils: 'non-blocking' forwards exceptions to the calling fiber.Ludovic Courtès2018-02-08
| | | | | * src/cuirass/utils.scm (%non-blocking): Forward exceptions to the calling fiber.
* database: Use argument binding in 'db-get-builds' queries.Ludovic Courtès2018-02-08
| | | | | | | | | That makes it safe from SQL injection. * src/cuirass/database.scm (db-get-builds): Rewrite to use question marks in SQL queries and binding through '%sqlite-exec'. * tests/database.scm ("database")["db-get-builds"]: Exercise 'WHERE' clauses.
* database: Handle binding directly in 'sqlite-exec'.Ludovic Courtès2018-02-08
| | | | | | | | | | | | | | | | | The new macro automatically takes care of inserting question marks in the SQL queries, which in turn guarantees that there are always as many question marks and arguments. * src/cuirass/database.scm (sqlite-exec): Rename to... (%sqlite-exec): ... this. (sqlite-exec/bind, sqlite-exec): New macros. (assq-refs): Remove. (db-add-specification): Use the new 'sqlite-exec' form. (db-get-specifications): Correctly deal with REV or TAG being #f. (db-add-derivation, db-get-derivation, db-add-evaluation) (db-add-build, db-update-build-status!, db-get-outputs) (db-get-build, db-get-stamp, db-add-stamp): Adjust to the new 'sqlite-exec' form.
* base: Account for derivations built behind our back.Ludovic Courtès2018-02-08
| | | | | | | | | | Previously any derivation not directly built by Cuirass would be considered as failed because 'handle-build-event' wouldn't see any build event. Here we just make sure the build status recorded in the database corresponds to reality. * src/cuirass/base.scm (update-build-statuses!): New procedure. (spawn-builds): Call it after 'build-derivations&'.
* base: Fix computation of build success/failure.Ludovic Courtès2018-02-08
| | | | | | * src/cuirass/base.scm (build-packages)[register]: Make 'db-add-build' a tail call. Fix computation of 'outs'.
* database: 'db-update-build-status!' keeps stoptime unchanged when nothing new.Ludovic Courtès2018-02-08
| | | | | | | * src/cuirass/database.scm (db-update-build-status!): And "AND status != ?" in SQL queries. * tests/database.scm ("database")["db-update-build-status!"]: Add call to 'db-update-build-status!'.
* base: Explicitly enable #:print-build-trace.Ludovic Courtès2018-02-08
| | | | | | | | This is purely "defensive programming" since the default value for 'print-build-trace' is currently #t. * src/cuirass/base.scm (with-store): Pass #:print-build-trace to 'set-build-options'.
* database: Cache prepared statements.Ludovic Courtès2018-02-08
| | | | | | | Suggested by Danny Milosavljevic. * src/cuirass/database.scm (sqlite-exec): Pass #:cache? to 'sqlite-prepare'.
* database: Use 'sqlite-bind' to avoid SQL injection.Danny Milosavljevic2018-02-08
| | | | | | | | | | | | | | | | | | | * src/cuirass/database.scm (%sqlite-exec): Remove. (sqlite-exec): Turn back into a procedure. Use 'sqlite-bind'. Add 'normalize' procedure and use it. (db-add-specification, db-add-derivation, db-get-derivation) (db-add-evaluation, db-add-build, db-update-build-status!) (db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL queries. * src/cuirass/base.scm (build-packages)[register]: Make #:log non-false. * tests/database.scm (make-dummy-job): Add #:job-name, #:system, #:nix-name, and #:eval-id. This is necessary because 'sqlite-bind' would now translate #f to a real NULL (before it would translate to the string "#f"...), and would thus report violations of the non-NULL constraint. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
* base: Remove useless 'log-message' calls.Ludovic Courtès2018-02-05
| | | | | * src/cuirass/base.scm (build-packages): Remove useless 'log-message' calls.
* http: /jobsets returns a list of jobsets.Ludovic Courtès2018-02-05
| | | | | | | Reported by Danny Milosavljevic. * src/cuirass/http.scm (url-handler) <"jobsets"> Return the result of 'db-get-specifications' as-is, not just the car.
* base: Log the number of builds performed.Ludovic Courtès2018-02-05
| | | | | * src/cuirass/base.scm (spawn-builds): Log the number of builds performed, not the number of remaining builds.
* base: Work around Fibers I/O scheduling bug.Ludovic Courtès2018-01-30
| | | | | | | | | | | Works around <https://github.com/wingo/fibers/issues/19>. The effect in practice would be that we'd usually not reach the 'close-pipe' call in 'evaluate', leaving zombie processes behind us, never executing the continuation, and additionally spinning fast on a sequence of epoll_wait/epoll_ctl calls. * src/cuirass/base.scm <top level>: Monkey-patch (fibers internal).
* base: Make build log processing non-blocking.Ludovic Courtès2018-01-29
| | | | | | | | | | | | | | | | | We used to have 'build-derivations' write to the custom binary port returned by 'build-event-output-port'. However, custom binary ports constitute continuation barriers, thereby preventing fibers from being suspended. To make build log processing non-blocking, we therefore invert this inversion of control and use a suspendable I/O procedure, 'read-line/non-blocking', when reading the build log. * src/cuirass/base.scm (read-line/non-blocking, process-build-log) (build-derivations&): New procedures. (%newline, build-event-output-port): Remove. (spawn-builds): Use 'build-derivations&' instead of 'build-derivations' with 'build-event-output-port'.
* http: Evaluate DB requests in 'non-blocking'.Ludovic Courtès2018-01-29
| | | | | * src/cuirass/http.scm (handle-builds-request): Wrap 'db-get-builds' into 'non-blocking'.
* cuirass: Log resource usage statistics regularly.Ludovic Courtès2018-01-29
| | | | | * src/cuirass/logging.scm (log-monitoring-stats): New procedure. * bin/cuirass.in (main): Add a fiber that calls it regularly.
* http: /api/queue returns builds sorted by status.Ludovic Courtès2018-01-29
| | | | | | * src/cuirass/database.scm (db-get-builds)[format-order-clause]: Add 'status+submission-time'. * src/cuirass/http.scm (url-handler) <"queue">: Use it.
* base: 'spawn-builds' shuffles jobs.Ludovic Courtès2018-01-29
| | | | | * src/cuirass/base.scm (shuffle-jobs): New procedure. (spawn-builds): Use it.
generated by cgit v1.2.3 (git 2.45.0) at 2024-11-22 02:18:48 +0000