summaryrefslogtreecommitdiff
path: root/src/cuirass/base.scm
Commit message (Collapse)AuthorAge
* utils: Change critical section terminology to worker threads.Christopher Baines2020-01-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As far as I'm aware, it's necessary to use a separate thread for interacting with SQLite as one of the threads used for fibers will be blocked while the SQLite query is running. This doesn't mean all queries have to be executed one at a time though, providing the queries are executed outside the threads used by fibers, and a single connection isn't used in multiple threads. These changes start to move in this direction, first by just changing the terminology. * src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change with-db-critical-section to with-db-worker-thread. * src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to with-db-worker-thread. (db-add-input, db-add-checkout, db-add-specification, db-remove-specification, db-get-inputs, db-get-specification, db-add-evaluation, db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output, db-add-build, db-update-build-status!, db-get-output, db-get-outputs, db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id, db-add-event, db-get-events, db-delete-events-with-ids-<=-to, db-get-pending-derivations, db-get-checkouts, db-get-evaluations, db-get-evaluations-build-summary, db-get-evaluations-id-max, db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max, db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change from using with-db-critical-section to with-db-worker-thread. (with-database): Change syntax rule to use make-worker-thread-channel, renaming from make-critical-section. * src/cuirass/utils.scm (%critical-section-args): Rename parameter to %worker-thread-args. (make-critical-section): Rename to make-worker-thread-channel, and adjust parameter and docstring. (call-with-critical-section): Rename to call-with-worker-thread and adjust parameter. (with-critical-section): Rename to with-worker-thread, and adjust to call call-with-worker-thread. * tests/database.scm (db-init): Use make-worker-thread-channel rather than make-critical-section. * tests/http.scm (db-init): Use make-worker-thread-channel rather than make-critical-section.
* base: 'build-derivations&' closes the build output port only once.Ludovic Courtès2019-11-03
| | | | | * src/cuirass/base.scm (build-derivations&): Remove extra 'close-port' call in 'guard' clause.
* http: Serve evaluation logs at /eval/ID/log/raw.Ludovic Courtès2019-10-23
| | | | | | | * src/cuirass/base.scm (evaluation-log-file): Export. * src/cuirass/http.scm (url-handler)[respond-gzipped-file]: New procedure. Add handler for /eval/ID/log/raw.
* base: Log the evaluation ID upon evaluation failure.Ludovic Courtès2019-10-23
| | | | | | * src/cuirass/base.scm (&evaluation-error)[id]: New field. (evaluate): Specify the 'id' field. (process-specs): Show the log file name upon evaluation failure.
* base: Write to 'evaluate' output to /var/log/cuirass.Ludovic Courtès2019-10-23
| | | | | | | | | | | This fixes a longstanding issue where evalution output would splatter over the /var/log/cuirass.log and be inscrutable. * src/cuirass/base.scm (%cuirass-state-directory): New variable. (evaluation-log-file): New procedure. (evaluate)[log-file, log-pipe]: New variables. Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into 'with-error-to-port'. Close 'log-pipe'.
* base: Use the new 'store-error' procedure names.Ludovic Courtès2019-08-17
| | | | | | | The 'nix-error' names are now deprecated in Guix. * src/cuirass/base.scm (build-derivations&): Use the new 'store-error' procedure names.
* base: Keep going when a Git error is raised.Ludovic Courtès2018-11-20
| | | | | * src/cuirass/base.scm (process-specs): Catch 'git-error around call to 'process'.
* base: Clear the "in progress" bit of failed evaluations.Ludovic Courtès2018-11-07
| | | | | | | Fixes <https://bugs.gnu.org/32734>. * src/cuirass/base.scm (evaluate): Call 'db-set-evaluation-done' upon '&evaluation-error'.
* base: Add missing space in message.Ludovic Courtès2018-11-07
| | | | * src/cuirass/base.scm (spawn-builds): Add missing space in message.
* base: Fix type error in 'update-build-statuses!'.Ludovic Courtès2018-11-07
| | | | | | | This is a followup to be489a26c0e6a5f23a48142a87728a0ec8bc3c9c. * src/cuirass/base.scm (update-build-statuses!): Use DRV, not (derivation-file-name DRV).
* base: Register GC roots for build results.Ludovic Courtès2018-11-07
| | | | | | | | | | | | | Fixes <https://bugs.gnu.org/33124>. * src/cuirass/base.scm (%gc-root-directory, %gc-root-ttl): New variables. (gc-root-expiration-time, register-gc-root): New procedures. (handle-build-event)[gc-roots]: New procedure. Upon 'build-succeeded' events, call 'register-gc-root' and 'maybe-remove-expired-cache-entries'. * bin/cuirass.in (show-help, %options): Add '--ttl'. (main): Parameterize %GC-ROOT-TTL. Create %GC-ROOT-DIRECTORY. * doc/cuirass.texi (Invocation): Document '--ttl'.
* base: Disable builder output.Ludovic Courtès2018-11-07
| | | | | * src/cuirass/base.scm (with-store): Pass #:build-verbosity to 'set-build-options'.
* base: Distinguish between 'failed' and 'failed-dependency'.Ludovic Courtès2018-10-08
| | | | | * src/cuirass/base.scm (update-build-statuses!): Call 'log-file' and use that to distinguish between 'failed' and 'failed-dependency'.
* database: Add a Checkouts table.Clément Lassieur2018-08-27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is used to know when a new evaluation must be triggered and to display input changes. * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-3.sql'. * bin/cuirass.in (main): Call DB-SET-EVALUATION-DONE at startup to clear 'in-progress' evaluations. * bin/evaluate.in (input-checkout, format-checkouts): Rename '#:name' to '#:input'. * doc/cuirass.texi (Stamps): Remove section. (Checkouts): New section. * src/cuirass/base.scm (fetch-input, fetch-inputs, compile-checkouts): Rename '#:name' to '#:input'. (evaluate): Remove the COMMITS argument. Add an EVAL-ID argument. Don't call DB-ADD-EVALUATION because it was called sooner. Remove the EVAL-ID argument to AUGMENT-JOB because it's a closure. (build-packages): Add an EVAL-ID argument. Call DB-SET-EVALUATION-DONE once all the derivations are registered. (process-specs): Replace the stamping mechanism by the primary key constraint of the Checkouts table: call "evaluate" only when DB-ADD-EVALUATION is true, which means that at least one checkout was added. Change the EVALUATE and BUILD-PACKAGES arguments accordingly. * src/cuirass/database.scm (db-add-stamp, db-get-stamp): Remove procedures. (db-set-evaluations-done, db-set-evaluation-done): New exported procedure. (db-add-checkout): New procedure that returns #f if a checkout with the same revision already exists. (db-add-evaluation): Replace the EVAL argument with a SPEC-NAME and a CHECKOUTS arguments. Insert the evaluation only if at least one checkout was inserted. Return #f otherwise. (db-get-checkouts): New procedure. (db-get-evaluations, db-get-evaluations-build-summary): Handle the 'in_progress' column, remove the 'commits' column. Return the result of DB-GET-CHECKOUTS as part of the evaluation. * src/cuirass/templates.scm (input-changes, evaluation-badges): New procedures. (evaluation-info-table): Rename "Commits" to "Input changes". Use INPUT-CHANGES to display the input changes that triggered the evaluation. Use EVALUATION-BADGES to display a message indicating that the evaluation is in progress. * src/schema.sql (Stamps): Remove table. (Checkouts): New table. (Evaluations): Remove the 'commits' column. Add an 'in_progress' column. * src/sql/upgrade-3.sql: New file with SQL queries to upgrade the database. * tests/database.scm (make-dummy-eval): Remove procedure. (make-dummy-checkouts): New procedure. ("sqlite-exec"): Remove the 'commits' column. Add the 'in_progress' column. ("db-update-build-status!", "db-get-builds", "db-get-pending-derivations"): Update the arguments of DB-ADD-EVALUATION accordingly. * tests/http.scm (hash-table=?): Add support for lists of hash tables. (evaluations-query-result): Replace '#:commits' with '#:checkouts'. Return a list instead of returning one element, for symmetry. ("fill-db"): Add a new input so that the second checkout can refer to it. Replace EVALUATION1 and EVALUATION2 with CHECKOUTS1 and CHECKOUTS2. Update the arguments of DB-ADD-EVALUATION accordingly. ("/api/queue?nr=100"): Take the CAR of the EVALUATIONS-QUERY-RESULT list to make it symmetrical with the other argument of HASH-TABLE=?.
* database: Serialize all database accesses in a thread.Clément Lassieur2018-08-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes <https://bugs.gnu.org/32234>. * bin/cuirass.in (main): Keep only one WITH-DATABASE call around all fibers. Remove all DB arguments. * src/cuirass/base.scm (evaluate, update-build-statuses!, spawn-builds, handle-build-event, build-packages): Remove all DB arguments. (clear-build-queue, cancel-old-builds): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments. (restart-builds): Remove the NON-BLOCKING call, remove all DB arguments. (process-specs): Remove all DB arguments, remove the WITH-DATABASE call. * src/cuirass/database.scm (%db-channel): New parameter. (with-db-critical-section): New macro. (db-add-input, db-add-specification, db-get-inputs, db-get-specifications, db-add-evaluation, db-add-build, db-update-build-status!, db-get-outputs, db-get-builds, db-get-build, db-get-pending-derivations, db-get-stamp, db-add-stamp, db-get-evaluations, db-get-evaluations-build-summary, db-get-evaluations-id-min, db-get-evaluations-id-max, db-get-builds-min, db-get-builds-max): Wrap in WITH-DB-CRITICAL-SECTION, remove all DB arguments. (with-database): Wrap BODY in PARAMETERIZE form that sets %DB-CHANNEL to the channel returned by MAKE-CRITICAL-SECTION. * src/cuirass/http.scm (handle-build-request, handle-builds-request): Remove all DB arguments. (url-handler): Remove all DB arguments, remove the DB-CHANNEL state, remove the WITH-CRITICAL-SECTION calls. (run-cuirass-server): Remove the DB arguments, remove the MAKE-CRITICAL-SECTION call. * src/cuirass/utils.scm (make-critical-section): Replace SPAWN-FIBER with CALL-WITH-NEW-THREAD. Wrap body in PARAMETERIZE form that clears CURRENT-FIBER. * tests/database.scm (with-temporary-database, "db-add-specification", "db-add-build", "db-update-build-status!", "db-get-builds", "db-get-pending-derivations"): Remove the DB arguments. ("db-init"): Set the %DB-CHANNEL parameter to the channel returned by MAKE-CRITICAL-SECTION, and return #t. ("database"): Set %DB-CHANNEL to #f during cleanup. * tests/http.scm ("db-init"): Set the %DB-CHANNEL parameter to the channel returned by MAKE-CRITICAL-SECTION, and return #t. ("cuirass-run", "fill-db"): Remove the DB arguments. ("http"): Set %DB-CHANNEL to #f during cleanup.
* database: Merge Derivations into Builds table.Clément Lassieur2018-08-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes <https://bugs.gnu.org/32190>. * Makefile.am (dist_sql_DATA): Add 'src/sql/upgrade-2.sql'. * doc/cuirass.texi (Derivations): Remove section. (Builds): Update accordingly. Add columns from the Derivations table. (Outputs): Replace 'id' with 'derivation'. * src/cuirass/base.scm (evaluate): Don't add jobs to the Derivations table. (build-packages): Add columns that were in the Derivations table before. Only build the derivations that were successfully registered, that is, those that didn't exist in the Builds table. Give a derivation instead of a build id to DB-GET-BUILD. Compute the number of failed jobs based on the derivations that were added to the table, instead of the jobs. * src/cuirass/database.scm (db-add-derivation, db-get-derivation): Remove exported procedures. (db-add-build): Catch SQLITE_CONSTRAINT_PRIMARYKEY error, which means that two jobs produced the same derivation, and return #f in that case. Add columns that were in the Derivations table before. Use 'derivation' as primary key for the Outputs table. (db-get-outputs): Use 'derivation' as identifier, instead of 'build-id'. (filters->order): Replace 'id' with 'rowid'. (db-get-builds): Add a 'derivation' filter. Replace 'id' with 'rowid'. Remove the 'INNER JOIN Derivations'. Replace Derivations with Builds. Return 'derivation' in first position to make it clear that it's the primary key. Pass DERIVATION instead of ID to DB-GET-OUTPUTS. (db-get-build): Allow to take a derivation as argument. Use NUMBER? to differentiate between derivations and ids. (db-get-pending-derivations): Remove the 'SELECT DISTINCT' clause now that derivations are unique. Remove the 'INNER JOIN Builds'. (db-get-evaluations-build-summary, db-get-builds-min, db-get-builds-max): Replace 'id' with 'rowid'. * src/schema.sql (Derivations): Remove table. (Outputs): Replace Builds.id with Builds.derivation. (Builds): Use 'derivation' as primary key. Remove the 'id' column. Add 'job_name', 'system', 'nix_name' columns that were in the Derivations table before. (Builds_Derivations_index): Rename to Builds_index. Update accordingly. (Derivations_index): Remove index. * src/sql/upgrade-2.sql: New file with SQL queries to upgrade the database. * tests/database.scm (make-dummy-job, make-dummy-derivation): Remove procedures. (make-dummy-build): Add columns that were in MAKE-DUMMY-DERIVATION. Get the DRV parameter to be mandatory because it's a primary key. (%id): Remove parameter. ("db-add-derivation", "db-get-derivation"): Remove tests. ("db-add-build"): Expect #f, because it adds twice the same derivation. Pass the derivation argument to MAKE-DUMMY-BUILD. ("db-update-build-status!"): Rename 'id' to 'derivation'. Pass the derivation argument to MAKE-DUMMY-BUILD. Remove the DB-ADD-DERIVATION call. ("db-get-builds", "db-get-pending-derivations"): Pass the derivation argument to MAKE-DUMMY-BUILD. Remove the DB-ADD-DERIVATION calls. * tests/http.scm ("fill-db"): Remove DERIVATION1 and DERIVATION2, and put their content in BUILD1 and BUILD2. Remove the DB-ADD-DERIVATION calls.
* Add support for multiple inputs.Clément Lassieur2018-07-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql. * bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that was used afterwards as %GUIX-PACKAGE-PATH. * bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path, spec-package-path, format-checkouts): New procedures. (%not-colon): Remove variable. (main): Take the load path, package path and PROC from the checkouts that result from the inputs. Format the checkouts before sending them to the procedure. Remove the LOAD-PATH argument. * doc/cuirass.texi (Overview, Database schema): Document the changes. * examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm, hello-subset.scm, random.scm}: Adapt to the new specification format. * examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT. (package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle the new format of its return value. * examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT. Rename the checkout from 'random (which is a specification) to 'cuirass (which is a checkout resulting from an input). * src/cuirass/base.scm (fetch-repository): Rename to fetch-input. Rename SPEC to INPUT. Return a checkout object instead of returning two values. (evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of SOURCE. Remove TOKENIZE and LOAD-PATH. Pass the CHECKOUTS instead of the SOURCE to "evaluate". Remove %GUIX-PACKAGE-PATH. Build the EVAL object instead of getting it from "evaluate". (compile?, fetch-inputs, compile-checkouts): New procedures. (process-specs): Fetch all inputs instead of only fetching one repository. The result of that fetching operation is a list of CHECKOUTS whose COMMITS are used as a STAMP. (%guix-package-path, set-guix-package-path): Remove them. * src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures. (db-add-specification, db-get-specifications): Adapt to the new specification format. Add/get all inputs as well. (db-add-evaluation): Rename REVISION to COMMITS. Store COMMITS as space separated commit hashes. (db-get-builds): Rename REPO_NAME to NAME. (db-get-stamp): Rename COMMIT to STAMP. Return #f when there is no STAMP. (db-add-stamp): Rename COMMIT to STAMP. Deal with DB-GET-STAMP's new return value. (db-get-evaluations): Rename REVISION to COMMITS. Tokenize COMMITS. * src/cuirass/utils.scm (%non-blocking): Export it. * src/schema.sql (Inputs): New table that refers to the Specifications table. (Specifications): Move input related fields to the Inputs table. Rename REPO_NAME to NAME. Rename ARGUMENTS to PROC_ARGS. Rename FILE to PROC_FILE. Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to the Inputs table. (Stamps): Rename REPO_NAME to NAME. (Evaluations): Rename REPO_NAME to NAME. Rename REVISION to COMMITS. (Specifications_index): Replace with Inputs_index. * src/sql/upgrade-1.sql: New file. * tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to the new specifications format. Rename REVISION to COMMITS. * tests/http.scm (evaluations-query-result, fill-db): Idem.
* base: 'spawn-builds' really builds by chunks.Ludovic Courtès2018-04-09
| | | | | | | | | Fixes a regression introduced in 074b9d02f1ca01007f39adbc019763027a51d9bd whereby we'd attempt to build all of DRV at once. * src/cuirass/base.scm (spawn-builds): Pass BATCH, not DRV, to 'build-derivations&' and to 'update-build-statuses!'.
* base: Pass the correct load path to the 'evaluate' command.Ludovic Courtès2018-04-07
| | | | | | | | | | The previous load path was potentially incorrect since commit 2fe7ff87e23b18d49bd33cffc4766b7eaa382054. * src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables. Assume #:load-path is colon-separated. Pass LOAD-PATH as the second argument to 'evaluate'. * doc/cuirass.texi (Database): Adjust documentation.
* base: Let sqlite handle deduplication of the list of pending derivations.Ludovic Courtès2018-04-05
| | | | | | | | | | | | | | | | | | | | | | | | | Previously we would make a SQL query that would return many build jobs, and then call 'delete-duplicates' on that. This was extremely wasteful because the list of returned by the query was huge leading to a heap of several tens of GiB on a big database, and 'delete-duplicates' would lead to more GC and it would take ages. Furthermore, since 'delete-duplicates' is written in C as of Guile 2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the kernel thread running the 'restart-builds' fiber would never schedule other fibers, which could lead to deadlocks--e.g., since fibers are scheduled on a circular shuffled list of kernel threads, once every N times, a web server fiber would be sent to that kernel thread and not be serviced. * src/cuirass/base.scm (shuffle-jobs): Remove. (shuffle-derivations): New procedure. (spawn-builds): Take a list of derivations instead of a list of jobs. (restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates' call. Remove done/remaining partitioning. (build-packages): Adjust to pass 'spawn-builds' a list of derivations. * bin/cuirass.in (main): Remove computation of PENDING. Remove second parameter in call to 'restart-builds'.
* base: Do not resort to Coreutils' "chmod".Ludovic Courtès2018-04-01
| | | | | * src/cuirass/base.scm (make-writable-copy)[chmod+w]: New procedure. Replace 'system*' call with 'file-system-fold' call.
* base: Make a writable copy of the checkout only when #:no-compile? is false.Ludovic Courtès2018-04-01
| | | | | | | | | | | | | | | This avoids copying things back and forth. * src/cuirass/base.scm (fetch-repository): Add #:writable-copy? parameter. Call 'make-writable-copy' when it's true. (copy-repository-cache): Remove. (make-writable-copy): New procedure. (evaluate): Add 'source' parameter and pass it to the 'evaluate' program. (process-specs): Define 'compile?'. Pass #:writable-copy? to 'fetch-repository'. Remove call to 'copy-repository-cache'. Remove computation of the checkout directory name. Pass CHECKOUT to 'evaluate'. * bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove computation of the checkout directory name.
* base: Add 'cancel-old-builds'.Ludovic Courtès2018-03-28
| | | | * src/cuirass/base.scm (cancel-old-builds): New procedure.
* 'with-store' and 'with-database' and written in terms of 'unwind-protect'.Ludovic Courtès2018-03-19
| | | | | * src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'. * src/cuirass/database.scm (with-database): Likewise.
* base: Catch errors in the 'process-build-log' handler.Ludovic Courtès2018-03-18
| | | | | | | | | | | Previously, when an exception was raised from 'handle-build-event' (e.g., a "database is locked" error), we'd throw, thereby leaving PORT open and we'd never read from it again. Thus, the corresponding 'guix-daemon' process would eventually get stuck in a 'write' call to that socket, and its build processes would stall. * src/cuirass/base.scm (exception-reporter): New procedure. (spawn-builds): Use it.
* base: Move database update from 'evaluate' process to the main process.Ludovic Courtès2018-03-01
| | | | | | | | | | | | Fixes <https://bugs.gnu.org/30618>. Reported by Andreas Enge <andreas@enge.fr>. * bin/evaluate.in (fill-job): Remove. (main): Remove 'database' command-line argument. Remove DB and its uses. Write an (evaluation EVAL JOBS) sexp. * src/cuirass/base.scm (evaluate)[augment-job]: New procedure. Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call 'db-add-evaluation' and 'db-add-derivation'.
* base: 'spawn-builds' correctly keeps track of remaining builds.Ludovic Courtès2018-03-01
| | | | | | | Fixes <https://bugs.gnu.org/30645>. Reported by Andreas Enge <andreas@enge.fr>. * src/cuirass/base.scm (spawn-builds): Fix TOTAL vs. COUNT mismatches.
* cuirass: Clear the build queue when starting.Ludovic Courtès2018-02-14
| | | | | * src/cuirass/base.scm (clear-build-queue): New procedure. * bin/cuirass.in (main): Call it.
* base: Account for derivations built behind our back.Ludovic Courtès2018-02-08
| | | | | | | | | | Previously any derivation not directly built by Cuirass would be considered as failed because 'handle-build-event' wouldn't see any build event. Here we just make sure the build status recorded in the database corresponds to reality. * src/cuirass/base.scm (update-build-statuses!): New procedure. (spawn-builds): Call it after 'build-derivations&'.
* base: Fix computation of build success/failure.Ludovic Courtès2018-02-08
| | | | | | * src/cuirass/base.scm (build-packages)[register]: Make 'db-add-build' a tail call. Fix computation of 'outs'.
* base: Explicitly enable #:print-build-trace.Ludovic Courtès2018-02-08
| | | | | | | | This is purely "defensive programming" since the default value for 'print-build-trace' is currently #t. * src/cuirass/base.scm (with-store): Pass #:print-build-trace to 'set-build-options'.
* database: Use 'sqlite-bind' to avoid SQL injection.Danny Milosavljevic2018-02-08
| | | | | | | | | | | | | | | | | | | * src/cuirass/database.scm (%sqlite-exec): Remove. (sqlite-exec): Turn back into a procedure. Use 'sqlite-bind'. Add 'normalize' procedure and use it. (db-add-specification, db-add-derivation, db-get-derivation) (db-add-evaluation, db-add-build, db-update-build-status!) (db-get-build, db-get-stamp, db-add-stamp): Use question marks in SQL queries. * src/cuirass/base.scm (build-packages)[register]: Make #:log non-false. * tests/database.scm (make-dummy-job): Add #:job-name, #:system, #:nix-name, and #:eval-id. This is necessary because 'sqlite-bind' would now translate #f to a real NULL (before it would translate to the string "#f"...), and would thus report violations of the non-NULL constraint. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
* base: Remove useless 'log-message' calls.Ludovic Courtès2018-02-05
| | | | | * src/cuirass/base.scm (build-packages): Remove useless 'log-message' calls.
* base: Log the number of builds performed.Ludovic Courtès2018-02-05
| | | | | * src/cuirass/base.scm (spawn-builds): Log the number of builds performed, not the number of remaining builds.
* base: Work around Fibers I/O scheduling bug.Ludovic Courtès2018-01-30
| | | | | | | | | | | Works around <https://github.com/wingo/fibers/issues/19>. The effect in practice would be that we'd usually not reach the 'close-pipe' call in 'evaluate', leaving zombie processes behind us, never executing the continuation, and additionally spinning fast on a sequence of epoll_wait/epoll_ctl calls. * src/cuirass/base.scm <top level>: Monkey-patch (fibers internal).
* base: Make build log processing non-blocking.Ludovic Courtès2018-01-29
| | | | | | | | | | | | | | | | | We used to have 'build-derivations' write to the custom binary port returned by 'build-event-output-port'. However, custom binary ports constitute continuation barriers, thereby preventing fibers from being suspended. To make build log processing non-blocking, we therefore invert this inversion of control and use a suspendable I/O procedure, 'read-line/non-blocking', when reading the build log. * src/cuirass/base.scm (read-line/non-blocking, process-build-log) (build-derivations&): New procedures. (%newline, build-event-output-port): Remove. (spawn-builds): Use 'build-derivations&' instead of 'build-derivations' with 'build-event-output-port'.
* base: 'spawn-builds' shuffles jobs.Ludovic Courtès2018-01-29
| | | | | * src/cuirass/base.scm (shuffle-jobs): New procedure. (spawn-builds): Use it.
* base: 'spawn-builds' logs the beginning of each new batch.Ludovic Courtès2018-01-29
| | | | * src/cuirass/base.scm (spawn-builds): Add 'log-message' call.
* base: 'build-event-output-port' no longer reads past the buffer.Ludovic Courtès2018-01-26
| | | | | | | | | Fixes a bug whereby 'build-event-output-port' would sometimes read more than COUNT from BV, which would usually result in a 'decoding-error' exception from 'utf8->string'. * src/cuirass/base.scm (build-event-output-port)[write!]: Use 'bytevector-range' to honor COUNT and OFFSET.
* base: Make repository fetching and compilation non-blocking.Ludovic Courtès2018-01-26
| | | | | | | | Previously these calls would effectively suspend execution of all the fibers until they had completed. * src/cuirass/base.scm (process-specs): Wrap 'fetch-repository' and 'compile' calls in 'non-blocking'.
* base: Do not restart builds that turn out to have succeeded already.Ludovic Courtès2018-01-26
| | | | | * src/cuirass/base.scm (restart-builds): Mark as succeeded the subset of VALID with at least one valid output.
* base: Do not pass bogus store file names to 'db-update-build-status!'.Ludovic Courtès2018-01-26
| | | | | | * src/cuirass/base.scm (handle-build-event)[valid?]: New procedure. Use it when handling 'build-started', 'build-succeeded', and 'build-failed' events.
* base: Delete duplicate builds when restarting them.Ludovic Courtès2018-01-25
| | | | | * src/cuirass/base.scm (build-derivation=?): New procedure. (restart-builds): Call 'delete-duplicates' on BUILDS.
* base: Remove unneeded import.Ludovic Courtès2018-01-25
| | | | * src/cuirass/base.scm: Remove unneeded #:use-module.
* base: Build derivations in batches.Ludovic Courtès2018-01-25
| | | | | | | | This works around a scalability issue in guix-daemon when passing a long list of derivations to 'build-derivations'. * src/cuirass/base.scm (spawn-builds): New procedure. (restart-builds, build-packages): Use it.
* base: Buffer input pipe of the 'evaluate' process.Ludovic Courtès2018-01-25
| | | | * src/cuirass/base.scm (read/non-blocking): Add 'setvbuf' call.
* base: Extract 'read/non-blocking'.Ludovic Courtès2018-01-25
| | | | | * src/cuirass/base.scm (read/non-blocking): New procedure. (evaluate): Use it instead of inline code.
* base: Be more verbose about restarted builds.Ludovic Courtès2018-01-24
| | | | | | | * src/cuirass/base.scm (handle-build-event): Remove unused 'log-port' parameter. (restart-builds): Wrap 'build-derivations' in 'guard'. Add 'log-message' calls.
* base: Populate the 'Outputs' table.Ludovic Courtès2018-01-24
| | | | | | | | | Fixes a regression introduced in d7306a4f48a666a008091bfeb94d1fdb32b46948 whereby 'register' would pass the empty list as #:outputs to 'db-add-build'. * src/cuirass/base.scm (build-packages)[register]: Always include outputs.
* base: Always set 'keep-going' when opening the store.Ludovic Courtès2018-01-23
| | | | | | | | Previously 'restart-builds' would lack it, for instance. * src/cuirass/base.scm (process-specs): Move 'set-build-options' call to... (with-store): ... here.
generated by cgit v1.2.3 (git 2.25.4) at 2024-06-28 06:25:05 +0000