summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* utils: Prevent critical section clients to talk to each other.Clément Lassieur2018-07-27
| | | | | | | | | * src/cuirass/utils.scm (make-critical-section): Put the modified message to the REPLY channel that was part of the initial message. (call-with-critical-section): Create a REPLY channel, add it to the sent message, get the modified message from that channel. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
* database: Use SQLite in Multi-thread mode.Clément Lassieur2018-07-20
| | | | | | | | | This disables mutexing on database connection and prepared statement objects, thus making us responsible for serializing access to database connections and prepared statements. It may result in a performance improvement. * src/cuirass/database.scm (db-init, db-open): Pass the SQLITE_OPEN_NOMUTEX flag to SQLITE-OPEN.
* Add support for multiple inputs.Clément Lassieur2018-07-16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Makefile.am (dist_sql_DATA): Add src/sql/upgrade-1.sql. * bin/cuirass.in (show-help, %options, main): Remove the LOAD-PATH option that was used afterwards as %GUIX-PACKAGE-PATH. * bin/evaluate.in (absolutize, input-checkout, spec-source, spec-load-path, spec-package-path, format-checkouts): New procedures. (%not-colon): Remove variable. (main): Take the load path, package path and PROC from the checkouts that result from the inputs. Format the checkouts before sending them to the procedure. Remove the LOAD-PATH argument. * doc/cuirass.texi (Overview, Database schema): Document the changes. * examples/{guix-jobs.scm, hello-git.scm, hello-singleton.scm, hello-subset.scm, random.scm}: Adapt to the new specification format. * examples/guix-track-git.scm (package->spec): Rename to PACKAGE->INPUT. (package->git-tracked): Replace FETCH-REPOSITORY with FETCH-INPUT and handle the new format of its return value. * examples/random-jobs.scm (make-random-jobs): Rename RANDOM to CHECKOUT. Rename the checkout from 'random (which is a specification) to 'cuirass (which is a checkout resulting from an input). * src/cuirass/base.scm (fetch-repository): Rename to fetch-input. Rename SPEC to INPUT. Return a checkout object instead of returning two values. (evaluate): Take a list of CHECKOUTS and COMMITS as arguments, instead of SOURCE. Remove TOKENIZE and LOAD-PATH. Pass the CHECKOUTS instead of the SOURCE to "evaluate". Remove %GUIX-PACKAGE-PATH. Build the EVAL object instead of getting it from "evaluate". (compile?, fetch-inputs, compile-checkouts): New procedures. (process-specs): Fetch all inputs instead of only fetching one repository. The result of that fetching operation is a list of CHECKOUTS whose COMMITS are used as a STAMP. (%guix-package-path, set-guix-package-path): Remove them. * src/cuirass/database.scm (db-add-input, db-get-inputs): New procedures. (db-add-specification, db-get-specifications): Adapt to the new specification format. Add/get all inputs as well. (db-add-evaluation): Rename REVISION to COMMITS. Store COMMITS as space separated commit hashes. (db-get-builds): Rename REPO_NAME to NAME. (db-get-stamp): Rename COMMIT to STAMP. Return #f when there is no STAMP. (db-add-stamp): Rename COMMIT to STAMP. Deal with DB-GET-STAMP's new return value. (db-get-evaluations): Rename REVISION to COMMITS. Tokenize COMMITS. * src/cuirass/utils.scm (%non-blocking): Export it. * src/schema.sql (Inputs): New table that refers to the Specifications table. (Specifications): Move input related fields to the Inputs table. Rename REPO_NAME to NAME. Rename ARGUMENTS to PROC_ARGS. Rename FILE to PROC_FILE. Add LOAD_PATH_INPUTS, PACKAGE_PATH_INPUTS and PROC_INPUT fields that refer to the Inputs table. (Stamps): Rename REPO_NAME to NAME. (Evaluations): Rename REPO_NAME to NAME. Rename REVISION to COMMITS. (Specifications_index): Replace with Inputs_index. * src/sql/upgrade-1.sql: New file. * tests/database.scm (example-spec, make-dummy-eval, sqlite-exec): Adapt to the new specifications format. Rename REVISION to COMMITS. * tests/http.scm (evaluations-query-result, fill-db): Idem.
* database: Call a specification 'jobset' instead of 'project'.Clément Lassieur2018-07-14
| | | | | | | | | | | | | | | | | This removes the possibility to filter specifications by branch, because branches were previously called 'jobset'. But it doesn't matter because later on, specifications will have as many branches as inputs. And people should filter by specification name instead. * doc/cuirass.texi (Build Information, Latest builds): Remove 'jobset', replace 'project' with 'jobset'. * src/cuirass/http.scm (build->hydra-build): Idem. * tests/database.scm (db-get-builds): Idem. * tests/http.scm (build-query-result, /api/latestbuilds?nr=1&jobset=guix, /api/latestbuilds?nr=1&jobset=gnu): Idem. * src/cuirass/database.scm (db-format-build, db-get-builds): Don't associate builds with branches (which were called 'jobset' afterwards). (db-get-builds): Remove the #:project filter.
* database: Add support for database upgrades.Clément Lassieur2018-07-14
| | | | | | | | * src/cuirass/database.scm (%package-sql-dir): New parameter. (db-load, db-schema-version, db-set-schema-version, latest-db-schema-version, schema-upgrade-file, db-upgrade): New procedures. (db-init): Set version corresponding to the existing upgrade-n.sql files. (db-open): If database exists, upgrade it.
* utils: Reset the Fiber dynamic environment in %NON-BLOCKING.Clément Lassieur2018-07-14
| | | | | | | | * src/cuirass/utils.scm (%non-blocking): Wrap body in PARAMETERIZE form that clears CURRENT-FIBER. So that PUT-MESSAGE doesn't try to suspend itself within CALL-WITH-NEW-THREAD. See https://lists.gnu.org/archive/html/guile-devel/2018-07/msg00009.html.
* evaluate: Use a generic key to identify Cuirass arguments.Clément Lassieur2018-06-18
| | | | | | | | | | | | | | | | | So that Cuirass specifications used to build 'guix-modular' can be named differently than "guix" and "guix-modular" (see Guix's build-aux/hydra/guix-modular.scm). The name is used as a primary key, so before that commit, it was also impossible to have several such specifications. This is a workaround, Cuirass should normally support several inputs per specification, as Hydra does. The specification's name would then only be used to identify it. See <https://bugs.gnu.org/31813>. * bin/evaluate.in (main): Replace custom NAME (passed to PROC) with 'guix'. Co-authored-by: Mathieu Othacehe <m.othacehe@gmail.com>
* http: Reverse evaluations list.Mathieu Othacehe2018-04-23
| | | | | * src/cuirass/database.scm (db-get-evaluations): Reverse returned evaluations list.
* http: Add /api/evaluations route.Mathieu Othacehe2018-04-19
| | | | | | * src/cuirass/database.scm (db-get-evaluations): New exported procedure. * src/cuirass/http.scm (url-handler): Add /api/evaluations route. * tests/http.scm ("http"): Add /api/evaluations test route.
* database: Adjust index to speed up /api/queue queries.Ludovic Courtès2018-04-10
| | | | | * src/schema.sql: Extend 'Builds_Derivations_index' to account for /api/queue queries.
* Use the 2.2 'setvbuf' API style.Ludovic Courtès2018-04-09
| | | | | * bin/cuirass.in (main): Use the 2.2 'setvbuf' API style. * examples/guix-track-git.scm (current-error-port): Likewise.
* base: 'spawn-builds' really builds by chunks.Ludovic Courtès2018-04-09
| | | | | | | | | Fixes a regression introduced in 074b9d02f1ca01007f39adbc019763027a51d9bd whereby we'd attempt to build all of DRV at once. * src/cuirass/base.scm (spawn-builds): Pass BATCH, not DRV, to 'build-derivations&' and to 'update-build-statuses!'.
* evaluate: Change '%load-path' once and for all.Ludovic Courtès2018-04-09
| | | | * bin/evaluate.in (main): Do not restore the original '%load-path'.
* evaluate: Leave GUILE_LOAD_PATH unchanged.Ludovic Courtès2018-04-09
| | | | | | | | | | | | Previously we'd systematically add the first argument to the search path. When that first argument was the empty string (as with the 'guix-modular' jobset), we'd thus add the current directory to the search path, even if the intent was to leave the load path unchanged. Furthermore, a3a7c09b06027bd30a96ae4607fa40bd790af840 changed load-path handling to be explicit in Scheme. * bin/evaluate.in: Remove GUILE_LOAD_PATH assignment.
* evaluate: Change directory to SOURCE.Ludovic Courtès2018-04-09
| | | | | * bin/evaluate.in (with-directory-excursion): Remove. (main): chdir to SOURCE.
* evaluate: Do not load Guix/Cuirass modules upfront.Ludovic Courtès2018-04-09
| | | | | | | | | | | | | | This avoids a situation whereby, when evaluating from a Guix checkout, we'd have already loaded slightly different and incompatible (guix …) modules. Hydra's 'hydra-eval-guile-jobs' implemented the same solution as in this patch already. * bin/evaluate.in: Remove use of (cuirass …) and (guix …) modules. (ref): New procedure. (with-directory-excursion): New macro. (main): Use 'ref'. Remove uses of Guix or Cuirass modules.
* base: Pass the correct load path to the 'evaluate' command.Ludovic Courtès2018-04-07
| | | | | | | | | | The previous load path was potentially incorrect since commit 2fe7ff87e23b18d49bd33cffc4766b7eaa382054. * src/cuirass/base.scm (evaluate)[tokenize, load-path]: New variables. Assume #:load-path is colon-separated. Pass LOAD-PATH as the second argument to 'evaluate'. * doc/cuirass.texi (Database): Adjust documentation.
* evaluate: Honor the given load path.Ludovic Courtès2018-04-07
| | | | | * bin/evaluate.in (main): Prepend LOAD-PATH to '%load-path' for the dynamic extend of the 'primitive-load' call.
* evaluate: Really support 'build-derivations' RPCs.Ludovic Courtès2018-04-07
| | | | | | | Fixes a thinko introduced in 1872dd95253b4805a00bfe5dee8d1a0ed90af149. * bin/evaluate.in (main): Make sure 'real-build-things' is bound.
* evaluate: Really pass arguments like Hydra.Ludovic Courtès2018-04-06
| | | | | * bin/evaluate.in (main): Use SPEC's #:name as a key in ARGS. * examples/random-jobs.scm (make-random-jobs): Adjust accordingly.
* base: Let sqlite handle deduplication of the list of pending derivations.Ludovic Courtès2018-04-05
| | | | | | | | | | | | | | | | | | | | | | | | | Previously we would make a SQL query that would return many build jobs, and then call 'delete-duplicates' on that. This was extremely wasteful because the list of returned by the query was huge leading to a heap of several tens of GiB on a big database, and 'delete-duplicates' would lead to more GC and it would take ages. Furthermore, since 'delete-duplicates' is written in C as of Guile 2.2.3, it is uninterruptible from Fiber's viewpoint. Consequently, the kernel thread running the 'restart-builds' fiber would never schedule other fibers, which could lead to deadlocks--e.g., since fibers are scheduled on a circular shuffled list of kernel threads, once every N times, a web server fiber would be sent to that kernel thread and not be serviced. * src/cuirass/base.scm (shuffle-jobs): Remove. (shuffle-derivations): New procedure. (spawn-builds): Take a list of derivations instead of a list of jobs. (restart-builds): Remove 'builds' parameter. Remove 'delete-duplicates' call. Remove done/remaining partitioning. (build-packages): Adjust to pass 'spawn-builds' a list of derivations. * bin/cuirass.in (main): Remove computation of PENDING. Remove second parameter in call to 'restart-builds'.
* database: Add 'db-get-pending-derivations'.Ludovic Courtès2018-04-05
| | | | | * src/cuirass/database.scm (db-get-pending-derivations): New procedure. * tests/database.scm ("database")["db-get-pending-derivations"]: New test.
* evaluate: Pass the file name and revision to the user procedure.Ludovic Courtès2018-04-05
| | | | | | | * bin/evaluate.in (main): Always pass an alist as the arguments to PROC, containing at least 'file-name' and 'revision'. * examples/random-jobs.scm (make-random-jobs): Display 'file-name' and 'revision' from ARGUMENTS.
* utils: Add critical sections.Ludovic Courtès2018-04-02
| | | | | | | | | | | * src/cuirass/utils.scm (make-critical-section) (call-with-critical-section): New procedures. (with-critical-section): New macro. * src/cuirass/http.scm (with-database-access): Remove. (handle-build-request, handle-builds-request, url-handler): Use 'with-critical-section' instead of 'with-database-access'. (run-cuirass-server): Remove 'spawn-fiber' call. Use 'make-critical-section' instead.
* base: Do not resort to Coreutils' "chmod".Ludovic Courtès2018-04-01
| | | | | * src/cuirass/base.scm (make-writable-copy)[chmod+w]: New procedure. Replace 'system*' call with 'file-system-fold' call.
* base: Make a writable copy of the checkout only when #:no-compile? is false.Ludovic Courtès2018-04-01
| | | | | | | | | | | | | | | This avoids copying things back and forth. * src/cuirass/base.scm (fetch-repository): Add #:writable-copy? parameter. Call 'make-writable-copy' when it's true. (copy-repository-cache): Remove. (make-writable-copy): New procedure. (evaluate): Add 'source' parameter and pass it to the 'evaluate' program. (process-specs): Define 'compile?'. Pass #:writable-copy? to 'fetch-repository'. Remove call to 'copy-repository-cache'. Remove computation of the checkout directory name. Pass CHECKOUT to 'evaluate'. * bin/evaluate.in (main): Replace 'cachedir' with 'source'. Remove computation of the checkout directory name.
* database: 'db-get-builds' honors 'status+submission-time' ordering again.Ludovic Courtès2018-03-29
| | | | | | | | | | | | | | | Fixes a regression introduced in 1bab5c4e56eb1849edc2cf0b23d433aeb2cac421 whereby the 'status+submission-time' order would no longer be honored. As a result, /api/queue would return the queue ordered by build IDs, making it largely useless. * src/cuirass/database.scm (db-get-builds): Remove 'order' and rename 'order-column-name' to 'order'. Add case for 'status+submission-time'. * tests/database.scm ("database")["db-get-builds"]: Move below "db-update-build-status!" test. Add case for the 'status+submission-time' order.
* http: Process client connections really concurrently, again.Ludovic Courtès2018-03-28
| | | | | | | | | | | | | | | | This reinstates c47dfdf82b4be62501a7932eaec4c124566a1829 and fixes the issues that led to the revert in b71f0cdca5aeb82e5eb24f54b32e3f09fee22bad. Before that, 'run-server' would force sequential processing of client requests one after another. * src/cuirass/http.scm (run-cuirass-server): Rewrite to use its own loop instead of 'run-server'. Spawn a database fiber. (with-database-access): New macro. (handle-build-request): Expect 'db-channel' and use 'with-database-access'. (handle-builds-request): Likewise. (url-handler): Likewise.
* base: Add 'cancel-old-builds'.Ludovic Courtès2018-03-28
| | | | * src/cuirass/base.scm (cancel-old-builds): New procedure.
* evaluate: Tolerate calls to 'build-things' during evaluations.Ludovic Courtès2018-03-28
| | | | * bin/evaluate.in (main): 'build-things' replacement no longer calls 'exit'.
* database: Set a 'busy_timeout' to handle concurrent accesses.Ludovic Courtès2018-03-25
| | | | | | | | | | | | Fixes a bug whereby some fibers would get a SQLITE_BUSY exception while accessing the database: see <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=30644#26>. Suggested by Danny Milosavljevic <dannym@scratchpost.org>. * src/cuirass/database.scm (wal-mode): Rename to... (set-db-options): ... this. Add call to 'sqlite-exec' for 'busy_timeout'.
* cuirass: Line-buffer stdout and stderr.Ludovic Courtès2018-03-25
| | | | * bin/cuirass.in (main): Add 'setvbuf' calls.
* 'with-store' and 'with-database' and written in terms of 'unwind-protect'.Ludovic Courtès2018-03-19
| | | | | * src/cuirass/base.scm (with-store): Rewrite using 'unwind-protect'. * src/cuirass/database.scm (with-database): Likewise.
* utils: Add 'unwind-protect'.Ludovic Courtès2018-03-19
| | | | * src/cuirass/utils.scm (unwind-protect): New macro.
* http: Correctly set #:timestamp for non-finished builds.Ludovic Courtès2018-03-18
| | | | | | | | | Fixes a regression introduced in f083282fd3bf813fda0b54ed33278d2d5325dfa1, whereby we'd return 0 as the timestamp for everything in /api/queue. * src/cuirass/http.scm (build->hydra-build): Make 'finished?' a Boolean. Move 'bool->int' call in #:finished definition.
* base: Catch errors in the 'process-build-log' handler.Ludovic Courtès2018-03-18
| | | | | | | | | | | Previously, when an exception was raised from 'handle-build-event' (e.g., a "database is locked" error), we'd throw, thereby leaving PORT open and we'd never read from it again. Thus, the corresponding 'guix-daemon' process would eventually get stuck in a 'write' call to that socket, and its build processes would stall. * src/cuirass/base.scm (exception-reporter): New procedure. (spawn-builds): Use it.
* base: Move database update from 'evaluate' process to the main process.Ludovic Courtès2018-03-01
| | | | | | | | | | | | Fixes <https://bugs.gnu.org/30618>. Reported by Andreas Enge <andreas@enge.fr>. * bin/evaluate.in (fill-job): Remove. (main): Remove 'database' command-line argument. Remove DB and its uses. Write an (evaluation EVAL JOBS) sexp. * src/cuirass/base.scm (evaluate)[augment-job]: New procedure. Use it. Adjust to read (evaluation EVAL JOBS) sexp. Call 'db-add-evaluation' and 'db-add-derivation'.
* base: 'spawn-builds' correctly keeps track of remaining builds.Ludovic Courtès2018-03-01
| | | | | | | Fixes <https://bugs.gnu.org/30645>. Reported by Andreas Enge <andreas@enge.fr>. * src/cuirass/base.scm (spawn-builds): Fix TOTAL vs. COUNT mismatches.
* http: Return build completion time as #:timestamp when completed.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/http.scm (build->hydra-build): Set #:timestamp to #:stoptime when BUILD is finished.
* database: Indent 'db-get-builds'.Ludovic Courtès2018-03-01
| | | | * src/cuirass/database.scm (db-get-builds): Reindent.
* database: Adjust style of 'assqx-ref'.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/database.scm (db-get-builds)[assqx-ref]: Rewrite with 'match'.
* http: 'request-parameters' always returns a list.Ludovic Courtès2018-03-01
| | | | | * src/cuirass/http.scm (request-parameters): Return the empty list when QUERY is #f.
* http: Fix interpretation of non-numerical parameters.Ludovic Courtès2018-03-01
| | | | | | | | | | Fixes a regression introduced in 593cb7be108ed97bca5371aad2e53fa8ce4817ba. * src/cuirass/http.scm (request-parameters): Fix fallback case in 'match' form. Previously it would return a procedure in this case, as returned by (const param), leading to a failure down the road in 'sqlite-bind-arguments' as could be seen by running tests/http.scm.
* logging: "Defensive programming" for 'log-monitoring-stats'.Ludovic Courtès2018-02-27
| | | | | | | | I've seen 'scandir' report #f once, even though that's theoretically impossible. * src/cuirass/logging.scm (log-monitoring-stats): Return '() if 'scandir' returns #f.
* database: Fix grouping in db-get-builds.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/database.scm (db-get-builds): Fix grouping.
* http: Interpret id and nr request-parameters as numbers.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/http.scm (request-parameters): Interpret id and nr parameters as numbers.
* database: db-get-builds: Remove debugging output.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/database.scm (db-get-builds): Remove debugging output. (db-get-build): Remove debugging output.
* http: Convert build-id URL part to number.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/http.scm (url-handler): Convert build-id URL part to number.
* database: db-get-builds: Inline output selection.Danny Milosavljevic2018-02-19
| | | | * src/cuirass/database.scm (db-get-builds): Inline output selection.
* database: Simplify 'db-get-builds'.Danny Milosavljevic2018-02-19
| | | | | * src/cuirass/database.scm (db-get-builds): Modify. (db-get-build): Modify.
generated by cgit v1.2.3 (git 2.25.4) at 2024-06-29 16:41:55 +0000