aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* utils: Handle errors in worker threads.handle-errors-in-worker-threadsChristopher Baines2020-02-05
| | | | | | | | | | | | | | | Previously, if an error occurred, the worker fiber simply never sends a reply. In the case of HTTP requests to Cuirass, where an exception occurs when performing a database query, the fiber handling the request blocks as it never gets a response. I think that this has the potential to cause the process to hit file descriptor limits, as the connections are never responded to. This is fixed by responding with the details of the exception, and then throwing it within the fiber that made the call. * src/cuirass/utils.scm (make-worker-thread-channel): Catch exceptions when calling proc. (call-with-worker-thread): Handle receiving exceptions from the worker thread.
* database: Enable running up to 4 database queries at once.allow-concurrent-database-queriesChristopher Baines2020-01-25
| | | | | | | | | | | | | The number of threads is copied from bin/cuirass.in. When you have at least two processors, this will allow database queries to be executed in parallel. With some crude testing using the Apache HTTP server benchmarking tool (ab from the httpd package), the max request latency does seem to drop when multiple threads are used, especially when the database queries are slow (I tested by adding usleep to the worker thread code). * src/cuirass/database.scm (with-database): Pass #:parallelism to make-worker-thread-channel.
* Enable make-worker-thread-channel to create multiple worker threads.Christopher Baines2020-01-25
| | | | | | | | This will allow running multiple threads, that all listen on the same channel, enabling processing multiple jobs at one time. * src/cuirass/utils.scm (make-worker-thread-channel): Add a #:parallelism argument, and create as many threads as the given parallelism.
* Adjust make-worker-thread-channel to take an initializer.Christopher Baines2020-01-25
| | | | | | | | | | | | | | | | | While this is a generic method, and initializer function will give the flexibility required to create multiple worker threads for performing SQLite queries, each with it's own database connection (as a result of calling the initializer once for each thread). Without this change, they'd all have to use the same connection, which would not work. * src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take an initializer, rather than arguments directly. * src/cuirass/database.scm (with-database): Adjust to call make-worker-thread-channel with an initializer. * tests/database.scm (db-init): Change to use make-worker-thread-channel initializer. * tests/http.scm (db-init): Change to use make-worker-thread-channel initializer.
* utils: Change critical section terminology to worker threads.Christopher Baines2020-01-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As far as I'm aware, it's necessary to use a separate thread for interacting with SQLite as one of the threads used for fibers will be blocked while the SQLite query is running. This doesn't mean all queries have to be executed one at a time though, providing the queries are executed outside the threads used by fibers, and a single connection isn't used in multiple threads. These changes start to move in this direction, first by just changing the terminology. * src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change with-db-critical-section to with-db-worker-thread. * src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to with-db-worker-thread. (db-add-input, db-add-checkout, db-add-specification, db-remove-specification, db-get-inputs, db-get-specification, db-add-evaluation, db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output, db-add-build, db-update-build-status!, db-get-output, db-get-outputs, db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id, db-add-event, db-get-events, db-delete-events-with-ids-<=-to, db-get-pending-derivations, db-get-checkouts, db-get-evaluations, db-get-evaluations-build-summary, db-get-evaluations-id-max, db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max, db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change from using with-db-critical-section to with-db-worker-thread. (with-database): Change syntax rule to use make-worker-thread-channel, renaming from make-critical-section. * src/cuirass/utils.scm (%critical-section-args): Rename parameter to %worker-thread-args. (make-critical-section): Rename to make-worker-thread-channel, and adjust parameter and docstring. (call-with-critical-section): Rename to call-with-worker-thread and adjust parameter. (with-critical-section): Rename to with-worker-thread, and adjust to call call-with-worker-thread. * tests/database.scm (db-init): Use make-worker-thread-channel rather than make-critical-section. * tests/http.scm (db-init): Use make-worker-thread-channel rather than make-critical-section.
* Alter the Builds table to have an id fieldimprove-handling-of-rowidsChristopher Baines2020-01-25
| | | | | | | | | | | | | | | The internal rowid's are used for builds as you can request builds by using the rowid in the URL. The motivation here is to enable running VACUUM operations in SQLite, without risking the rowid's for Builds changing. It would be bad if they change, as they're used in the URL's for builds. * src/schema.sql (Builds): Add id column. * src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to UNIQUE constraint. * src/sql/upgrade-6.sql: New file. * Makefile.am (dist_sql_DATA): Add it.
* database: Don't return rowid from db-add-input.Christopher Baines2020-01-24
| | | | | | | As it is unused from where db-add-input is called. * src/cuirass/database.scm (db-add-input): Don't call and return (last-insert-rowid).
* build: Allow builds with Guile 3.0.Ludovic Courtès2020-01-18
| | | | | * configure.ac: Add "3.0" to 'GUILE_PKG'. * README: Mention it.
* http: Add missing import of (ice-9 threads) in test.Ludovic Courtès2020-01-18
| | | | | | | Using the core binding for 'call-with-new-thread' was deprecated in 2.2 and is removed in 3.0. * tests/http.scm: Use (ice-9 threads).
* Support returning build information by output.Christopher Baines2020-01-16
| | | | | | | | | | | | | | | | | | | | | | | | Being able to take a derivation and query the build information is useful, but in cases where there are multiple derivations that produce the same outputs, the probability of getting the data back from Cuirass is reduced. This is because Cuirass might not have build the exact derivation you have, but a different derivation that produces the same outputs (this can commonly happen when a related fixed output derivation changes). Cuirass doesn't store derivations if they produce the same outputs as a derivation it already knows about, so it can't determine if this is the case. Therefore, provide a way of querying build results by output, rather than derivation. The motivation behind this is to make it easier to import build information in to the Guix Data Service. * src/cuirass/database.scm (db-get-output): New procedure. * src/cuirass/http.scm (respond-output-not-found): New procedure. (request-path-components): Handle /output/… requests. * doc/cuirass.texi (Build information): Mention that you can get build information by output.
* Support publishing evaluation eventsChristopher Baines2020-01-16
| | | | | | | * src/cuirass/database.scm (db-add-evaluation): Record the creation of new evaluations as events. (db-set-evaluation-done): Record when evaluations finish as an event. * src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
* Support publishing build eventsChristopher Baines2020-01-16
| | | | | | | | | | | | | | | | | | | | | | | | | Add a table to store events, which have a type and a JSON blob. These can be used to record changes, this commit inserts events when new builds are created, and when the status of builds change. The EventsOutbox table is then used to track when events have been sent out. This is done through the new cuirass-send-events script. * Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events. .gitignore: Add bin/cuirass-send-events. (dist_pkgmodule_DATA): Add src/cuirass/send-events.scm. (dist_sql_DATA): Add src/sql/upgrade-5.sql. (EXTRA_DIST): bin/cuirass-send-events.in. (bin/cuirass-send-events): New rule. * bin/cuirass-send-events.in: New file. * src/cuirass/send-events.scm: New file. * src/sql/upgrade-5.sql: New file. * src/cuirass/database.scm (changes-count): New procedure. (db-update-build-status!): Call db-add-event after updating the build status. (db-add-event): New procedure. (db-add-build): Insert an event when a new build is inserted. (db-delete-events-with-ids-<=-to): New procedure. * src/schema.sql (Events): New table.
* base: 'build-derivations&' closes the build output port only once.Ludovic Courtès2019-11-03
| | | | | * src/cuirass/base.scm (build-derivations&): Remove extra 'close-port' call in 'guard' clause.
* tests: Remove 405 test.Ricardo Wurmus2019-10-30
| | | | | | This is a follow-up to commit c88a7c006ef164015e5c3fb1544b8112bf6ececf. * tests/http.scm: Remove test for invalid POST to /build/1.
* http: Add route for /admin.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/http.scm (url-handler): Handle /admin route.
* bin: Remove redundant lambda.Ricardo Wurmus2019-10-30
| | | | * bin/cuirass.in (main): Remove redundant lambda wrapping in for-each.
* http: Handle specification deletion and addition.Ricardo Wurmus2019-10-30
| | | | | * src/cuirass/http.scm (url-handler): Handle /admin/specifications/add, /admin/specifications/delete/*, and /admin/specifications.
* templates: specifications-table: Add admin features.Ricardo Wurmus2019-10-30
| | | | | * src/cuirass/templates.scm (specifications-table): Handle optional ADMIN? argument.
* database: Add db-remove-specification.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/database.scm (db-remove-specification): New procedure.
* http: Be explicit about accepted HTTP methods.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/http.scm (url-handler): Match on HTTP method.
* templates: Failed evaluations link to their log.Ludovic Courtès2019-10-23
| | | | | * src/cuirass/templates.scm (evaluation-badges): Link to /eval/ID/log/raw in the 'zero?' case.
* http: Serve evaluation logs at /eval/ID/log/raw.Ludovic Courtès2019-10-23
| | | | | | | * src/cuirass/base.scm (evaluation-log-file): Export. * src/cuirass/http.scm (url-handler)[respond-gzipped-file]: New procedure. Add handler for /eval/ID/log/raw.
* base: Log the evaluation ID upon evaluation failure.Ludovic Courtès2019-10-23
| | | | | | * src/cuirass/base.scm (&evaluation-error)[id]: New field. (evaluate): Specify the 'id' field. (process-specs): Show the log file name upon evaluation failure.
* base: Write to 'evaluate' output to /var/log/cuirass.Ludovic Courtès2019-10-23
| | | | | | | | | | | This fixes a longstanding issue where evalution output would splatter over the /var/log/cuirass.log and be inscrutable. * src/cuirass/base.scm (%cuirass-state-directory): New variable. (evaluation-log-file): New procedure. (evaluate)[log-file, log-pipe]: New variables. Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into 'with-error-to-port'. Close 'log-pipe'.
* http: Support fetching builds by derivation.Christopher Baines2019-10-02
| | | | | | | | | There's a one to one mapping of build id to derivation, so allow querying by derivation file name as well. I'm looking at this as I'm interested in getting build information in to the Guix Data Service. * src/cuirass/http.scm (url-handler): Support fetching builds by numeric id or the derivation file name.
* templates: Fix type errors in build details page.Ludovic Courtès2019-09-23
| | | | | | | This is a followup to 858b6b8c8f2ae7b1ddaf4ae363147121be1f1fe8. * src/cuirass/templates.scm (build-details)[blocking-outputs]: Fix type errors when dealing with the result of 'derivation-build-plan'.
* http: Pass 'build-search-results-table' a list, not a vector.Ludovic Courtès2019-08-27
| | | | | * src/cuirass/http.scm (url-handler): Call 'vector->list' on the result of 'handle-builds-search-request', as expected by 'build-search-results-table'.
* http: Pass 'build-eval-table' a list, not a vector.Ludovic Courtès2019-08-27
| | | | | * src/cuirass/http.scm (url-handler): Call 'vector->list' on the result of 'handle-builds-request', as expected by 'build-eval-table'.
* http: Fix type error.Ricardo Wurmus2019-08-27
| | | | | | | This is a follow-up to commit c6f4fa5f577e4752d845fa3ce17c68fcd2079904. * src/cuirass/http.scm (url-handler): SPECIFICATIONS-TABLE expects a list, not a vector.
* templates: Use 'derivation-build-plan'.Ludovic Courtès2019-08-17
| | | | | * src/cuirass/templates.scm (build-details): Use 'derivation-build-plan' instead of 'derivation-prerequisites-to-build'.
* doc: Guile 2.2 is needed.Ludovic Courtès2019-08-17
| | | | | | This is a followup to ee11ba1d93b24753eafcd77eef8543e5cac5ba4c. * README: Ask for Guile 2.2.
* Switch to Guile-JSON 3.x.Ludovic Courtès2019-08-17
| | | | | | | | | | | | | | | | | | | | | | | Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on until now: it maps JSON dictionaries to alists (instead of hash tables), and JSON arrays to vectors (instead of lists). This commit is about adjusting all the existing code to this new mapping. * src/cuirass/http.scm (evaluation->json-object): New procedure. (handle-builds-request): Pass the result through 'list->vector'. (handle-builds-search-request): Likewise. (url-handler): Likewise for /jobsets, /specifications, /api/evaluations, and /build. For /api/evaluations, use 'evaluation->json-object'. * src/cuirass/utils.scm (object->json-scm): Add 'vector?' case. * tests/http.scm (hash-table-keys, hash-table=?): Remove. (evaluations-query-result): Use vectors for JSON arrays. ("object->json-string"): Expects alists instead of hash tables. ("/build/1"): Use 'lset=' instead of 'hash-table=?'. ("/api/latestbuilds?nr=1&jobset=guix"): Likewise, and expect alists instead of hash tables. ("/api/latestbuilds?nr=1&jobset=gnu"): Likewise. ("/api/evaluations?nr=1"): Likewise. * README: Mention Guile-JSON 3.x.
* http: Import (ice-9 format).Ludovic Courtès2019-08-17
| | | | * src/cuirass/http.scm: Add missing include of (ice-9 format).
* base: Use the new 'store-error' procedure names.Ludovic Courtès2019-08-17
| | | | | | | The 'nix-error' names are now deprecated in Guix. * src/cuirass/base.scm (build-derivations&): Use the new 'store-error' procedure names.
* Separate web interface.Ricardo Wurmus2019-08-10
| | | | | | | * bin/cuirass.in (show-help): Document "--web" option. (%options): Default to running without web interface. (main): Either run the web interface or build packages. * README: Mention the "--web" option.
* Fix templates to handle all statuses.Robert Vollmert2019-08-07
| | | | | * src/cuirass/templates.scm: Factor out class and title helpers for build statuses, and handle all the statuses.
* database: query->bind-arguments: Process ^ and $ characters.Ricardo Wurmus2019-06-20
| | | | | * src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters in query strings.
* templates: If applicable, disable "Last" and "First" pagination links.Danny Milosavljevic2019-06-19
| | | | | * src/cuirass/templates.scm (pagination): Disable "Last" and "First" if applicable.
* templates: Add link titles to evaluation badges.Tobias Geerinckx-Rice2019-06-19
| | | | * src/cuirass/templates.scm (evaluation-badges): Add ‘title’ attributes.
* templates: Expand search input on focus.Ricardo Wurmus2019-06-19
| | | | | * src/cuirass/templates.scm (html-page): Include CSS for widening the search input field on focus.
* cuirass: Restore search query.Ricardo Wurmus2019-06-19
| | | | | | | | * src/cuirass/templates.scm (search-form): Take the query as an argument and set it as the value (unless #F). (html-page): Optionally take a value for QUERY as an argument. * src/cuirass/http.scm (url-handler): Let "search" handler pass QUERY to HTML-PAGE.
* database: query->bind-arguments: Ignore invalid query terms.Ricardo Wurmus2019-06-19
| | | | | * src/cuirass/database.scm (query->bind-arguments): Ignore invalid query terms.
* templates: Display logs of blocking dependencies.Ricardo Wurmus2019-06-19
| | | | | * src/cuirass/templates.scm (build-details): Determine failed outputs of prerequisites and link to their logs.
* http: Show number of builds.Ricardo Wurmus2019-06-19
| | | | | * src/cuirass/database.scm (db-get-evaluation-summary): New procedure. * src/cuirass/http.scm (url-handler): Display number of builds in tabs.
* cuirass: Remove tabs for canceled, dependency failed, and other failed.Ricardo Wurmus2019-06-19
| | | | | | All of them are included in "failed". * src/cuirass/http.scm (url-handler): Remove extraneous tabs.
* cuirass: Adjust query for scheduled builds.Ricardo Wurmus2019-06-19
| | | | * src/cuirass/http.scm (url-handler): Use "pending" instead of "scheduled".
* cuirass: Display builds in separate tabs.Ricardo Wurmus2019-06-19
| | | | | | * src/cuirass/http.scm (url-handler): Split result for "eval" page in tabs. * src/cuirass/templates.scm (build-eval-table): Remove border from table head; Remove lead paragraph.
* http: Fix navigation URL for build details page.Ricardo Wurmus2019-06-19
| | | | * src/cuirass/http.scm (url-handler): Use "jobset" instead of "spec".
* templates: Link to "details" page in build tables.Ricardo Wurmus2019-06-19
| | | | | * src/cuirass/templates.scm (build-eval-table, build-search-results-table): Link ID to build details URL.
* http: Handle /build/<id>/details URL.Ricardo Wurmus2019-06-18
| | | | | * src/cuirass/http.scm (url-handler): Add handler for /build/<id>/details. * src/cuirass/templates.scm (build-details): New procedure.
generated by cgit v1.2.3 (git 2.45.0) at 2024-11-21 15:03:37 +0000