summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
* Begin tuning db-get-builds for performancedb-get-builds-performance-tuningChristopher Baines2020-05-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | Now that I actually have some experience using SQLite from writing the Guix Build Coordinator, I can actually see potential ways to improve bits of Cuirass. The intention here is to start to address the performance of db-get-builds, which I think can be quite slow. This commit does several things, the big change is to try and construct a simpler query for SQLite. I'm not confident that SQLite's query planner can look past handling the NULL parameters, so I think it could be helpful to try and create a simpler query, both to avoid that problem if it exists, but also move the complexity in to Guile code, which I think is a bit more manageable. The way ordering is handled is also changed. Order is one of the filters, although it's not a filter, and some of the other filters also influenced the order. I think there are things still to fix/improve with the handling of ordering, but at least this commit just has the ordering happen once in the query. If this direction sounds OK, I think the next things to do is do a bit more testing, both to check uses of this query still work as intended, but also to take those specific queries generated by db-get-builds, and check what SQLite says when running them with EXPLAIN QUERY PLAN, as that should inform further improvements, and what indexes to create.
* build-aux: Update for Guile 3.Christopher Baines2020-05-25
| | | | | * build-aux/guix.scm (arguments): Change 2.2 to 3.0. (inputs): Change guile@2.2 to guile.
* templates: Evaluation page links to VCS web view.Ludovic Courtès2020-04-17
| | | | | | * src/cuirass/templates.scm (%vcs-web-views): New variable. (commit-hyperlink): New procedure. (evaluation-build-table): Use it in the input/commit table.
* templates: Evaluation page shows each input and commit.Ludovic Courtès2020-04-17
| | | | | | | | * src/cuirass/database.scm (db-get-inputs, db-get-checkouts): Export. * src/cuirass/http.scm (evaluation-html-page): Pass #:checkouts and #:inputs to 'evaluation-build-table'. * src/cuirass/templates.scm (evaluation-build-table): Add #:checkouts and #:inputs. Emit a table with "Input" and "Commit" columns.
* database: Add missing (ice-9 threads) import.Ludovic Courtès2020-04-17
| | | | | | This is required on Guile 3 for 'current-processor-count'. * src/cuirass/database.scm: Use (ice-9 threads).
* templates: 'time->string' doesn't show timezone for old builds.Ludovic Courtès2020-04-17
| | | | * src/cuirass/templates.scm (time->string): Remove ~z for old builds.
* templates: Fix typo in 'time->string'.Ludovic Courtès2020-04-17
| | | | | | | This is a followup to a1d353b1527fd1a2bfa2a1cbb2f0ff5c79d6e973. Until now we'd never display the year. * src/cuirass/templates.scm (time->string): Check for (= year current).
* templates: html-page: Fix indentation.Ricardo Wurmus2020-04-15
| | | | * src/cuirass/templates.scm (html-page): Fix indentation.
* templates: Display search hints.Ricardo Wurmus2020-04-15
| | | | | | | | * src/static/css/cuirass.css: New file. * Makefile.am (dist_css_DATA): Add it. * src/cuirass/http.scm (%file-white-list): Add css/cuirass.css. * src/cuirass/templates.scm (search-form): Add search-hints div. (html-page): Load cuirass.css.
* database: Support "status:" keyword in queries.Ricardo Wurmus2020-04-15
| | | | | | * src/cuirass/database.scm (query->bind-arguments): Handle "status" query. (db-get-builds-by-search, db-get-builds-query-min, db-get-builds-query-max): Add status filter.
* http: Move "/eval" page to (cuirass templates).Ludovic Courtès2020-02-24
| | | | | | | * src/cuirass/http.scm (url-handler): Move inline code for ('GET "eval" id) to... (evaluation-html-page): ... here. New procedure. * src/cuirass/templates.scm (evaluation-build-table): New procedure.
* templates: Build page links to corresponding evaluation.Ludovic Courtès2020-02-24
| | | | | * src/cuirass/templates.scm (build-details): Define 'evaluation'. Add "Evaluation" row with a link to EVALUATION.
* database: 'db-get-builds' returns each build's evaluation ID.Ludovic Courtès2020-02-24
| | | | | * src/cuirass/database.scm (db-get-builds): Fetch 'Builds.evaluation' and return it as #:eval-id.
* utils: Handle errors in worker threads.handle-errors-in-worker-threadsChristopher Baines2020-02-05
| | | | | | | | | | | | | | | Previously, if an error occurred, the worker fiber simply never sends a reply. In the case of HTTP requests to Cuirass, where an exception occurs when performing a database query, the fiber handling the request blocks as it never gets a response. I think that this has the potential to cause the process to hit file descriptor limits, as the connections are never responded to. This is fixed by responding with the details of the exception, and then throwing it within the fiber that made the call. * src/cuirass/utils.scm (make-worker-thread-channel): Catch exceptions when calling proc. (call-with-worker-thread): Handle receiving exceptions from the worker thread.
* database: Enable running up to 4 database queries at once.allow-concurrent-database-queriesChristopher Baines2020-01-25
| | | | | | | | | | | | | The number of threads is copied from bin/cuirass.in. When you have at least two processors, this will allow database queries to be executed in parallel. With some crude testing using the Apache HTTP server benchmarking tool (ab from the httpd package), the max request latency does seem to drop when multiple threads are used, especially when the database queries are slow (I tested by adding usleep to the worker thread code). * src/cuirass/database.scm (with-database): Pass #:parallelism to make-worker-thread-channel.
* Enable make-worker-thread-channel to create multiple worker threads.Christopher Baines2020-01-25
| | | | | | | | This will allow running multiple threads, that all listen on the same channel, enabling processing multiple jobs at one time. * src/cuirass/utils.scm (make-worker-thread-channel): Add a #:parallelism argument, and create as many threads as the given parallelism.
* Adjust make-worker-thread-channel to take an initializer.Christopher Baines2020-01-25
| | | | | | | | | | | | | | | | | While this is a generic method, and initializer function will give the flexibility required to create multiple worker threads for performing SQLite queries, each with it's own database connection (as a result of calling the initializer once for each thread). Without this change, they'd all have to use the same connection, which would not work. * src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take an initializer, rather than arguments directly. * src/cuirass/database.scm (with-database): Adjust to call make-worker-thread-channel with an initializer. * tests/database.scm (db-init): Change to use make-worker-thread-channel initializer. * tests/http.scm (db-init): Change to use make-worker-thread-channel initializer.
* utils: Change critical section terminology to worker threads.Christopher Baines2020-01-25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | As far as I'm aware, it's necessary to use a separate thread for interacting with SQLite as one of the threads used for fibers will be blocked while the SQLite query is running. This doesn't mean all queries have to be executed one at a time though, providing the queries are executed outside the threads used by fibers, and a single connection isn't used in multiple threads. These changes start to move in this direction, first by just changing the terminology. * src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change with-db-critical-section to with-db-worker-thread. * src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to with-db-worker-thread. (db-add-input, db-add-checkout, db-add-specification, db-remove-specification, db-get-inputs, db-get-specification, db-add-evaluation, db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output, db-add-build, db-update-build-status!, db-get-output, db-get-outputs, db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id, db-add-event, db-get-events, db-delete-events-with-ids-<=-to, db-get-pending-derivations, db-get-checkouts, db-get-evaluations, db-get-evaluations-build-summary, db-get-evaluations-id-max, db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max, db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change from using with-db-critical-section to with-db-worker-thread. (with-database): Change syntax rule to use make-worker-thread-channel, renaming from make-critical-section. * src/cuirass/utils.scm (%critical-section-args): Rename parameter to %worker-thread-args. (make-critical-section): Rename to make-worker-thread-channel, and adjust parameter and docstring. (call-with-critical-section): Rename to call-with-worker-thread and adjust parameter. (with-critical-section): Rename to with-worker-thread, and adjust to call call-with-worker-thread. * tests/database.scm (db-init): Use make-worker-thread-channel rather than make-critical-section. * tests/http.scm (db-init): Use make-worker-thread-channel rather than make-critical-section.
* Alter the Builds table to have an id fieldimprove-handling-of-rowidsChristopher Baines2020-01-25
| | | | | | | | | | | | | | | The internal rowid's are used for builds as you can request builds by using the rowid in the URL. The motivation here is to enable running VACUUM operations in SQLite, without risking the rowid's for Builds changing. It would be bad if they change, as they're used in the URL's for builds. * src/schema.sql (Builds): Add id column. * src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to UNIQUE constraint. * src/sql/upgrade-6.sql: New file. * Makefile.am (dist_sql_DATA): Add it.
* database: Don't return rowid from db-add-input.Christopher Baines2020-01-24
| | | | | | | As it is unused from where db-add-input is called. * src/cuirass/database.scm (db-add-input): Don't call and return (last-insert-rowid).
* build: Allow builds with Guile 3.0.Ludovic Courtès2020-01-18
| | | | | * configure.ac: Add "3.0" to 'GUILE_PKG'. * README: Mention it.
* http: Add missing import of (ice-9 threads) in test.Ludovic Courtès2020-01-18
| | | | | | | Using the core binding for 'call-with-new-thread' was deprecated in 2.2 and is removed in 3.0. * tests/http.scm: Use (ice-9 threads).
* Support returning build information by output.Christopher Baines2020-01-16
| | | | | | | | | | | | | | | | | | | | | | | | Being able to take a derivation and query the build information is useful, but in cases where there are multiple derivations that produce the same outputs, the probability of getting the data back from Cuirass is reduced. This is because Cuirass might not have build the exact derivation you have, but a different derivation that produces the same outputs (this can commonly happen when a related fixed output derivation changes). Cuirass doesn't store derivations if they produce the same outputs as a derivation it already knows about, so it can't determine if this is the case. Therefore, provide a way of querying build results by output, rather than derivation. The motivation behind this is to make it easier to import build information in to the Guix Data Service. * src/cuirass/database.scm (db-get-output): New procedure. * src/cuirass/http.scm (respond-output-not-found): New procedure. (request-path-components): Handle /output/… requests. * doc/cuirass.texi (Build information): Mention that you can get build information by output.
* Support publishing evaluation eventsChristopher Baines2020-01-16
| | | | | | | * src/cuirass/database.scm (db-add-evaluation): Record the creation of new evaluations as events. (db-set-evaluation-done): Record when evaluations finish as an event. * src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
* Support publishing build eventsChristopher Baines2020-01-16
| | | | | | | | | | | | | | | | | | | | | | | | | Add a table to store events, which have a type and a JSON blob. These can be used to record changes, this commit inserts events when new builds are created, and when the status of builds change. The EventsOutbox table is then used to track when events have been sent out. This is done through the new cuirass-send-events script. * Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events. .gitignore: Add bin/cuirass-send-events. (dist_pkgmodule_DATA): Add src/cuirass/send-events.scm. (dist_sql_DATA): Add src/sql/upgrade-5.sql. (EXTRA_DIST): bin/cuirass-send-events.in. (bin/cuirass-send-events): New rule. * bin/cuirass-send-events.in: New file. * src/cuirass/send-events.scm: New file. * src/sql/upgrade-5.sql: New file. * src/cuirass/database.scm (changes-count): New procedure. (db-update-build-status!): Call db-add-event after updating the build status. (db-add-event): New procedure. (db-add-build): Insert an event when a new build is inserted. (db-delete-events-with-ids-<=-to): New procedure. * src/schema.sql (Events): New table.
* base: 'build-derivations&' closes the build output port only once.Ludovic Courtès2019-11-03
| | | | | * src/cuirass/base.scm (build-derivations&): Remove extra 'close-port' call in 'guard' clause.
* tests: Remove 405 test.Ricardo Wurmus2019-10-30
| | | | | | This is a follow-up to commit c88a7c006ef164015e5c3fb1544b8112bf6ececf. * tests/http.scm: Remove test for invalid POST to /build/1.
* http: Add route for /admin.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/http.scm (url-handler): Handle /admin route.
* bin: Remove redundant lambda.Ricardo Wurmus2019-10-30
| | | | * bin/cuirass.in (main): Remove redundant lambda wrapping in for-each.
* http: Handle specification deletion and addition.Ricardo Wurmus2019-10-30
| | | | | * src/cuirass/http.scm (url-handler): Handle /admin/specifications/add, /admin/specifications/delete/*, and /admin/specifications.
* templates: specifications-table: Add admin features.Ricardo Wurmus2019-10-30
| | | | | * src/cuirass/templates.scm (specifications-table): Handle optional ADMIN? argument.
* database: Add db-remove-specification.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/database.scm (db-remove-specification): New procedure.
* http: Be explicit about accepted HTTP methods.Ricardo Wurmus2019-10-30
| | | | * src/cuirass/http.scm (url-handler): Match on HTTP method.
* templates: Failed evaluations link to their log.Ludovic Courtès2019-10-23
| | | | | * src/cuirass/templates.scm (evaluation-badges): Link to /eval/ID/log/raw in the 'zero?' case.
* http: Serve evaluation logs at /eval/ID/log/raw.Ludovic Courtès2019-10-23
| | | | | | | * src/cuirass/base.scm (evaluation-log-file): Export. * src/cuirass/http.scm (url-handler)[respond-gzipped-file]: New procedure. Add handler for /eval/ID/log/raw.
* base: Log the evaluation ID upon evaluation failure.Ludovic Courtès2019-10-23
| | | | | | * src/cuirass/base.scm (&evaluation-error)[id]: New field. (evaluate): Specify the 'id' field. (process-specs): Show the log file name upon evaluation failure.
* base: Write to 'evaluate' output to /var/log/cuirass.Ludovic Courtès2019-10-23
| | | | | | | | | | | This fixes a longstanding issue where evalution output would splatter over the /var/log/cuirass.log and be inscrutable. * src/cuirass/base.scm (%cuirass-state-directory): New variable. (evaluation-log-file): New procedure. (evaluate)[log-file, log-pipe]: New variables. Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into 'with-error-to-port'. Close 'log-pipe'.
* http: Support fetching builds by derivation.Christopher Baines2019-10-02
| | | | | | | | | There's a one to one mapping of build id to derivation, so allow querying by derivation file name as well. I'm looking at this as I'm interested in getting build information in to the Guix Data Service. * src/cuirass/http.scm (url-handler): Support fetching builds by numeric id or the derivation file name.
* templates: Fix type errors in build details page.Ludovic Courtès2019-09-23
| | | | | | | This is a followup to 858b6b8c8f2ae7b1ddaf4ae363147121be1f1fe8. * src/cuirass/templates.scm (build-details)[blocking-outputs]: Fix type errors when dealing with the result of 'derivation-build-plan'.
* http: Pass 'build-search-results-table' a list, not a vector.Ludovic Courtès2019-08-27
| | | | | * src/cuirass/http.scm (url-handler): Call 'vector->list' on the result of 'handle-builds-search-request', as expected by 'build-search-results-table'.
* http: Pass 'build-eval-table' a list, not a vector.Ludovic Courtès2019-08-27
| | | | | * src/cuirass/http.scm (url-handler): Call 'vector->list' on the result of 'handle-builds-request', as expected by 'build-eval-table'.
* http: Fix type error.Ricardo Wurmus2019-08-27
| | | | | | | This is a follow-up to commit c6f4fa5f577e4752d845fa3ce17c68fcd2079904. * src/cuirass/http.scm (url-handler): SPECIFICATIONS-TABLE expects a list, not a vector.
* templates: Use 'derivation-build-plan'.Ludovic Courtès2019-08-17
| | | | | * src/cuirass/templates.scm (build-details): Use 'derivation-build-plan' instead of 'derivation-prerequisites-to-build'.
* doc: Guile 2.2 is needed.Ludovic Courtès2019-08-17
| | | | | | This is a followup to ee11ba1d93b24753eafcd77eef8543e5cac5ba4c. * README: Ask for Guile 2.2.
* Switch to Guile-JSON 3.x.Ludovic Courtès2019-08-17
| | | | | | | | | | | | | | | | | | | | | | | Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on until now: it maps JSON dictionaries to alists (instead of hash tables), and JSON arrays to vectors (instead of lists). This commit is about adjusting all the existing code to this new mapping. * src/cuirass/http.scm (evaluation->json-object): New procedure. (handle-builds-request): Pass the result through 'list->vector'. (handle-builds-search-request): Likewise. (url-handler): Likewise for /jobsets, /specifications, /api/evaluations, and /build. For /api/evaluations, use 'evaluation->json-object'. * src/cuirass/utils.scm (object->json-scm): Add 'vector?' case. * tests/http.scm (hash-table-keys, hash-table=?): Remove. (evaluations-query-result): Use vectors for JSON arrays. ("object->json-string"): Expects alists instead of hash tables. ("/build/1"): Use 'lset=' instead of 'hash-table=?'. ("/api/latestbuilds?nr=1&jobset=guix"): Likewise, and expect alists instead of hash tables. ("/api/latestbuilds?nr=1&jobset=gnu"): Likewise. ("/api/evaluations?nr=1"): Likewise. * README: Mention Guile-JSON 3.x.
* http: Import (ice-9 format).Ludovic Courtès2019-08-17
| | | | * src/cuirass/http.scm: Add missing include of (ice-9 format).
* base: Use the new 'store-error' procedure names.Ludovic Courtès2019-08-17
| | | | | | | The 'nix-error' names are now deprecated in Guix. * src/cuirass/base.scm (build-derivations&): Use the new 'store-error' procedure names.
* Separate web interface.Ricardo Wurmus2019-08-10
| | | | | | | * bin/cuirass.in (show-help): Document "--web" option. (%options): Default to running without web interface. (main): Either run the web interface or build packages. * README: Mention the "--web" option.
* Fix templates to handle all statuses.Robert Vollmert2019-08-07
| | | | | * src/cuirass/templates.scm: Factor out class and title helpers for build statuses, and handle all the statuses.
* database: query->bind-arguments: Process ^ and $ characters.Ricardo Wurmus2019-06-20
| | | | | * src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters in query strings.
generated by cgit v1.2.3 (git 2.25.4) at 2024-05-16 04:59:40 +0000