aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAge
* Use a non-blocking socket for store communication.Ludovic Courtès2020-10-27
| | | | | | | | Set the store socket as non-blocking so that fibers communicating with the store don't get blocked as described here: https://issues.guix.gnu.org/43565. * src/cuirass/base.scm (with-store): Set the store socket as non-blocking. (build-derivations&): Unset current-read-waiter and current-write-waiter.
* Use with-db-writer-worker-thread/force macro.Mathieu Othacehe2020-10-15
| | | | | * src/cuirass/database.scm (db-register-builds): Use "with-db-writer-worker-thread/force" macro.
* Use the writer worker for all write queries.Mathieu Othacehe2020-10-15
| | | | | | | | | | | | | | | | | * .dir-locals.el: Add "with-queue-writer-worker". * bin/cuirass.in: Modify "with-queue-writer-worker" scope to include the web-server operations. * src/cuirass/database.scm (with-db-writer-worker-thread): Export it. (with-db-writer-worker-thread/force): New macro. (db-add-input, db-add-checkout, db-add-specification, db-remove-specification, db-add-evaluation, db-abort-pending-evaluations, db-set-evaluation-status, db-set-evaluation-time, db-add-output, db-add-build-product, db-add-event, db-delete-events-with-ids-<=-to): Use "with-db-writer-worker-thread" or "with-db-writer-worker-thread/force" instead of "with-db-worker-thread". * src/cuirass/metrics.scm (db-update-metrics): Ditto. * tests/database.scm ("db-init"): Set "%db-writer-channel". * tests/http.scm ("db-init"): Ditto. * tests/metrics.scm ("db-init"): Ditto.
* Remove unused procedure.Mathieu Othacehe2020-10-14
| | | | * src/cuirass/base.scm (cancel-old-builds): Remove it.
* Fix tests.Mathieu Othacehe2020-10-14
| | | | | | | | | | | This is a follow-up of b67f38a7b91c8605a3ae9eba1e2bd3da4b579622. * src/cuirass/database.scm (catch-sqlite-error): New macro. (SQLITE_CONSTRAINT_PRIMARYKEY, SQLITE_CONSTRAINT_UNIQUE, %db-writer-channel): New variables. * tests/database.scm (with-temporary-database): Set "%db-writer-channel". (db-add-build-with-fixed-output): Catch sqlite error. (db-get-pending-derivations): Do not add builds with duplicated outputs.
* Optimize SQLite database.Mathieu Othacehe2020-10-14
| | | | * src/cuirass/database.scm (set-db-options): Optimize database parameters.
* Queue write operations.Mathieu Othacehe2020-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | SQLite only allows one concurrent write query operation. Having multiple database workers calling "db-update-build-status!", will thus increase worker starvation. Every write operation will also be done is a single transaction. For those reasons, create a database worker dedicated to write queries. Have this worker queue work and issue all the queued work queries in a single transaction. * .dir-locals.el: Add with-db-writer-worker-thread. * src/cuirass/database.scm (with-queue-writer-worker): Rename "with-registration-workers" macro. (%db-writer-channel): Rename "%db-registration-channel" variable. (with-queue-writer-worker): Rename "with-registration-workers". (db-register-builds): Use "with-db-writer-worker-thread" instead of "with-db-registration-worker-thread". (db-update-build-status!): Ditto * src/cuirass/utils.scm (make-worker-thread-channel): Add "queue-size" and "queue-proc" arguments. (call-with-worker-thread): Add "options" argument. * bin/cuirass.in (main): Use "with-queue-writer-worker" instead of "with-registration-workers". Modify the macro scope to include all the possible write operations.
* Update metrics in a single transaction.Mathieu Othacehe2020-10-14
| | | | | * src/cuirass/metrics.scm (db-update-metrics): Update all metrics in a single transaction.
* Register new builds in a single transaction.Mathieu Othacehe2020-10-14
| | | | | | | | | | | Registering an evaluation can cause a large number of write queries using individual transactions. This is really sub-optimal as commiting a transaction will often result in a database sync to the disk. * src/cuirass/database.scm (db-add-build): Do not create a new transaction for each new build, instead ... (db-register-builds): ..., create a single transaction per evaluation registration.
* Add missing copyrights.Mathieu Othacehe2020-10-09
| | | | | * src/cuirass/metrics.scm: Add missing copyright. * src/cuirass/watchdog.scm: Ditto.
* Optimize build query.Mathieu Othacehe2020-10-07
| | | | | | | | | | | | | | | | | | | | | | | Make sure that db-get-builds only performs one query and add build products to the result. Also add indexes so that all "order" filters are covered. Limit the maximum of build objects returned in "/api/latestbuilds" and "/api/queue" to 1000. Fixes: <https://issues.guix.gnu.org/43826>. * src/sql/upgrade-15.sql: New file. * Makefile (dist_sql_DATA): Add it. * src/cuirass/database.scm (db-get-builds-by-search): Add "buildproducts" to the returned build objects. (db-get-builds): Remove unused filters. Join Outputs and BuildProducts table in the query. Add "buildproducts" to the returned build objects. * src/cuirass/http.scm (build->hydra-build): Use "buildproducts" from "db-get-builds" procedure. (url-handler): Forbid limit parameters greater than 1000 and use "buildproducts" from "db-get-builds" procedure. * src/schema.sql (Builds_stoptime, Builds_stoptime_id, Builds_status_ts_id): New indexes.
* Add index on builds table.Mathieu Othacehe2020-10-06
| | | | | | * src/sql/upgrade-14.sql: New file. * src/schema.sql (dist_sql_DATA): Add it. * src/schema.sql (Builds_timestamp_stoptime): New index.
* Remove unused modules.Mathieu Othacehe2020-10-06
| | | | * src/cuirass/database.scm: Remove unused modules.
* Do not allow full search on nix_name field.Mathieu Othacehe2020-10-05
| | | | | | | | | | | | | | | | | | Searching for something like "%a%" can take up to ten minutes. Remove the usage of special characters "^" and "$" in queries. Instead, always add a "%" character at the end of the query. Also add an index on the nix_name field. Fixes: <https://issues.guix.gnu.org/43791>. * src/sql/upgrade-13.sql: New file. * Makefile.am (dist_sql_DATA): Add it. * src/schema.sql (Builds_nix_name): New index. * src/cuirass/database.scm (query->bind-arguments): Remove support for "^" and "$" special characters. Instead make sure that the query does not contain any "%" character and prefix the query by "%". (db-get-builds-by-search): Remove an useless query nesting level. * src/cuirass/templates.scm (search-form): Adapt the search help message accordingly.
* Remove another par-map call.Mathieu Othacehe2020-10-03
| | | | | | | | This is a follow-up of 761443bca6178b4ac299a8bd368d1cac4abda5f8. * src/cuirass/base.scm (fetch-inputs): Use map to replace the previous par-map call. (compile-checkouts): Also replace par-map call by map.
* Do not use par-map in fiberized context.Mathieu Othacehe2020-10-03
| | | | | | | Fixes: <https://issues.guix.gnu.org/43757>. * src/cuirass/base.scm (fetch-inputs): Call par-map from a separate thread as this call is not suspendable.
* Add a timeout on database worker query send.Mathieu Othacehe2020-10-03
| | | | | | | | | | | | | | The number of database worker is limited, and can introduce some starvation. Print a message each time a two seconds timeout expires while sending an SQL query to a worker. * src/cuirass/utils.scm (put-message-with-timeout): New procedure. (call-with-worker-thread): Rename timeout and timeout-proc arguments to receive-timeout and receive-timeout-proc arguments. Add two new arguments: send-timeout and send-timeout-proc. Use put-message-with-timeout to honor them. * src/cuirass/database.scm (with-db-worker-thread): Adapt accordingly and pass send-timeout and send-timeout-proc arguments.
* Improve logging message.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/base.scm (build-packages): Improve the logging message.
* Print evaluation number.Mathieu Othacehe2020-10-03
| | | | | * src/cuirass/base.scm (build-packages): Print the id of the registered evaluation.
* Print the caller name in timeout message.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/database.scm (with-db-worker-thread): Print the caller name.
* Add date to SQL queries log file.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/logging.scm (log-query): Add date.
* Do not browse the store during registration.Mathieu Othacehe2020-10-01
| | | | | | * src/cuirass/database.scm (db-register-builds): Remove store argument and assume that 'log and 'outputs properties are provided by the evaluation. * src/cuirass/base.scm (build-packages): Adapt accordingly.
* Add evaluation database workers.Mathieu Othacehe2020-10-01
| | | | | | | | | | | | | | Evaluation registration involves running a large number of SQL queries. This can cause some database worker starvation as well as some contention. To avoid this issue, spawn database workers dedicated to evaluation registration. * src/cuirass/database.scm (%db-registration-channel): New variable. (with-db-registration-worker-thread, with-registration-workers): New macros. (with-db-worker-thread-no-timeout): Remove it. (db-register-builds): Run registration in dedicated database workers using "with-db-registration-worker-thread" macro. * bin/cuirass.in (main): Spawn database registration workers by calling "with-registration-workers" macro.
* Register all new outputs in one worker.Mathieu Othacehe2020-10-01
| | | | | | | | | | | | Make sure that all registration queries are done from within a single database worker. Otherwise, when builds from multiple evaluations are registered at the same time, some contention occurs communicating with workers. * src/cuirass/base.scm (new-outputs?, build-packages): Move build registration to ... * src/cuirass/database.scm (db-register-builds): ... this new procedure. (with-db-worker-thread-no-timeout): New procedure. Use it in "db-register-builds" to avoid timeout messages.
* Expect nix-name and system to be part of the evaluation.Mathieu Othacehe2020-10-01
| | | | | | * src/cuirass/base.scm (evaluate): Expect 'nix-name and 'system properties to be provided by the evaluation. (build-packages): Use the eval-id provided as argument.
* metrics: Optimize queries.Mathieu Othacehe2020-09-29
| | | | | | | | Avoid full Builds table scans. * src/cuirass/metrics.scm (db-average-build-start-time-per-eval, db-average-build-complete-time-per-eval, db-evaluation-completion-speed): Rewrite query to avoid full Builds table scans.
* Add watchdog support.Mathieu Othacehe2020-09-29
| | | | | | | * src/cuirass/watchdog.scm: New file. * Makefile.am (dist_pkgmodule_DATA): Add it. * src/cuirass/utils.scm (with-timeout, get-message-with-timeout): Export them. * bin/cuirass.in (main): Start the watchdog.
* Optimize database queries.Mathieu Othacehe2020-09-28
| | | | | | | | | | | | | Avoid full scans of Builds table that can be very time consuming by rewriting some queries and using new indexes. * src/sql/upgrade-12.sql: New file. * Makefile.am (dist_sql_DATA): Add it. * src/schema.sql (Builds_evaluation_index, Evaluations_status_index, Evaluations_specification_index): New indexes. * src/cuirass/database.scm (db-get-evaluations-build-summary, db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan and use the new indexes.
* Limit builds insertion queries.Mathieu Othacehe2020-09-28
| | | | | | | | | | | | | | | | | | | | | Once the evaluation is over, the new builds are registered. This registration tries to insert a new build for each derivation returned by the evaluation phase. If the new build does not add a new output, the insertion query is then rollbacked. This means that there are at least as many insertion queries as new derivations. SQlite allows at most one writer at a time, and even though we are using WAL mode, performing a lot of insertions will reduce the reading perforances. When multiple evaluations are performed in parallel, the large number of concurrent insertion queries also causes contention. To avoid those issues, check first in the "Outputs" table which derivations are already registered. This means that most of the insertion queries will be replaced by reading queries, that are much less expensive and more suitable for Cuirass concurrent implementation. * src/cuirass/base.scm (new-outputs?): New procedure. (build-packages): Use it to insert only builds registering new outputs.
* Add SQL query logging support.Mathieu Othacehe2020-09-24
| | | | | | | | | * bin/cuirass.in (show-help): Document "--log-queries" option. (%options): Add it. (main): Enable query logging if the above option is set. * src/cuirass/database.scm (db-log-queries): New procedure. * src/cuirass/logging.scm (query-logging-port): New parameter. (log-query): New procedure.
* Fix GC race-condition in spawn-builds.Mathieu Othacehe2020-09-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a derivation is GC'd just before calling "spawn-builds", "build-derivations: will throw an exception that will be catched. Then, "update-build-statuses!" will call "derivation-path->output-paths" that will throw another exception because the derivation does not exit. This exception is not handled, causing Cuirass to crash. 2020-09-18T10:41:18 batch of builds (partially) failed: build of `/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv' failed (status: 1) Backtrace: In ice-9/boot-9.scm: 1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) In unknown file: 10 (apply-smob/0 #<thunk 7ffff5cbd4a0>) In ice-9/boot-9.scm: 718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>) In ice-9/eval.scm: 619:8 8 (_ #(#(#<directory (guile-user) 7ffff58f8f00>))) In ice-9/boot-9.scm: 2806:4 7 (save-module-excursion _) 4351:12 6 (_) In cuirass/base.scm: 562:10 5 (spawn-builds #<store-connection 256.99 7fffec03f910> _ #:max-batch-size _) In srfi/srfi-1.scm: 634:9 4 (for-each #<procedure update! (drv)> ("/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")) In cuirass/base.scm: 474:4 3 (update! "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv") In guix/derivations.scm: 552:17 2 (derivation-path->output-paths "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv") In ice-9/ports.scm: 440:11 1 (call-with-input-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" #<procedure read-derivation (drv-port #:optional read-derivation-from-file)> #:binary _ …) In unknown file: 0 (open-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" "r" #:encoding #f #:guess-encoding #f) * src/cuirass/base.scm (update-build-statuses!): Catch "derivation-path->output-paths" exceptions and set the build status to "failed".
* metrics: Order evaluation metrics by ascending evaluation id.Mathieu Othacehe2020-09-17
| | | | | * src/cuirass/http.scm (metrics-page): Order 'average-eval-build-start-time and 'evaluation-completion-speed metrics per ascending evaluation id.
* metrics: Rename "Builds speed" section to "Builds completion".Mathieu Othacehe2020-09-17
| | | | | * src/cuirass/templates.scm (global-metrics-content): Rename "Builds speed" section to "Builds completion".
* metrics: Add graph legends.Mathieu Othacehe2020-09-17
| | | | * src/cuirass/templates.scm (global-metrics-content): Add legends.
* templates: Do not display chart title.Mathieu Othacehe2020-09-17
| | | | * src/cuirass/templates.scm (make-line-chart): Do not display chart title.
* metrics: Display more metrics.Mathieu Othacehe2020-09-17
| | | | | | | | | | | * src/cuirass/http.scm (url-handler): Move metric page creation to ... (metrics-page): ... this new procedure. Pass average evaluation build start time and evaluation completion speed to "global-metrics-content". * src/cuirass/templates.scm (make-line-chart): Add "time-x-axes?", "xaxes-labels", "x-label" and "y-label" arguents. (global-metrics-content): Add "avg-eval-build-start-time" and "eval-completion-speed" arguments. Create and display two new charts from those metrics.
* metrics: Change evaluation-completion-speed unit.Mathieu Othacehe2020-09-17
| | | | | | * src/cuirass/metrics.scm (db-evaluation-completion-speed): Change the completion speed unit to builds/hour. (%metrics): Adapt comment.
* metrics: Add evaluation related metrics.Mathieu Othacehe2020-09-16
| | | | | | | | | | | | | * src/cuirass/metrics.scm (db-average-build-start-time-per-eval, db-average-build-complete-time-per-eval, db-evaluation-completion-speed, db-latest-evaluations): New procedures. (%metrics): Add 'average-eval-build-start-time, 'average-eval-build-complete-time, 'evaluation-completion-speed. (db-update-metrics): Update evaluation related metrics for the evaluations added the past three days. * tests/metrics.scm ("average-eval-build-start-time", "average-eval-build-complete-time", "evaluation-completion-speed"): Add new tests.
* metrics: Add 'percentage-failed-eval-per-spec support.Mathieu Othacehe2020-09-15
| | | | | | | | | | | | * src/cuirass/metrics.scm (db-percentage-failed-eval-per-spec): New procedure. (%metrics): Add 'percentage-failure-10-last-eval-per-spec, 'percentage-failure-100-last-eval-per-spec and 'percentage-failed-eval-per-spec metrics. (db-update-metrics): Add them. * src/cuirass/templates.scm (global-metrics-content): Add "percentage-failed-eval" argument. Use it to display the "percentage of failed evaluations" table. * src/cuirass/templates.scm (url-handler): Adapt accordingly.
* metrics: Fix db-average-eval-duration-per-spec.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/metrics.scm (db-average-eval-duration-per-spec): Order results by descending rowid and adjust docstring.
* metrics: Warn about metric insertion.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/metrics.scm: Add a warning about metric insertion in %metrics list.
* metrics: Add 'new-derivations-per-day support.Mathieu Othacehe2020-09-15
| | | | | | | | | | | | * src/cuirass/metrics.scm (db-new-derivations-previous-day): New procedure. (%metrics): Add 'new-derivations-per-day. (db-update-metrics): Add it. * src/cuirass/templates.scm (make-line-chart): Add support for multiple datasets. Also add "interpolation?" and "legend?" options. (global-metrics-content): Add "new-derivations-per-day" argument. Adapt "Builds per day" chart so that it uses two datasets from 'new-derivations-per-day and 'builds-per-day metrics. * src/cuirass/templates.scm (url-handler): Adapt accordingly.
* metrics: Change 'builds-per-day computation.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/metrics.scm (db-builds-previous-day): Only take into account the builds that are created and processed during the previous day.
* metrics: Add pending builds support.Mathieu Othacehe2020-09-15
| | | | | | | | | | | * src/cuirass/metrics.scm (db-pending-builds, db-current-day-timestamp): New procedures. (%metrics): Add 'pending-builds. (db-update-metrics): Add it. * src/cuirass/templates.scm (global-metrics-content): Add "pending-builds" argument. Display pending builds in a line chart. * src/cuirass/http.scm (url-handler): Adapt "global-metrics-content" call accordingly.
* Do not override onload callback.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/templates.scm (make-line-chart): Do not override onload callback as that would prevent us from adding multiple charts.
* metrics: Rename 'builds-previous-day to 'builds-per-day.Mathieu Othacehe2020-09-15
| | | | | | | * src/cuirass/metrics.scm (%metrics): Rename 'builds-previous-day to 'builds-per-day. (db-update-metrics): Adapt accordingly. * src/cuirass/http.scm (url-handler): Ditto.
* metrics: Order by descending rowid by default.Mathieu Othacehe2020-09-15
| | | | | | | * src/cuirass/metrics.scm (db-get-metrics-with-id): Order by descending rowid by default. * src/cuirass/http.scm (url-handler): Use the default order for "db-get-metrics-with-id".
* Add metrics support.Mathieu Othacehe2020-09-14
| | | | | | | | | | | | | | * Makefile.am: Add "upgrade-11.sql", "chart.js" and "metrics.js". * bin/cuirass.in (main): Add 'metrics fiber. * src/cuirass/http.scm (%file-white-list): Add "js/chart.js". (url-handler): Add "/metrics" route. * src/cuirass/metrics.scm: New file. * src/cuirass/templates.scm (html-page): Add a "Global metrics" sub-menu to "Status" dropdown menu. (make-line-chart, global-metrics-content): New procedures. * src/schema.sql (Metrics): New table. * src/sql/upgrade-11.sql: New file. * src/static/js/chart.js: New file.
* database: Export procedures.Mathieu Othacehe2020-09-14
| | | | | * src/cuirass/database.scm (last-insert-rowid, expect-one-row, %sqlite-exec): Export them.
* Add a status field to Evaluation table.Mathieu Othacehe2020-09-10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The Evaluation table currently has an 'in_progress' field. Distinction between succeeded and failed evaluations are based on the presence of Builds records for the evaluation. It it also not possible to distinguish aborted evaluations from failed evaluations. Rename 'in_progress' field to 'status'. The 'status' field can be equal to 'started', 'succeeded', 'failed' or 'aborted'. * src/cuirass/database.scm (evaluation-status): New exported enumeration. (db-set-evaluations-done, db-set-evaluation-done): Remove them. (db-abort-pending-evaluations, db-set-evaluation-status): New exported procedures. (db-add-evaluation, db-get-builds, db-get-evaluations, db-get-evaluations-build-summary, db-get-evaluation-summary): Adapt to use 'status' field instead of 'in_progress' field. * src/cuirass/templates.scm (evaluation-badges): Ditto. * src/schema.sql (Evaluations): Rename 'in_progress' field to 'status'. * src/sql/upgrade-10.sql: New file. * bin/cuirass.in (main): Use "db-abort-pending-evaluations" instead of "db-set-evaluations-done". * src/cuirass/base.scm (evaluate): Use "db-set-evaluation-status" instead of "db-set-evaluations-done". (build-packages): Use "db-set-evaluation-status" instead of "db-set-evaluation-done". * tests/database.scm (sqlite-exec): Adapt accordingly. * tests/http.scm (evaluations-query-result): Ditto.
generated by cgit v1.2.3 (git 2.50.1) at 2025-10-21 04:55:37 +0000