aboutsummaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAge
* Fix using sqlite-reset in a few queriessqlite-statement-reset-fixesChristopher Baines2020-12-06
| | | | | | | | | | Rather than calling sqlite-reset before reading the rows, call sqlite-reset after. I think this is important to stop SQLite getting stuck because the statement is still live even though it's not being read from after these procedures return. Assuming I've got the ordering right, I think using sqlite-fold-right directly is simpler code as well.
* Remove redundant reset callsChristopher Baines2020-12-06
| | | | | sqlite-prepare will reset cached statements before returning them, so these sqlite-reset calls are redundant.
* Don't finalise cached statementsChristopher Baines2020-12-06
| | | | | All these statements are cached, but that's pointless if they're finalized after use, because they'll need recreating on next use.
* database: Fix builds pagination.Mathieu Othacehe2020-12-02
| | | | * src/cuirass/database.scm (db-get-builds): Fix pagination.
* database: Rewrite db-get-builds-max and db-get-builds-min.Mathieu Othacehe2020-12-02
| | | | | * src/cuirass/database.scm (db-get-builds-min, db-get-builds-max): Rewrite SQL queries without nesting.
* Use specification for periodic builds.Mathieu Othacehe2020-11-25
| | | | | | | | | | | | | | When the periodic argument is passed to a job, it will be registered only if the time difference between the current time and the registration time of the last build is greater than the specified period. Make sure that the last job that is searched is part of the same specification. * src/sql/upgrade-16.sql: New file. * Makefile.am (dist_sql_DATA): Add it. * src/schema.sql (Builds_job_name_timestamp): New index. * src/cuirass/database.scm (db-get-time-since-previous-build): Add "specification" argument, (db-register-builds): pass it.
* Fix interactive checkout.Mathieu Othacehe2020-11-25
| | | | | | | This is a work-around: https://issues.guix.gnu.org/44742. * src/cuirass/base.scm (fetch-inputs): Set "current-error-port" parameter to a void port to prevent the fetch progression bar to appear.
* Add periodical build support.Mathieu Othacehe2020-11-25
| | | | | | * src/cuirass/database.scm (db-get-time-since-previous-build): New procedure, (db-register-builds): if the period argument is set, only register builds which last registration is older than the specified period.
* Revert "Use a non-blocking socket for store communication."Mathieu Othacehe2020-11-02
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 387909454c835c994414aa740a2d33d288064158 that causes: 2020-11-02T11:05:08 fatal: uncaught exception 'wrong-type-arg' in 'build' fiber! 2020-11-02T11:05:08 exception arguments: ("struct-vtable" "Wrong type argument in position 1 (expecting struct): ~S" (#f) (#f)) In ice-9/boot-9.scm: 1731:15 12 (with-exception-handler #<procedure 7fb1a93f9930 at ic…> …) 1736:10 11 (with-exception-handler _ _ #:unwind? _ # _) 718:2 10 (call-with-prompt ("break") #<procedure 7fb1ab76f440 a…> …) 718:2 9 (call-with-prompt ("continue") #<procedure 7fb1ab77084…> …) In ice-9/eval.scm: 619:8 8 (_ #(#(#<directory (guile-user) 7fb1ac680f00> #<var…> …))) In srfi/srfi-1.scm: 634:9 7 (for-each #<procedure 7fb1a9525900 at cuirass/base.scm…> …) In ice-9/boot-9.scm: 1731:15 6 (with-exception-handler #<procedure 7fb1a95a94e0 at ic…> …) 1669:16 5 (raise-exception _ #:continuable? _) 1764:13 4 (_ #<&compound-exception components: (#<&assertion-fail…>) In cuirass/utils.scm: 319:8 3 (_ _ . _) In ice-9/boot-9.scm: 1731:15 2 (with-exception-handler #<procedure 7fb1ab2e3720 at ic…> …) In cuirass/utils.scm: 320:22 1 (_) In unknown file: 0 (make-stack #t) ERROR: In procedure make-stack: In procedure struct-vtable: Wrong type argument in position 1 (expecting struct): #f
* Use a non-blocking socket for store communication.Ludovic Courtès2020-10-27
| | | | | | | | Set the store socket as non-blocking so that fibers communicating with the store don't get blocked as described here: https://issues.guix.gnu.org/43565. * src/cuirass/base.scm (with-store): Set the store socket as non-blocking. (build-derivations&): Unset current-read-waiter and current-write-waiter.
* Use with-db-writer-worker-thread/force macro.Mathieu Othacehe2020-10-15
| | | | | * src/cuirass/database.scm (db-register-builds): Use "with-db-writer-worker-thread/force" macro.
* Use the writer worker for all write queries.Mathieu Othacehe2020-10-15
| | | | | | | | | | | | | | | | | * .dir-locals.el: Add "with-queue-writer-worker". * bin/cuirass.in: Modify "with-queue-writer-worker" scope to include the web-server operations. * src/cuirass/database.scm (with-db-writer-worker-thread): Export it. (with-db-writer-worker-thread/force): New macro. (db-add-input, db-add-checkout, db-add-specification, db-remove-specification, db-add-evaluation, db-abort-pending-evaluations, db-set-evaluation-status, db-set-evaluation-time, db-add-output, db-add-build-product, db-add-event, db-delete-events-with-ids-<=-to): Use "with-db-writer-worker-thread" or "with-db-writer-worker-thread/force" instead of "with-db-worker-thread". * src/cuirass/metrics.scm (db-update-metrics): Ditto. * tests/database.scm ("db-init"): Set "%db-writer-channel". * tests/http.scm ("db-init"): Ditto. * tests/metrics.scm ("db-init"): Ditto.
* Remove unused procedure.Mathieu Othacehe2020-10-14
| | | | * src/cuirass/base.scm (cancel-old-builds): Remove it.
* Fix tests.Mathieu Othacehe2020-10-14
| | | | | | | | | | | This is a follow-up of b67f38a7b91c8605a3ae9eba1e2bd3da4b579622. * src/cuirass/database.scm (catch-sqlite-error): New macro. (SQLITE_CONSTRAINT_PRIMARYKEY, SQLITE_CONSTRAINT_UNIQUE, %db-writer-channel): New variables. * tests/database.scm (with-temporary-database): Set "%db-writer-channel". (db-add-build-with-fixed-output): Catch sqlite error. (db-get-pending-derivations): Do not add builds with duplicated outputs.
* Optimize SQLite database.Mathieu Othacehe2020-10-14
| | | | * src/cuirass/database.scm (set-db-options): Optimize database parameters.
* Queue write operations.Mathieu Othacehe2020-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | SQLite only allows one concurrent write query operation. Having multiple database workers calling "db-update-build-status!", will thus increase worker starvation. Every write operation will also be done is a single transaction. For those reasons, create a database worker dedicated to write queries. Have this worker queue work and issue all the queued work queries in a single transaction. * .dir-locals.el: Add with-db-writer-worker-thread. * src/cuirass/database.scm (with-queue-writer-worker): Rename "with-registration-workers" macro. (%db-writer-channel): Rename "%db-registration-channel" variable. (with-queue-writer-worker): Rename "with-registration-workers". (db-register-builds): Use "with-db-writer-worker-thread" instead of "with-db-registration-worker-thread". (db-update-build-status!): Ditto * src/cuirass/utils.scm (make-worker-thread-channel): Add "queue-size" and "queue-proc" arguments. (call-with-worker-thread): Add "options" argument. * bin/cuirass.in (main): Use "with-queue-writer-worker" instead of "with-registration-workers". Modify the macro scope to include all the possible write operations.
* Update metrics in a single transaction.Mathieu Othacehe2020-10-14
| | | | | * src/cuirass/metrics.scm (db-update-metrics): Update all metrics in a single transaction.
* Register new builds in a single transaction.Mathieu Othacehe2020-10-14
| | | | | | | | | | | Registering an evaluation can cause a large number of write queries using individual transactions. This is really sub-optimal as commiting a transaction will often result in a database sync to the disk. * src/cuirass/database.scm (db-add-build): Do not create a new transaction for each new build, instead ... (db-register-builds): ..., create a single transaction per evaluation registration.
* Add missing copyrights.Mathieu Othacehe2020-10-09
| | | | | * src/cuirass/metrics.scm: Add missing copyright. * src/cuirass/watchdog.scm: Ditto.
* Optimize build query.Mathieu Othacehe2020-10-07
| | | | | | | | | | | | | | | | | | | | | | | Make sure that db-get-builds only performs one query and add build products to the result. Also add indexes so that all "order" filters are covered. Limit the maximum of build objects returned in "/api/latestbuilds" and "/api/queue" to 1000. Fixes: <https://issues.guix.gnu.org/43826>. * src/sql/upgrade-15.sql: New file. * Makefile (dist_sql_DATA): Add it. * src/cuirass/database.scm (db-get-builds-by-search): Add "buildproducts" to the returned build objects. (db-get-builds): Remove unused filters. Join Outputs and BuildProducts table in the query. Add "buildproducts" to the returned build objects. * src/cuirass/http.scm (build->hydra-build): Use "buildproducts" from "db-get-builds" procedure. (url-handler): Forbid limit parameters greater than 1000 and use "buildproducts" from "db-get-builds" procedure. * src/schema.sql (Builds_stoptime, Builds_stoptime_id, Builds_status_ts_id): New indexes.
* Add index on builds table.Mathieu Othacehe2020-10-06
| | | | | | * src/sql/upgrade-14.sql: New file. * src/schema.sql (dist_sql_DATA): Add it. * src/schema.sql (Builds_timestamp_stoptime): New index.
* Remove unused modules.Mathieu Othacehe2020-10-06
| | | | * src/cuirass/database.scm: Remove unused modules.
* Do not allow full search on nix_name field.Mathieu Othacehe2020-10-05
| | | | | | | | | | | | | | | | | | Searching for something like "%a%" can take up to ten minutes. Remove the usage of special characters "^" and "$" in queries. Instead, always add a "%" character at the end of the query. Also add an index on the nix_name field. Fixes: <https://issues.guix.gnu.org/43791>. * src/sql/upgrade-13.sql: New file. * Makefile.am (dist_sql_DATA): Add it. * src/schema.sql (Builds_nix_name): New index. * src/cuirass/database.scm (query->bind-arguments): Remove support for "^" and "$" special characters. Instead make sure that the query does not contain any "%" character and prefix the query by "%". (db-get-builds-by-search): Remove an useless query nesting level. * src/cuirass/templates.scm (search-form): Adapt the search help message accordingly.
* Remove another par-map call.Mathieu Othacehe2020-10-03
| | | | | | | | This is a follow-up of 761443bca6178b4ac299a8bd368d1cac4abda5f8. * src/cuirass/base.scm (fetch-inputs): Use map to replace the previous par-map call. (compile-checkouts): Also replace par-map call by map.
* Do not use par-map in fiberized context.Mathieu Othacehe2020-10-03
| | | | | | | Fixes: <https://issues.guix.gnu.org/43757>. * src/cuirass/base.scm (fetch-inputs): Call par-map from a separate thread as this call is not suspendable.
* Add a timeout on database worker query send.Mathieu Othacehe2020-10-03
| | | | | | | | | | | | | | The number of database worker is limited, and can introduce some starvation. Print a message each time a two seconds timeout expires while sending an SQL query to a worker. * src/cuirass/utils.scm (put-message-with-timeout): New procedure. (call-with-worker-thread): Rename timeout and timeout-proc arguments to receive-timeout and receive-timeout-proc arguments. Add two new arguments: send-timeout and send-timeout-proc. Use put-message-with-timeout to honor them. * src/cuirass/database.scm (with-db-worker-thread): Adapt accordingly and pass send-timeout and send-timeout-proc arguments.
* Improve logging message.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/base.scm (build-packages): Improve the logging message.
* Print evaluation number.Mathieu Othacehe2020-10-03
| | | | | * src/cuirass/base.scm (build-packages): Print the id of the registered evaluation.
* Print the caller name in timeout message.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/database.scm (with-db-worker-thread): Print the caller name.
* Add date to SQL queries log file.Mathieu Othacehe2020-10-03
| | | | * src/cuirass/logging.scm (log-query): Add date.
* Do not browse the store during registration.Mathieu Othacehe2020-10-01
| | | | | | * src/cuirass/database.scm (db-register-builds): Remove store argument and assume that 'log and 'outputs properties are provided by the evaluation. * src/cuirass/base.scm (build-packages): Adapt accordingly.
* Add evaluation database workers.Mathieu Othacehe2020-10-01
| | | | | | | | | | | | | | Evaluation registration involves running a large number of SQL queries. This can cause some database worker starvation as well as some contention. To avoid this issue, spawn database workers dedicated to evaluation registration. * src/cuirass/database.scm (%db-registration-channel): New variable. (with-db-registration-worker-thread, with-registration-workers): New macros. (with-db-worker-thread-no-timeout): Remove it. (db-register-builds): Run registration in dedicated database workers using "with-db-registration-worker-thread" macro. * bin/cuirass.in (main): Spawn database registration workers by calling "with-registration-workers" macro.
* Register all new outputs in one worker.Mathieu Othacehe2020-10-01
| | | | | | | | | | | | Make sure that all registration queries are done from within a single database worker. Otherwise, when builds from multiple evaluations are registered at the same time, some contention occurs communicating with workers. * src/cuirass/base.scm (new-outputs?, build-packages): Move build registration to ... * src/cuirass/database.scm (db-register-builds): ... this new procedure. (with-db-worker-thread-no-timeout): New procedure. Use it in "db-register-builds" to avoid timeout messages.
* Expect nix-name and system to be part of the evaluation.Mathieu Othacehe2020-10-01
| | | | | | * src/cuirass/base.scm (evaluate): Expect 'nix-name and 'system properties to be provided by the evaluation. (build-packages): Use the eval-id provided as argument.
* metrics: Optimize queries.Mathieu Othacehe2020-09-29
| | | | | | | | Avoid full Builds table scans. * src/cuirass/metrics.scm (db-average-build-start-time-per-eval, db-average-build-complete-time-per-eval, db-evaluation-completion-speed): Rewrite query to avoid full Builds table scans.
* Add watchdog support.Mathieu Othacehe2020-09-29
| | | | | | | * src/cuirass/watchdog.scm: New file. * Makefile.am (dist_pkgmodule_DATA): Add it. * src/cuirass/utils.scm (with-timeout, get-message-with-timeout): Export them. * bin/cuirass.in (main): Start the watchdog.
* Optimize database queries.Mathieu Othacehe2020-09-28
| | | | | | | | | | | | | Avoid full scans of Builds table that can be very time consuming by rewriting some queries and using new indexes. * src/sql/upgrade-12.sql: New file. * Makefile.am (dist_sql_DATA): Add it. * src/schema.sql (Builds_evaluation_index, Evaluations_status_index, Evaluations_specification_index): New indexes. * src/cuirass/database.scm (db-get-evaluations-build-summary, db-get-evaluation-summary): Rewrite queries to avoid full Builds table scan and use the new indexes.
* Limit builds insertion queries.Mathieu Othacehe2020-09-28
| | | | | | | | | | | | | | | | | | | | | Once the evaluation is over, the new builds are registered. This registration tries to insert a new build for each derivation returned by the evaluation phase. If the new build does not add a new output, the insertion query is then rollbacked. This means that there are at least as many insertion queries as new derivations. SQlite allows at most one writer at a time, and even though we are using WAL mode, performing a lot of insertions will reduce the reading perforances. When multiple evaluations are performed in parallel, the large number of concurrent insertion queries also causes contention. To avoid those issues, check first in the "Outputs" table which derivations are already registered. This means that most of the insertion queries will be replaced by reading queries, that are much less expensive and more suitable for Cuirass concurrent implementation. * src/cuirass/base.scm (new-outputs?): New procedure. (build-packages): Use it to insert only builds registering new outputs.
* Add SQL query logging support.Mathieu Othacehe2020-09-24
| | | | | | | | | * bin/cuirass.in (show-help): Document "--log-queries" option. (%options): Add it. (main): Enable query logging if the above option is set. * src/cuirass/database.scm (db-log-queries): New procedure. * src/cuirass/logging.scm (query-logging-port): New parameter. (log-query): New procedure.
* Fix GC race-condition in spawn-builds.Mathieu Othacehe2020-09-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a derivation is GC'd just before calling "spawn-builds", "build-derivations: will throw an exception that will be catched. Then, "update-build-statuses!" will call "derivation-path->output-paths" that will throw another exception because the derivation does not exit. This exception is not handled, causing Cuirass to crash. 2020-09-18T10:41:18 batch of builds (partially) failed: build of `/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv' failed (status: 1) Backtrace: In ice-9/boot-9.scm: 1736:10 11 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _) In unknown file: 10 (apply-smob/0 #<thunk 7ffff5cbd4a0>) In ice-9/boot-9.scm: 718:2 9 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>) In ice-9/eval.scm: 619:8 8 (_ #(#(#<directory (guile-user) 7ffff58f8f00>))) In ice-9/boot-9.scm: 2806:4 7 (save-module-excursion _) 4351:12 6 (_) In cuirass/base.scm: 562:10 5 (spawn-builds #<store-connection 256.99 7fffec03f910> _ #:max-batch-size _) In srfi/srfi-1.scm: 634:9 4 (for-each #<procedure update! (drv)> ("/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv")) In cuirass/base.scm: 474:4 3 (update! "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv") In guix/derivations.scm: 552:17 2 (derivation-path->output-paths "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv") In ice-9/ports.scm: 440:11 1 (call-with-input-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" #<procedure read-derivation (drv-port #:optional read-derivation-from-file)> #:binary _ …) In unknown file: 0 (open-file "/gnu/store/zrmxzjf025nc89a7vdy5i94zavprc7fs-emacs-guix-0.5.2-2.58a840d.drv" "r" #:encoding #f #:guess-encoding #f) * src/cuirass/base.scm (update-build-statuses!): Catch "derivation-path->output-paths" exceptions and set the build status to "failed".
* metrics: Order evaluation metrics by ascending evaluation id.Mathieu Othacehe2020-09-17
| | | | | * src/cuirass/http.scm (metrics-page): Order 'average-eval-build-start-time and 'evaluation-completion-speed metrics per ascending evaluation id.
* metrics: Rename "Builds speed" section to "Builds completion".Mathieu Othacehe2020-09-17
| | | | | * src/cuirass/templates.scm (global-metrics-content): Rename "Builds speed" section to "Builds completion".
* metrics: Add graph legends.Mathieu Othacehe2020-09-17
| | | | * src/cuirass/templates.scm (global-metrics-content): Add legends.
* templates: Do not display chart title.Mathieu Othacehe2020-09-17
| | | | * src/cuirass/templates.scm (make-line-chart): Do not display chart title.
* metrics: Display more metrics.Mathieu Othacehe2020-09-17
| | | | | | | | | | | * src/cuirass/http.scm (url-handler): Move metric page creation to ... (metrics-page): ... this new procedure. Pass average evaluation build start time and evaluation completion speed to "global-metrics-content". * src/cuirass/templates.scm (make-line-chart): Add "time-x-axes?", "xaxes-labels", "x-label" and "y-label" arguents. (global-metrics-content): Add "avg-eval-build-start-time" and "eval-completion-speed" arguments. Create and display two new charts from those metrics.
* metrics: Change evaluation-completion-speed unit.Mathieu Othacehe2020-09-17
| | | | | | * src/cuirass/metrics.scm (db-evaluation-completion-speed): Change the completion speed unit to builds/hour. (%metrics): Adapt comment.
* metrics: Add evaluation related metrics.Mathieu Othacehe2020-09-16
| | | | | | | | | | | | | * src/cuirass/metrics.scm (db-average-build-start-time-per-eval, db-average-build-complete-time-per-eval, db-evaluation-completion-speed, db-latest-evaluations): New procedures. (%metrics): Add 'average-eval-build-start-time, 'average-eval-build-complete-time, 'evaluation-completion-speed. (db-update-metrics): Update evaluation related metrics for the evaluations added the past three days. * tests/metrics.scm ("average-eval-build-start-time", "average-eval-build-complete-time", "evaluation-completion-speed"): Add new tests.
* metrics: Add 'percentage-failed-eval-per-spec support.Mathieu Othacehe2020-09-15
| | | | | | | | | | | | * src/cuirass/metrics.scm (db-percentage-failed-eval-per-spec): New procedure. (%metrics): Add 'percentage-failure-10-last-eval-per-spec, 'percentage-failure-100-last-eval-per-spec and 'percentage-failed-eval-per-spec metrics. (db-update-metrics): Add them. * src/cuirass/templates.scm (global-metrics-content): Add "percentage-failed-eval" argument. Use it to display the "percentage of failed evaluations" table. * src/cuirass/templates.scm (url-handler): Adapt accordingly.
* metrics: Fix db-average-eval-duration-per-spec.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/metrics.scm (db-average-eval-duration-per-spec): Order results by descending rowid and adjust docstring.
* metrics: Warn about metric insertion.Mathieu Othacehe2020-09-15
| | | | | * src/cuirass/metrics.scm: Add a warning about metric insertion in %metrics list.
generated by cgit v1.2.3 (git 2.45.0) at 2024-11-22 07:44:52 +0000