| Commit message (Expand) | Author | Age |
|---|
| * | Begin tuning db-get-builds for performance•••Now that I actually have some experience using SQLite from writing the Guix
Build Coordinator, I can actually see potential ways to improve bits of
Cuirass.
The intention here is to start to address the performance of db-get-builds,
which I think can be quite slow.
This commit does several things, the big change is to try and construct a
simpler query for SQLite. I'm not confident that SQLite's query planner can
look past handling the NULL parameters, so I think it could be helpful to try
and create a simpler query, both to avoid that problem if it exists, but also
move the complexity in to Guile code, which I think is a bit more manageable.
The way ordering is handled is also changed. Order is one of the filters,
although it's not a filter, and some of the other filters also influenced the
order. I think there are things still to fix/improve with the handling of
ordering, but at least this commit just has the ordering happen once in the
query.
If this direction sounds OK, I think the next things to do is do a bit more
testing, both to check uses of this query still work as intended, but also to
take those specific queries generated by db-get-builds, and check what SQLite
says when running them with EXPLAIN QUERY PLAN, as that should inform further
improvements, and what indexes to create.
db-get-builds-performance-tuning | Christopher Baines | 2020-05-25 |
| * | build-aux: Update for Guile 3.•••* build-aux/guix.scm (arguments): Change 2.2 to 3.0.
(inputs): Change guile@2.2 to guile.
| Christopher Baines | 2020-05-25 |
| * | templates: Evaluation page links to VCS web view.•••* src/cuirass/templates.scm (%vcs-web-views): New variable.
(commit-hyperlink): New procedure.
(evaluation-build-table): Use it in the input/commit table.
| Ludovic Courtès | 2020-04-17 |
| * | templates: Evaluation page shows each input and commit.•••* src/cuirass/database.scm (db-get-inputs, db-get-checkouts): Export.
* src/cuirass/http.scm (evaluation-html-page): Pass #:checkouts and
#:inputs to 'evaluation-build-table'.
* src/cuirass/templates.scm (evaluation-build-table): Add #:checkouts
and #:inputs. Emit a table with "Input" and "Commit" columns.
| Ludovic Courtès | 2020-04-17 |
| * | database: Add missing (ice-9 threads) import.•••This is required on Guile 3 for 'current-processor-count'.
* src/cuirass/database.scm: Use (ice-9 threads).
| Ludovic Courtès | 2020-04-17 |
| * | templates: 'time->string' doesn't show timezone for old builds.•••* src/cuirass/templates.scm (time->string): Remove ~z for old builds.
| Ludovic Courtès | 2020-04-17 |
| * | templates: Fix typo in 'time->string'.•••This is a followup to a1d353b1527fd1a2bfa2a1cbb2f0ff5c79d6e973.
Until now we'd never display the year.
* src/cuirass/templates.scm (time->string): Check for (= year current).
| Ludovic Courtès | 2020-04-17 |
| * | templates: html-page: Fix indentation.•••* src/cuirass/templates.scm (html-page): Fix indentation.
| Ricardo Wurmus | 2020-04-15 |
| * | templates: Display search hints.•••* src/static/css/cuirass.css: New file.
* Makefile.am (dist_css_DATA): Add it.
* src/cuirass/http.scm (%file-white-list): Add css/cuirass.css.
* src/cuirass/templates.scm (search-form): Add search-hints div.
(html-page): Load cuirass.css.
| Ricardo Wurmus | 2020-04-15 |
| * | database: Support "status:" keyword in queries.•••* src/cuirass/database.scm (query->bind-arguments): Handle "status" query.
(db-get-builds-by-search, db-get-builds-query-min, db-get-builds-query-max):
Add status filter.
| Ricardo Wurmus | 2020-04-15 |
| * | http: Move "/eval" page to (cuirass templates).•••* src/cuirass/http.scm (url-handler): Move inline code for ('GET "eval" id)
to...
(evaluation-html-page): ... here. New procedure.
* src/cuirass/templates.scm (evaluation-build-table): New procedure.
| Ludovic Courtès | 2020-02-24 |
| * | templates: Build page links to corresponding evaluation.•••* src/cuirass/templates.scm (build-details): Define 'evaluation'. Add
"Evaluation" row with a link to EVALUATION.
| Ludovic Courtès | 2020-02-24 |
| * | database: 'db-get-builds' returns each build's evaluation ID.•••* src/cuirass/database.scm (db-get-builds): Fetch 'Builds.evaluation'
and return it as #:eval-id.
| Ludovic Courtès | 2020-02-24 |
| * | utils: Handle errors in worker threads.•••Previously, if an error occurred, the worker fiber simply never sends a
reply. In the case of HTTP requests to Cuirass, where an exception occurs when
performing a database query, the fiber handling the request blocks as it never
gets a response. I think that this has the potential to cause the process to
hit file descriptor limits, as the connections are never responded to.
This is fixed by responding with the details of the exception, and then
throwing it within the fiber that made the call.
* src/cuirass/utils.scm (make-worker-thread-channel): Catch exceptions when
calling proc.
(call-with-worker-thread): Handle receiving exceptions from the worker thread.
handle-errors-in-worker-threads | Christopher Baines | 2020-02-05 |
| * | database: Enable running up to 4 database queries at once.•••The number of threads is copied from bin/cuirass.in. When you have at least
two processors, this will allow database queries to be executed in parallel.
With some crude testing using the Apache HTTP server benchmarking tool (ab
from the httpd package), the max request latency does seem to drop when
multiple threads are used, especially when the database queries are slow (I
tested by adding usleep to the worker thread code).
* src/cuirass/database.scm (with-database): Pass #:parallelism to
make-worker-thread-channel.
allow-concurrent-database-queries | Christopher Baines | 2020-01-25 |
| * | Enable make-worker-thread-channel to create multiple worker threads.•••This will allow running multiple threads, that all listen on the same channel,
enabling processing multiple jobs at one time.
* src/cuirass/utils.scm (make-worker-thread-channel): Add a #:parallelism
argument, and create as many threads as the given parallelism.
| Christopher Baines | 2020-01-25 |
| * | Adjust make-worker-thread-channel to take an initializer.•••While this is a generic method, and initializer function will give the
flexibility required to create multiple worker threads for performing SQLite
queries, each with it's own database connection (as a result of calling the
initializer once for each thread). Without this change, they'd all have to use
the same connection, which would not work.
* src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take
an initializer, rather than arguments directly.
* src/cuirass/database.scm (with-database): Adjust to call
make-worker-thread-channel with an initializer.
* tests/database.scm (db-init): Change to use make-worker-thread-channel
initializer.
* tests/http.scm (db-init): Change to use make-worker-thread-channel
initializer.
| Christopher Baines | 2020-01-25 |
| * | utils: Change critical section terminology to worker threads.•••As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.
This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.
These changes start to move in this direction, first by just changing the
terminology.
* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
| Christopher Baines | 2020-01-25 |
| * | Alter the Builds table to have an id field•••The internal rowid's are used for builds as you can request builds by using
the rowid in the URL.
The motivation here is to enable running VACUUM operations in SQLite, without
risking the rowid's for Builds changing. It would be bad if they change, as
they're used in the URL's for builds.
* src/schema.sql (Builds): Add id column.
* src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to
UNIQUE constraint.
* src/sql/upgrade-6.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
improve-handling-of-rowids | Christopher Baines | 2020-01-25 |
| * | database: Don't return rowid from db-add-input.•••As it is unused from where db-add-input is called.
* src/cuirass/database.scm (db-add-input): Don't call and
return (last-insert-rowid).
| Christopher Baines | 2020-01-24 |
| * | build: Allow builds with Guile 3.0.•••* configure.ac: Add "3.0" to 'GUILE_PKG'.
* README: Mention it.
| Ludovic Courtès | 2020-01-18 |
| * | http: Add missing import of (ice-9 threads) in test.•••Using the core binding for 'call-with-new-thread' was deprecated in 2.2
and is removed in 3.0.
* tests/http.scm: Use (ice-9 threads).
| Ludovic Courtès | 2020-01-18 |
| * | Support returning build information by output.•••Being able to take a derivation and query the build information is useful, but
in cases where there are multiple derivations that produce the same outputs,
the probability of getting the data back from Cuirass is reduced.
This is because Cuirass might not have build the exact derivation you have,
but a different derivation that produces the same outputs (this can commonly
happen when a related fixed output derivation changes).
Cuirass doesn't store derivations if they produce the same outputs as a
derivation it already knows about, so it can't determine if this is the
case. Therefore, provide a way of querying build results by output, rather
than derivation.
The motivation behind this is to make it easier to import build information in
to the Guix Data Service.
* src/cuirass/database.scm (db-get-output): New procedure.
* src/cuirass/http.scm (respond-output-not-found): New procedure.
(request-path-components): Handle /output/… requests.
* doc/cuirass.texi (Build information): Mention that you can get build
information by output.
| Christopher Baines | 2020-01-16 |
| * | Support publishing evaluation events•••* src/cuirass/database.scm (db-add-evaluation): Record the creation of new
evaluations as events.
(db-set-evaluation-done): Record when evaluations finish as an event.
* src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
| Christopher Baines | 2020-01-16 |
| * | Support publishing build events•••Add a table to store events, which have a type and a JSON blob. These can be
used to record changes, this commit inserts events when new builds are
created, and when the status of builds change.
The EventsOutbox table is then used to track when events have been sent
out. This is done through the new cuirass-send-events script.
* Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events.
.gitignore: Add bin/cuirass-send-events.
(dist_pkgmodule_DATA): Add src/cuirass/send-events.scm.
(dist_sql_DATA): Add src/sql/upgrade-5.sql.
(EXTRA_DIST): bin/cuirass-send-events.in.
(bin/cuirass-send-events): New rule.
* bin/cuirass-send-events.in: New file.
* src/cuirass/send-events.scm: New file.
* src/sql/upgrade-5.sql: New file.
* src/cuirass/database.scm (changes-count): New procedure.
(db-update-build-status!): Call db-add-event after updating the build status.
(db-add-event): New procedure.
(db-add-build): Insert an event when a new build is inserted.
(db-delete-events-with-ids-<=-to): New procedure.
* src/schema.sql (Events): New table.
| Christopher Baines | 2020-01-16 |
| * | base: 'build-derivations&' closes the build output port only once.•••* src/cuirass/base.scm (build-derivations&): Remove extra 'close-port'
call in 'guard' clause.
| Ludovic Courtès | 2019-11-03 |
| * | tests: Remove 405 test.•••This is a follow-up to commit c88a7c006ef164015e5c3fb1544b8112bf6ececf.
* tests/http.scm: Remove test for invalid POST to /build/1.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Add route for /admin.•••* src/cuirass/http.scm (url-handler): Handle /admin route.
| Ricardo Wurmus | 2019-10-30 |
| * | bin: Remove redundant lambda.•••* bin/cuirass.in (main): Remove redundant lambda wrapping in for-each.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Handle specification deletion and addition.•••* src/cuirass/http.scm (url-handler): Handle /admin/specifications/add,
/admin/specifications/delete/*, and /admin/specifications.
| Ricardo Wurmus | 2019-10-30 |
| * | templates: specifications-table: Add admin features.•••* src/cuirass/templates.scm (specifications-table): Handle optional ADMIN?
argument.
| Ricardo Wurmus | 2019-10-30 |
| * | database: Add db-remove-specification.•••* src/cuirass/database.scm (db-remove-specification): New procedure.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Be explicit about accepted HTTP methods.•••* src/cuirass/http.scm (url-handler): Match on HTTP method.
| Ricardo Wurmus | 2019-10-30 |
| * | templates: Failed evaluations link to their log.•••* src/cuirass/templates.scm (evaluation-badges): Link to
/eval/ID/log/raw in the 'zero?' case.
| Ludovic Courtès | 2019-10-23 |
| * | http: Serve evaluation logs at /eval/ID/log/raw.•••* src/cuirass/base.scm (evaluation-log-file): Export.
* src/cuirass/http.scm (url-handler)[respond-gzipped-file]: New
procedure.
Add handler for /eval/ID/log/raw.
| Ludovic Courtès | 2019-10-23 |
| * | base: Log the evaluation ID upon evaluation failure.•••* src/cuirass/base.scm (&evaluation-error)[id]: New field.
(evaluate): Specify the 'id' field.
(process-specs): Show the log file name upon evaluation failure.
| Ludovic Courtès | 2019-10-23 |
| * | base: Write to 'evaluate' output to /var/log/cuirass.•••This fixes a longstanding issue where evalution output would splatter
over the /var/log/cuirass.log and be inscrutable.
* src/cuirass/base.scm (%cuirass-state-directory): New variable.
(evaluation-log-file): New procedure.
(evaluate)[log-file, log-pipe]: New variables.
Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into
'with-error-to-port'. Close 'log-pipe'.
| Ludovic Courtès | 2019-10-23 |
| * | http: Support fetching builds by derivation.•••There's a one to one mapping of build id to derivation, so allow querying by
derivation file name as well. I'm looking at this as I'm interested in getting
build information in to the Guix Data Service.
* src/cuirass/http.scm (url-handler): Support fetching builds by numeric id or
the derivation file name.
| Christopher Baines | 2019-10-02 |
| * | templates: Fix type errors in build details page.•••This is a followup to 858b6b8c8f2ae7b1ddaf4ae363147121be1f1fe8.
* src/cuirass/templates.scm (build-details)[blocking-outputs]: Fix type
errors when dealing with the result of 'derivation-build-plan'.
| Ludovic Courtès | 2019-09-23 |
| * | http: Pass 'build-search-results-table' a list, not a vector.•••* src/cuirass/http.scm (url-handler): Call 'vector->list' on the result
of 'handle-builds-search-request', as expected by 'build-search-results-table'.
| Ludovic Courtès | 2019-08-27 |
| * | http: Pass 'build-eval-table' a list, not a vector.•••* src/cuirass/http.scm (url-handler): Call 'vector->list' on the result
of 'handle-builds-request', as expected by 'build-eval-table'.
| Ludovic Courtès | 2019-08-27 |
| * | http: Fix type error.•••This is a follow-up to commit c6f4fa5f577e4752d845fa3ce17c68fcd2079904.
* src/cuirass/http.scm (url-handler): SPECIFICATIONS-TABLE expects a list, not
a vector.
| Ricardo Wurmus | 2019-08-27 |
| * | templates: Use 'derivation-build-plan'.•••* src/cuirass/templates.scm (build-details): Use 'derivation-build-plan'
instead of 'derivation-prerequisites-to-build'.
| Ludovic Courtès | 2019-08-17 |
| * | doc: Guile 2.2 is needed.•••This is a followup to ee11ba1d93b24753eafcd77eef8543e5cac5ba4c.
* README: Ask for Guile 2.2.
| Ludovic Courtès | 2019-08-17 |
| * | Switch to Guile-JSON 3.x.•••Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on
until now: it maps JSON dictionaries to alists (instead of hash tables),
and JSON arrays to vectors (instead of lists). This commit is about
adjusting all the existing code to this new mapping.
* src/cuirass/http.scm (evaluation->json-object): New procedure.
(handle-builds-request): Pass the result through 'list->vector'.
(handle-builds-search-request): Likewise.
(url-handler): Likewise for /jobsets, /specifications, /api/evaluations,
and /build. For /api/evaluations, use 'evaluation->json-object'.
* src/cuirass/utils.scm (object->json-scm): Add 'vector?' case.
* tests/http.scm (hash-table-keys, hash-table=?): Remove.
(evaluations-query-result): Use vectors for JSON arrays.
("object->json-string"): Expects alists instead of hash tables.
("/build/1"): Use 'lset=' instead of 'hash-table=?'.
("/api/latestbuilds?nr=1&jobset=guix"): Likewise, and expect alists
instead of hash tables.
("/api/latestbuilds?nr=1&jobset=gnu"): Likewise.
("/api/evaluations?nr=1"): Likewise.
* README: Mention Guile-JSON 3.x.
| Ludovic Courtès | 2019-08-17 |
| * | http: Import (ice-9 format).•••* src/cuirass/http.scm: Add missing include of (ice-9 format).
| Ludovic Courtès | 2019-08-17 |
| * | base: Use the new 'store-error' procedure names.•••The 'nix-error' names are now deprecated in Guix.
* src/cuirass/base.scm (build-derivations&): Use the new 'store-error'
procedure names.
| Ludovic Courtès | 2019-08-17 |
| * | Separate web interface.•••* bin/cuirass.in (show-help): Document "--web" option.
(%options): Default to running without web interface.
(main): Either run the web interface or build packages.
* README: Mention the "--web" option.
| Ricardo Wurmus | 2019-08-10 |
| * | Fix templates to handle all statuses.•••* src/cuirass/templates.scm: Factor out class and title helpers
for build statuses, and handle all the statuses.
| Robert Vollmert | 2019-08-07 |
| * | database: query->bind-arguments: Process ^ and $ characters.•••* src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters
in query strings.
| Ricardo Wurmus | 2019-06-20 |