| Commit message (Expand) | Author | Age |
|---|
| * | database: Enable running up to 4 database queries at once.•••The number of threads is copied from bin/cuirass.in. When you have at least
two processors, this will allow database queries to be executed in parallel.
With some crude testing using the Apache HTTP server benchmarking tool (ab
from the httpd package), the max request latency does seem to drop when
multiple threads are used, especially when the database queries are slow (I
tested by adding usleep to the worker thread code).
* src/cuirass/database.scm (with-database): Pass #:parallelism to
make-worker-thread-channel.
allow-concurrent-database-queries | Christopher Baines | 2020-01-25 |
| * | Enable make-worker-thread-channel to create multiple worker threads.•••This will allow running multiple threads, that all listen on the same channel,
enabling processing multiple jobs at one time.
* src/cuirass/utils.scm (make-worker-thread-channel): Add a #:parallelism
argument, and create as many threads as the given parallelism.
| Christopher Baines | 2020-01-25 |
| * | Adjust make-worker-thread-channel to take an initializer.•••While this is a generic method, and initializer function will give the
flexibility required to create multiple worker threads for performing SQLite
queries, each with it's own database connection (as a result of calling the
initializer once for each thread). Without this change, they'd all have to use
the same connection, which would not work.
* src/cuirass/utils.scm (make-worker-thread-channel): Change procedure to take
an initializer, rather than arguments directly.
* src/cuirass/database.scm (with-database): Adjust to call
make-worker-thread-channel with an initializer.
* tests/database.scm (db-init): Change to use make-worker-thread-channel
initializer.
* tests/http.scm (db-init): Change to use make-worker-thread-channel
initializer.
| Christopher Baines | 2020-01-25 |
| * | utils: Change critical section terminology to worker threads.•••As far as I'm aware, it's necessary to use a separate thread for interacting
with SQLite as one of the threads used for fibers will be blocked while the
SQLite query is running.
This doesn't mean all queries have to be executed one at a time though,
providing the queries are executed outside the threads used by fibers, and a
single connection isn't used in multiple threads.
These changes start to move in this direction, first by just changing the
terminology.
* src/cuirass/base.scm (clear-build-queue, cancel-old-builds): Change
with-db-critical-section to with-db-worker-thread.
* src/cuirass/database.scm (with-db-critical-section): Rename syntax rule to
with-db-worker-thread.
(db-add-input, db-add-checkout, db-add-specification, db-remove-specification,
db-get-inputs, db-get-specification, db-add-evaluation,
db-set-evaluations-done, db-set-evaluation-done, db-add-derivation-output,
db-add-build, db-update-build-status!, db-get-output, db-get-outputs,
db-get-builds-by-search, db-get-builds, db-get-build derivation-or-id,
db-add-event, db-get-events, db-delete-events-with-ids-<=-to,
db-get-pending-derivations, db-get-checkouts, db-get-evaluations,
db-get-evaluations-build-summary, db-get-evaluations-id-max,
db-get-evaluation-summary, db-get-builds-query-min, db-get-builds-query-max,
db-get-builds-min, db-get-builds-max, db-get-evaluation-specification): Change
from using with-db-critical-section to
with-db-worker-thread.
(with-database): Change syntax rule to use make-worker-thread-channel,
renaming from make-critical-section.
* src/cuirass/utils.scm (%critical-section-args): Rename parameter to
%worker-thread-args.
(make-critical-section): Rename to make-worker-thread-channel, and adjust
parameter and docstring.
(call-with-critical-section): Rename to call-with-worker-thread and adjust
parameter.
(with-critical-section): Rename to with-worker-thread, and adjust to call
call-with-worker-thread.
* tests/database.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
* tests/http.scm (db-init): Use make-worker-thread-channel rather than
make-critical-section.
| Christopher Baines | 2020-01-25 |
| * | Alter the Builds table to have an id field•••The internal rowid's are used for builds as you can request builds by using
the rowid in the URL.
The motivation here is to enable running VACUUM operations in SQLite, without
risking the rowid's for Builds changing. It would be bad if they change, as
they're used in the URL's for builds.
* src/schema.sql (Builds): Add id column.
* src/curiass/dataabse.scm (db-add-build): Change PRIMARYKEY constraint to
UNIQUE constraint.
* src/sql/upgrade-6.sql: New file.
* Makefile.am (dist_sql_DATA): Add it.
improve-handling-of-rowids | Christopher Baines | 2020-01-25 |
| * | database: Don't return rowid from db-add-input.•••As it is unused from where db-add-input is called.
* src/cuirass/database.scm (db-add-input): Don't call and
return (last-insert-rowid).
| Christopher Baines | 2020-01-24 |
| * | build: Allow builds with Guile 3.0.•••* configure.ac: Add "3.0" to 'GUILE_PKG'.
* README: Mention it.
| Ludovic Courtès | 2020-01-18 |
| * | http: Add missing import of (ice-9 threads) in test.•••Using the core binding for 'call-with-new-thread' was deprecated in 2.2
and is removed in 3.0.
* tests/http.scm: Use (ice-9 threads).
| Ludovic Courtès | 2020-01-18 |
| * | Support returning build information by output.•••Being able to take a derivation and query the build information is useful, but
in cases where there are multiple derivations that produce the same outputs,
the probability of getting the data back from Cuirass is reduced.
This is because Cuirass might not have build the exact derivation you have,
but a different derivation that produces the same outputs (this can commonly
happen when a related fixed output derivation changes).
Cuirass doesn't store derivations if they produce the same outputs as a
derivation it already knows about, so it can't determine if this is the
case. Therefore, provide a way of querying build results by output, rather
than derivation.
The motivation behind this is to make it easier to import build information in
to the Guix Data Service.
* src/cuirass/database.scm (db-get-output): New procedure.
* src/cuirass/http.scm (respond-output-not-found): New procedure.
(request-path-components): Handle /output/… requests.
* doc/cuirass.texi (Build information): Mention that you can get build
information by output.
| Christopher Baines | 2020-01-16 |
| * | Support publishing evaluation events•••* src/cuirass/database.scm (db-add-evaluation): Record the creation of new
evaluations as events.
(db-set-evaluation-done): Record when evaluations finish as an event.
* src/cuirass/http.scm (url-handler): Add a new /api/evaluation-events page.
| Christopher Baines | 2020-01-16 |
| * | Support publishing build events•••Add a table to store events, which have a type and a JSON blob. These can be
used to record changes, this commit inserts events when new builds are
created, and when the status of builds change.
The EventsOutbox table is then used to track when events have been sent
out. This is done through the new cuirass-send-events script.
* Makefile.am (bin_SCRIPTS): Add bin/cuirass-send-events.
.gitignore: Add bin/cuirass-send-events.
(dist_pkgmodule_DATA): Add src/cuirass/send-events.scm.
(dist_sql_DATA): Add src/sql/upgrade-5.sql.
(EXTRA_DIST): bin/cuirass-send-events.in.
(bin/cuirass-send-events): New rule.
* bin/cuirass-send-events.in: New file.
* src/cuirass/send-events.scm: New file.
* src/sql/upgrade-5.sql: New file.
* src/cuirass/database.scm (changes-count): New procedure.
(db-update-build-status!): Call db-add-event after updating the build status.
(db-add-event): New procedure.
(db-add-build): Insert an event when a new build is inserted.
(db-delete-events-with-ids-<=-to): New procedure.
* src/schema.sql (Events): New table.
| Christopher Baines | 2020-01-16 |
| * | base: 'build-derivations&' closes the build output port only once.•••* src/cuirass/base.scm (build-derivations&): Remove extra 'close-port'
call in 'guard' clause.
| Ludovic Courtès | 2019-11-03 |
| * | tests: Remove 405 test.•••This is a follow-up to commit c88a7c006ef164015e5c3fb1544b8112bf6ececf.
* tests/http.scm: Remove test for invalid POST to /build/1.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Add route for /admin.•••* src/cuirass/http.scm (url-handler): Handle /admin route.
| Ricardo Wurmus | 2019-10-30 |
| * | bin: Remove redundant lambda.•••* bin/cuirass.in (main): Remove redundant lambda wrapping in for-each.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Handle specification deletion and addition.•••* src/cuirass/http.scm (url-handler): Handle /admin/specifications/add,
/admin/specifications/delete/*, and /admin/specifications.
| Ricardo Wurmus | 2019-10-30 |
| * | templates: specifications-table: Add admin features.•••* src/cuirass/templates.scm (specifications-table): Handle optional ADMIN?
argument.
| Ricardo Wurmus | 2019-10-30 |
| * | database: Add db-remove-specification.•••* src/cuirass/database.scm (db-remove-specification): New procedure.
| Ricardo Wurmus | 2019-10-30 |
| * | http: Be explicit about accepted HTTP methods.•••* src/cuirass/http.scm (url-handler): Match on HTTP method.
| Ricardo Wurmus | 2019-10-30 |
| * | templates: Failed evaluations link to their log.•••* src/cuirass/templates.scm (evaluation-badges): Link to
/eval/ID/log/raw in the 'zero?' case.
| Ludovic Courtès | 2019-10-23 |
| * | http: Serve evaluation logs at /eval/ID/log/raw.•••* src/cuirass/base.scm (evaluation-log-file): Export.
* src/cuirass/http.scm (url-handler)[respond-gzipped-file]: New
procedure.
Add handler for /eval/ID/log/raw.
| Ludovic Courtès | 2019-10-23 |
| * | base: Log the evaluation ID upon evaluation failure.•••* src/cuirass/base.scm (&evaluation-error)[id]: New field.
(evaluate): Specify the 'id' field.
(process-specs): Show the log file name upon evaluation failure.
| Ludovic Courtès | 2019-10-23 |
| * | base: Write to 'evaluate' output to /var/log/cuirass.•••This fixes a longstanding issue where evalution output would splatter
over the /var/log/cuirass.log and be inscrutable.
* src/cuirass/base.scm (%cuirass-state-directory): New variable.
(evaluation-log-file): New procedure.
(evaluate)[log-file, log-pipe]: New variables.
Call 'spawn-fiber' with a logging fiber. Wrap 'open-pipe*' call into
'with-error-to-port'. Close 'log-pipe'.
| Ludovic Courtès | 2019-10-23 |
| * | http: Support fetching builds by derivation.•••There's a one to one mapping of build id to derivation, so allow querying by
derivation file name as well. I'm looking at this as I'm interested in getting
build information in to the Guix Data Service.
* src/cuirass/http.scm (url-handler): Support fetching builds by numeric id or
the derivation file name.
| Christopher Baines | 2019-10-02 |
| * | templates: Fix type errors in build details page.•••This is a followup to 858b6b8c8f2ae7b1ddaf4ae363147121be1f1fe8.
* src/cuirass/templates.scm (build-details)[blocking-outputs]: Fix type
errors when dealing with the result of 'derivation-build-plan'.
| Ludovic Courtès | 2019-09-23 |
| * | http: Pass 'build-search-results-table' a list, not a vector.•••* src/cuirass/http.scm (url-handler): Call 'vector->list' on the result
of 'handle-builds-search-request', as expected by 'build-search-results-table'.
| Ludovic Courtès | 2019-08-27 |
| * | http: Pass 'build-eval-table' a list, not a vector.•••* src/cuirass/http.scm (url-handler): Call 'vector->list' on the result
of 'handle-builds-request', as expected by 'build-eval-table'.
| Ludovic Courtès | 2019-08-27 |
| * | http: Fix type error.•••This is a follow-up to commit c6f4fa5f577e4752d845fa3ce17c68fcd2079904.
* src/cuirass/http.scm (url-handler): SPECIFICATIONS-TABLE expects a list, not
a vector.
| Ricardo Wurmus | 2019-08-27 |
| * | templates: Use 'derivation-build-plan'.•••* src/cuirass/templates.scm (build-details): Use 'derivation-build-plan'
instead of 'derivation-prerequisites-to-build'.
| Ludovic Courtès | 2019-08-17 |
| * | doc: Guile 2.2 is needed.•••This is a followup to ee11ba1d93b24753eafcd77eef8543e5cac5ba4c.
* README: Ask for Guile 2.2.
| Ludovic Courtès | 2019-08-17 |
| * | Switch to Guile-JSON 3.x.•••Guile-JSON 3.x is incompatible with Guile-JSON 1.x, which we relied on
until now: it maps JSON dictionaries to alists (instead of hash tables),
and JSON arrays to vectors (instead of lists). This commit is about
adjusting all the existing code to this new mapping.
* src/cuirass/http.scm (evaluation->json-object): New procedure.
(handle-builds-request): Pass the result through 'list->vector'.
(handle-builds-search-request): Likewise.
(url-handler): Likewise for /jobsets, /specifications, /api/evaluations,
and /build. For /api/evaluations, use 'evaluation->json-object'.
* src/cuirass/utils.scm (object->json-scm): Add 'vector?' case.
* tests/http.scm (hash-table-keys, hash-table=?): Remove.
(evaluations-query-result): Use vectors for JSON arrays.
("object->json-string"): Expects alists instead of hash tables.
("/build/1"): Use 'lset=' instead of 'hash-table=?'.
("/api/latestbuilds?nr=1&jobset=guix"): Likewise, and expect alists
instead of hash tables.
("/api/latestbuilds?nr=1&jobset=gnu"): Likewise.
("/api/evaluations?nr=1"): Likewise.
* README: Mention Guile-JSON 3.x.
| Ludovic Courtès | 2019-08-17 |
| * | http: Import (ice-9 format).•••* src/cuirass/http.scm: Add missing include of (ice-9 format).
| Ludovic Courtès | 2019-08-17 |
| * | base: Use the new 'store-error' procedure names.•••The 'nix-error' names are now deprecated in Guix.
* src/cuirass/base.scm (build-derivations&): Use the new 'store-error'
procedure names.
| Ludovic Courtès | 2019-08-17 |
| * | Separate web interface.•••* bin/cuirass.in (show-help): Document "--web" option.
(%options): Default to running without web interface.
(main): Either run the web interface or build packages.
* README: Mention the "--web" option.
| Ricardo Wurmus | 2019-08-10 |
| * | Fix templates to handle all statuses.•••* src/cuirass/templates.scm: Factor out class and title helpers
for build statuses, and handle all the statuses.
| Robert Vollmert | 2019-08-07 |
| * | database: query->bind-arguments: Process ^ and $ characters.•••* src/cuirass/database.scm (query->bind-arguments): Handle ^ and $ characters
in query strings.
| Ricardo Wurmus | 2019-06-20 |
| * | templates: If applicable, disable "Last" and "First" pagination links.•••* src/cuirass/templates.scm (pagination): Disable "Last" and "First"
if applicable.
| Danny Milosavljevic | 2019-06-19 |
| * | templates: Add link titles to evaluation badges.•••* src/cuirass/templates.scm (evaluation-badges): Add ‘title’ attributes.
| Tobias Geerinckx-Rice | 2019-06-19 |
| * | templates: Expand search input on focus.•••* src/cuirass/templates.scm (html-page): Include CSS for widening the search
input field on focus.
| Ricardo Wurmus | 2019-06-19 |
| * | cuirass: Restore search query.•••* src/cuirass/templates.scm (search-form): Take the query as an argument and
set it as the value (unless #F).
(html-page): Optionally take a value for QUERY as an argument.
* src/cuirass/http.scm (url-handler): Let "search" handler pass QUERY to
HTML-PAGE.
| Ricardo Wurmus | 2019-06-19 |
| * | database: query->bind-arguments: Ignore invalid query terms.•••* src/cuirass/database.scm (query->bind-arguments): Ignore invalid query
terms.
| Ricardo Wurmus | 2019-06-19 |
| * | templates: Display logs of blocking dependencies.•••* src/cuirass/templates.scm (build-details): Determine failed outputs of
prerequisites and link to their logs.
| Ricardo Wurmus | 2019-06-19 |
| * | http: Show number of builds.•••* src/cuirass/database.scm (db-get-evaluation-summary): New procedure.
* src/cuirass/http.scm (url-handler): Display number of builds in tabs.
| Ricardo Wurmus | 2019-06-19 |
| * | cuirass: Remove tabs for canceled, dependency failed, and other failed.•••All of them are included in "failed".
* src/cuirass/http.scm (url-handler): Remove extraneous tabs.
| Ricardo Wurmus | 2019-06-19 |
| * | cuirass: Adjust query for scheduled builds.•••* src/cuirass/http.scm (url-handler): Use "pending" instead of "scheduled".
| Ricardo Wurmus | 2019-06-19 |
| * | cuirass: Display builds in separate tabs.•••* src/cuirass/http.scm (url-handler): Split result for "eval" page in tabs.
* src/cuirass/templates.scm (build-eval-table): Remove border from table head;
Remove lead paragraph.
| Ricardo Wurmus | 2019-06-19 |
| * | http: Fix navigation URL for build details page.•••* src/cuirass/http.scm (url-handler): Use "jobset" instead of "spec".
| Ricardo Wurmus | 2019-06-19 |
| * | templates: Link to "details" page in build tables.•••* src/cuirass/templates.scm (build-eval-table, build-search-results-table):
Link ID to build details URL.
| Ricardo Wurmus | 2019-06-19 |
| * | http: Handle /build/<id>/details URL.•••* src/cuirass/http.scm (url-handler): Add handler for /build/<id>/details.
* src/cuirass/templates.scm (build-details): New procedure.
| Ricardo Wurmus | 2019-06-18 |
| * | database: db-get-specifications: Order specs by name.•••* src/cuirass/database.scm (db-get-specifications): Order rows by "name".
| Ricardo Wurmus | 2019-06-18 |