aboutsummaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAge
...
* Better report exceptions in query-build-server-substitutesChristopher Baines2023-06-09
|
* Detach the database setup from the main guix-data-service processChristopher Baines2023-06-09
| | | | | This will allow restarting them independently, leaving it up to the operator to ensure that all processes are compatible.
* Query for outputs when build events arriveChristopher Baines2023-06-09
| | | | This will keep the substitute information more up to date.
* Make the build event handling code more genericChristopher Baines2023-06-06
| | | | So that triggering a check for substitutes can be integrated in.
* Extend the invalid parameter handling for more typesChristopher Baines2023-05-22
|
* Have render-compare/package-derivations provide invalid param infoChristopher Baines2023-05-22
| | | | | | So that this can be used by the qa-frontpage. This should be improved and generalised.
* Fix ignoring canceled buildsChristopher Baines2023-05-18
| | | | | The previous changes only affected searching for package derivations, and they also didn't work.
* Ignore canceled builds when querying package derivationsChristopher Baines2023-05-18
| | | | | This will help when using this to submit builds, since you won't end up ignoring derivations with canceled builds.
* Add Guile internal time metricsChristopher Baines2023-05-14
| | | | As this will help identify when the service restarts.
* Ensure the known and unknown keys appearChristopher Baines2023-05-09
|
* Remove redundant match-lambda in select-package-output-availability-for-revisionChristopher Baines2023-05-09
|
* Set lock_timeout for some data deletion transactionsChristopher Baines2023-05-09
| | | | | As these can cause deadlocks. This will probably cause errors, so some retrying will need to be added.
* Use the package_derivations system id in a queryChristopher Baines2023-05-04
| | | | | Rather than the derivations system id, as this helps PostgreSQL run the query faster.
* Further tweak fetching narinfosChristopher Baines2023-04-28
| | | | | Move the batching to the database, which should reduce memory usage while removing the limit on the number of fetched narinfos.
* Tweak fetching narinfosChristopher Baines2023-04-28
| | | | | Currently I'm seeing failures due to guile-gnutls not supporting suspendable ports (write_wait_fd), so batch the requested outputs to try and avoid this.
* Set request timeouts for the thread poolsChristopher Baines2023-04-27
| | | | | The request timeout should ensure that the operations don't back up if the thread pool is overloaded.
* Support request timeouts in the thread poolChristopher Baines2023-04-27
|
* Change par-mapper' to use fetch-result-of-defered-thunksChristopher Baines2023-04-27
| | | | So the fetch-result-of-defered-thunk procedure can be removed.
* Split the thread pool used for database connectionsChristopher Baines2023-04-27
| | | | | | | | In to two thread pools, a default one, and one reserved for essential functionality. There are some pages that use slow queries, so this should help stop those pages block other operations.
* Include distribution counts table in data deletionChristopher Baines2023-04-07
|
* Improve performance of select-fixed-output-package-derivations-in-revisionChristopher Baines2023-03-11
|
* Fix query in get-count-for-next-levelChristopher Baines2023-03-09
|
* Defer backfilling derivation distribution counts until laterChristopher Baines2023-03-09
| | | | After the migrations have run.
* Reorder some vacuum operations when loading revisionsChristopher Baines2023-03-09
|
* Avoid a recursive CTE for finding blocked builds where possibleChristopher Baines2023-03-09
| | | | | | Use the new approach of looking up the distribution of the derivations, and building a non recursive query specifically for this revision. This should avoid PostgreSQL picking a poor plan for performing the query.
* Store the distribution of derivations related to packagesChristopher Baines2023-03-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This might be generally useful, but I've been looking at it as it offers a way to try and improve query performance when you want to select all the derivations related to the packages for a revision. The data looks like this (for a specified system and target): ┌───────┬───────┐ │ level │ count │ ├───────┼───────┤ │ 15 │ 2 │ │ 14 │ 3 │ │ 13 │ 3 │ │ 12 │ 3 │ │ 11 │ 14 │ │ 10 │ 25 │ │ 9 │ 44 │ │ 8 │ 91 │ │ 7 │ 1084 │ │ 6 │ 311 │ │ 5 │ 432 │ │ 4 │ 515 │ │ 3 │ 548 │ │ 2 │ 2201 │ │ 1 │ 21162 │ │ 0 │ 22310 │ └───────┴───────┘ Level 0 reflects the number of packages. Level 1 is similar as you have all the derivations for the package origins. The remaining levels contain less packages since it's mostly just derivations involved in bootstrapping. When using a recursive CTE to collect all the derivations, PostgreSQL assumes that the each derivation has the same number of inputs, and this leads to a large overestimation of the number of derivations per a revision. This in turn can lead to PostgreSQL picking a slower way of running the query. When it's known how many new derivations you should see at each level, it's possible to inform PostgreSQL this by using LIMIT's at various points in the query. This reassures the query planner that it's not going to be handling lots of rows and helps it make better decisions about how to execute the query.
* Avoid logging when there are no processes to wait forChristopher Baines2023-03-09
|
* Fix counting derivations when there are lotsChristopher Baines2023-03-09
|
* Add an extra index on blocked_buildsChristopher Baines2023-03-05
| | | | I think this might help with queries that don't use the build_server_id.
* Tweak the comparision queryChristopher Baines2023-03-05
| | | | This seems to generate better plans.
* Remove peekChristopher Baines2023-03-05
|
* Set current-guix-package when computing system test derivationsChristopher Baines2023-02-28
| | | | This is a bit ugly, but might speed up computing derivations for system tests.
* Speed up deleting blocked_builds entriesChristopher Baines2023-02-27
|
* Move the delete-unreferenced-derivations advisory lockChristopher Baines2023-02-27
| | | | To better prevent two processes running at the same time.
* Comparing package derivations: Fix inconsistent verbage.Leo Famulari2023-02-17
| | | | | | | | | | Harmonize "Build change" options between the selection menu and the documentation * guix-data-service/web/compare/html.scm (compare/package-derivations): Replace "Still broken" with "Still failing" in the "Build change" help text. Signed-off-by: Christopher Baines <mail@cbaines.net>
* Include some useful numbers on the package derivations comparisonChristopher Baines2023-02-15
| | | | | As it's frequently useful to know how many packages/builds some change has affected.
* Remove redundant postgresql connection when deleting derivationsChristopher Baines2023-02-14
|
* Delete blocked_builds entries when deleting derivationsChristopher Baines2023-02-14
|
* Delete git_branches entriesChristopher Baines2023-02-14
| | | | When deleting data for a branch.
* Avoid errors dropping partition tables if they don't existChristopher Baines2023-02-13
|
* Improve dropping package_derivation_by_guix_revision_range partitionsChristopher Baines2023-02-13
|
* Adjust render procedures to not use procedures for responsesChristopher Baines2023-02-09
| | | | | | | The newer Guile Fibers web server will use the chunked transfer encoding when a procedure is used and the content length is unspecified. This is good for large responses, but unnecessary here. Also, there's a bug with the charset so these changes to respond with correctly encoded bytevectors to avoid that.
* Tweak behaviour when the response body is a procedureChristopher Baines2023-02-09
| | | | | | | | | Newer versions of Guile Fibers will now use chunked encoding when a procedure is used (and no content length is set). This is good, but not always what is wanted, and there's also an issue with the port encoding. This commit switches to responding with a string/bytevector when more appropriate, plus explicitly setting the port encoding where that's needed.
* Allow skipping processing system testsChristopher Baines2023-02-08
| | | | | | | | | | | Generating system test derivations are difficult, since you generally need to do potentially expensive builds for the system you're generating the system tests for. You might not want to disable grafts for instance because you might be trying to test whatever the test is testing in the context of grafts being enabled. I'm looking at skipping the system tests on data.guix.gnu.org, because they're not used and quite expensive to compute.
* Skip some metrics that apply for each branchChristopher Baines2023-02-02
| | | | | | | As data.qa.guix.gnu.org has lots of branches and 100,000+ metrics, and this is causing Prometheus to time out fetching the metrics. I'm not sure there's much value in these metrics, so cut them out for now.
* Enable database connection instrumentation in the serverChristopher Baines2023-01-01
|
* Support instrumenting the number of database connectionsChristopher Baines2023-01-01
| | | | Since this is now quite dynamic, it's useful to have a metric for it.
* Pull the metrics registry out of the controllerChristopher Baines2023-01-01
| | | | | This will allow for instrumenting low level database functionality, before anything starts using the database.
* Expose metrics from pg_statsChristopher Baines2022-11-29
|
* Only sometimes attempt manually computing n_distinct valuesChristopher Baines2022-11-29
| | | | For derivation_inputs.
generated by cgit v1.2.3 (git 2.45.0) at 2024-11-29 00:25:41 +0000