aboutsummaryrefslogtreecommitdiff
path: root/guix-data-service
Commit message (Collapse)AuthorAge
* Allow skipping processing system testsChristopher Baines2023-02-08
| | | | | | | | | | | Generating system test derivations are difficult, since you generally need to do potentially expensive builds for the system you're generating the system tests for. You might not want to disable grafts for instance because you might be trying to test whatever the test is testing in the context of grafts being enabled. I'm looking at skipping the system tests on data.guix.gnu.org, because they're not used and quite expensive to compute.
* Skip some metrics that apply for each branchChristopher Baines2023-02-02
| | | | | | | As data.qa.guix.gnu.org has lots of branches and 100,000+ metrics, and this is causing Prometheus to time out fetching the metrics. I'm not sure there's much value in these metrics, so cut them out for now.
* Enable database connection instrumentation in the serverChristopher Baines2023-01-01
|
* Support instrumenting the number of database connectionsChristopher Baines2023-01-01
| | | | Since this is now quite dynamic, it's useful to have a metric for it.
* Pull the metrics registry out of the controllerChristopher Baines2023-01-01
| | | | | This will allow for instrumenting low level database functionality, before anything starts using the database.
* Expose metrics from pg_statsChristopher Baines2022-11-29
|
* Only sometimes attempt manually computing n_distinct valuesChristopher Baines2022-11-29
| | | | For derivation_inputs.
* Guard against divide by 0 in update-derivation-outputs-statisticsChristopher Baines2022-11-28
|
* Do derivation inputs and outputs housekeeping at the end of each jobChristopher Baines2022-11-28
| | | | | | This should help with query performance, as the recursive queries using derivation_inputs and derivation_outputs are particularly sensitive to the n_distinct values for these tables.
* Use advisory locks to avoid deadlocks during data deletionChristopher Baines2022-11-28
| | | | | In the case where multiple data deleting processes end up running at the same time.
* Improve the inferior cleanup when computing package derivationsChristopher Baines2022-11-24
|
* Skip the derivation linterChristopher Baines2022-11-24
| | | | | | | | | | | | | | | | | And remove the chunking of derivation lint warnings. The derivation linter computes the derivation for each packages supported systems, but there are two problems with the approach. By doing this for each package in turn, it forces inefficient uses of caches, since most of the cached data is only relevant to a single system. More importantly though, because the work of checking one package is dependent on it's supported systems, it's unpredictable how much work will happen, and this will tend to increase as more packages support more systems. I think especially because of this last point, it's not worth attempting to keep running the derivation linter at the moment, because it doesn't seem sustainable. I can't see an way to run it that's futureproof and won't break at some point in the future when packages in Guix support more systems.
* Fix calling insert-blocked-buildsChristopher Baines2022-11-20
|
* Render a branch not found page if the branch doesn't existChristopher Baines2022-11-19
|
* Fix quasiquotingChristopher Baines2022-11-19
|
* Better guard against exceptions in the build event handlersChristopher Baines2022-11-19
|
* Fix closing thread postgresql connectionsChristopher Baines2022-11-17
|
* Include more information about invalid query parametersChristopher Baines2022-11-17
| | | | | | | In the /compare response. This should enable qa.guix.gnu.org to detect when the base revision for a comparison is unknown.
* Drop the chunk size when gathering lint warningsChristopher Baines2022-11-14
| | | | To try and bring the peak memory usage down.
* Include blocked_builds information in comparison responsesChristopher Baines2022-11-14
| | | | | This will make it easier to tell when a scheduled build is yet to start, and can't start due to a missing dependency.
* Remove Build status field from blocking builds pageChristopher Baines2022-11-12
| | | | As this is unused.
* Make backfilling blocked_builds a bit smarterChristopher Baines2022-11-12
| | | | And drop the chunk size.
* Handle deleting from blocked_builds when builds are scheduledChristopher Baines2022-11-12
| | | | As scheduling a build might unblock others.
* View scheduled builds like succeeded builds in terms of blockingChristopher Baines2022-11-12
| | | | | | | | This means that an output is viewed to not be blocking if it has a scheduled build, just as if it has a succeeded build. Also, scheduling builds will unblock blocked builds. This is helpful as it means that it reduces noise for blocking builds.
* Tweak backfilling the blocked buildsChristopher Baines2022-11-12
|
* Use latest_build_status rather than build_statusChristopher Baines2022-11-12
| | | | In various places in the blocked-builds module.
* Spawn specific PostgreSQL connections for the blocked_builds updatesChristopher Baines2022-11-12
| | | | So that the queries don't get cancelled by the statement timeout.
* Have insert-blocked-builds cache when the partitions existChristopher Baines2022-11-11
| | | | To make it more efficient.
* Rework insert-blocked-builds to make it more efficientChristopher Baines2022-11-11
| | | | This also fixes a typo in the partition name.
* Stop using exception handling when inserting blocked_builds entriesChristopher Baines2022-11-11
| | | | As it doesn't work in a transaction.
* Add a blocking builds pageChristopher Baines2022-11-11
|
* Add support for incrementally tracking blocked buildsChristopher Baines2022-11-11
| | | | | | | | | This will hopefully provide a less expensive way of finding out if a scheduled build is probably blocked by other builds failing or being canceled. By working this out when the build events are recieved, it should be more feasible to include information about whether builds are likely blocked or not in various places (e.g. revision comparisons).
* Log heap size when computing system testsChristopher Baines2022-11-06
|
* Add more granular logging for computing system test derivationsChristopher Baines2022-11-06
|
* Close postgresql connections when the thread pool thread is idleChristopher Baines2022-10-23
| | | | | | I think the idle connections associated with idle threads are still taking up memory, so especially now that you can configure an arbitrary number of threads (and thus connections), I think it's good to close them regularly.
* Fix JSON response for unknown revisionsChristopher Baines2022-10-17
|
* Expose Guile GC metricsChristopher Baines2022-10-09
| | | | | As the guix-data-service process seems to be using excessive amounts of memory, and this will be useful to track it.
* Make it possible to increase the number of thread pool threadsChristopher Baines2022-10-02
| | | | And double the default to 16.
* Log delays in the thread pool channelChristopher Baines2022-10-02
| | | | As I think with lots of requests, this could become a bottleneck.
* Insert guix revision lint warnings in chunksChristopher Baines2022-09-17
| | | | To avoid long running queries.
* Improve chunking when inserting derivation inputsChristopher Baines2022-09-17
| | | | | Chunk the values inserted in the query, rather than the derivations involved, as this is more consistent.
* Reduce some chunk sizesChristopher Baines2022-09-17
|
* Chunk inserting guix revision package derivation entriesChristopher Baines2022-09-15
|
* Further reduce some chunk sizesChristopher Baines2022-09-15
|
* Chunk the data for some queries in insert-missing-data-and-return-all-idsChristopher Baines2022-09-15
| | | | | This helps to avoid queries getting logged as slow just because of the amount of data.
* Format some queries generated in insert-missing-data-and-return-all-idsChristopher Baines2022-09-14
|
* Reduce some chunk sizesChristopher Baines2022-09-14
| | | | As these queries are still slow enough to be logged.
* Implement the JSON response for package reproducibilityChristopher Baines2022-09-14
|
* Implement the JSON response for package substitute availabilityChristopher Baines2022-09-14
|
* Speed up finding the locales for a revisionChristopher Baines2022-09-14
|