diff options
| author | Christopher Baines <mail@cbaines.net> | 2023-03-09 08:29:39 +0000 |
|---|---|---|
| committer | Christopher Baines <mail@cbaines.net> | 2023-03-09 08:29:39 +0000 |
| commit | e39c9da028a9a29a0212cdb0287b0046bb786c32 (patch) | |
| tree | 27260624f5437663a87afeca1376a2c3463401fa /scripts | |
| parent | 55059558e7ba636fb46d84058b1d73a372ce0dd8 (diff) | |
| download | data-service-e39c9da028a9a29a0212cdb0287b0046bb786c32.tar data-service-e39c9da028a9a29a0212cdb0287b0046bb786c32.tar.gz | |
Store the distribution of derivations related to packages
This might be generally useful, but I've been looking at it as it offers a way
to try and improve query performance when you want to select all the
derivations related to the packages for a revision.
The data looks like this (for a specified system and target):
┌───────┬───────┐
│ level │ count │
├───────┼───────┤
│ 15 │ 2 │
│ 14 │ 3 │
│ 13 │ 3 │
│ 12 │ 3 │
│ 11 │ 14 │
│ 10 │ 25 │
│ 9 │ 44 │
│ 8 │ 91 │
│ 7 │ 1084 │
│ 6 │ 311 │
│ 5 │ 432 │
│ 4 │ 515 │
│ 3 │ 548 │
│ 2 │ 2201 │
│ 1 │ 21162 │
│ 0 │ 22310 │
└───────┴───────┘
Level 0 reflects the number of packages. Level 1 is similar as you have all
the derivations for the package origins. The remaining levels contain less
packages since it's mostly just derivations involved in bootstrapping.
When using a recursive CTE to collect all the derivations, PostgreSQL assumes
that the each derivation has the same number of inputs, and this leads to a
large overestimation of the number of derivations per a revision. This in turn
can lead to PostgreSQL picking a slower way of running the query.
When it's known how many new derivations you should see at each level, it's
possible to inform PostgreSQL this by using LIMIT's at various points in the
query. This reassures the query planner that it's not going to be handling
lots of rows and helps it make better decisions about how to execute the
query.
Diffstat (limited to 'scripts')
| -rw-r--r-- | scripts/guix-data-service.in | 8 |
1 files changed, 8 insertions, 0 deletions
diff --git a/scripts/guix-data-service.in b/scripts/guix-data-service.in index 7653562..d8b9b5a 100644 --- a/scripts/guix-data-service.in +++ b/scripts/guix-data-service.in @@ -37,6 +37,7 @@ (guix-data-service config) (guix-data-service database) (guix-data-service substitutes) + (guix-data-service model guix-revision-package-derivation) (guix-data-service web server) (guix-data-service web controller) (guix-data-service web nar controller)) @@ -217,6 +218,13 @@ (pid-file (assq-ref opts 'pid-file))) + (call-with-new-thread + (lambda () + (with-postgresql-connection-per-thread + "backfill" + (lambda () + (backfill-guix-revision-package-derivation-distribution-counts))))) + (when pid-file (call-with-output-file pid-file (lambda (port) |