| Commit message (Expand) | Author | Age |
|---|
| * | Use vector-fold rather than map to construct input-derivations•••This avoids append-map and vector->list so might be faster and use less
memory.
| Christopher Baines | 2025-03-10 |
| * | Improve package deduplication•••Handle cases where there are up to 4 packages with the same name and version
plus some having replacements. This is currently the case with glibc.
| Christopher Baines | 2025-02-25 |
| * | Update the package derivations table in a transaction | Christopher Baines | 2025-02-20 |
| * | Make knots update changes | Christopher Baines | 2025-02-11 |
| * | Update use of resource pools | Christopher Baines | 2025-02-06 |
| * | Name the temporary threads | Christopher Baines | 2025-02-04 |
| * | Remove some knots workarounds | Christopher Baines | 2025-02-03 |
| * | Update exception handling in load-new-guix-revision | Christopher Baines | 2025-02-03 |
| * | Add a slightly crude method to ignore systems and targets•••While processing a revision. It would be good to also record what systems and
targets are in the platforms so it's clear what data is missing, but that can
be added later.
| Christopher Baines | 2025-02-03 |
| * | Force add resources parallelism to 1 for some resource pools•••This'll be the default in coming revisions of knots, and is neccessary for
pools being used to share a single object.
| Christopher Baines | 2025-02-02 |
| * | Remove use of call-with-utility-thread•••As I think it's no longer necessary, and the PostgreSQL connection is being
handled here incorrectly.
| Christopher Baines | 2025-01-31 |
| * | Work around the resource pool needing a destructor | Christopher Baines | 2025-01-31 |
| * | Update Guile Knots | Christopher Baines | 2025-01-14 |
| * | Report on the GC when processing revisions | Christopher Baines | 2024-12-29 |
| * | Parallelise inserting derivation counts | Christopher Baines | 2024-12-29 |
| * | Update Guile Knots | Christopher Baines | 2024-12-29 |
| * | Use knots•••A library of extracted Guile Fibers patterns and utilities.
| Christopher Baines | 2024-12-15 |
| * | Add a comment | Christopher Baines | 2024-12-09 |
| * | Log differently when using the load-new-guix-revision-inserts lock | Christopher Baines | 2024-12-09 |
| * | Avoid getting the load-new-guix-revision-inserts lock twice | Christopher Baines | 2024-12-09 |
| * | Fix wrong number of values from channel-derivations-by-system->guix-store-item | Christopher Baines | 2024-12-09 |
| * | Rework loading revision data•••These changes were motivated by switching to a mechanism of loading data that
isn't dependent on the big advisory lock that prevents more than one revision
from being processed at a time.
Since INSERT ... RETURNING id; is used, this can block if another transaction
inserts the same data, and then cause an error when that transaction
commits. The solution is to use ON CONFLICT DO NOTHING, but you have to handle
the case when the INSERT doesn't return an id since the other transaction has
inserted it.
This commit rewrites insert-missing-data-and-return-all-ids to do as described
above, as well as being more efficient in how existing data is detected and to
use more vectors. Other utilities for inserting data are added as well.
| Christopher Baines | 2024-12-09 |
| * | Add in call-with-worker-thread to try to avoid sort problems | Christopher Baines | 2024-11-29 |
| * | Move log message | Christopher Baines | 2024-11-29 |
| * | Don't lookup #f in the hash table | Christopher Baines | 2024-11-29 |
| * | Insert guix_revision_package_derivation_distribution_counts later•••As this should speed up processing revisions.
| Christopher Baines | 2024-11-26 |
| * | Don't crash when linters crash•••This is currently happening because some linters try to evaluate parts of
packages for cross-building to aarch64-linux-gnu, but not all packages support
that and some crash in this case.
I'm not quite sure what the correct behaviour should be, but maybe the data
service needs to try and handle these crashes rather than not processing the
entire revision.
| Christopher Baines | 2024-11-26 |
| * | Restore calling update-derivation-ids-hash-table!•••In derivation-file-names->derivation-ids. This is important for performance
since it avoids reading derivations with an entry already in the database.
| Christopher Baines | 2024-11-25 |
| * | Change to logging the filtered derivations•••As that's what's relevant here.
| Christopher Baines | 2024-11-25 |
| * | Allow using the heap profiler in the inferior | Christopher Baines | 2024-11-08 |
| * | More efficiently create some vectors•••And instrument the inferior memory after creating the gds-inferior-packages
vector.
| Christopher Baines | 2024-11-08 |
| * | Do more when cleaning inferiors•••Not sure if this actually helps, but it might do.
| Christopher Baines | 2024-11-08 |
| * | Add log-tag when inserting channel instance derivations | Christopher Baines | 2024-11-08 |
| * | Always assume that resource pool waiters will stick around•••As I think this is a more efficient design.
| Christopher Baines | 2024-11-05 |
| * | Move the inferior cleanup to the end of each system+target | Christopher Baines | 2024-11-04 |
| * | Refactor | Christopher Baines | 2024-11-03 |
| * | Use a separate db connection in channel-derivations-by-system-promise•••As this can run in parallel with something else using conn.
| Christopher Baines | 2024-11-02 |
| * | Finish chasing the call-with-resource-pool bug•••This took a while to find as process-job would just get stuck, and this wasn't
directly related to any particular change, just that more fibers increased the
chance of hitting it.
This commit includes lots of the things I changed while debugging.
| Christopher Baines | 2024-10-31 |
| * | Add a comment | Christopher Baines | 2024-10-27 |
| * | Rework inserting derivations•••To add more parallelism.
| Christopher Baines | 2024-10-27 |
| * | Use a common derivation-ids-hash-table•••To allow for calling derivation-file-names->derivation-ids in parallel across
multiple fibers, using the PostgreSQL connection fiber to perform atomic
operations.
| Christopher Baines | 2024-10-27 |
| * | Include the package version in the "error when computing" messages•••As this helps when there are lots of packages with the same name.
| Christopher Baines | 2024-10-27 |
| * | Ensure the guix derivations are stored prior to attempting to substitute | Christopher Baines | 2024-10-27 |
| * | Fix extract-information-from message | Christopher Baines | 2024-10-27 |
| * | Increase parallelism when loading revisions | Christopher Baines | 2024-10-27 |
| * | Use a dedicated fiber for heap size monitoring | Christopher Baines | 2024-10-27 |
| * | Fix free-disk-space call•••As the WAL file might not exist.
| Christopher Baines | 2024-10-13 |
| * | Make the WAL threshold more flexible•••So that the WAL can grow more when there's sufficient space. When the
inferiors are closed it takes time to restart them, so doing this less should
speed up processing revisions.
| Christopher Baines | 2024-10-10 |
| * | Add retrying around inserting channel instance derivations | Christopher Baines | 2024-08-29 |
| * | Stop using profiles for channel instances•••Just use the store item instead. This should avoid issues where the profile
gets GC'd.
| Christopher Baines | 2024-08-27 |