| Commit message (Expand) | Author | Age |
|---|
| ... | |
| * | Remove localhost from the Sqitch config•••This means it'll connect over the socket, like the application.
| Christopher Baines | 2019-07-16 |
| * | Prioritise processing the latest revisions for branches•••Reserve some capacity to process revisions which are the tip of a branch. This
should reduce the time between new revisions appearing, and then being
processed.
| Christopher Baines | 2019-07-16 |
| * | Record jobs as failed when they are killed•••So that they aren't retried again and again.
| Christopher Baines | 2019-07-14 |
| * | Kill long running load new revision jobs•••There are some revisions of Guix which take forever to process (or days at
least). To avoid jobs being processed forever, kill them after they've been
running for a while (default 24 hours).
| Christopher Baines | 2019-07-12 |
| * | Switch to processing jobs in parallel•••This should speed up processing new revisions, reduce latency between finding
out about new revisions and processing them, as well as help manage memory
usage, by processing each job in a process that then exits.
| Christopher Baines | 2019-07-12 |
| * | Add a process-job command•••This allows easily processing an individual job by id. This may be useful to
use manually, but also when processing jobs in parallel, as forking doesn't
work well with the libpq library used by squee.
| Christopher Baines | 2019-07-12 |
| * | Obtain a lock when inserting the data for a new revision•••This is working towards running the jobs in parallel. Each job looks at the
records in the database, and adds missing ones. If other jobs, running in
different transactions insert the same missing records at the same time, this
could cause an error.
Therefore, to just avoid this problem, lock before inserting the data. This
will allow the jobs to be processed in parallel, and it shouldn't have too
much of an effect on performance, as the slow bit is outside of the
transaction.
| Christopher Baines | 2019-07-12 |
| * | Lock around the channel fetch when loading new revisions•••This is in preparation for running jobs in parallel. The channels code in Guix
uses a cached copy of the Git repository. Multiple jobs can't concurrently
access this without causing issues, so use an advisory lock to ensure that
only one job is using the repository at a time.
| Christopher Baines | 2019-07-12 |
| * | Add a couple of functions for PostgreSQL advisory locks•••Use symbol-hash to convert a symbol to the number for the lock. I'm hoping
this is OK, and it seems to be stable.
| Christopher Baines | 2019-07-12 |
| * | Fix a bug where the list of 0 licenses wasn't handled correctly•••This happened for a package with #f as the licenses. That's incorrect, but try
to handle it without erroring.
| Christopher Baines | 2019-07-12 |
| * | Fix the name for the jobs-load-new-guix-revision test module | Christopher Baines | 2019-07-12 |
| * | Associate a name with database connections•••This helps when working out which connection to the database is doing what.
| Christopher Baines | 2019-07-12 |
| * | Fix using defined? to check for the open-inferior/container function•••It doesn't work as intended unless the module is also specified, so do that.
| Christopher Baines | 2019-07-09 |
| * | Fix issue inserting the empty log row•••If the job started, and then was restarted, the row will already exist. So
don't error on a conflict.
| Christopher Baines | 2019-07-09 |
| * | Move around some of the load new revision code•••To better separate the code that needs to happen after a lock has been
acquired to allow concurrently loading revisions without concurrent insertion
issues.
| Christopher Baines | 2019-07-07 |
| * | Tweak how logs are stored•••Previously, the query for the jobs page was really slow, as it checked the
load_new_guix_revision_job_log_parts table for each job, doing a sequential
scan through the potentially large table.
Adding an index didn't seem to help, as the query planner would belive the
query could return loads of rows, where actually, all that needed checking is
whether a single row existed with a given job_id.
To avoid adding the index to the load_new_guix_revision_job_log_parts table,
and fighting with the query planner, this commit changes the
load_new_guix_revision_job_logs table to include a blank entry for jobs which
are currently being processed. This is inserted at the start of the job, and
then updated at the end to combine and replace all the parts.
This all means that the jobs page should render quickly now.
| Christopher Baines | 2019-07-07 |
| * | Add some missing arguments to view-revision-packages | Christopher Baines | 2019-07-07 |
| * | Add buttons to scroll the page on the job page•••As the page can be quite long, so jumping to the top and bottom is really
useful.
| Christopher Baines | 2019-06-24 |
| * | Iterate the log viewing•••Replace the Guile-side HTML escaping with a less complete, but hopefully
faster PostgreSQL side HTML escaping approach.
Also, allow reading part of the log, by default, the last 1,000,000
characters, as this should render quickly.
| Christopher Baines | 2019-06-24 |
| * | Add a parse-number function to the query-parameters module•••Which does the same thing as parse-result-limit, which may have an overly
specific name.
| Christopher Baines | 2019-06-24 |
| * | Direct the build output to the logging-port•••So that this is logged.
| Christopher Baines | 2019-06-22 |
| * | Stop closing the log-port port•••This part of the soft-port seems to be called, but I don't know why, and
trying to close the output port causes issues.
| Christopher Baines | 2019-06-22 |
| * | Add a job page•••Which shows the output for that job.
| Christopher Baines | 2019-06-22 |
| * | Record the output from loading new revisions to the database•••So that it can easily be shown through the web interface. There's two tables
being used. One which temporarily stores the output as it's output while the
job is running, and other which stores the whole log once the job has
finished.
| Christopher Baines | 2019-06-22 |
| * | Re-arrange some of the load new revision code•••Try to isolate the code that inserts in to the database, so that the relevant
tables can be locked during this time.
| Christopher Baines | 2019-06-21 |
| * | Add a couple of functions around timing•••That don't rely on timing a single function.
| Christopher Baines | 2019-06-21 |
| * | Log the time loading new revisions takes | Christopher Baines | 2019-06-21 |
| * | Set #:fallback? to #t•••If a substitute fails, just build it anyway.
| Christopher Baines | 2019-06-21 |
| * | Remove the revision labels from the links•••As that seems neater.
| Christopher Baines | 2019-06-20 |
| * | Remove the compare form from the index page•••It isn't very useful, as you have to know the commits. Now that there's Git
branch information, it should be possible to access this in a more useful way.
| Christopher Baines | 2019-06-20 |
| * | Move the derivations off the compare page•••To the compare/derivations page. Previously, the compare/derivations page was
comparing more than the derivations, notably the package metadata. This change
avoids that, and also reduces the information overload on the compare page.
| Christopher Baines | 2019-06-20 |
| * | Don't have all the inputs use a monospace font•••Only those where it's appropriate for the data, like commit hashes.
| Christopher Baines | 2019-06-20 |
| * | Process jobs starting with the most recent•••Conventionally, you'd process the oldest job in a queue, but at the moment, it
would be more useful to have recent information more promptly, and fill in the
historical gaps later. I'm not sure this'll always be the case, but for now,
flip the order in which jobs are processed.
| Christopher Baines | 2019-06-19 |
| * | Use revision labels on the index page as well•••Also flip the branch and revision columns around, and add date information to
the branch column.
| Christopher Baines | 2019-06-19 |
| * | Add labels to revisions on the branches and branch pages•••And always link to the revision page. Now the label let's you know if you
should expect information or not for that revision.
| Christopher Baines | 2019-06-19 |
| * | Improve the Unknown revision page•••Make it similar to the revision page, as some of the information, like the
link to the relevant CGit instance might be available.
| Christopher Baines | 2019-06-19 |
| * | Use line buffering for the input and output ports•••As these are used for logging, which is done on a line by line basis. Remove
the now redundant calls to (force-output).
| Christopher Baines | 2019-06-17 |
| * | Time everywhere a derivation is built when loading a new revision•••As this is useful information.
| Christopher Baines | 2019-06-17 |
| * | Add a new page listing jobs | Christopher Baines | 2019-06-17 |
| * | Fix indicating if a revision is queued for processing | Christopher Baines | 2019-06-17 |
| * | Remove some peek calls that have crept in | Christopher Baines | 2019-06-16 |
| * | Link to the latest processed revision for a branch page | Christopher Baines | 2019-06-16 |
| * | Make some pages around revisions more generic•••So that they can also be used for the /branch/foo/latest-processed-revision
pages. The content is the same, but the title, link, and some of the links on
the page are different.
| Christopher Baines | 2019-06-16 |
| * | Improve the error message when a load revision job fails•••It's useful to know the commit.
| Christopher Baines | 2019-06-15 |
| * | Support accessing the latest processed revision for a branch•••This makes is easier to get the latest data for a branch in a single request,
rather than making one request to find the latest revision, then another to
get the data.
| Christopher Baines | 2019-06-13 |
| * | Display a backtrace for errors in channel->derivation-file-name•••To help with debugging.
| Christopher Baines | 2019-06-13 |
| * | Catch the guix available when built•••And use it for the hosting the inferiors, rather than computing the guix
package at runtime. This simplifies the behaviour when the Guix Data Service
is deployed as a Guix package.
| Christopher Baines | 2019-06-13 |
| * | Rework creating /etc when building the derivation for a revision•••Within an inferior. Just use the presence of open-inferior/container directly,
as this is more explicit.
| Christopher Baines | 2019-06-13 |
| * | Don't show failing jobs on the index page•••I'm not sure quite how to treat these yet, but for now, just stick with the
previous behaviour of showing revisions which haven't been processed yet,
along with the ones that have been successfully processed.
| Christopher Baines | 2019-06-13 |
| * | Display the request in one go•••Otherwise, concurrent requests can cause the output to be interleaved.
| Christopher Baines | 2019-06-13 |