| Commit message (Collapse) | Author | Age |
|
|
|
|
|
| |
By moving looking up derivation outputs with no build to before the
transaction that writes to the database, as this will allow for more
concurrency.
|
|
|
|
|
| |
As I think this could help with reliability. Once the Guile GnuTLS bindings
are updated, this can potentially be increased.
|
|
|
|
|
| |
Such that the retry happens with a fresh slot (and the associated tracking
information).
|
| |
|
|
|
|
| |
If there's a queue.
|
|
|
|
|
|
|
| |
This is useful when builds finish quickly since there could be more than 2
idle threads, and then threads start stopping. This way, each thread waits 20
seconds before stopping, which should be enough time for new builds to be
fetched.
|
| |
|
|
|
|
|
| |
Include the jobs from the post build queue too, since they'll still be
allocated.
|
| |
|
| |
|
|
|
|
| |
To actually use max-parallel-builds.
|
|
|
|
|
|
| |
This means that the main work queue can start more jobs while others are being
finished off, which is particularly important now that the parallelism of
uploading outputs is limited.
|
| |
|
|
|
|
|
| |
And report the progress periodically. This can be a bottleneck if the upload
speed is slow, and the machine is fast at building things.
|
| |
|
| |
|
|
|
|
|
| |
A fixed threshold should make things line up when there's multiple ports in
use.
|
| |
|
|
|
|
|
| |
Since time has been spent building them, so wait longer before giving up
submitting the outputs.
|
|
|
|
|
| |
If the connection closes before all the data's been received. Most of a nar
file isn't useful.
|
| |
|
| |
|
|
|
|
| |
A broken one was committed previously.
|
|
|
|
|
|
|
|
| |
Don't compress then send, since I think compression can be slower than
sending, so doing both at the same time is probably faster. Add
make-chunked-output-port* which might be more efficient than the Guile chunked
output port, will disable garbage collection to avoid issues with GnuTLS and
will try to force the garbage collector to run if there's garbage building up.
|
|
|
|
| |
Bring more stuff inside one with-gc-protection block.
|
|
|
|
| |
As this reduces the GC disabling/enabling.
|
| |
|
|
|
|
|
|
|
| |
These procedures actually increment/decrement a counter, so gc-enable might
not enable garbage collection if gc-disable has been called twice in a
row. dynamic-wind should ensure that gc-enable is always called after
gc-disable, even if thunk raises an exception for example.
|
|
|
|
|
|
|
|
| |
Even if they aren't requeued, the agent should learn about the job again from
the coordinator.
I'm mostly removing this, because I'm seeing agents seemingly process the same
job twice at the same time, and I wonder if it's related.
|
| |
|
| |
|
| |
|
|
|
|
|
| |
Since the gc breaking gnutls problem can occur for these requests probably as
well.
|
|
|
|
| |
I think this works better.
|
|
|
|
|
|
|
|
|
| |
This was put in to try and prevent the crashes inside gnutls, but was
ineffective since the actual trigger for the issues is garbage collection,
rather than parallel requests.
There might be some benefit from limiting request parallelism in the future,
but that can be thought through then.
|
|
|
|
|
|
|
|
|
| |
Guile's garbage collector interferes with Guile+gnutls, which means that
sending files while the garbage collector is active is difficult.
These changes try to work around this by disabling the garbage collector just
as the data is being written, then enabling it again. I think this helps to
work around the issue.
|
| |
|
| |
|
| |
|
|
|
|
|
| |
Prefer upfront compression, as this might reduce GC activity while sending the
data.
|
| |
|
|
|
|
|
|
| |
This code was copied from Guile, but this seems like a deficiency. I can't
imagine a case where you'd be processing chunked data, and just want to
pretend you've got to the end, when you haven't.
|
|
|
|
|
| |
Until actually storing the build, since the build might not actually be
submitted if there's a build for those outputs already.
|
|
|
|
| |
As this is clearer than '().
|
|
|
|
| |
If the derivation doesn't exist.
|
|
|
|
| |
Just use it when uploading files.
|
|
|
|
| |
I thought this would be on by default, but apparently not.
|
| |
|
| |
|
| |
|