Commit message (Collapse) | Author | Age | |
---|---|---|---|
* | Increase upload mutex waiting time | Christopher Baines | 2021-08-07 |
| | |||
* | Report on uploads regularly | Christopher Baines | 2021-08-07 |
| | | | | If there's a queue. | ||
* | Support thread-stop-delay for the work queues | Christopher Baines | 2021-06-20 |
| | | | | | | | This is useful when builds finish quickly since there could be more than 2 idle threads, and then threads start stopping. This way, each thread waits 20 seconds before stopping, which should be enough time for new builds to be fetched. | ||
* | Tweak some indentation | Christopher Baines | 2021-06-19 |
| | |||
* | agent: Tweak the number of builds to fetch. | Christopher Baines | 2021-06-19 |
| | | | | | Include the jobs from the post build queue too, since they'll still be allocated. | ||
* | Increase the thread start delay | Christopher Baines | 2021-06-10 |
| | |||
* | Increase some agent max silent time values | Christopher Baines | 2021-06-10 |
| | |||
* | Fix the thread-start-delay time calculation | Christopher Baines | 2021-06-10 |
| | | | | To actually use max-parallel-builds. | ||
* | Perform post build actions outside the main work queue | Christopher Baines | 2021-06-09 |
| | | | | | | This means that the main work queue can start more jobs while others are being finished off, which is particularly important now that the parallelism of uploading outputs is limited. | ||
* | Support the work queue matching threads to jobs | Christopher Baines | 2021-06-09 |
| | |||
* | Limit the parallelism of output uploads | Christopher Baines | 2021-06-08 |
| | | | | | And report the progress periodically. This can be a bottleneck if the upload speed is slow, and the machine is fast at building things. | ||
* | Support reporting bytes sent when submitting outputs | Christopher Baines | 2021-06-08 |
| | |||
* | Support reporting bytes sent with streaming HTTP requests | Christopher Baines | 2021-06-08 |
| | |||
* | Ensure there's data in backtraces | Christopher Baines | 2021-06-07 |
| | |||
* | Switch to a lower gc threshold in make-chunked-output-port* | Christopher Baines | 2021-06-07 |
| | | | | | A fixed threshold should make things line up when there's multiple ports in use. | ||
* | Refactor the code around work queues | Christopher Baines | 2021-05-31 |
| | |||
* | Retry more when sending outputs | Christopher Baines | 2021-05-30 |
| | | | | | Since time has been spent building them, so wait longer before giving up submitting the outputs. | ||
* | Don't pretend that chunked inputs have been read | Christopher Baines | 2021-05-30 |
| | | | | | If the connection closes before all the data's been received. Most of a nar file isn't useful. | ||
* | Improve the reveived output message | Christopher Baines | 2021-05-30 |
| | |||
* | Fix closing the custom chunked output port | Christopher Baines | 2021-05-30 |
| | |||
* | Fix the make-chunked-output-port* implementation | Christopher Baines | 2021-05-30 |
| | | | | A broken one was committed previously. | ||
* | Further tweak sending chunked HTTP requests | Christopher Baines | 2021-05-29 |
| | | | | | | | | Don't compress then send, since I think compression can be slower than sending, so doing both at the same time is probably faster. Add make-chunked-output-port* which might be more efficient than the Guile chunked output port, will disable garbage collection to avoid issues with GnuTLS and will try to force the garbage collector to run if there's garbage building up. | ||
* | Tweak how the GC protection works when closing the port | Christopher Baines | 2021-05-28 |
| | | | | Bring more stuff inside one with-gc-protection block. | ||
* | Increase the size of one of the buffers for sending files | Christopher Baines | 2021-05-28 |
| | | | | As this reduces the GC disabling/enabling. | ||
* | Add more gc protection | Christopher Baines | 2021-05-28 |
| | |||
* | Use dyanmic-wind when changing the GC configuration | Christopher Baines | 2021-05-28 |
| | | | | | | | These procedures actually increment/decrement a counter, so gc-enable might not enable garbage collection if gc-disable has been called twice in a row. dynamic-wind should ensure that gc-enable is always called after gc-disable, even if thunk raises an exception for example. | ||
* | Don't automatically requeue jobs | Christopher Baines | 2021-05-28 |
| | | | | | | | | Even if they aren't requeued, the agent should learn about the job again from the coordinator. I'm mostly removing this, because I'm seeing agents seemingly process the same job twice at the same time, and I wonder if it's related. | ||
* | Add a space in coordinator-handle-failed-request | Christopher Baines | 2021-05-28 |
| | |||
* | Improve retrying for missing log files | Christopher Baines | 2021-05-28 |
| | |||
* | Have with-gc-protection return the thunk return values properly | Christopher Baines | 2021-05-28 |
| | |||
* | Use GC protection for normal requests to the coordinator as well | Christopher Baines | 2021-05-28 |
| | | | | | Since the gc breaking gnutls problem can occur for these requests probably as well. | ||
* | Increase the buffer size for sending outputs and log files | Christopher Baines | 2021-05-28 |
| | | | | I think this works better. | ||
* | Get rid of the request mutex | Christopher Baines | 2021-05-28 |
| | | | | | | | | | This was put in to try and prevent the crashes inside gnutls, but was ineffective since the actual trigger for the issues is garbage collection, rather than parallel requests. There might be some benefit from limiting request parallelism in the future, but that can be thought through then. | ||
* | Tune sending files over HTTP | Christopher Baines | 2021-05-28 |
| | | | | | | | | | Guile's garbage collector interferes with Guile+gnutls, which means that sending files while the garbage collector is active is difficult. These changes try to work around this by disabling the garbage collector just as the data is being written, then enabling it again. I think this helps to work around the issue. | ||
* | Fix datastore-find-derivation-for-output | Christopher Baines | 2021-05-27 |
| | |||
* | Fix part of the create_outputs migration | Christopher Baines | 2021-05-27 |
| | | | | | It was completely broken, since it would just delete all entries from the unbuilt outputs table. | ||
* | Fix datastore-list-unbuilt-derivation-outputs | Christopher Baines | 2021-05-27 |
| | |||
* | Fix datastore-list-build-outputs for the new outputs table | Christopher Baines | 2021-05-27 |
| | |||
* | Reduce the threshold for compressing nars on the fly | Christopher Baines | 2021-05-26 |
| | | | | | Prefer upfront compression, as this might reduce GC activity while sending the data. | ||
* | Remove stale log files | Christopher Baines | 2021-05-26 |
| | |||
* | Don't ignore premature termination for chunked requests | Christopher Baines | 2021-05-26 |
| | | | | | | This code was copied from Guile, but this seems like a deficiency. I can't imagine a case where you'd be processing chunked data, and just want to pretend you've got to the end, when you haven't. | ||
* | Delay storing derivations in the database | Christopher Baines | 2021-05-21 |
| | | | | | Until actually storing the build, since the build might not actually be submitted if there's a build for those outputs already. | ||
* | Return #f from datastore-find-derivation-outputs if none are found | Christopher Baines | 2021-05-21 |
| | | | | As this is clearer than '(). | ||
* | Allow datastore-find-derivation-system to return #f | Christopher Baines | 2021-05-21 |
| | | | | If the derivation doesn't exist. | ||
* | Switch to a default 1min load average of 3/4 of the processor count | Christopher Baines | 2021-05-21 |
| | | | | As this will be a better default on machines with more cores. | ||
* | Drop the request mutex for most requests | Christopher Baines | 2021-05-21 |
| | | | | Just use it when uploading files. | ||
* | Enable foreign key checking | Christopher Baines | 2021-05-21 |
| | | | | I thought this would be on by default, but apparently not. | ||
* | Introduce a systems table | Christopher Baines | 2021-05-21 |
| | |||
* | Introduce an outputs table | Christopher Baines | 2021-05-21 |
| | |||
* | Line buffer the queue builds script output | Christopher Baines | 2021-05-14 |
| | | | | So that the output is more consistent. |