aboutsummaryrefslogtreecommitdiff
path: root/guix-build-coordinator/utils.scm
Commit message (Collapse)AuthorAge
* Support thread-stop-delay for the work queuesChristopher Baines2021-06-20
| | | | | | | This is useful when builds finish quickly since there could be more than 2 idle threads, and then threads start stopping. This way, each thread waits 20 seconds before stopping, which should be enough time for new builds to be fetched.
* Support the work queue matching threads to jobsChristopher Baines2021-06-09
|
* Support reporting bytes sent with streaming HTTP requestsChristopher Baines2021-06-08
|
* Switch to a lower gc threshold in make-chunked-output-port*Christopher Baines2021-06-07
| | | | | A fixed threshold should make things line up when there's multiple ports in use.
* Refactor the code around work queuesChristopher Baines2021-05-31
|
* Don't pretend that chunked inputs have been readChristopher Baines2021-05-30
| | | | | If the connection closes before all the data's been received. Most of a nar file isn't useful.
* Fix closing the custom chunked output portChristopher Baines2021-05-30
|
* Fix the make-chunked-output-port* implementationChristopher Baines2021-05-30
| | | | A broken one was committed previously.
* Further tweak sending chunked HTTP requestsChristopher Baines2021-05-29
| | | | | | | | Don't compress then send, since I think compression can be slower than sending, so doing both at the same time is probably faster. Add make-chunked-output-port* which might be more efficient than the Guile chunked output port, will disable garbage collection to avoid issues with GnuTLS and will try to force the garbage collector to run if there's garbage building up.
* Tweak how the GC protection works when closing the portChristopher Baines2021-05-28
| | | | Bring more stuff inside one with-gc-protection block.
* Increase the size of one of the buffers for sending filesChristopher Baines2021-05-28
| | | | As this reduces the GC disabling/enabling.
* Add more gc protectionChristopher Baines2021-05-28
|
* Use dyanmic-wind when changing the GC configurationChristopher Baines2021-05-28
| | | | | | | These procedures actually increment/decrement a counter, so gc-enable might not enable garbage collection if gc-disable has been called twice in a row. dynamic-wind should ensure that gc-enable is always called after gc-disable, even if thunk raises an exception for example.
* Don't automatically requeue jobsChristopher Baines2021-05-28
| | | | | | | | Even if they aren't requeued, the agent should learn about the job again from the coordinator. I'm mostly removing this, because I'm seeing agents seemingly process the same job twice at the same time, and I wonder if it's related.
* Have with-gc-protection return the thunk return values properlyChristopher Baines2021-05-28
|
* Use GC protection for normal requests to the coordinator as wellChristopher Baines2021-05-28
| | | | | Since the gc breaking gnutls problem can occur for these requests probably as well.
* Increase the buffer size for sending outputs and log filesChristopher Baines2021-05-28
| | | | I think this works better.
* Tune sending files over HTTPChristopher Baines2021-05-28
| | | | | | | | | Guile's garbage collector interferes with Guile+gnutls, which means that sending files while the garbage collector is active is difficult. These changes try to work around this by disabling the garbage collector just as the data is being written, then enabling it again. I think this helps to work around the issue.
* Don't ignore premature termination for chunked requestsChristopher Baines2021-05-26
| | | | | | This code was copied from Guile, but this seems like a deficiency. I can't imagine a case where you'd be processing chunked data, and just want to pretend you've got to the end, when you haven't.
* Simplify the work queue loopChristopher Baines2021-05-14
|
* Buffer the network socket used for streaming http requestsChristopher Baines2021-05-13
| | | | As this helps improve throughput.
* Stop agent threads only when 2 or more are idleChristopher Baines2021-04-23
| | | | | | If the jobs are really quick, I think the one running thread keeps stopping and starting, and that stops the agent starting more threads. I think this change might help.
* Add Guile GC related metricsChristopher Baines2021-03-25
| | | | | I'm seeing mmap(PROT_NONE) failed crashes, and maybe these metrics will help in understanding what's going on.
* Move request-query-parameters to utilsChristopher Baines2021-02-28
|
* Prepare for the addition of a (guix substitutes) moduleChristopher Baines2021-02-23
| | | | | | | If and when this happens, some proceduces will be moved. This change might avoid things breaking. https://issues.guix.gnu.org/45409
* Avoid some threads and locks when running on the hurdChristopher Baines2021-02-15
| | | | I've see the process hang on the hurd, and I think this might help.
* Tweak trying to safely load the (guix narinfo) moduleChristopher Baines2021-02-10
| | | | | I think the eval-when thing might help given narinfo-references is a syntax thing, rather than a normal procedure.
* Try to fix using narinfo-referencesChristopher Baines2021-02-09
|
* Fix compatability with newer versions of GuixChristopher Baines2021-02-09
| | | | | With respect to narinfo-references, which moved from (guix scripts substitute) to (guix narinfo).
* Don't use with-exception-handler with (backtrace)Christopher Baines2021-01-22
| | | | | | | | | | | With with-exception-handler being called with #:unwind? #f (implicitly). This breaks Guile internals used by (backtrace) [1], meaning you get a different exception/backtrace when Guile itself breaks. This should avoid the "string->number: Wrong type argument in position 1 (expecting string): #f" exception I've been haunted by for the last year. 1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46009
* Tweak agent queue timingsChristopher Baines2021-01-12
| | | | | Wait slightly longer between starting new threads, and give more time for new jobs to arrive for running threads.
* Work on the agent thread handling some moreChristopher Baines2021-01-06
| | | | | Various changes, hopefully improvements. Inactive threads should stop promptly, and new threads should start promptly when new builds arrive.
* Stop agent threads if there's no work within some time periodChristopher Baines2021-01-02
| | | | So that the number of threads can decrease without new jobs arriving.
* Don't start new agent threads if there are no jobsChristopher Baines2021-01-02
|
* Only use lookup-narinfos on one thread at a timeChristopher Baines2021-01-02
| | | | In the agent, as I doubt the new connection caching in Guix is thread safe.
* Work around caching broken connections in GuixChristopher Baines2021-01-01
| | | | Introduced in be5a75ebb5988b87b2392e2113f6590f353dd6cd.
* Have the agent fetch builds based on the current number of threadsChristopher Baines2020-12-31
| | | | | | This means that if the agent is only processing 2 builds at a time, it'll only fetch up to two builds, rather than whatever maximum it would fetch. This avoids fetching builds unnecessarily.
* Rate limit the agent starting threads to process buildsChristopher Baines2020-12-31
| | | | | | | | | | Allow rate limiting new worker threads starting in the agent. Currently if the running jobs is limited by system load, lots of jobs start, the load goes up, then the jobs gradually finish, and once the load decreases, lots of jobs start again, and the cycle repeats. Rate limiting the starting of new threads might help to soften the jobs all starting at once.
* Alter the work queue to allow for a dynamic thread countChristopher Baines2020-12-30
| | | | | | This means that rather than having threads, just sleep, the number of threads running can change. One good effect of this is that a bounds can be easily put on the number of threads (like a lower bound).
* Remove unnecessary nproc usageChristopher Baines2020-12-24
|
* Further tweak exception handling around has-substitutes-no-cache?Christopher Baines2020-12-24
|
* Add better logging to has-substiutes-no-cache?Christopher Baines2020-12-24
|
* Improve some loggingChristopher Baines2020-12-24
| | | | I think exceptions are happening, but the existing logging isn't working.
* Avoid starting builds if the system load is highChristopher Baines2020-12-23
|
* Improve the guix-daemon claims a substitute is unavailable messagesChristopher Baines2020-12-23
| | | | Include the substitute servers that should be providing the substitute.
* Better format the duration in call-with-time-loggingChristopher Baines2020-12-16
|
* Add in timeouts around fetching substitutesChristopher Baines2020-12-05
| | | | As I think this sometimes hangs.
* Fix agent confusion over how many builds are runningChristopher Baines2020-12-04
| | | | | The previous code was less than ideal, this simpler and avoids less messy state.
* Change "or" to "of" in retry-on-errorChristopher Baines2020-12-02
|
* Try to reduce errors when deleting substitute cache filesChristopher Baines2020-11-30
|
generated by cgit v1.2.3 (git 2.25.4) at 2024-06-09 04:26:18 +0000