Commit message (Collapse) | Author | Age | |
---|---|---|---|
* | Support reporting bytes sent when submitting outputs | Christopher Baines | 2021-06-08 |
| | |||
* | Support reporting bytes sent with streaming HTTP requests | Christopher Baines | 2021-06-08 |
| | |||
* | Switch to a lower gc threshold in make-chunked-output-port* | Christopher Baines | 2021-06-07 |
| | | | | | A fixed threshold should make things line up when there's multiple ports in use. | ||
* | Refactor the code around work queues | Christopher Baines | 2021-05-31 |
| | |||
* | Retry more when sending outputs | Christopher Baines | 2021-05-30 |
| | | | | | Since time has been spent building them, so wait longer before giving up submitting the outputs. | ||
* | Don't pretend that chunked inputs have been read | Christopher Baines | 2021-05-30 |
| | | | | | If the connection closes before all the data's been received. Most of a nar file isn't useful. | ||
* | Improve the reveived output message | Christopher Baines | 2021-05-30 |
| | |||
* | Fix closing the custom chunked output port | Christopher Baines | 2021-05-30 |
| | |||
* | Fix the make-chunked-output-port* implementation | Christopher Baines | 2021-05-30 |
| | | | | A broken one was committed previously. | ||
* | Further tweak sending chunked HTTP requests | Christopher Baines | 2021-05-29 |
| | | | | | | | | Don't compress then send, since I think compression can be slower than sending, so doing both at the same time is probably faster. Add make-chunked-output-port* which might be more efficient than the Guile chunked output port, will disable garbage collection to avoid issues with GnuTLS and will try to force the garbage collector to run if there's garbage building up. | ||
* | Tweak how the GC protection works when closing the port | Christopher Baines | 2021-05-28 |
| | | | | Bring more stuff inside one with-gc-protection block. | ||
* | Increase the size of one of the buffers for sending files | Christopher Baines | 2021-05-28 |
| | | | | As this reduces the GC disabling/enabling. | ||
* | Add more gc protection | Christopher Baines | 2021-05-28 |
| | |||
* | Use dyanmic-wind when changing the GC configuration | Christopher Baines | 2021-05-28 |
| | | | | | | | These procedures actually increment/decrement a counter, so gc-enable might not enable garbage collection if gc-disable has been called twice in a row. dynamic-wind should ensure that gc-enable is always called after gc-disable, even if thunk raises an exception for example. | ||
* | Don't automatically requeue jobs | Christopher Baines | 2021-05-28 |
| | | | | | | | | Even if they aren't requeued, the agent should learn about the job again from the coordinator. I'm mostly removing this, because I'm seeing agents seemingly process the same job twice at the same time, and I wonder if it's related. | ||
* | Add a space in coordinator-handle-failed-request | Christopher Baines | 2021-05-28 |
| | |||
* | Improve retrying for missing log files | Christopher Baines | 2021-05-28 |
| | |||
* | Have with-gc-protection return the thunk return values properly | Christopher Baines | 2021-05-28 |
| | |||
* | Use GC protection for normal requests to the coordinator as well | Christopher Baines | 2021-05-28 |
| | | | | | Since the gc breaking gnutls problem can occur for these requests probably as well. | ||
* | Increase the buffer size for sending outputs and log files | Christopher Baines | 2021-05-28 |
| | | | | I think this works better. | ||
* | Get rid of the request mutex | Christopher Baines | 2021-05-28 |
| | | | | | | | | | This was put in to try and prevent the crashes inside gnutls, but was ineffective since the actual trigger for the issues is garbage collection, rather than parallel requests. There might be some benefit from limiting request parallelism in the future, but that can be thought through then. | ||
* | Tune sending files over HTTP | Christopher Baines | 2021-05-28 |
| | | | | | | | | | Guile's garbage collector interferes with Guile+gnutls, which means that sending files while the garbage collector is active is difficult. These changes try to work around this by disabling the garbage collector just as the data is being written, then enabling it again. I think this helps to work around the issue. | ||
* | Fix datastore-find-derivation-for-output | Christopher Baines | 2021-05-27 |
| | |||
* | Fix datastore-list-unbuilt-derivation-outputs | Christopher Baines | 2021-05-27 |
| | |||
* | Fix datastore-list-build-outputs for the new outputs table | Christopher Baines | 2021-05-27 |
| | |||
* | Reduce the threshold for compressing nars on the fly | Christopher Baines | 2021-05-26 |
| | | | | | Prefer upfront compression, as this might reduce GC activity while sending the data. | ||
* | Remove stale log files | Christopher Baines | 2021-05-26 |
| | |||
* | Don't ignore premature termination for chunked requests | Christopher Baines | 2021-05-26 |
| | | | | | | This code was copied from Guile, but this seems like a deficiency. I can't imagine a case where you'd be processing chunked data, and just want to pretend you've got to the end, when you haven't. | ||
* | Delay storing derivations in the database | Christopher Baines | 2021-05-21 |
| | | | | | Until actually storing the build, since the build might not actually be submitted if there's a build for those outputs already. | ||
* | Return #f from datastore-find-derivation-outputs if none are found | Christopher Baines | 2021-05-21 |
| | | | | As this is clearer than '(). | ||
* | Allow datastore-find-derivation-system to return #f | Christopher Baines | 2021-05-21 |
| | | | | If the derivation doesn't exist. | ||
* | Drop the request mutex for most requests | Christopher Baines | 2021-05-21 |
| | | | | Just use it when uploading files. | ||
* | Enable foreign key checking | Christopher Baines | 2021-05-21 |
| | | | | I thought this would be on by default, but apparently not. | ||
* | Introduce a systems table | Christopher Baines | 2021-05-21 |
| | |||
* | Introduce an outputs table | Christopher Baines | 2021-05-21 |
| | |||
* | Simplify the work queue loop | Christopher Baines | 2021-05-14 |
| | |||
* | Buffer the network socket used for streaming http requests | Christopher Baines | 2021-05-13 |
| | | | | As this helps improve throughput. | ||
* | Use a bigger buffer when uploading logs | Christopher Baines | 2021-05-13 |
| | | | | As I think this might make it faster. | ||
* | Increase the basic build allocation strategy default planned builds | Christopher Baines | 2021-04-27 |
| | | | | | Mostly so the coordinator for building patches plans more builds at once, since the allocator is very slow at the moment. | ||
* | Stop agent threads only when 2 or more are idle | Christopher Baines | 2021-04-23 |
| | | | | | | If the jobs are really quick, I think the one running thread keeps stopping and starting, and that stops the agent starting more threads. I think this change might help. | ||
* | Handle receiving outputs as a bytevector | Christopher Baines | 2021-04-23 |
| | | | | This can happen if the request doesn't arrive in chunks. | ||
* | Remove non-existent export | Christopher Baines | 2021-04-20 |
| | |||
* | Make some SQLite related improvements | Christopher Baines | 2021-04-20 |
| | | | | | Don't keep database connections around forever as this relates to cached query plans, and also run the optimize pragma when closing connections. | ||
* | Support destructors and lifetimes for worker threads | Christopher Baines | 2021-04-20 |
| | |||
* | Support finding blocking builds not including cancelled ones | Christopher Baines | 2021-04-16 |
| | | | | As including cancelled ones can make the query much slower. | ||
* | Handle receiving logs as bytevectors | Christopher Baines | 2021-04-09 |
| | | | | I think this can happen if the log doesn't arrive as a chunked HTTP request. | ||
* | Always join against the derivations table | Christopher Baines | 2021-04-04 |
| | | | | | As that's now needed for the derivation name in datastore-list-failed-builds-with-blocking-count. | ||
* | Truncate the WAL if it exceeds 100MiB | Christopher Baines | 2021-04-04 |
| | | | | As database performance seems to start to drop off around this point. | ||
* | Fix tagging builds | Christopher Baines | 2021-04-02 |
| | |||
* | Always include derivation join | Christopher Baines | 2021-04-02 |
| |