From 00753f7038234a0f5a79be3ec9ab949840a18743 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= Date: Mon, 17 Apr 2017 23:13:40 +0200 Subject: publish: Add '--cache' and '--workers'. Fixes . Reported by . These options allow nars to be "baked" off-line and cached instead of being compressed on the fly. As a side-effect, this allows us to provide a 'Content-Length' header for nars. * guix/scripts/publish.scm (show-help, %options): Add '--cache' and '--workers'. (%default-options): Add 'workers'. (nar-cache-file, narinfo-cache-file, run-single-baker): New procedures. (single-baker): New macro. (render-narinfo/cached, bake-narinfo+nar) (render-nar/cached): New procedures. (make-request-handler): Add #:cache and #:pool parameters and honor them. (run-publish-server): Likewise. (guix-publish): Honor '--cache' and '--workers'. * tests/publish.scm ("with cache"): New test. * doc/guix.texi (Invoking guix publish): Document it. --- doc/guix.texi | 46 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index fd3483ee5d..bbb2ba732d 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6522,6 +6522,13 @@ archive}), the daemon may download substitutes from it: guix-daemon --substitute-urls=http://example.org:8080 @end example +By default, @command{guix publish} compresses archives on the fly as it +serves them. This ``on-the-fly'' mode is convenient in that it requires +no setup and is immediately available. However, when serving lots of +clients, we recommend using the @option{--cache} option, which enables +caching of the archives before they are sent to clients---see below for +details. + As a bonus, @command{guix publish} also serves as a content-addressed mirror for source files referenced in @code{origin} records (@pxref{origin Reference}). For instance, assuming @command{guix @@ -6559,10 +6566,43 @@ disable compression. The range 1 to 9 corresponds to different gzip compression levels: 1 is the fastest, and 9 is the best (CPU-intensive). The default is 3. -Compression occurs on the fly and the compressed streams are not +Unless @option{--cache} is used, compression occurs on the fly and +the compressed streams are not cached. Thus, to reduce load on the machine that runs @command{guix -publish}, it may be a good idea to choose a low compression level, or to -run @command{guix publish} behind a caching proxy. +publish}, it may be a good idea to choose a low compression level, to +run @command{guix publish} behind a caching proxy, or to use +@option{--cache}. Using @option{--cache} has the advantage that it +allows @command{guix publish} to add @code{Content-Length} HTTP header +to its responses. + +@item --cache=@var{directory} +@itemx -c @var{directory} +Cache archives and meta-data (@code{.narinfo} URLs) to @var{directory} +and only serve archives that are in cache. + +When this option is omitted, archives and meta-data are created +on-the-fly. This can reduce the available bandwidth, especially when +compression is enabled, since this may become CPU-bound. Another +drawback of the default mode is that the length of archives is not known +in advance, so @command{guix publish} does not add a +@code{Content-Length} HTTP header to its responses, which in turn +prevents clients from knowing the amount of data being downloaded. + +Conversely, when @option{--cache} is used, the first request for a store +item (@i{via} a @code{.narinfo} URL) returns 404 and triggers a +background process to @dfn{bake} the archive---computing its +@code{.narinfo} and compressing the archive, if needed. Once the +archive is cached in @var{directory}, subsequent requests succeed and +are served directly from the cache, which guarantees that clients get +the best possible bandwidth. + +The ``baking'' process is performed by worker threads. By default, one +thread per CPU core is created, but this can be customized. See +@option{--workers} below. + +@item --workers=@var{N} +When @option{--cache} is used, request the allocation of @var{N} worker +threads to ``bake'' archives. @item --ttl=@var{ttl} Produce @code{Cache-Control} HTTP headers that advertise a time-to-live -- cgit v1.2.3