summaryrefslogtreecommitdiff
path: root/guix/scripts/offload.scm
AgeCommit message (Collapse)Author
2018-06-15offload: Fix error message in 'guix offload test'.Ludovic Courtès
Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com> in <https://bugs.gnu.org/31824>. * guix/scripts/offload.scm (assert-node-has-guix): Fix typo in failure message; add missing argument.
2018-06-14offload: Gracefully handle invalid results from 'machines.scm'.Ludovic Courtès
* guix/scripts/offload.scm (build-machines): Check the result of FILE. Ignore it if it's not a list of <build-machine>.
2018-06-11offload: Honor the build timeout internally.Ludovic Courtès
* guix/scripts/offload.scm (call-with-timeout): New procedure. (with-timeout): New macro. (process-request): Use it around 'transfer-and-offload' call.
2018-01-15offload: Look at machine loads for the past minute.Ludovic Courtès
Previously we were looking at the load of the past 5 minutes, which means that, after a build, we could end up waiting for 5 minutes for that metric to be low enough. * guix/scripts/offload.scm (machine-load): Compute RAW based on ONE, not FIVE.
2018-01-12offload: 'test' reports Guile and module errors more nicely.Ludovic Courtès
Fixes <https://bugs.gnu.org/26008>. Reported by Myles English <mylesenglish@gmail.com>. * guix/ssh.scm (retrieve-files*): Move error reporting to... (report-guile-error, report-module-error): ... here. New procedures. * guix/scripts/offload.scm (assert-node-repl): Use 'report-guile-error'. (assert-node-has-guix): Explicitly check for 'use-modules' first. Use 'report-module-error'.
2018-01-12offload: 'test' gracefully handles 'node-repl-error'.Ludovic Courtès
Fixes <https://bugs.gnu.org/28057>. Reported by Ricardo Wurmus <rekado@elephly.net>. * guix/scripts/offload.scm (assert-node-has-guix): Catch 'node-repl-error' and call 'leave'.
2018-01-12offload: Fix regression in file retrieval.Ludovic Courtès
This fixes a regression in 'retrieve-files*' introduced in 896fec476f728183b331cbb6e2afb891207b4205, whereby (guix scripts offload) would not read the initial sexp now sent by the remote host via 'store-export-channel'. This would effectively prevent file retrieval entirely when offloading. * guix/ssh.scm (retrieve-files*): New procedure, like former 'retrieve-files' but with an extra #:import parameter. (retrieve-files): Rewrite in terms of 'retrieve-files*'. (file-retrieval-port): Make private. * guix/scripts/offload.scm (transfer-and-offload): Pass #:import to 'retrieve-files*'. (retrieve-files*): Remove.
2017-12-16guix: offload: Add "status" sub-command.Ricardo Wurmus
* guix/scripts/offload.scm (check-machine-status): New procedure. (guix-offload): Call it when the argument is "status". * doc/guix.texi (Daemon Offload Setup): Document it.
2017-12-01Do not set '%fresh-auto-compile'.Ludovic Courtès
* guix/scripts/offload.scm (build-machines): Comment out '(set! %fresh-auto-compile #t)' since with Guile 2.2.3 it could lead to an actual rebuild of everything that gets loaded from there on. See <https://bugs.gnu.org/29226>. * guix/ui.scm (load*): Likewise.
2017-10-12offload: Reduce the number of calls to 'machine-load'.Ludovic Courtès
Previously we would call 'machine-load' once per machine, which was very costly when there were many machines. Now we arrange to call it only once on average (when all the machines have the same 'speed' value). * guix/scripts/offload.scm (random-seed, shuffle): New procedures. (choose-build-machine)[machines+slots+loads]: Rename to... [machines+slots]: ... this. Remove load from the tuples therein. [undecorate]: Adjust accordingly. [machine-less-loaded-or-faster?]: Remove. [machine-faster?]: New procedure. Sort MACHINES+SLOTS according to 'machine-faster?'. Call 'machine-load?' as the last thing.
2017-07-25offload: Fix potential file descriptor and memory leak.Ludovic Courtès
The '%slots' list could grow indefinitely; in practice though, guix-daemon is likely to restart 'guix offload' often enough. * guix/scripts/offload.scm (%slots): Remove. (choose-build-machine): Don't 'set!' %SLOTS. Return the acquired slot as a second value. (process-request): Adjust accordingly. Release the returned slot after 'transfer-and-offload'.
2017-07-25offload: Disconnect sessions created by 'machine-load'.Ludovic Courtès
This fixes a memory leak that can be seen by running: (map (lambda _ (machine-load m)) (iota 1000)) * guix/scripts/offload.scm (machine-load): Add call to 'disconnect!'.
2017-06-12derivations: Introduce 'read-derivation-from-file'.Ludovic Courtès
This avoids the open/fstat/close syscalls upon a cache hit that we had with the previous idiom: (call-with-input-file file read-derivation) where caching happened in 'read-derivation' itself. * guix/derivations.scm (%read-derivation): Rename to... (read-derivation): ... this. (read-derivation-from-file): New procedure. (derivation-prerequisites, substitution-oracle) (derivation-prerequisites-to-build): (derivation-path->output-path, derivation-path->output-paths): (derivation-path->base16-hash, map-derivation): Use 'read-derivation-from-file' instead of (call-with-input-file … read-derivation). * guix/grafts.scm (item->deriver): Likewise. * guix/scripts/build.scm (log-url, options->things-to-build): Likewise. * guix/scripts/graph.scm (file->derivation): Remove. (derivation-dependencies, %derivation-node-type): Use 'read-derivation-from-file' instead. * guix/scripts/offload.scm (guix-offload): Likewise. * guix/scripts/perform-download.scm (guix-perform-download): Likewise. * guix/scripts/publish.scm (load-derivation): Remove. (narinfo-string): Use 'read-derivation-from-file'.
2017-05-28scripts: Set thread names.Ludovic Courtès
This allows 'guix publish' threads as well as 'guix substitute' and 'guix offload' processes to be properly labeled in 'top', 'pstree', etc. * guix/workers.scm (worker-thunk): Add #:thread-name parameter and honor it. (make-pool): Likewise. * guix/scripts/publish.scm (http-write): Add calls to 'set-thread-name' in bodies of 'call-with-new-thread'. (guix-publish): Call 'set-thread-name'. Pass #:thread-name to 'make-pool'. * guix/scripts/offload.scm (guix-offload): Call 'set-thread-name'. * guix/scripts/substitute.scm (guix-substitute): Likewise.
2017-05-03ui: Rename '_' to 'G_'.Ludovic Courtès
This avoids collisions with '_' when the latter is used as a 'match' pattern for instance. See <https://lists.gnu.org/archive/html/guix-devel/2017-04/msg00464.html>. * guix/ui.scm: Rename '_' to 'G_'. * po/guix/Makevars (XGETTEXT_OPTIONS): Adjust accordingly. * build-aux/compile-all.scm (warnings): Remove 'format'. * gnu/packages.scm, gnu/services.scm, gnu/services/shepherd.scm, gnu/system.scm, gnu/system/shadow.scm, guix/gnupg.scm, guix/http-client.scm, guix/import/cpan.scm, guix/import/elpa.scm, guix/import/pypi.scm, guix/nar.scm, guix/scripts.scm, guix/scripts/archive.scm, guix/scripts/authenticate.scm, guix/scripts/build.scm, guix/scripts/challenge.scm, guix/scripts/container.scm, guix/scripts/container/exec.scm, guix/scripts/copy.scm, guix/scripts/download.scm, guix/scripts/edit.scm, guix/scripts/environment.scm, guix/scripts/gc.scm, guix/scripts/graph.scm, guix/scripts/hash.scm, guix/scripts/import.scm, guix/scripts/import/cpan.scm, guix/scripts/import/cran.scm, guix/scripts/import/crate.scm, guix/scripts/import/elpa.scm, guix/scripts/import/gem.scm, guix/scripts/import/gnu.scm, guix/scripts/import/hackage.scm, guix/scripts/import/nix.scm, guix/scripts/import/pypi.scm, guix/scripts/import/stackage.scm, guix/scripts/lint.scm, guix/scripts/offload.scm, guix/scripts/pack.scm, guix/scripts/package.scm, guix/scripts/perform-download.scm, guix/scripts/publish.scm, guix/scripts/pull.scm, guix/scripts/refresh.scm, guix/scripts/size.scm, guix/scripts/substitute.scm, guix/scripts/system.scm, guix/ssh.scm, guix/upstream.scm: Use 'G_' instead of '_'. Most of this change was obtained by running: "sed -i -e's/(_ "/(G_ "/g' `find -name \*.scm`".
2017-04-21offload: Avoid using '_' as a 'match' pattern.Ludovic Courtès
* guix/scripts/offload.scm (host-key->type+key, machine-load) (process-request, guix-offload): Do not use '_' as a 'match' pattern.
2016-12-31Add (guix ssh) module.Ludovic Courtès
* guix/scripts/offload.scm (connect-to-remote-daemon) (store-import-channel, store-export-channel, send-files) (retrieve-files): Move to (guix ssh). (nonce): Add optional 'name' parameter and use it. (retrieve-files*): New procedure. (transfer-and-offload): Use it instead of 'retrieve-files', and add first parameter to 'send-files'. (assert-node-can-import): Likewise. (assert-node-can-export): Use 'retrieve-files' instead of 'store-export-channel'. * guix/ssh.scm: New file. * configure.ac: Use 'GUIX_CHECK_GUILE_SSH' and define 'HAVE_GUILE_SSH' Automake conditional. * Makefile.am (MODULES) [HAVE_GUILE_SSH]: Add guix/ssh.scm.
2016-12-09offload: Allow testing machines that match a regexp.Ludovic Courtès
* guix/scripts/offload.scm (check-machine-availability): Add 'pred' parameter and honor it. (guix-offload): for the "test" sub-command, accept an extra 'regexp' parameter. Pass a second argument to 'check-machine-availability'.
2016-12-09offload: Test each machine only once.Ludovic Courtès
* guix/scripts/offload.scm (check-machine-availability)[build-machine=?]: New procedure. Add call to 'delete-duplicates'.
2016-12-09offload: Do not read ~/.ssh/known_hosts.Ludovic Courtès
* guix/scripts/offload.scm (open-ssh-session): Pass #:knownhosts to 'make-session'.
2016-12-06offload: Increase the connection timeout.Ludovic Courtès
* guix/scripts/offload.scm (open-ssh-session): Set #:timeout to 10.
2016-12-06offload: Send the build log to the right file descriptor.Ludovic Courtès
This fixes a regression introduced in 21531add3205e400707c8fbfd841845f9a71863a whereby the build log would no longer be sent to FD 4, thereby leading the daemon to not see the build log. * guix/scripts/offload.scm (transfer-and-offload): Parameterize CURRENT-BUILD-OUTPUT-PORT.
2016-12-05offload: Fix plural of some messages.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Use 'N_' for possibly plural message. Write "store item" instead of "store file". (retrieve-files): Likewise.
2016-12-05offload: Make the compression methods configurable.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[compression] [compression-level]: New fields. (open-ssh-session): Honor them. * doc/guix.texi (Daemon Offload Setup): Document them.
2016-12-05offload: Add "test" sub-command.Ludovic Courtès
* guix/scripts/offload.scm (assert-node-repl, assert-node-has-guix) (nonce, assert-node-can-import, assert-node-can-export) (check-machine-availability): New procedures. (%random-state): New variable. (guix-offload): Add case for "test". * doc/guix.texi (Daemon Offload Setup): Document it. Remove obsolete bit about remote invocation of 'guix build'.
2016-12-01offload: Do not abort when a machine is unreachable.Ludovic Courtès
* guix/scripts/offload.scm (machine-load): Wrap 'open-ssh-session' call in 'false-if-exception'; return +inf.0 if it returns #f.
2016-12-01offload: Gracefully report connection failures.Ludovic Courtès
* guix/scripts/offload.scm (open-ssh-session): Check the return value of 'connect!'. Call 'leave' when it's not 'ok.
2016-12-01offload: Warn about the lack of zlib support.Ludovic Courtès
* guix/scripts/offload.scm (guix-offload): Print a warning when 'zlib-support?' returns false.
2016-12-01offload: Remove redundant call to 'topologically-sorted' in 'send-file'.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Remove call to 'topologically-sorted'.
2016-11-26offload: Call 'machine-load' only once per machine.Ludovic Courtès
This fixes a longstanding issue where 'choose-build-machine' would make on average O(N log(N)) calls to 'machine-load', plus an extra call for the selected machine, instead of N calls. * guix/scripts/offload.scm (machine-load): Add comment. (machine-power-factor, machine-less-loaded-or-faster?): Remove. (choose-build-machine)[machines+slots]: Rename to... [machines+slots+loads]: ... this. [undecorate]: Adjust accordingly. [machine-less-loaded-or-faster?]: New procedure. Remove extra 'machine-load' call in body.
2016-11-25offload: Drop 'remote-pipe'.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Remove. (machine-load): Use 'open-remote-pipe*' instead of 'remote-pipe'.
2016-11-25offload: Rewrite to make direct RPCs to the remote daemon.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field. (connect-to-remote-daemon): New procedure. (%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove. (transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and RPCs over SSH. (store-import-channel, store-export-channel): New procedures. (send-files, retrieve-files): Rewrite using these.
2016-11-25offload: Remove 'with-nar-error-handling' macro.Ludovic Courtès
* guix/scripts/offload.scm (with-nar-error-handling): Remove. (guix-offload): Use 'with-error-handling' instead.
2016-11-25offload: Reuse SSH session during 'transfer-and-offload'.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Replace 'machine' parameter with 'session'. Remove 'open-ssh-session' call. (register-gc-root): Replace 'machine' with 'session'. Use ' session-get' instead of 'build-machine-name'. (remove-gc-roots, offload, send-files, retrieve-files): Likewise. (transfer-and-offload): Add 'open-ssh-session' call. Handle 'offload' errors here. (machine-load): Add call to 'open-ssh-session'.
2016-11-25offload: Use Guile-SSH instead of GNU lsh.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove. [host-key, host-key-type]: New fields. (%lsh-command, %lshg-command, user-lsh-private-key): Remove. (user-openssh-private-key, private-key-from-file*): New procedures. (host-key->type+key, open-ssh-session): New procedures. (remote-pipe): Remove 'mode' parameter. Rewrite in terms of 'open-ssh-session' etc. Update users. (send-files)[missing-files]: Rewrite using the bidirectional channel port. Remove call to 'call-with-compressed-output-port'. (retrieve-files): Remove call to 'call-with-decompressed-port'. (machine-load): Remove exit status logic. * doc/guix.texi (Requirements): Mention Guile-SSH. (Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the default value on each @item line. * m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro. * config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set 'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
2016-10-19Use (ice-9 binary-ports) instead of (rnrs io ports).Ludovic Courtès
This reduces the closure of (guix ui) from 123 to 106 modules. * guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io ports). (map-derivation)[substitute-file]: Use 'read-string' instead of 'get-string-all'. * guix/ftp-client.scm: Likewise. * guix/hash.scm: Likewise. * guix/http-client.scm: Likewise. * guix/pki.scm (ensure-acl, current-acl): Likewise. * guix/scripts/archive.scm (authorize-key)[read-key]: Likewise. * guix/scripts/authenticate.scm (read-canonical-sexp) (read-hash-data): Likewise. * guix/scripts/download.scm: Likewise. * guix/scripts/offload.scm (register-gc-root, remove-gc-roots) (send-files): Likewise. * guix/scripts/publish.scm (lazy-read-file-sexp): Likewise. * guix/scripts/refresh.scm: Likewise. * guix/scripts/substitute.scm (check-acl-initialized): Likewise. * guix/serialization.scm (read-maybe-utf8-string): Likewise. * guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of 'flush-output-port'. * guix/store.scm (process-stderr): Likewise. * guix/tests.scm: Likewise. * guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports) for 'make-custom-binary-input-port'.
2016-09-20ui: Do not shadow '_' where it's used as a literal syntax match.Ludovic Courtès
Fixes compilation with Guile 2.1. Reported by Mu Lei. * guix/ui.scm (report-load-error) (warn-about-load-error, read/eval-package-expression): Use 'rest' instead of '_' as the pattern variable name. * gnu/packages.scm (%find-package): Likewise. * guix/scripts/build.scm (transform-package-inputs): Likewise. * guix/scripts/hash.scm (guix-hash): Likewise. * guix/scripts/import/gnu.scm (%options, guix-import-gnu): Likewise. * guix/scripts/import/nix.scm (guix-import-nix): Likewise. * guix/scripts/offload.scm (build-machines): Likewise. * guix/scripts/refresh.scm (%options): Likewise. * guix/scripts/substitute.scm (narinfo-signature->canonical-sexp): Likewise.
2016-05-31offload: Use (guix build syscalls).Ludovic Courtès
This is a followup to 4e0ea3eb288c2143b44bf324c64047762c72d3b3. * guix/scripts/offload.scm: Use (guix build syscalls).
2015-09-25offload: Use gzip instead of xz for compression on the master.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Use gzip --fast instead of xz.
2015-07-09offload: Add 'ssh-options' field to <build-machine>.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[ssh-options]: New field. (remote-pipe): Use it. (send-files): Likewise.
2015-06-17offload: Fix sorting bug in 'choose-build-machine'.Mark H Weaver
* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Return the boolean result of pred instead of the best machine+slot.
2015-02-05offload: Better report failure to create the GC root directory.Ludovic Courtès
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>. * guix/scripts/offload.scm (register-gc-root)[script]: Replace 'false-if-exception' with a finer-grain 'system-error handler. Provide the name of MACHINE in 'leave' error message.
2015-02-05offload: Warn about SSH client issues.Ludovic Courtès
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>. * guix/scripts/offload.scm (remote-pipe): Remove unneeded 'catch'. (machine-load): Check the exit value upon (close-pipe pipe). Call 'warning' when it is non-zero.
2014-10-29offload: Remove mutual exclusion on transfers.Ludovic Courtès
Suggested by Mark H. Weaver <mhw@netris.org> at <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00352.html>. * guix/scripts/offload.scm (transfer-and-offload): Remove uses of 'with-machine-lock'.
2014-10-09Break module cycle involving (guix store) and (guix ui).Ludovic Courtès
Before, there was a cycle along the lines of: (guix store) -> (guix nar) -> (guix ui) -> (guix store) This caused problems, as discussed at: http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html This patch removes cycles in the (guix ...) modules. * guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents, read-contents, %archive-version-1, write-file, restore-file): Move to... * guix/serialization.scm: ... here. * guix/store.scm: Remove dependency on (guix nar). * guix/scripts/hash.scm, guix/scripts/offload.scm, guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm, tests/substitute-binary.scm: Adjust accordingly.
2014-09-20offload: Use a total order when sorting available machines.Ludovic Courtès
* guix/scripts/offload.scm (machine-less-loaded?, machine-faster?): Remove. (machine-power-factor): New procedure. (machine-less-loaded-or-faster?): Use it.
2014-09-20offload: Try another machine when the "best" machine is overloaded.Ludovic Courtès
* guix/scripts/offload.scm (choose-build-machine): When BEST is overloaded, try the other machines.
2014-08-29offload: Ignore EEXIST when registering a .drv as a GC root.Ludovic Courtès
Fixes <http://bugs.gnu.org/18115>. Reported by Mark H Weaver <mhw@netris.org>. * guix/scripts/offload.scm (register-gc-root)[script]: Wrap 'symlink' call in "catch 'system-error", and ignore EEXIST errors.
2014-08-29offload: Ignore unreachable machines.Ludovic Courtès
Fixes <http://bugs.gnu.org/18070>. Reported by Andreas Enge <andreas@enge.fr>. * guix/scripts/offload.scm (remote-pipe): Augment docstring. (machine-load): Return +inf.0 instead of 1 if MACHINE does not respond or responds badly.
2014-06-27guix {system,offload}: Improve reporting of syntax errors.Ludovic Courtès
* guix/scripts/system.scm (read-operating-system) <catch handler>: Add case for 'syntax-error'. Correct message for default case. * guix/scripts/offload.scm (build-machines) <catch handler>: Add case for 'syntax-error'. * tests/guix-system.sh: New file. * Makefile.am (SH_TESTS): Add it.