summaryrefslogtreecommitdiff
path: root/guix/scripts/offload.scm
AgeCommit message (Collapse)Author
2016-11-26offload: Call 'machine-load' only once per machine.Ludovic Courtès
This fixes a longstanding issue where 'choose-build-machine' would make on average O(N log(N)) calls to 'machine-load', plus an extra call for the selected machine, instead of N calls. * guix/scripts/offload.scm (machine-load): Add comment. (machine-power-factor, machine-less-loaded-or-faster?): Remove. (choose-build-machine)[machines+slots]: Rename to... [machines+slots+loads]: ... this. [undecorate]: Adjust accordingly. [machine-less-loaded-or-faster?]: New procedure. Remove extra 'machine-load' call in body.
2016-11-25offload: Drop 'remote-pipe'.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Remove. (machine-load): Use 'open-remote-pipe*' instead of 'remote-pipe'.
2016-11-25offload: Rewrite to make direct RPCs to the remote daemon.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[daemon-socket]: New field. (connect-to-remote-daemon): New procedure. (%gc-root-file, register-gc-root, remove-gc-roots, offload): Remove. (transfer-and-offload): Rewrite using 'connect-to-remote-daemon' and RPCs over SSH. (store-import-channel, store-export-channel): New procedures. (send-files, retrieve-files): Rewrite using these.
2016-11-25offload: Remove 'with-nar-error-handling' macro.Ludovic Courtès
* guix/scripts/offload.scm (with-nar-error-handling): Remove. (guix-offload): Use 'with-error-handling' instead.
2016-11-25offload: Reuse SSH session during 'transfer-and-offload'.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Replace 'machine' parameter with 'session'. Remove 'open-ssh-session' call. (register-gc-root): Replace 'machine' with 'session'. Use ' session-get' instead of 'build-machine-name'. (remove-gc-roots, offload, send-files, retrieve-files): Likewise. (transfer-and-offload): Add 'open-ssh-session' call. Handle 'offload' errors here. (machine-load): Add call to 'open-ssh-session'.
2016-11-25offload: Use Guile-SSH instead of GNU lsh.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[ssh-options]: Remove. [host-key, host-key-type]: New fields. (%lsh-command, %lshg-command, user-lsh-private-key): Remove. (user-openssh-private-key, private-key-from-file*): New procedures. (host-key->type+key, open-ssh-session): New procedures. (remote-pipe): Remove 'mode' parameter. Rewrite in terms of 'open-ssh-session' etc. Update users. (send-files)[missing-files]: Rewrite using the bidirectional channel port. Remove call to 'call-with-compressed-output-port'. (retrieve-files): Remove call to 'call-with-decompressed-port'. (machine-load): Remove exit status logic. * doc/guix.texi (Requirements): Mention Guile-SSH. (Daemon Offload Setup): Document 'host-key' and 'private-key'. Show the default value on each @item line. * m4/guix.m4 (GUIX_CHECK_GUILE_SSH): New macro. * config-daemon.ac: Use 'GUIX_CHECK_GUILE_SSH'. Set 'HAVE_DAEMON_OFFLOAD_HOOK' as a function of that.
2016-10-19Use (ice-9 binary-ports) instead of (rnrs io ports).Ludovic Courtès
This reduces the closure of (guix ui) from 123 to 106 modules. * guix/derivations.scm: Use (ice-9 binary-ports) instead of (rnrs io ports). (map-derivation)[substitute-file]: Use 'read-string' instead of 'get-string-all'. * guix/ftp-client.scm: Likewise. * guix/hash.scm: Likewise. * guix/http-client.scm: Likewise. * guix/pki.scm (ensure-acl, current-acl): Likewise. * guix/scripts/archive.scm (authorize-key)[read-key]: Likewise. * guix/scripts/authenticate.scm (read-canonical-sexp) (read-hash-data): Likewise. * guix/scripts/download.scm: Likewise. * guix/scripts/offload.scm (register-gc-root, remove-gc-roots) (send-files): Likewise. * guix/scripts/publish.scm (lazy-read-file-sexp): Likewise. * guix/scripts/refresh.scm: Likewise. * guix/scripts/substitute.scm (check-acl-initialized): Likewise. * guix/serialization.scm (read-maybe-utf8-string): Likewise. * guix/scripts/hash.scm (guix-hash): Use 'force-output' instead of 'flush-output-port'. * guix/store.scm (process-stderr): Likewise. * guix/tests.scm: Likewise. * guix/utils.scm: Use (ice-9 binary-ports) and autoload (rnrs io ports) for 'make-custom-binary-input-port'.
2016-09-20ui: Do not shadow '_' where it's used as a literal syntax match.Ludovic Courtès
Fixes compilation with Guile 2.1. Reported by Mu Lei. * guix/ui.scm (report-load-error) (warn-about-load-error, read/eval-package-expression): Use 'rest' instead of '_' as the pattern variable name. * gnu/packages.scm (%find-package): Likewise. * guix/scripts/build.scm (transform-package-inputs): Likewise. * guix/scripts/hash.scm (guix-hash): Likewise. * guix/scripts/import/gnu.scm (%options, guix-import-gnu): Likewise. * guix/scripts/import/nix.scm (guix-import-nix): Likewise. * guix/scripts/offload.scm (build-machines): Likewise. * guix/scripts/refresh.scm (%options): Likewise. * guix/scripts/substitute.scm (narinfo-signature->canonical-sexp): Likewise.
2016-05-31offload: Use (guix build syscalls).Ludovic Courtès
This is a followup to 4e0ea3eb288c2143b44bf324c64047762c72d3b3. * guix/scripts/offload.scm: Use (guix build syscalls).
2015-09-25offload: Use gzip instead of xz for compression on the master.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Use gzip --fast instead of xz.
2015-07-09offload: Add 'ssh-options' field to <build-machine>.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>)[ssh-options]: New field. (remote-pipe): Use it. (send-files): Likewise.
2015-06-17offload: Fix sorting bug in 'choose-build-machine'.Mark H Weaver
* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Return the boolean result of pred instead of the best machine+slot.
2015-02-05offload: Better report failure to create the GC root directory.Ludovic Courtès
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>. * guix/scripts/offload.scm (register-gc-root)[script]: Replace 'false-if-exception' with a finer-grain 'system-error handler. Provide the name of MACHINE in 'leave' error message.
2015-02-05offload: Warn about SSH client issues.Ludovic Courtès
Suggested by Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de>. * guix/scripts/offload.scm (remote-pipe): Remove unneeded 'catch'. (machine-load): Check the exit value upon (close-pipe pipe). Call 'warning' when it is non-zero.
2014-10-29offload: Remove mutual exclusion on transfers.Ludovic Courtès
Suggested by Mark H. Weaver <mhw@netris.org> at <http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00352.html>. * guix/scripts/offload.scm (transfer-and-offload): Remove uses of 'with-machine-lock'.
2014-10-09Break module cycle involving (guix store) and (guix ui).Ludovic Courtès
Before, there was a cycle along the lines of: (guix store) -> (guix nar) -> (guix ui) -> (guix store) This caused problems, as discussed at: http://lists.gnu.org/archive/html/guix-devel/2014-10/msg00109.html This patch removes cycles in the (guix ...) modules. * guix/nar.scm (&nar-error, &nar-read-error, dump, write-contents, read-contents, %archive-version-1, write-file, restore-file): Move to... * guix/serialization.scm: ... here. * guix/store.scm: Remove dependency on (guix nar). * guix/scripts/hash.scm, guix/scripts/offload.scm, guix/scripts/substitute-binary.scm, tests/nar.scm, tests/store.scm, tests/substitute-binary.scm: Adjust accordingly.
2014-09-20offload: Use a total order when sorting available machines.Ludovic Courtès
* guix/scripts/offload.scm (machine-less-loaded?, machine-faster?): Remove. (machine-power-factor): New procedure. (machine-less-loaded-or-faster?): Use it.
2014-09-20offload: Try another machine when the "best" machine is overloaded.Ludovic Courtès
* guix/scripts/offload.scm (choose-build-machine): When BEST is overloaded, try the other machines.
2014-08-29offload: Ignore EEXIST when registering a .drv as a GC root.Ludovic Courtès
Fixes <http://bugs.gnu.org/18115>. Reported by Mark H Weaver <mhw@netris.org>. * guix/scripts/offload.scm (register-gc-root)[script]: Wrap 'symlink' call in "catch 'system-error", and ignore EEXIST errors.
2014-08-29offload: Ignore unreachable machines.Ludovic Courtès
Fixes <http://bugs.gnu.org/18070>. Reported by Andreas Enge <andreas@enge.fr>. * guix/scripts/offload.scm (remote-pipe): Augment docstring. (machine-load): Return +inf.0 instead of 1 if MACHINE does not respond or responds badly.
2014-06-27guix {system,offload}: Improve reporting of syntax errors.Ludovic Courtès
* guix/scripts/system.scm (read-operating-system) <catch handler>: Add case for 'syntax-error'. Correct message for default case. * guix/scripts/offload.scm (build-machines) <catch handler>: Add case for 'syntax-error'. * tests/guix-system.sh: New file. * Makefile.am (SH_TESTS): Add it.
2014-04-14offload: '{send,receive}-files' wait for completion of the transfer.Ludovic Courtès
Fixes situations where the remote 'guix build' is invoked before the .drv has been completely copied, as reported at <https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00174.html>. In some cases 'send-files' would return before the other end is done importing the files, and so the subsequent 'guix build' invocation would just miss the .drv file it refers to. * guix/utils.scm (call-with-decompressed-port): Don't close PORT. (call-with-compressed-output-port): Likewise. * tests/utils.scm ("compressed-output-port + decompressed-port"): Adjust accordingly. * guix/scripts/offload.scm (send-files): Add explicit (close-pipe pipe) call. (retrieve-files): Likewise.
2014-04-14offload: Better synchronize with remote invocation of 'guix archive --missing'.Ludovic Courtès
* guix/scripts/offload.scm (send-files)[missing-files]: Call 'waitpid' after reading all of MISSING.
2014-04-08offload: Remove all the GC roots in case of multiple-output derivations.Ludovic Courtès
* guix/scripts/offload.scm (remove-gc-root): Rename to... (remove-gc-roots): ... this. [builder]: Use 'scandir' and remove all the files starting with %GC-ROOT-FILE. (transfer-and-offload): Adjust to renaming; remove 'false-if-exception' wraps.
2014-04-08offload: Bail out when failing to register a GC root on the build machine.Ludovic Courtès
* guix/scripts/offload.scm (register-gc-root): Call 'leave' when 'close-pipe' returns non-zero.
2014-04-03offload: Prevent the '.drv' and build result from being GC'd.Ludovic Courtès
Before that, there was a small time window during which the GC could wipe the .drv (before 'guix build' has been called), or the build result (before 'retrieve-files' has started.) * guix/scripts/offload.scm (remote-pipe): Add #:quote? parameter and honor it. (%gc-root-file): New variable. (register-gc-root, remove-gc-root): New procedures. (offload): Adjust comment. Run 'guix build' with '-r %GC-ROOT-FILE'. (transfer-and-offload): Call 'register-gc-root' before sending (derivation-file-name DRV). Call 'remove-gc-root' after the call to 'offload' or 'retrieve-files'. (send-files): Call 'remote-pipe' with #:quote? #f. (retrieve-files): Likewise.
2014-03-31offload: Exit with code 100 upon build failures.Ludovic Courtès
* guix/scripts/offload.scm (transfer-and-offload): Exit with code 100 upon build failure.
2014-03-26offload: Remove erroneous 'close-pipe' call.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Remove 'close-pipe' call from 'guard' handler ('pipe' here referred to Guile's 'pipe' procedure.)
2014-03-26offload: Wait for the processes involved in 'guix archive --missing'.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Keep the second return value of 'filtered-port'. Call 'waitpid' on it.
2014-03-26offload: Allow one transfer in each direction simultaneously.Ludovic Courtès
* guix/scripts/offload.scm (transfer-and-offload): Use 'upload' lock instead of 'bandwidth' around 'send-files' calls, and 'download' lock around 'retrieve-files' call.
2014-03-26offload: Disable SSH-level compression.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Remove '-z' lsh command line argument. This makes transfers almost an order of magnitude slower. OpenSSH's ssh(1) man page notes: "Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks." See also <http://www.spikelab.org/blog/transfer-largedata-scp-tarssh-tarnc-compared.html>.
2014-03-24offload: Compress files being sent/retrieved.Ludovic Courtès
* guix/scripts/offload.scm (send-files): Add "xz -dc |" to the remote pipe command. Pass PIPE through 'call-with-compressed-output-port'. Remove 'close-pipe' call. (retrieve-files): Add "| xz -c" to the remote pipe command. Pass PIPE through 'call-with-decompressed-port'. Remove 'close-pipe' call.
2014-03-19offload: Send build logs to file descriptor 4.Ludovic Courtès
* guix/scripts/offload.scm (with-error-to-port): New macro. (remote-pipe): Add #:error-port parameter. Use 'with-error-to-port' around 'open-pipe*' call. (build-log-port): New procedure. (offload): Change #:log-port to default to (build-log-port). Call 'remote-pipe' with #:error-port LOG-PORT.
2014-03-13offload: Fix 'choose-build-machine' for several machines.Ludovic Courtès
* guix/scripts/offload.scm (choose-build-machine)[undecorate]: Turn into a two-argument procedure.
2014-03-13offload: Convert the port number to a string when invoking lsh.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe, send-files): Pass the result of 'build-machine-port' to 'number->string'.
2014-03-13offload: Allow build machines to specify a port number.Ludovic Courtès
* guix/scripts/offload.scm (<build-machine>): Add 'port' field. (remote-pipe, send-files): Use lsh's '-p' option when invoking it.
2014-03-09offload: Honor absolute build timeouts.Ludovic Courtès
* guix/scripts/offload.scm (offload): Remove default value for 'build-timeout'. Pass '--timeout' to the remote 'guix build' process. (transfer-and-offload, process-request): Remove default value for 'build-timeout'.
2014-03-09offload: Move macro definitions before use.Ludovic Courtès
* guix/scripts/offload.scm (lock-file, unlock-file, with-file-lock, with-machine-lock, machine-slot-file, acquire-build-slot, release-build-slot): Move definitions above their first use.
2014-03-09offload: Distinguish between 'decline' and 'postpone'.Ludovic Courtès
* guix/scripts/offload.scm (transfer-and-offload): New procedure, with core formerly in 'process-request'. (choose-build-machine): Remove 'requirements' parameter. (process-request): Reply 'decline' when none of MACHINES matches the requirements, and 'postpone' when MACHINES are busy.
2014-03-08offload: Prevent locked files from being GC'd.Ludovic Courtès
* guix/scripts/offload.scm (%slots): New variable. (choose-build-machine): Add SLOT to '%slots'.
2014-03-08offload: Fix thinko.Ludovic Courtès
* guix/scripts/offload.scm (choose-build-machine)[machine+slots]: Use 'filter-map', not 'filter'.
2014-03-08offload: Make 'parallel-builds' a hard limit.Ludovic Courtès
* guix/scripts/offload.scm (machine-choice-lock-file, machine-slot-file, acquire-build-slot, release-build-slot): New procedures. (choose-build-machine): Operate with (machine-choice-lock-file) taken. Acquire a build slot for each of MACHINES. Release those not used.
2014-03-08offload: Further generalize lock files.Ludovic Courtès
* guix/scripts/offload.scm (lock-machine, unlock-machine): Remove. (lock-file, unlock-file): New procedures. (with-file-lock): New macro. (with-machine-lock): Rewrite in terms of 'with-file-lock'.
2014-03-08offload: Generalize the machine lock mechanism.Ludovic Courtès
* guix/scripts/offload.scm (lock-machine): Add 'hint' parameter. (unlock-machine): Remove 'machine' parameter. (with-machine-lock): Add 'hint' parameter, and pass it down. (process-request): Adjust uses of 'with-machine-lock' to pass the 'bandwidth hint.
2014-03-06offload: Serialize file transfers to build machines.Ludovic Courtès
* guix/scripts/offload.scm (machine-lock-file, lock-machine, unlock-machine): New procedures. (with-machine-lock): New macro. (process-request): Wrap 'send-files' and 'retrieve-files' calls in 'with-machine-lock'.
2014-03-01offload: Comment out attempt to set up an lsh gateway.Ludovic Courtès
* guix/scripts/offload.scm (open-ssh-gateway): Comment out. (process-request): Remove call to 'open-ssh-gateway' and to 'kill'.
2014-03-01offload: 'remote-pipe' uses the right SSH key.Ludovic Courtès
* guix/scripts/offload.scm (remote-pipe): Pass -i when invoking %LSHG-COMMAND.
2014-03-01offload: Take the target machine load into account.Ludovic Courtès
* guix/scripts/offload.scm (machine-load, machine-less-loaded?, machine-less-loaded-or-faster?): New procedures. (choose-build-machine): Use 'machine-less-loaded-or-faster?' when sorting. Return the head of MACHINES unless it's loaded is >= 2.
2014-02-02offload: Have 'build-machines' honor its argument.Ludovic Courtès
* guix/scripts/offload.scm (build-machines): Honor FILE.
2014-01-27offload: Do not try to retrieve anything upon build failure.Ludovic Courtès
* guix/scripts/offload.scm (offload): Add 'log-port' keyword parameter. Handle log display here. Return the result of (close-pipe pipe). (process-request): Adjust 'offload' call site accordingly. Call 'retrieve-files' only when 'offload' returns zero; exit when 'offload' returns non-zero.
generated by cgit v1.2.3 (git 2.45.2) at 2024-07-31 17:20:10 +0000