summaryrefslogtreecommitdiff
path: root/guix/build/download.scm
AgeCommit message (Collapse)Author
2024-03-09download: Honor ‘GUIX_DOWNLOAD_METHODS’ environment variable.Ludovic Courtès
This replaces ‘GUIX_DOWNLOAD_FALLBACK_TEST’ and allows you to test various download methods, like so: GUIX_DOWNLOAD_METHODS=nar guix build guile-gcrypt -S --check GUIX_DOWNLOAD_METHODS=disarchive guix build hello -S --check * guix/build/download.scm (%download-methods): New variable. (download-method-enabled?): New procedure. (url-fetch): Define ‘initial-uris’; honor ‘download-method-enabled?’. Call ‘disarchive-fetch/any’ only when the 'disarchive method is enabled. * guix/build/git.scm (git-fetch-with-fallback): Honor ‘download-method-enabled?’. * guix/download.scm (%download-methods): New variable. (%download-fallback-test): Remove. (built-in-download): Add #:download-methods parameter and honor it. (url-fetch*): Pass #:content-addressed-mirrors and #:disarchive-mirrors unconditionally. * guix/git-download.scm (git-fetch/in-band*): Pass “git url” unconditionally. (git-fetch/built-in): Likewise. Pass “download-methods”. * guix/bzr-download.scm (bzr-fetch)[build]: Honor ‘download-method-enabled?’. Pass ‘GUIX_DOWNLOAD_METHODS’ to #:env-vars. * guix/cvs-download.scm (cvs-fetch)[build]: Honor ‘download-method-enabled?’. Pass ‘GUIX_DOWNLOAD_METHODS’ to #:env-vars. * guix/hg-download.scm (hg-fetch): Honor ‘download-method-enabled?’. Pass #:env-vars to ‘gexp->derivation’. * guix/scripts/perform-download.scm (perform-download): Honor “download-methods” from DRV. Parameterize ‘%download-methods’ before calling ‘url-fetch’. (perform-git-download): Likewise. * guix/svn-download.scm (svn-fetch): Honor ‘download-method-enabled?’. Pass ‘GUIX_DOWNLOAD_METHODS’ to #:env-vars. (svn-multi-fetch): Likewise. Change-Id: Ia3402e17f0303dfa964bdc761265efe8a1dd69ab
2022-08-03download: Do not wrap TLS port on GnuTLS >= 3.7.7.Ludovic Courtès
The custom input/output port wrapping the TLS session record port would introduce overhead, and it would also prevent its uses in a non-blocking context--e.g., with Fibers. The port close mechanism added in GnuTLS 3.7.7 allows us to get rid of that wrapper. * guix/build/download.scm (wrap-record-port-for-gnutls<3.7.7): New procedure, with code formerly in 'tls-wrap'. (tls-wrap): Check for 'set-session-record-port-close!' and use it when available; otherwise call 'wrap-record-port-for-gnutls<3.7.7'.
2022-08-01download: 'url-fetch' closes its HTTP/HTTPS port.Ludovic Courtès
* guix/build/download.scm (url-fetch)[fetch]: In the http/https case, close PORT before returning.
2022-03-03download: Fix typo in procedure name.Ludovic Courtès
* guix/build/download.scm (make-credendials-with-ca-trust-files): Rename to... (make-credentials-with-ca-trust-files): ... this.
2022-03-03download: Load X.509 certificates only once.Ludovic Courtès
Previously we'd load /etc/ssl/certs/*.pem (or similar) every time 'http-fetch' is called. * guix/build/download.scm (make-credendials-with-ca-trust-files): Wrap in 'mlambda'.
2022-03-03download: Remove obsolete bit from docstring.Ludovic Courtès
This is a followup to f4cde9ac4aedb516c050a30fd999673da434bfa0. * guix/build/download.scm (open-connection-for-uri): Remove bit about 'close-connection' from the docstring.
2021-10-14download: Print URL of the Disarchive spec being read.Ludovic Courtès
* guix/build/download.scm (disarchive-fetch/any): Show URI of the Disarchive spec.
2021-10-14download: Add missing autoload.Ludovic Courtès
This is a followup to 6d02a994f911a75e3a223a22c05c2939cdfed2b5, which left '%verify-swh-certificate?' unbound. * guix/build/download.scm: Autoload for '%verify-swh-certificate'.
2021-10-07download: Honor #:verify-certificate? for SWH downloads.Ludovic Courtès
Previously, the SWH + Disarchive fallback could fail with: Trying to use Disarchive to assemble /gnu/store/…-ucsim-0.6-pre68.tar.gz... Assembling the directory ucsim-0.6-pre68 Downloading /gnu/store/…-ucsim-0.6-pre68.tar.gz from Software Heritage... X.509 certificate of 'archive.softwareheritage.org' could not be verified: signer-not-found invalid Could not resolve directory reference This will no longer be the case since 'guix perform-download' passes #:verify-certificate? #f. * guix/build/download.scm (disarchive-fetch/any): Parameterize '%verify-swh-certificate?'.
2021-09-22download: Fall back to web.archive.org as a very last resort.Ludovic Courtès
Suggested by Florian Pelz <pelzflorian@pelzflorian.de>. * guix/build/download.scm (internet-archive-uri): New procedure. (url-fetch): Append it to the list of URIs after CONTENT-ADDRESSED-URIS.
2021-09-14download: Disarchive mirrors can be URL-returning procedures.Ludovic Courtès
As discussed at <https://issues.guix.gnu.org/47336#16>. * guix/build/download.scm (url-fetch)[disarchive-uris]: Accept MIRROR as a procedure. * guix/download.scm (%disarchive-mirrors): Add comment. This change can only be made once a 'guix perform-download' that understands procedures is widely deployed.
2021-06-25download: 'tls-wrap' retries handshake upon non-fatal errors.Ludovic Courtès
Fixes <https://bugs.gnu.org/49223>. Reported by Domagoj Stolfa <ds815@gmx.com>. * guix/build/download.scm (tls-wrap): Retry up to 5 times when 'handshake' throws a non-fatal error.
2021-05-05download: Restore error reporting.Timothy Sample
Normal error reporting was disrupted by the introduction of Disarchive in commit fbc2a52a32ddc664db8ebab420c2e17b1432c744. In particular, running 'guix download' would succeed with a partially downloaded file. * guix/build/download.scm (disarchive-fetch/any): Return '#f' when Disarchive cannot be found, the specification cannot be found, or Disarchive fails due to an error.
2021-04-29download: Use Disarchive as a last resort.Timothy Sample
This is a fixed version of 66b14dccdd0d83c875ce3a8d50ceab8b6f0a3ce2, which was reverted in e74250c3c535b75dd2225a26df51febb7ed94654. * guix/download.scm (%disarchive-mirrors): New variable. (%disarchive-mirror-file): New variable. (built-in-download): Add 'disarchive-mirrors' keyword argument and pass its value along to the 'builtin:download' derivation. (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. * guix/scripts/perform-download.scm (perform-download): Read Disarchive mirrors from the environment and pass them to 'url-fetch'. * guix/build/download.scm (disarchive-fetch/any): New procedure. (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to make a list of URIs, and use the new procedure to fetch the file if all other methods fail. * build-aux/build-self.scm (build-program)[select?]: Exclude '(guix build download)'. * guix/self.scm (compiled-guix)[*core-modules*]: Add 'guile-json' to the list of extensions.
2021-04-28Revert "download: Use Disarchive as a last resort."Timothy Sample
This reverts commit 66b14dccdd0d83c875ce3a8d50ceab8b6f0a3ce2, which broke 'guix pull'.
2021-04-27download: Use Disarchive as a last resort.Timothy Sample
* guix/download.scm (%disarchive-mirrors): New variable. (%disarchive-mirror-file): New variable. (built-in-download): Add 'disarchive-mirrors' keyword argument and pass its value along to the 'builtin:download' derivation. (url-fetch): Pass '%disarchive-mirror-file' to 'built-in-download'. * guix/scripts/perform-download.scm (perform-download): Read Disarchive mirrors from the environment and pass them to 'url-fetch'. * guix/build/download.scm (disarchive-fetch/any): New procedure. (url-fetch): Add 'disarchive-mirrors' keyword argument, use it to make a list of URIs, and use the new procedure to fetch the file if all other methods fail.
2021-03-19download: 'tls-wrap' treats premature TLS termination as EOF.Ludovic Courtès
This is a backport of Guile commit 076276c4f580368b4106316a77752d69c8f1494a. * guix/build/download.scm (tls-wrap)[read!]: Wrap 'get-bytevector-n!' call in 'catch' and handle 'error/premature-termination' GnuTLS errors.
2021-03-11download: 'tls-wrap' returns an unbuffered custom port.Ludovic Courtès
Partly fixes <https://bugs.gnu.org/46967>. * guix/build/download.scm (tls-wrap)[unbuffered]: New procedure. Pass the result of 'make-custom-binary-input/output-port' to 'unbuffered'.
2021-03-11download: 'tls-wrap' avoids intermediate buffer.Ludovic Courtès
* guix/build/download.scm (tls-wrap)[read!]: Read straight into BV instead of calling 'get-bytevector-some' and 'unget-bytevector'.
2020-03-22download: Delete the output file upon failure.Ludovic Courtès
This allows ENOSPC conditions to be properly reported as such rather than as a hash mismatch due to the availability of a truncated file. Fixes <https://bugs.gnu.org/39993>. Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>. * guix/build/download.scm (url-fetch): In the failure case, delete FILE.
2020-03-11download: Remove (web http) workarounds no longer relevant.Ludovic Courtès
* guix/build/download.scm <top level>: Remove workarounds for <https://bugs.gnu.org/23421> and for <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00070.html>.
2020-01-07download, git: Refer to the right module in 'module-use!' call.Ludovic Courtès
This fixes a regression introduced in 6a7c4636d4dec47eefa03c95da5a1315bd0e0413. * guix/build/download.scm (load-gnutls): Call 'resolve-module' instead of 'current-module'. * guix/git.scm (load-git-submodules): Likewise.
2020-01-06Adjust module autoloads.Ludovic Courtès
In Guile < 2.9.7, autoloading a module would give you access to all its bindings. In future versions, autoloading a module gives access only to the listed bindings, as per #:select (see <https://bugs.gnu.org/38895>). This commit adjusts autoloads to the new semantics, allowing Guix to be built with Guile 2.9.7/2.9.8. * guix/build/download.scm <top level>: Remove call to 'module-autoload!'. (load-gnutls): New procedure. (tls-wrap): Call it. * guix/git.scm <top level>: Remove call to 'module-autoload!'. (load-git-submodules): New procedure. (update-submodules): Call it instead of 'resolve-interface'. * gnu/bootloader/grub.scm: Replace #:autoload with #:use-module. * gnu/packages.scm: Likewise. * gnu/packages/ssh.scm: Likewise. * gnu/packages/tex.scm: Likewise. * gnu/services/cuirass.scm: Likewise. * gnu/services/mcron.scm: Likewise. * guix/lint.scm: Augment list of bindings in #:autoload. * guix/scripts/build.scm: Likewise. * guix/scripts/gc.scm: Likewise. * guix/scripts/pack.scm: Likewise. * guix/scripts/publish.scm: Likewise. * guix/scripts/pull.scm: Likewise. * guix/utils.scm: Remove unnecessary #:autoload clauses; replace one of them with #:use-module.
2020-01-03download: Do not leak file descriptors on TLS ports.Ludovic Courtès
Fixes <https://bugs.gnu.org/20145>. * guix/build/download.scm (%tls-ports, register-tls-record-port): Remove. (tls-wrap): Remove call to 'register-tls-record-port'. Return a custom binary input/output port instead. This is a backport of what Guile 2.2's (web client) module has been doing. (close-connection): Define as an alias for 'close-port'. * guix/http-client.scm (http-fetch): Remove #:keep-alive? parameter, which was ignored and unused. Pass #:keep-alive? #f to 'http-get'. * guix/lint.scm (probe-uri): Use 'close-port' instead of 'close-connection'. * guix/scripts/substitute.scm (http-multiple-get): Likewise.
2019-12-26download: Enable TLS 1.3.Marius Bakke
This reverts commit e4ee84202633636b4c8cef4a332f0c74912a3b23. * guix/build/download.scm (tls-wrap): Dot not disable TLS 1.3.
2019-11-18download: Load *.crt certificate bundles when *.pem files are missing.Ludovic Courtès
Fixes <https://bugs.gnu.org/38254>. * guix/build/download.scm (make-credendials-with-ca-trust-files): Look for *.crt files under DIRECTORY when *.pem files cannot be found.
2019-10-21download: Honor /etc/ssl/certs when 'SSL_CERT_DIR' is not set.Ludovic Courtès
* guix/build/download.scm (%x509-certificate-directory): Use "/etc/ssl/certs" as a last resort. This ensures, for instance, that 'guix download' honors system-wide certificates when SSL_CERT_DIR is unset.
2019-05-13download: Support 'https_proxy'.宋文武
* guix/build/download.scm (setup-http-tunnel): New procedure. (open-connection-for-uri): Honor the 'https_proxy' environment variable.
2019-01-26download: Ask not to use TLS 1.3.Ludovic Courtès
Works around <https://bugs.gnu.org/34102>. Reported by Marius Bakke <mbakke@fastmail.com>. * guix/build/download.scm (tls-wrap): Add "-VERS-TLS1.3" to the priority string when (gnutls-version) is not prefixed by "3.5".
2019-01-09Remove most uses of the _IO*F constants.Ludovic Courtès
These constants, for use with 'setvbuf', were deprecated in Guile 2.2 and disappeared in Guile 3.0. Here we keep these constants in build-side code where removing them is not feasible. * guix/build/download-nar.scm (download-nar): Adjust 'setvbuf' calls to the Guile 2.2+ API. * guix/build/download.scm (open-socket-for-uri): Likewise. (open-connection-for-uri, url-fetch): Likewise. * guix/build/make-bootstrap.scm (make-stripped-libc): Likewise. * guix/build/union.scm (setvbuf) [guile-2.0]: New conditional wrapper. (union-build): Adjust to new API. * guix/ftp-client.scm (ftp-open, ftp-list, ftp-retr): Likewise. * guix/http-client.scm (http-fetch): Likewise. * guix/inferior.scm (proxy): Likewise. * guix/scripts/substitute.scm (fetch, http-multiple-get): Likewise. * guix/self.scm (compiled-modules): Likewise. * guix/ssh.scm (remote-daemon-channel, store-import-channel) (store-export-channel): Likewise. * guix/ui.scm (initialize-guix): Likewise. * tests/publish.scm (http-get-port): Likewise. * guix/store.scm (%newlines): Adjust comment.
2019-01-09download: Remove Guile 2.0 workaround.Ludovic Courtès
* guix/build/download.scm: Remove Guile 2.0 workaround.
2019-01-09maint: Remove 'cond-expand' forms for Guile 2.0.Ludovic Courtès
Note: Leave 'cond-expand' forms used in the build-side modules that can run on %BOOTSTRAP-GUILE, which is currently Guile 2.0. * guix/build/compile.scm: Move 'use-modules' clause from 'cond-expand' to 'define-module' form. (%default-optimizations): Remove 'cond-expand'. * guix/build/download.scm (tls-wrap): Remove 'cond-expand'. * guix/build/syscalls.scm: Remove 'cond-expand' form around '%set-automatic-finalization-enabled?!' and 'without-automatic-finalization'. * guix/inferior.scm (port->inferior): Remove 'cond-expand'. * guix/scripts/pack.scm (wrapped-package)[build]: Remove 'cond-expand'. * guix/status.scm (build-event-output-port): Remove 'cond-expand'. * guix/store.scm (open-inet-socket): Remove 'cond-expand'. * guix/ui.scm (install-locale): Remove 'cond-expand'. * tests/status.scm ("current-build-output-port, UTF-8 + garbage"): Remove 'cond-expand'. * tests/store.scm ("current-build-output-port, UTF-8 + garbage"): Remove 'cond-expand'.
2018-09-27perform-download: Optionally report a "download-progress" trace.Ludovic Courtès
* guix/scripts/perform-download.scm (perform-download): Add #:print-build-trace? and pass it to 'url-fetch'. (guix-perform-download): Define 'print-build-trace?' and pass it to 'perform-download'. * guix/build/download.scm (ftp-fetch): Add #:print-build-trace? and honor it. (url-fetch): Likewise. * nix/libstore/builtins.cc (builtinDownload): Set _NIX_OPTIONS environment variable.
2018-02-26build: Require Guile >= 2.0.13.Ludovic Courtès
* README, configure.ac, doc/guix.texi (Requirements): Increase minimum Guile version from 2.0.9 to 2.0.13. * config-daemon.ac: Remove use of 'GUIX_CHECK_UNBUFFERED_CBIP'. * m4/guix.m4 (GUIX_CHECK_UNBUFFERED_CBIP): Remove. * guix/build/download.scm (current-http-proxy): Remove. * guix/build/syscalls.scm (%libc-errno-pointer, errno): Remove. (syscall->procedure): Use #:return-errno unconditionally. * guix/hash.scm (open-sha256-input-port)[unbuffered]: Remove outdated comment. * guix/http-client.scm (when-guile<=2.0.5-or-otherwise-broken): Remove. <top level>: Remove 'when-guile<=2.0.5-or-otherwise-broken' block. * guix/scripts/substitute.scm (fetch): Remove 'guile-version>?' conditional. * tests/hash.scm (supports-unbuffered-cbip?): Remove. <top level>: Remove 'test-skip' call.
2018-01-28download: Fix return value of the url-fetch procedure.Maxim Cournoyer
Fixes <https://bugs.gnu.org/30270>. Regression was introduced by commit 347fa4aebf. * guix/build/download.scm (url-fetch): Return `file' instead of #t upon success. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2017-12-10utils: Fix cond-expand for Guile 2.0.Eric Bavier
* guix/build/download.scm (tls-wrap): Use 'guile-2.2' feature instead.
2017-11-16download: Improve efficiency of 'write-request' over TLS.Ludovic Courtès
This is another instance of <https://bugs.gnu.org/22966>. The Microsoft-IIS/7.5 server at static.nvd.nist.gov would sometimes hang when receiving our requests byte by byte. * guix/build/download.scm (tls-wrap) [!guile-2.0]: Add 'setvbuf' call.
2017-11-14download: Pass the timeout to 'ftp-retr'.Ludovic Courtès
This ensures the timeout applies when connecting to the port returned by PASV. * guix/ftp-client.scm (ftp-list): Add #:timeout parameter. Use 'connect*' instead of 'connect' and pass TIMEOUT. (ftp-retr): Likewise. * guix/build/download.scm (ftp-fetch): Pass TIMEOUT to 'ftp-retr'.
2017-11-11download: Work around more bogus HTTP handling in Guile 2.2 <= 2.2.2.Ludovic Courtès
Reported by Mark H Weaver <mhw@netris.org> at <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00190.html>. * guix/build/download.scm (guile-2.2) [write-request-line]: Backport Guile commit 6ad28ae3bc6a6d9e95ab7d70510d12c97673a143.
2017-11-10download: Work around bogus HTTP handling in Guile 2.2 <= 2.2.2.Ludovic Courtès
Reported by Konrad Hinsen <konrad.hinsen@fastmail.net> at <https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00070.html>. * guix/build/download.scm (write-request-line) [guile-2.2]: New procedure.
2017-10-19Add (guix progress).Ludovic Courtès
Among other things, this removes (guix utils), (guix ui), (guix config), etc. from the closure of (guix build download), as was the case since 798648515b77507c242752457b4dc17c155bad6e. * guix/utils.scm (<progress-reporter>, call-with-progress-reporter): Move to... * guix/progress.scm: ... here. New file. * Makefile.am (MODULES): Add it. * guix/build/download.scm (current-terminal-columns) (nearest-exact-integer, duration->seconds, seconds->string) (byte-count->string, progress-bar, string-pad-middle) (rate-limited, progress-reporter/file, dump-port*) (time-monotonic): Move to progress.scm. * guix/scripts/download.scm: Adjust accordingly. * guix/scripts/substitute.scm: Likewise.
2017-10-19download: Make 'http-fetch' public.Ludovic Courtès
* guix/build/download.scm (http-fetch): Remove 'file' parameter. Change to return an input port and the content-length. Make public. (url-fetch): Adjust accordingly.
2017-10-19download: Remove old-Guile leftovers.Ludovic Courtès
This is a followup to 36626c556ed75219bce196ac93d148f6b9af984c. * guix/build/download.scm (http-fetch): Rename 'port-or-bv' to 'port'. Assume (port? port) is always true, and remove other branch.
2017-09-20download: Don't report the progress too fast.宋文武
* guix/utils.scm (<progress-reporter>): New record type. (call-with-progress-reporter): New procedure. * guix/build/download.scm (dump-port*, rate-limited, progress-reporter/file): New procedures. (ftp-fetch, http-fetch): Use 'dump-port*'. (progress-proc): Remove procedure. * guix/scripts/substitute.scm (progress-report-port): Rewrite in terms of <progress-reporter>. (process-substitution): Adjust accordingly.
2017-09-05Handle the same HTTP redirects everywhere.Tobias Geerinckx-Rice
* guix/build/download.scm (http-fetch): Complete the hard-coded list of HTTP redirection status codes. * guix/http-client.scm (http-fetch): Likewise. * guix/scripts/lint.scm (probe-uri): Likewise.
2017-05-30download: Work around GnuTLS bug with UTF-8 certificate file names.Ludovic Courtès
Reported by Mark H Weaver <mhw@netris.org> at <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26948#17>. * guix/build/download.scm (set-certificate-credentials-x509-trust-file!*): New procedure. (make-credendials-with-ca-trust-files): Use it instead of 'set-certificate-credentials-x509-trust-file!'.
2017-05-02download: Continue handshake upon TLS warning alerts.Ludovic Courtès
This allows us to download from site such as <https://fusionforge.int-evry.fr> where the server does not recognize the server name passed via the 'server_name' extension. * guix/build/download.scm (tls-wrap): Catch 'gnutls-error' around 'handshake'. Upon ERROR/WARNING-ALERT-RECEIVED, print a message and call 'handshake'.
2017-04-22cache: Work around 'time-monotonic' bug in Guile 2.2.2.Ludovic Courtès
* guix/cache.scm (time-monotonic) [guile-2.2]: New variable. * tests/cache.scm (time-monotonic) [guile-2.2]: Likewise. * guix/build/download.scm (time-monotonic) [guile-2.2]: Adjust comment: it's a 2.2.2 bug.
2017-04-22download: Work around Guile 2.2 bug with 'time-monotonic' objects.Ludovic Courtès
* guix/build/download.scm (time-monotonic) [guile-2.2]: New variable.
2017-03-28download: Properly parse ftp:// URLs that lack user info.Ludovic Courtès
Fixes a regression introduced in a4f542341511f33ece18d16b68118214da8143ec. Reported by Hartmut Goebel. * guix/build/download.scm (ftp-fetch): Account for the case where 'uri-userinfo' returns #f. Remove the case where it returns "" since that cannot happen.
generated by cgit v1.2.3 (git 2.44.0) at 2024-07-01 06:35:38 +0000