summaryrefslogtreecommitdiff
path: root/guix/store/database.scm
AgeCommit message (Collapse)Author
2024-04-07store: database: Use correct function name in assertion.Wojtek Kosior
* guix/store/database.scm (register-valid-path): Replace "sqlite-register" with "register-valid-path" as argument to `assert-integer'. Change-Id: Id93687e90d0a806d715006ca0b2498a1d10cfba6 Signed-off-by: Christopher Baines <mail@cbaines.net>
2024-04-03store: database: Rename a couple of procedures.Christopher Baines
These names should be more descriptive. * guix/store/database.scm (path-id): Rename to select-valid-path-id. (sqlite-register): Rename to register-valid-path. (register-items): Update accordingly. Change-Id: I6d4a14d4cde9d71ab34d6ffdbfbfde51b2c0e1db
2024-04-03store: database: Refactor sqlite-register.Christopher Baines
The update-or-insert procedure name was unhelpfully generic, and these changes should improve the code readability. * guix/store/database.scm (update-or-insert): Remove procedure and inline functionality in to sqlite-register. Change-Id: Ifab0cdb7972d095460cc1f79b8b2f0e9b958059c
2024-04-03store: database: Stop finalizing prepared statements.Christopher Baines
Especially since we're asking for these to be cached. Management of prepared statements isn't trivial, since you don't want to keep them forever as this can lead to poor query performance, but I don't think that finalizing them immediately is the right solution. Change-Id: I61706b4d09d771835bb8f074b8f6a6ee871f5e2d * guix/store/database.scm (sqlite-step-and-reset): New procedure. (last-insert-row, path-id, update-or-insert, add-references): Don't finalize prepared statements. Change-Id: I2a2c6deb43935d67df9e43000a5105343d72b3e6
2024-04-03store: database: Inline SQL to where it's used.Christopher Baines
This makes the code easier to read, as you don't have to keep jumping between the two places. * guix/store/database.scm (path-id-sql, update-sql, insert-sql, add-reference-sql): Remove variables. (path-id, update-or-insert, add-references): Include SQL. Change-Id: I53b4ab973be8d0cd10a0f35ba25972f1c9680353
2024-04-03store: database: Remove with-statement and associated code.Christopher Baines
I think using dynamic-wind to finalize all statements is the wrong approach. Firstly it would be good to allow reseting statements rather than finalizing them. Then for the problem of handling errors, the approach I've settled on in the build coordinator is to close the database connection, since that'll trigger guile-sqlite3 to finalize all the cached statements. This reverts commit 5d6e2255286e591def122ec2f4a3cbda497fea21. * .dir-locals.el (scheme-mode): Remove with-statement. * guix/store/database.scm (call-with-statement): Remove procedure. (with-statement): Remove syntax rule. (call-with-transaction, last-insert-row-id, path-id, update-or-insert, add-references): Don't use with-statement. Change-Id: I2fd976b3f12ec8105cc56350933a953cf53647e8
2024-04-03store: database: Remove call-with-savepoint and associated code.Christopher Baines
While care does need to be taken with making updates or inserts to the ValidPaths table, I think that trying to ensure this within update-or-insert is the wrong approach. Instead, when working with the store database, only one connection should be used to make changes to the database and those changes should happen in transactions that ideally begin immediately. This reverts commit 37545de4a3bf59611c184b31506fe9a16abe4c8b. * .dir-locals.el (scheme-mode): Remove entries for call-with-savepoint and call-with-retrying-savepoint. * guix/store/database.scm (call-with-savepoint, call-with-retrying-savepoint): Remove procedures. (update-or-insert): Remove use of call-with-savepoint. Change-Id: I2f986e8623d8235a90c40d5f219c1292c1ab157b
2023-03-13guix: Strip #:use-module lists.Ludovic Courtès
This was obtained by setting up this environment: guix shell -D guix --with-input=guile@3.0.9=guile-next \ --with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2 -- make -j5 then adding 'unused-module' to (@@ (guix build compiler) %warnings), building, and checking all the "unused module" warnings and removing those that were definitely unused.
2022-11-06guix gc: Add '--vacuum-database'.Efraim Flashner
* guix/scripts/gc.scm (show-help, %options): Add '--vacuum-database'. * guix/store/database.scm (vacuum-database): New procedure. * doc/guix.texi (Invoking guix gc): Document the option.
2021-01-31database: Validate #:nar-size and #:time when registering store items.Ludovic Courtès
* guix/store/database.scm (assert-integer): New procedure. (update-or-insert): Use it to validate NAR-SIZE and TIME. * tests/store-database.scm ("sqlite-register with incorrect size"): New test.
2021-01-31database: Remove workarounds unnecessary with guile-sqlite3 0.1.2.Ludovic Courtès
* guix/store/database.scm (sqlite-exec, sqlite-finalize): Remove.
2020-12-15database: Honor 'SOURCE_DATE_EPOCH'.Ludovic Courtès
* guix/store/database.scm (timestamp): New procedure. (sqlite-register): Use it as the default for #:time. (register-items): Likewise for #:registeration-time.
2020-12-15database: Remove 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Remove. * tests/store-database.scm ("register-path"): Rename to... ("register-items"): ... this, and use 'register-items' instead of 'register-path'. ("register-path, directory"): Rename to... ("register-items, directory"): ... this, and use 'register-items' instead of 'register-path'. ("register-path with unregistered references"): Rename to... ("sqlite-register with unregistered references"): ... this.
2020-12-15system: 'init' does not recompute the hash of each store item.Ludovic Courtès
Fixes <https://bugs.gnu.org/44760>. Previously, the 'register-path' call would re-traverse ITEM to compute its nar hash, even though that hash is already known in the initial store. This patch also avoids repeated opening/closing of the database. * guix/store/database.scm (call-with-database): Export. * guix/scripts/system.scm (copy-item): Add 'db' parameter. Call 'sqlite-register' instead of 'register-path'. (copy-closure): Remove redundant call to 'references*'. Call 'call-with-database' and pass the database to 'copy-item'.
2020-12-15database: Remove #:deduplicate? and #:reset-timestamps? from 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Remove #:deduplicate? and #:reset-timestamps?. * guix/scripts/system.scm (copy-item): Adjust accordingly. * tests/store-database.scm ("register-path") ("register-path, directory"): Call 'reset-timestamps'.
2020-12-15database: Remove #:deduplicate? from 'register-items'.Ludovic Courtès
It is now up to the caller to deduplicate store contents. * guix/store/database.scm (register-items): Remove #:deduplicate? parameter and call to 'deduplicate'. (register-path): Call 'deduplicate' when #:deduplicate? is true. * gnu/build/image.scm (register-closure): Adjust call accordingly. * gnu/build/vm.scm (register-closure): Likewise. * guix/nar.scm (finalize-store-file): Likewise. * guix/scripts/pack.scm (store-database): Likewise.
2020-12-15database: Remove #:reset-timestamps? from 'register-items'.Ludovic Courtès
The assumption now is that the caller took care of resetting timestamps and permissions. * guix/store/database.scm (register-items): Remove #:reset-timestamps? parameter and the call to 'reset-timestamps'. (register-path): Adjust accordingly and add call to 'reset-timestamps'. * gnu/build/image.scm (register-closure): Remove #:reset-timestamps? parameter to 'register-items'. * gnu/build/vm.scm (register-closure): Likewise. * guix/nar.scm (finalize-store-file): Adjust accordingly. * guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-12-08database: Remove unnecessary module imports.Ludovic Courtès
* guix/store/database.scm: Remove unnecessary imports added in 4b9eecd322e566783369795ebea63a479b51f486.
2020-09-14database: register-items: reduce transaction scope.Christopher Baines
It was made transactional in a4678c6ba18d8dbd79d931f80426eebf61be7ebe, with the reasoning to prevent broken intermediate states from being visible. I think this means something like an entry being in ValidPaths, but the Refs not being inserted. Using a transaction for this makes sense, but I think using one single transaction for the whole register-items call is unnecessary to avoid broken states from being visible, and could block other writes to the store database while register-items is running. Because the deduplication and resetting timestamps happens within the transaction as well, even though these things don't involve the database, writes to the database will still be blocked while this is happening. To reduce the potential for register-items to block other writers to the database for extended periods, this commit moves the transaction to just wrap the call to sqlite-register. This is the one place where writes occur, so that should prevent the broken intermediate states issue above. The one difference this will make is some of the registered items will be visible to other connections while others may be still being added. I think this is OK, as it's equivalent to just registering different items. * guix/store/database.scm (register-items): Reduce transaction scope. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-09-14database: document extra registration requirements.Caleb Ristvedt
It's necessary that store items be locked and protected from garbage collection while they are being registered. This documents that. * guix/store/database.scm (register-path, register-items): document GC protection and locking requirements. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2020-07-11image: Do not set journal_model=WAL for the Hurd.Jan (janneke) Nieuwenhuizen
This fixes <https://bugs.gnu.org/42151>. * gnu/system/images/hurd.scm (hurd-initialize-root-partition): Use #:wal-mode #f in call to ... * gnu/build/image.scm (initialize-root-partition): ... this, add #:wal-mode? parameter, pass it to ... (register-closure): ... this, add #:wal-mode? parameter, pass it to ... * guix/store/database.scm (with-database): ... this, add #:wal-mode? parameter, pass it to ... (call-with-database): ... this, add #:wal-mode? parameter; when set to #f, do not set journal_model=WAL.
2020-06-18database: 'register-items' takes an open database.Ludovic Courtès
* guix/store/database.scm (store-database-directory) (store-database-file): New procedures. (call-with-database): Add call to 'mkdir-p'. (register-items): Add 'db' parameter and remove #:state-directory and #:schema. (register-path): Use 'store-database-file' and 'with-database', and parameterize SQL-SCHEMA. * gnu/build/image.scm (register-closure): Likewise. * gnu/build/vm.scm (register-closure): Likewise. * guix/scripts/pack.scm (store-database)[build]: Likewise.
2020-06-10database: separate transaction-handling and retry-handling.Caleb Ristvedt
Previously call-with-transaction would both retry when SQLITE_BUSY errors were thrown and do what its name suggested (start and rollback/commit a transaction). This changes it to do only what its name implies, which simplifies its implementation. Retrying is provided by the new call-with-SQLITE_BUSY-retrying procedure. * guix/store/database.scm (call-with-transaction): no longer restarts, new #:restartable? argument controls whether "begin" or "begin immediate" is used. (call-with-SQLITE_BUSY-retrying, call-with-retrying-transaction, call-with-retrying-savepoint): new procedures. (register-items): use call-with-retrying-transaction to preserve old behavior. * .dir-locals.el (call-with-retrying-transaction, call-with-retrying-savepoint): add indentation information.
2020-06-10database: ensure update-or-insert is run within a transactionCaleb Ristvedt
update-or-insert can break if an insert occurs between when it decides whether to update or insert and when it actually performs that operation. Putting the check and the update/insert operation in the same transaction ensures that the update/insert will only succeed if no other write has occurred in the middle. * guix/store/database.scm (call-with-savepoint): new procedure. (update-or-insert): use call-with-savepoint to ensure the read and the insert/update occur within the same transaction.
2020-06-10database: rewrite query procedures in terms of with-statement.Caleb Ristvedt
Most of our queries would fail to finalize their statements properly if sqlite returned an error during their execution. This resolves that, and also makes them somewhat more concise as a side-effect. This also makes some small changes to improve certain queries where behavior was strange or overly verbose. * guix/store/database.scm (call-with-statement): new procedure. (with-statement): new macro. (last-insert-row-id, path-id, update-or-insert, add-references): rewrite to use with-statement. (update-or-insert): factor last-insert-row-id out of the end of both branches. (add-references): remove pointless last-insert-row-id call. * .dir-locals.el (with-statement): add indenting information.
2020-06-10database: work around guile-sqlite3 bug preventing statement resetCaleb Ristvedt
guile-sqlite3 provides statement caching, making it unnecessary for sqlite to keep re-preparing statements that are frequently used. Unfortunately it doesn't quite emulate the semantics of sqlite_finalize properly, because it doesn't cause a commit if the statement being finalized is the last "active" statement (see https://notabug.org/guile-sqlite3/guile-sqlite3/issues/12). We work around this by wrapping sqlite-finalize with our own version that ensures sqlite-reset is called, which does The Right Thing™. * guix/store/database.scm (sqlite-finalize): new procedure that shadows the sqlite-finalize from (sqlite3).
2020-04-11database: 'reset-timestamps' can optionally preserve permissions.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Add #:preserve-permissions? and honor it.
2019-03-07database: Make 'register-items' transactional.Caleb Ristvedt
* guix/store/database.scm (SQLITE_BUSY, register-output-sql): New variables. (add-references): Don't try finalizing after each use, only after all the uses (otherwise a finalized statement would be used if #:cache? was #f). (call-with-transaction): New procedure. (register-items): Use call-with-transaction to prevent broken intermediate states from being visible. * .dir-locals.el (call-with-transaction): indent it. Signed-off-by: Ludovic Courtès <ludo@gnu.org>
2018-12-21database: Use "write-ahead log" mode and set a long "busy timeout".Ludovic Courtès
This should avoid "database is locked" errors when there's a lot of concurrency, for instance when offloading simultaneously a lot of builds. * guix/store/database.scm (call-with-database): Add two 'sqlite-exec' calls to set 'journal_mode' and 'busy_timeout'.
2018-11-23Update Guile-SQLite3 URL everywhere.Ludovic Courtès
* README: Update Guile-SQLite3 URL. * doc/guix.texi (Requirements): Likewise. * guix/store/database.scm (sqlite-exec): Likewise. * m4/guix.m4 (GUIX_CHECK_GUILE_SQLITE3): Likewise.
2018-11-13nar: Access the database instead of connecting to the daemon.Ludovic Courtès
* guix/store/database.scm (%default-database-file): New variable. (path-id): Export. * guix/nar.scm (finalize-store-file): Use 'with-database' instead of 'with-store', and use 'path-id' instead of 'valid-path?'.
2018-09-23database: Register each store item only once.Ludovic Courtès
Fixes <https://bugs.gnu.org/32600>. Reported by Leo Famulari. * guix/store/database.scm (register-items): Check whether TO-REGISTER is in DB by calling 'path-id', and skip the reset-timestamps, registration, and deduplication phases when it is.
2018-09-23database: 'register-items' shows a progress bar.Ludovic Courtès
* guix/store/database.scm (register-items): Add #:log-port. Use 'progress-reporter/bar' to show a progress report. (register-path): Pass #:log-port to 'register-items'.
2018-07-20database: Reset timestamps to one second after the Epoch.Ludovic Courtès
Previously, store items registered in the database by this code (for instance, store items retrieved by 'guix offload' and passed to 'restore-file-set') would have an mtime of 0 instead of 1. This would cause problems for things like .go files: Guile would consider them to be older than the corresponding .scm file, and consequently it would ignore them and possibly use another (incorrect) .go file. Reported by Ricardo Wurmus. * guix/store/database.scm (reset-timestamps): Pass 1, not 0, to 'utime'. * tests/store-database.scm ("register-path"): Check the mtime of FILE and REF.
2018-07-03database: 'reset-timestamps' now correctly handles symlinks.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Use 'utime' with AT_SYMLINK_NOFOLLOW for symlinks.
2018-06-14database: Allow for deterministic database construction.Ludovic Courtès
Fixes <https://bugs.gnu.org/21073>. * guix/store/database.scm (sqlite-register): Add #:time. (%epoch): New variable. (register-items): Add #:registration-time. Pass #:time to 'sqlite-register'. * gnu/build/install.scm (register-closure): Pass #:registration-time.
2018-06-14install: Use 'reset-timestamps' from (guix store database).Ludovic Courtès
* gnu/build/install.scm (reset-timestamps): Remove. * gnu/build/vm.scm: Use 'reset-timestamps' from (guix store database).
2018-06-14database: Add 'register-items'.Ludovic Courtès
* guix/build/store-copy.scm (store-info): Export. * guix/store/database.scm (register-items): New procedure. (register-path): Implement in terms of 'register-items'. * gnu/build/install.scm (register-closure): Use 'register-items' instead of 'for-each' and 'register-path'.
2018-06-14database: 'sqlite-register' takes a database, not a file name.Ludovic Courtès
* guix/store/database.scm (sqlite-register): Remove #:db-file and add 'db' parameter. Remove #:schema and 'parameterize'. (register-path): Wrap 'sqlite-register' call in 'with-database' and in 'parameterize'. * tests/store-database.scm ("new database") ("register-path with unregistered references"): Adjust accordingly.
2018-06-14database: 'reset-timestamps' sets file permissions as well.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): Add 'chmod' calls.
2018-06-14database: Replace existing entries in Refs.Ludovic Courtès
* guix/store/database.scm (add-reference-sql): Add "OR REPLACE".
2018-06-14database: Add #:reset-timestamps? to 'register-path'.Ludovic Courtès
* guix/store/database.scm (register-path): Add #:reset-timestamps? and honor it.
2018-06-14database: Remove extra SQL parameter in 'update-or-insert'.Ludovic Courtès
* guix/store/database.scm (update-or-insert): Remove extra #:path parameter.
2018-06-14database: 'register-path' creates the database directory if needed.Ludovic Courtès
* guix/store/database.scm (register-path): Call 'mkdir-p'.
2018-06-14database: Provide a way to specify the schema location.Ludovic Courtès
* guix/store/database.scm (sqlite-register): Add #:schema. Parameterize 'sql-schema' based on this. (register-path): Add #:schema and pass it to 'sqlite-register'.
2018-06-14database: Fail registration when encountering unregistered references.Ludovic Courtès
* guix/store/database.scm (add-reference-sql): Remove nested SELECT. (add-references): Expect REFERENCES to be a list of ids. (sqlite-register): Call 'path-id' for each of REFERENCES and pass it to 'add-references'. * tests/store-database.scm ("register-path with unregistered references"): New test.
2018-06-14database: 'with-database' can now initialize new databases.Ludovic Courtès
* nix/libstore/schema.sql: Rename to... * guix/store/schema.sql: ... this. * Makefile.am (nobase_dist_guilemodule_DATA): Add it. * nix/local.mk (%D%/libstore/schema.sql.hh): Adjust accordingly. * guix/store/database.scm (sql-schema): New variable. (sqlite-exec, initialize-database, call-with-database): New procedures. (with-database): Rewrite in terms of 'call-with-database'. * tests/store-database.scm ("new database"): New test. * guix/self.scm (compiled-guix)[*core-modules*]: Add 'schema.sql' to #:extra-files.
2018-06-01Add (guix store deduplication).Caleb Ristvedt
* guix/store/database.scm (register-path): Add #:deduplicate? and call 'deduplicate' when it's true. (counting-wrapper-port, nar-sha256): Move to... * guix/store/deduplication.scm: ... here. New file. * tests/store-deduplication.scm: New file. * Makefile.am (STORE_MODULES): Add deduplication.scm. (SCM_TESTS) [HAVE_GUILE_SQLITE3]: Add store-deduplication.scm. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
2018-06-01database: 'register-path' resets timestamps.Ludovic Courtès
* guix/store/database.scm (reset-timestamps): New procedure. (register-path): Use it.
2018-06-01Add (gnu store database).Caleb Ristvedt
* guix/config.scm.in (%store-database-directory): New variable. * guix/store/database.scm: New file. * tests/store-database.scm: New file. * Makefile.am (STORE_MODULES): New variable. (MODULES, MODULES_NOT_COMPILED): Adjust accordingly. (SCM_TESTS) [HAVE_GUILE_SQLITE3]: Add tests/store-database.scm. Co-authored-by: Ludovic Courtès <ludo@gnu.org>
generated by cgit v1.2.3 (git 2.44.0) at 2024-06-29 06:09:30 +0000