MirrorSave
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,790 Commits 14 Branches 49 Tags 90 MiB
Commit Graph

2385 Commits

Author SHA1 Message Date
Michael Eischer b8f409723d make timeout for slow requests configurable 2024-08-26 14:14:43 +02:00
Michael Eischer 311b27ced8 restic: cleanup redundant code in test case 2024-08-25 23:18:55 +02:00
Michael Eischer 43b36ad2b0 restore: test timestamps for macOS resource forks are restored correctly 2024-08-25 23:18:55 +02:00
Michael Eischer 2e55209b34 restic: restore timestamps after extended attributes 
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
 2024-08-25 23:18:55 +02:00
Michael Eischer 5cca6e66be
Merge pull request #4981 from konidev20/fix-gh-4934-cleanup-removed-snaphots-from-cache 
cache: clear snapshot files from cache during load index
 2024-08-16 19:04:59 +00:00
Srigovind Nayak 88174cd0a4
cache: remove redundant index file cleanup 
addressing code review comments
 2024-08-17 00:21:49 +05:30
Srigovind Nayak b7d014b685
Revert "repository: removed redundant prepareCache method from Repository" 
This reverts commit 720609f8ba.
 2024-08-17 00:18:13 +05:30
Michael Eischer 74d3f92cc7
Merge pull request #4993 from MichaelEischer/fix-timeout-error 
backend: return correct error on upload/request timeout
 2024-08-15 22:07:37 +02:00
Michael Eischer 80f24584a5
Merge pull request #4998 from zmanda/ea_vss_fix 
Fix extended attributes handling for VSS snapshots
 2024-08-15 19:51:35 +00:00
Michael Eischer 8e00158c34
Merge pull request #5000 from deining/fix-typo 
Fix typos
 2024-08-15 19:42:14 +00:00
Michael Eischer 36b5580c1c
Merge pull request #4989 from plant99/progress-bar-for-restore-verify 
restore: Add progress bar to 'restore --verify'
 2024-08-15 19:34:05 +00:00
aneesh-n 19f487750e
Add test cases and handle volume GUID paths 
Gracefully handle errors while checking for EA and add debug logs.
 2024-08-11 19:25:58 -06:00
Shivashis Padhi f1407afd1f
restore: Add progress bar to 'restore --verify' 2024-08-11 22:25:21 +02:00
Andreas Deininger 4401265e36 Fix typos 2024-08-11 21:38:15 +02:00
Srigovind Nayak 5fd984ba6f
cache: add test for the automated cache clear to cache backend 2024-08-11 23:41:07 +05:30
Srigovind Nayak 720609f8ba
repository: removed redundant prepareCache method from Repository 
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
 2024-08-11 23:41:07 +05:30
Srigovind Nayak a23e7bfb82
cache: check for context cancellation before clearing cache 2024-08-11 23:41:07 +05:30
Srigovind Nayak f66624f5bf
cache: backend add List method and a cache clear functionality 
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.

cache: fix ids set initialisation with NewIDSet()
 2024-08-11 23:40:52 +05:30
aneesh-n 849c441455
Gracefully handle invalid prepared volume names 2024-08-11 01:48:25 -06:00
aneesh-n 1d392a36f9
Fix extended attributes handling for VSS snapshots 2024-08-11 01:23:47 -06:00
Michael Eischer 3f5e2160de
Merge pull request #4938 from MichaelEischer/bump-go-version 
Bump go version to 1.21
 2024-08-10 19:57:59 +02:00
Michael Eischer 283225f15f
Merge pull request #4994 from MichaelEischer/less-flaky-watchdog-test 
backend: tweak timeouts to make watchdog timeout test less flaky
 2024-08-10 19:35:38 +02:00
Michael Eischer 400ae55940 replace deprecated usages of math/rand 2024-08-10 19:34:49 +02:00
Michael Eischer 0b19f6cf5a Switch back to sha256 from the std library 
The std library now also supports the sha assembly instructions on
ARM64. Thus, sha256-simd no longer provides a performance benefit.
 2024-08-10 19:16:10 +02:00
Michael Eischer ad48751adb bump required go version to 1.21 2024-08-10 19:16:10 +02:00
Michael Eischer 86390b453d
Merge pull request #4980 from zmanda/unsupported-ea-handling 
Skip ExtendedAttribute processing in Windows for volumes that do not support EA
 2024-08-10 17:11:12 +00:00
Michael Eischer fa35e72214 backend: tweak timeouts to make watchdog timeout test less flaky 2024-08-10 19:08:03 +02:00
Michael Eischer 05571286b2
Merge pull request #4946 from mikix/json-errors 
Improve error handling in --json mode
 2024-08-10 17:04:56 +00:00
aneesh-n 18e9d71d7a
Fix review comments 2024-08-10 10:38:04 -06:00
Michael Eischer 853a686994 backend: return correct error on upload/request timeout 2024-08-10 18:06:24 +02:00
Michael Eischer a164789321
Merge pull request #4977 from MichaelEischer/warn-on-incomplete-metadata 
backup: warn but store store item if extended metadata is incomplete
 2024-08-10 17:59:25 +02:00
aneesh-n 71632a8197
Handle extended length paths 2024-08-05 16:03:43 -06:00
aneesh-n 85639f5159
Add handling for relative paths, vss paths, UNC paths 2024-08-04 13:19:13 -06:00
aneesh-n c13725b5d0
Check EA support only for volumes, files and dirs 2024-08-04 11:05:40 -06:00
aneesh-n 89712f6640
Formatted 2024-08-04 10:36:13 -06:00
aneesh-n 9dedba6dfc
Address review comments 2024-08-04 10:23:39 -06:00
aneesh-n 8c8a066c0e
Correct the bitmask for fileSupportsExtendedAttributes and add link 2024-08-03 18:06:47 -06:00
aneesh-n 56113a8da7
Skip EA processing for volumes that do not support EA 2024-08-03 16:03:30 -06:00
Michael Eischer 73c9780321 backup: store but warn if extended metadata for item is incomplete 
Files were not included in the backup if the extended metadata for the
file could not be read. This is rather drastic. Instead settle on
returning a warning but still including the file in the backup.
 2024-08-03 22:25:29 +02:00
Michael Terry 88f59fc2d6 json: switch backup and restore errors from string to struct types 
This keeps backwards compatibility with the previous empty structs.
And maybe we'd want to put other fields into the inner struct later,
rather than the outer message.
 2024-08-03 16:21:44 -04:00
Michael Eischer a48baf6f3a
Merge pull request #4976 from textaligncenter/backup-panic 
fix panic in fs_reader
 2024-08-03 19:56:15 +00:00
Michael Terry a376323331 restore: print JSON versions of errors in --json mode 
Previously, they were printed as freeform text.

This also adds a ui.Terminal interface to make writing
tests easier and also adds a few tests.
 2024-08-03 15:18:46 -04:00
Michael Eischer e622135e7e
Merge pull request #4974 from MichaelEischer/fix-cache-debug-log 
cache: fix confusing debug log
 2024-08-03 21:12:32 +02:00
textaligncenter d8ea178e69 fix panic in fs_reader 2024-08-03 19:08:04 +00:00
Michael Terry ad2585af67 backup: show actual error strings in --json mode 
Previously, an error JSON fragment would look like:
{"message_type": "error", "error": {}}

This is because encoding/json cannot marshal an error interface.
Instead, we now call .Error() to get the string value.
 2024-08-03 15:07:52 -04:00
Michael Terry f4bdfea1c9 backup: print scanner errors to stderr, not stdout 2024-08-03 15:07:52 -04:00
Michael Eischer d2f7c5a9c6
Merge pull request #4978 from konidev20/fix-gh-4949-repair-index-spurious-index 
rewrite: skip saving empty indexes during MasterIndex.Rewrite
 2024-08-03 18:53:57 +00:00
Srigovind Nayak 068d5b95c3
rewrite: skip saving empty indexes during MasterIndex.Rewrite 2024-08-03 23:34:59 +05:30
Michael Eischer d4db5a364e
Merge pull request #4958 from mikix/restore-errors 
restore: clean up error handling when restoring metadata
 2024-08-03 17:58:54 +00:00
Michael Eischer 4266dca1b6 cache: fix confusing debug log 2024-08-03 18:51:38 +02:00
Michael Eischer d407abb50f
Merge pull request #4960 from MichaelEischer/reduce-cancelation-delays 
Add more checks for canceled contexts
 2024-08-03 18:23:20 +02:00
Michael Eischer ae1cb889dd Add more checks for canceled contexts 2024-07-31 19:30:47 +02:00
Michael Terry 6a97833337 restore: clean up error handling when restoring metadata 
- Fix a logic error that instead of reporting the *first*
  metadata-setting error that appears, we were instead reporting the
  *last* error (and only if the lchown call failed!).
- Don't show any errors when setting metadata for files in non-root
  mode (things like timestamps, attributes). Previously, only lchown
  errors were skipped. But other kinds of attribute errors make sense
  to skip as well. The code path happened to work correctly before
  because of the above logic error. But once that was fixed, this
  change needed to happen too.
 2024-07-30 19:27:34 -04:00
Michael Eischer a9be986782 restorer: add minimal long path handling test 2024-07-29 21:11:47 +02:00
Michael Eischer 62c4a5e9a0 fs: fix long path handling for ResetPermissions, ClearAttributes and security descriptors 2024-07-29 20:10:58 +02:00
Michael Eischer 729e009485 replace all uses of target in relation to backup 2024-07-24 19:40:53 +02:00
Michael Eischer fa5ff0873a
Merge pull request #4925 from MichaelEischer/fix-restore-truncation 
restore: Fix truncation of uptodate files
 2024-07-21 12:28:12 +02:00
Michael Eischer 0dcac90bea restorer: don't track already uptodate blobs 2024-07-21 12:03:56 +02:00
Michael Eischer 10efa77103 restorer: add test for file truncation case 2024-07-21 12:03:56 +02:00
Michael Eischer 6f8e17a463 restorer: minor code cleanups 2024-07-21 12:03:56 +02:00
Michael Eischer 5d6ce59a8d restorer: also truncate files if their content is already uptodate 
Files for which no blobs have to be restored, still have to be truncated
to the correct size. Take a file with content "foobar" that should be
replaced by restore with content "foo". The first three bytes are
already uptodate, such that no data has to be written. As file
truncation normally happens when writing data, a special case is
necessary.

This no blobs written special case is unified with the empty file
special case.
 2024-07-21 12:03:56 +02:00
Michael Eischer 59c880fce3 restorer: only show size in text output for files 2024-07-20 12:46:28 +02:00
Alex Johnson 3bf2927006 Update snapshot summary on rewrite 
Signed-off-by: Alex Johnson <hello@alex-johnson.net>
 2024-07-16 12:06:50 -05:00
Michael Eischer dcfffd7779 restore: extend overwrite test for small files 2024-07-14 11:30:41 +02:00
Michael Eischer 44e3610b32 restore: progress bar total on windows 2024-07-14 11:21:11 +02:00
Michael Eischer 98cfb2c4c8 restore: test progress reporting for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer 26aa65e0d4 restore: add regression test for corrupt in-place restore of large file 2024-07-14 11:21:11 +02:00
Michael Eischer 2833b2f699 restore: fix progress bar for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer c77b2d5ca2 restore: avoid long cancelation delay for frequently used blobs 2024-07-14 11:21:11 +02:00
Michael Eischer f64191da9c restore: improve reporting of cancelation errors 2024-07-14 11:21:08 +02:00
Michael Eischer 2971a769da restore: fix corrupt restore of partially up to date files 2024-07-12 22:27:00 +02:00
Michael Eischer 40e5163114 restore: properly cancel file verification 2024-07-12 22:26:22 +02:00
Michael Eischer 31887ea9aa restore: fix hang on command cancelation 2024-07-12 22:24:35 +02:00
Michael Eischer 375c572c4f
Merge pull request #4884 from MichaelEischer/lock-and-not-exist-error-codes 
Return different exit code if repo is locked or does not exist
 2024-07-12 21:05:52 +02:00
Michael Eischer 8731667156 restore: do not delete target if it is a file 
This is implicitly achieved by creating the target directory as the
first step, which will fail if a file already exist in its place.
 2024-07-10 22:08:49 +02:00
Michael Eischer 94fdca08c4 return exit code 10 if repository does not exist 2024-07-10 21:46:26 +02:00
Michael Eischer d8dbc71deb
Merge pull request #4908 from MichaelEischer/improve-anonymous-s3 
S3: Improve anonymous access
 2024-07-10 20:19:23 +02:00
Michael Eischer f74e70cc36 s3: forbid anonymous authentication unless explicitly requested 2024-07-10 20:10:27 +02:00
Michael Eischer 4b364940aa s3: use http client with configured timeouts for s3 IAM communication 
The default client has no timeouts configured opening network
connections. Thus, if 169.254.169.254 is inaccessible, then the client
would wait for until the operating system gives up, which will take
several minutes.
 2024-07-07 11:32:40 +02:00
Michael Eischer a2a2401a68 s3: prevent repeated credential queries with anonymous authentication 2024-07-07 11:31:04 +02:00
Michael Eischer aedead2823 ui/termstatus: fix clearing status lines 
To clear the status lines, they should be set to an empty array to
prevent future updates of those lines. Setting the status lines to an
array containing an empty string is wrong as this causes the output to
continuously add that empty status line after each message.
 2024-07-06 11:27:35 +02:00
Michael Eischer 868219aad1 restore: test --dry-run plus --delete 2024-07-05 22:38:39 +02:00
Michael Eischer aa8e18cf32 restore: add deletions to progress output 2024-07-05 22:38:39 +02:00
Michael Eischer f4b15fdd96 restore: allow deleting a directory to replace it with a file 
When the `--delete` option is specified, recursively delete directories
that should be replaced with a file.
 2024-07-05 22:38:39 +02:00
Michael Eischer 168fc09d5f restore: use case insensitive file name comparison on windows 2024-07-05 22:38:39 +02:00
Michael Eischer 013a6156bd restore: remove unused parameter from SelectFilter 2024-07-05 22:38:39 +02:00
Michael Eischer ac44bdf6dd restore: add --delete option to remove files that are not in snapshot 2024-07-05 22:38:39 +02:00
Michael Eischer 144e2a451f restore: track expected filenames in a folder 2024-07-05 21:03:35 +02:00
Michael Eischer d762f4ee64 restore: simplfy selectFilter arguments 2024-07-05 21:03:35 +02:00
Michael Eischer a9a60f77ce restore: optimize memory usage 2024-07-05 21:03:35 +02:00
Michael Eischer 1369658a32 archiver: extract Readdirnames to fs package 2024-07-05 21:03:34 +02:00
Michael Eischer ae978d60cc restore: pass action enum to restore progress 2024-07-05 20:41:27 +02:00
Michael Eischer 83351f42e3 restore: add dry-run support 2024-07-05 20:41:27 +02:00
Michael Eischer c47bf33884 restore: print processed files at maximum verbosity 2024-07-05 20:41:27 +02:00
Michael Eischer c9a4a95848 check: suggest using `repair packs` to repair truncated pack files 
Previously, that help message was only shown for running `check
--read-data`.
 2024-07-05 20:04:25 +02:00
Viktor Szépe ac00229386 Fix typos 2024-07-03 20:02:06 +02:00
Michael Eischer 132965fd5c
Merge pull request #4864 from MichaelEischer/restore-delete 
restore: improve file type mismatch handling
 2024-06-26 21:27:57 +02:00
Michael Eischer 9817126341
Merge pull request #4863 from MichaelEischer/safer-repo-init2 
repository: prevent initialization if a snapshot exists
 2024-06-26 21:11:11 +02:00
Michael Eischer c01b655229 backup: fix reading xattrs on Windows for root disks as regular user 
Unprivileged users cannot open the root disks with write permissions for
xattrs. Thus, only request read permissions for reading the xattrs.
 2024-06-26 20:13:50 +02:00
Michael Eischer 20a6c526e5 restic: also fix file path when reading windows xattrs 2024-06-14 22:40:25 +02:00
Michael Eischer b80aa7b1cc repository: prevent initialization if a snapshot exists 2024-06-14 20:37:01 +02:00
Michael Eischer deca7d08ac restorer: cleanup unexpected xattrs on windows 2024-06-14 20:17:06 +02:00
Michael Eischer 9572b7224f restorer: windows test fixes 2024-06-13 23:20:09 +02:00
Michael Eischer ca41c8fd11 restore: use fs function wrappers 
This ensures proper path handling on Windows.
 2024-06-13 22:40:37 +02:00
Michael Eischer 4d6042fe95 restore: remove unexpected xattrs from files 2024-06-13 22:21:00 +02:00
Michael Eischer ebbd4e26d7 restorer: allow directory to replace existing file 2024-06-13 21:57:48 +02:00
Michael Eischer ac729db3ce restorer: fix overwriting of special file types 
An attempt to replace an existing file with a hardlink previously ended
with a missing file.

Remove an existing file before trying to restore a special node. This
generalizes the existing behavior for symlinks to all special node
types.
 2024-06-13 21:56:18 +02:00
Michael Eischer c7902b7724 restorer: cleanup overwrite tests 2024-06-13 21:55:33 +02:00
Michael Eischer c598a751c2 restore: fine-grained sparse support for windows 2024-06-13 21:55:33 +02:00
Michael Eischer d265ec64f2 restore: correctly handle existing hardlinks 
With hardlinks there's no efficient way to detect which files are linked
with each other. Thus, just start from scratch when restore has to
modify a hardlinked file.
 2024-06-13 21:55:33 +02:00
Michael Eischer f19b69af25 restore: allow overwrite to replace empty directories and symlinks 
With an already existing file tree an old directory or symlink may exist
in a place where restore wants to create a new file. Thus, check for
unexpected file types and clean up if necessary.
 2024-06-13 21:55:33 +02:00
Michael Eischer f1c76a8286 restore: fix corrupted sparse files 2024-06-13 21:17:30 +02:00
Michael Eischer 1eccd6504b restore: test restore only changed parts functionality 2024-06-13 21:17:30 +02:00
Michael Eischer 5c3709e17a restore: add --overwrite=if-changed to skip files if their mtime&size matches 
--overwrite=always still checks the file content
 2024-06-13 21:17:30 +02:00
Michael Eischer a66658b4c9 restore: only restore changed file parts 
For files that should be overwritten, first check whether their content
matches the snapshot and only restore diverging parts.
 2024-06-13 21:17:30 +02:00
Michael Eischer 62e0e5e8d8 restore: cleanup options handling 2024-06-12 22:53:53 +02:00
Michael Eischer 8733cd21b9 restore: fix file encryption on windows 
The assumption that a file is not readonly is no longer valid.
 2024-06-12 22:53:53 +02:00
Michael Eischer 8aa3ab6ef8 restore: check file type in verify 2024-06-12 22:53:53 +02:00
Michael Eischer 7f7c995977 fix linter warnings 2024-06-12 22:36:52 +02:00
Michael Eischer ba53a2abb5 test overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer e47e08a688 restorer: separately track skipped files 2024-06-12 22:36:52 +02:00
Michael Eischer 64b7b6b975 restore/ui: refactor for extensibility 2024-06-12 22:36:52 +02:00
Michael Eischer 6a4ae9d6b1 restore: configurable overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer a23cb3a428 restore: reduce memory usage 2024-06-12 22:36:52 +02:00
Michael Eischer 2b50c2606c restorer: use options struct 2024-06-12 22:36:52 +02:00
Michael Eischer 0fcd89f892 restorer: remove special case for blobs with many occurrences 
Loading blobs by now is no longer prone to timeouts when processing
takes a long time.
 2024-06-12 22:36:52 +02:00
Michael Eischer fd2ff464a2 restorer: remove stale comment 2024-06-12 22:36:52 +02:00
Michael Eischer 607daeed4f restore: move nil pointer check into restoreui 2024-06-12 22:36:52 +02:00
Michael Eischer 30320a249a restore: let filerestorer also handle empty files 
This get's rid of the corresponding special cases.
 2024-06-12 22:36:52 +02:00
Michael Eischer c166ad7daf restore: factor out file creation helper 2024-06-12 22:36:52 +02:00
Michael Eischer da4512738a
Merge pull request #4853 from MichaelEischer/safer-repo-init 
init: double check that no repository exists yet
 2024-06-09 11:27:45 +02:00
Michael Eischer 8440b94159
Merge pull request #4807 from zmanda/windows-extendedattribs 
Back up and restore Extended Attributes on Windows NTFS
 2024-06-09 09:18:04 +00:00
Michael Eischer bab760369f repository: double check that there is not repository before init 
Apparently, calling `Stat` on the config file can be unreliable for some
backends.
 2024-06-09 00:05:32 +02:00
Michael Eischer 78485160fc
Merge pull request #4849 from fthoma/table-tcwidth 
Use character display width for table padding
 2024-06-07 19:53:14 +00:00
Michael Eischer 7cd324fe26 ui/table: avoid duplicate table cell width calculation 2024-06-07 21:45:40 +02:00
Michael Eischer edd3e214c2 ui/table: fix width calculation of multi-line column headers 2024-06-07 21:44:49 +02:00
Michael Eischer 3f878aa8e7
Merge pull request #4845 from greatroar/errors 
Fix error handling bug + clean up error messages
 2024-06-07 17:07:07 +00:00
Michael Eischer 112de8526d
Merge pull request #4852 from MichaelEischer/master 
crypto: cleanup ciphertext verification error
 2024-06-07 19:06:08 +02:00
Michael Eischer 0a70bbcea5
Merge pull request #4844 from MichaelEischer/improve-timeout-error 
backend: Improve timeout error message
 2024-06-07 19:05:39 +02:00
aneesh-n 4f053da06a
Fix test case by correcting cleanup order 2024-06-05 23:10:29 -06:00
aneesh-n 2101dfe448
Add missing return to fix failing test 2024-06-05 22:40:21 -06:00
aneesh-n 7a48c9ebd7
Fix review comments 2024-06-05 16:06:57 -06:00
Michael Eischer b82ecc6425 crypto: cleanup ciphertext verification error 2024-06-05 22:33:20 +02:00
Florian Thoma e9de9684f4
Use character display width for table padding 
Using len(...) for table cell padding produced wrong results for unicode
chracters leading to misaligned tables. Implementation changed to take
the actual terminal display width into consideration.
 2024-06-05 09:33:15 +02:00
Leo R. Lundgren b2bbbe805f azure: Improve error message in azure.Create() 2024-06-03 23:37:17 +02:00
Michael Eischer 660679c2f6
Merge pull request #4835 from MichaelEischer/fix-list-cancel 
backend/retry: do not log final error if context was canceled
 2024-06-01 19:17:30 +02:00
Michael Eischer db2398f35b backend: increase request progress timeout to 5 minutes 
Apparently, 2 minutes are too short in some cases and can result in
canceled List requests.
 2024-06-01 19:01:51 +02:00
Michael Eischer 6bf3d4859f backend: improve error on http request timeout 
Now yields a "request timeout" error instead of "context canceled".
 2024-06-01 18:52:39 +02:00
greatroar 4a874000b7 gs: Replace some errors.Wrap calls 
The first one in Create is already a WithStack error. The rest were
referencing code that hasn't existed for quite some time. Note that
errors from Google SDKs tends to start with "google:" or "googleapi:".

Also, use restic/internal/errors.
 2024-06-01 15:11:06 +02:00
Michael Eischer 38654a3bd7 backend/retry: do not log final error if context was canceled 
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
 2024-05-30 18:48:52 +02:00
Michael Eischer 2280fbfd2e
Merge pull request #4810 from konidev20/fix-gh-4768-add-custom-user-agent-for-http-client 
Allow custom User-Agent to be specified for outgoing requests
 2024-05-30 16:34:52 +00:00
Srigovind Nayak de7b418bbe http: allow custom `User-Agent` for outgoing HTTP requests 2024-05-30 15:38:06 +02:00
Michael Eischer 6869bdaaa8 backup: implement --skip-if-unchanged 2024-05-30 15:14:31 +02:00
Michael Eischer 7b4f81d964
Merge pull request #4829 from greatroar/termstatus 
termstatus: Do formatting in Message, not Terminal
 2024-05-30 11:57:44 +00:00
Michael Eischer 52b3fee42b
Merge pull request #4818 from MichaelEischer/move-code 
Move cache / index / hashing / pack packages
 2024-05-30 13:54:31 +02:00
greatroar 46f04bc5f0 termstatus: Do formatting in Message, not Terminal 2024-05-27 19:05:39 +02:00
Michael Eischer 1c6067d93d bloblru: variable name cleanup 2024-05-26 12:42:46 +02:00
Michael Eischer 21ce03cff2 bloblru: move defer outside critical section 2024-05-26 12:38:20 +02:00
Michael Eischer 0ad0b7ca7c bloblru: fix race condition that can compute value multiple times 2024-05-26 12:37:24 +02:00
Michael Eischer 7ea508e7b8
Merge pull request #4815 from greatroar/termstatus 
Termstatus refactoring
 2024-05-25 12:08:53 +00:00
Michael Eischer 496e57f956 hashing: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 5e0ea8fcfa pack: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 50ec408302 index: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 8e5d7d719c cache: move to backend package 2024-05-24 23:04:06 +02:00
Michael Eischer 3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer 462b82a060 index: reduce size of compressed indexes 
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
 2024-05-24 22:18:14 +02:00
Michael Eischer 77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer 2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer 93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer 860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer 027cc64737 repository: fix prune heuristic to allow resuming interrupted runs 
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
 2024-05-24 21:33:17 +02:00
Michael Eischer e52033a8bd index: slightly reduce Rewrite concurrency 
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
 2024-05-24 21:33:17 +02:00
Michael Eischer 57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer 2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer 5f7b48e65f index: replace Save() method with Rewrite and SaveFallback 
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
 2024-05-24 21:33:17 +02:00
Michael Eischer ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer 72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer 68fa0e0305 prune: no longer disable automatic index updates 
this allows prune to resume an interrupted prune run.
 2024-05-24 21:33:17 +02:00
Michael Eischer 9aa0c90fb2 index: remove supersedes field 
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
 2024-05-24 21:33:17 +02:00
Michael Eischer 76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer 04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 550d1eeac3 repository: remove SaveIndex from interface 
The method is now only indirectly accessible via Prune or RepairIndex.
 2024-05-24 21:33:17 +02:00
Michael Eischer fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer 447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer 6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer 864995271e repository: unwrap BlobHandle parameters of LookupBlob 
The method now uses the same parameters as LookupBlobSize.
 2024-05-24 21:33:17 +02:00
Michael Eischer 1266a4932f repository: fix parameter order of LookupBlobSize 
All methods should use blobType followed by ID.
 2024-05-24 21:33:17 +02:00
Michael Eischer e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer 0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer 0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer 8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer 3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer 57f9739573 forget: Add --unsafe-allow-remove-all option 
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
 2024-05-24 20:45:33 +02:00
Michael Eischer c0e1f36830 forget: refuse deleting the last snapshot in a snapshot group 
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
 2024-05-24 20:45:33 +02:00
Michael Eischer d106ad6921 restic: regenerate snapshot keep policy golden test files 2024-05-24 20:45:33 +02:00
Michael Eischer e4a48085ae backend/retry: feature flag new retry behavior 2024-05-24 20:24:02 +02:00
Michael Eischer b1266867d2 repository: wait max 1 minutes for lock removal if context is canceled 
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
 2024-05-24 20:24:02 +02:00
Michael Eischer 98709a4372 retry: reduce total number of retries 
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions

The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.

Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
 2024-05-24 20:24:02 +02:00