Commit Graph

2387 Commits

Author SHA1 Message Date
Damien Clark 4795143d6d cache: fix race condition in cache cleanup
Fix multiple restic processes executing concurrently and racing to remove obsolete snapshots.

Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2024-09-14 18:07:46 +02:00
Michael Eischer 34fe73ea42 fs: retry preallocate on Linux if interrupted by signal 2024-09-07 16:39:40 +02:00
Srigovind Nayak b69c6408a6
forget: make oldest snapshot marker more strict
Now, a snapshot is only marked as oldest if it's the last in the list AND its values matches the last seen value for that bucket.

Also, updated the corresponding golden files for the tests.
2024-09-07 15:07:23 +05:30
Srigovind Nayak d656a50852
forget: update tests to reflect specific reasons for keeping oldest snapshots in a group 2024-09-07 15:07:23 +05:30
Srigovind Nayak 87f30bc787
forget: indicate why the oldest snapshot in a group is kept
When the oldest snapshot in the
list is retained, the reason is now prefixed with "oldest" to clearly
indicate why it's being kept.
2024-09-07 15:07:23 +05:30
Michael Eischer 9a6059eb71
Merge pull request #5032 from dropbigfish/master
chore: fix some function name comments
2024-09-01 21:52:26 +00:00
dropbigfish 6f9513d88c chore: fix some function names
Signed-off-by: dropbigfish <fillfish@foxmail.com>
2024-09-01 00:54:39 +08:00
Michael Eischer b91ef3f1ff fs: remove dead code 2024-08-31 18:40:36 +02:00
Michael Eischer e2bce1b9ee fs: move WindowsAttributes definition back to restic package 2024-08-31 18:40:36 +02:00
Michael Eischer ebdd946ac1 fs: unexport nodeRestoreTimestamps 2024-08-31 18:40:36 +02:00
Michael Eischer 2aa1e2615b fs: fix comments 2024-08-31 18:40:36 +02:00
Michael Eischer 6c16733dfd fs: remove unused methods from File interface 2024-08-31 18:40:36 +02:00
Michael Eischer f0329bb4e6 fs: replace statT with ExtendedFileInfo 2024-08-31 18:40:36 +02:00
Michael Eischer 6d3a5260d3 fs: unexport a several windows functions 2024-08-31 18:40:36 +02:00
Michael Eischer cf051e777a fs: remove Readdir method from File interface 2024-08-31 18:20:41 +02:00
Michael Eischer cc7f99125a minimize usage of internal/fs in tests 2024-08-31 18:20:41 +02:00
Michael Eischer 65a7157383 mount: use os instead of fs package 2024-08-31 18:20:41 +02:00
Michael Eischer 24f4e780f1 backend: consistently use os package for filesystem access
The go std library should be good enough to manage the files in the
backend and cache folders.
2024-08-31 18:20:40 +02:00
Michael Eischer ca1e5e10b6 add proper constants for node type 2024-08-31 18:20:01 +02:00
Michael Eischer 7bb92dc7bd archiver: use ExtendedStat from FS interface
With this change, NodeFromFileInfo is the last function that bypasses
the FS interface in the archiver.
2024-08-31 18:05:09 +02:00
Michael Eischer e79dca644e fs: unexport DeviceID 2024-08-31 18:04:53 +02:00
Michael Eischer 70fbad6623 archiver: minimize imports 2024-08-31 18:04:37 +02:00
Michael Eischer 6fd5d5f2d5 archiver: move helper functions to combine rejects 2024-08-31 18:04:22 +02:00
Michael Eischer f1585af0f2 move include/exclude options to filter package 2024-08-31 18:04:07 +02:00
Michael Eischer 41c031a19e backup: move RejectFuncs to archiver package 2024-08-31 18:03:35 +02:00
Michael Eischer f9dbcd2531 backup: convert reject funcs to use FS interface
Depending on parameters the paths in a snapshot do not directly
correspond to real paths on the filesystem. Therefore, reject funcs must
use the FS interface to work correctly.
2024-08-31 18:03:02 +02:00
Michael Eischer c6fae0320e archiver: hide implementation details 2024-08-31 17:52:45 +02:00
Michael Eischer 507842b614 fs: remove Open method from FS interface 2024-08-31 17:37:25 +02:00
Michael Eischer 263709da8c fs: unexport isListxattrPermissionError 2024-08-31 17:37:25 +02:00
Michael Eischer 80ed863aab repository: remove redundant cleanup code
The temp files used by the packer manager are either delete after
creation (unix) or marked as delete on close (windows). Thus, no
explicit cleanup is necessary.
2024-08-31 17:37:25 +02:00
Michael Eischer 0ddb4441d7 fs: clean up helper functions 2024-08-31 17:37:25 +02:00
Michael Eischer fc549c9462 cleanup imports 2024-08-31 17:37:25 +02:00
Michael Eischer b9b32e5647 restic: extract Node filesystem code to fs package 2024-08-31 17:37:25 +02:00
Michael Eischer a2e54eac64 restic: simplify nodeCreateFileAt
The code to write the file content is never used.
2024-08-31 17:37:25 +02:00
Michael Eischer 5644079707 restic: prepare extraction of fs code from Node 2024-08-31 17:37:25 +02:00
Michael Eischer 97f696b937 backend: remove dead code 2024-08-31 17:25:24 +02:00
Michael Eischer af989aab4e backend/layout: unexport fields and simplify rest layout 2024-08-31 17:25:24 +02:00
Michael Eischer 6024597028 drop support for s3legacy layout 2024-08-31 17:25:24 +02:00
Michael Eischer 943b6ccfba index: remove support for legacy index format 2024-08-31 17:12:43 +02:00
Michael Eischer a5533344f9
Merge pull request #5028 from MichaelEischer/windows-allow-specifying-volumes
backup: support specifying volume instead of path on Windows
2024-08-31 16:43:20 +02:00
Michael Eischer ddf35a60ad
Merge pull request #5026 from MichaelEischer/fix-handling-invalid-filenames
cache: Fix handling of invalid filenames
2024-08-31 16:42:13 +02:00
Michael Eischer 4fcedb4bae backup: support specifying volume instead of path on Windows
"C:" (volume name) versus "C:\" (path)
2024-08-30 11:35:43 +02:00
Michael Eischer a0f2dfbc19
Merge pull request #5019 from MichaelEischer/fix-windows-sd-race
backup: Fix spurious "A Required Privilege Is Not Held by the Client" error
2024-08-29 16:59:06 +02:00
Michael Eischer 0aadfe32bb
Merge pull request #5018 from MichaelEischer/rest-retry-http2-goaway
rest: improve handling of HTTP2 goaway
2024-08-29 16:58:04 +02:00
Michael Eischer 7bbf75237d
Merge pull request #5014 from MichaelEischer/configurable-slow-request-timeout
Make timeout for slow requests configurable
2024-08-29 16:52:24 +02:00
Michael Eischer 8eff4e0e5c cache: correctly ignore files whose filename is no ID
this can for example be the case for temporary files created by the
backend implementation.
2024-08-29 16:32:15 +02:00
Michael Eischer 9c70794886 fs: fix error handling for retried get/set of security descriptor
The retry code path did not filter `ERROR_NOT_SUPPORTED`. Just call the
original function a second time to correctly follow the low privilege
code path.
2024-08-26 19:36:43 +02:00
Michael Eischer 6fbfccc2d3 fs: fix race condition in get/set security descriptor
Calling `Load()` twice for an atomic variable can return different
values each time. This resulted in trying to read the security
descriptor with high privileges, but then not entering the code path to
switch to low privileges when another thread has already done so
concurrently.
2024-08-26 19:31:21 +02:00
Michael Eischer 2296fdf668 lock: introduce short delay between failed locking retries
Failed locking attempts were immediately retried up to three times
without any delay between the retries. If a lock file is not found while
checking for other locks, with the reworked backend retries there is no
delay between those retries. This is a problem if a backend requires a
few seconds to reflect file deletions in the file listings. To work
around this problem, introduce a short exponentially increasing delay
between the retries. The number of retries is now increased to 4. This
results in delays of 5, 10 and 20 seconds between the retries.
2024-08-26 16:31:42 +02:00
Michael Eischer 89d216ca76
Merge pull request #5011 from MichaelEischer/fix-canceled-retry
backend/retry: don't trip circuit breaker if context is canceled
2024-08-26 16:30:03 +02:00
Michael Eischer e24dd5a162 backend/retry: don't trip circuit breaker if context is canceled
When the context used for a load operation is canceled, then the result
is always an error independent of whether the file could be retrieved
from the backend. Do not false positively trip the circuit breaker in
this case.

The old behavior was problematic when trying to lock a repository. When
`Lock.checkForOtherLocks` listed multiple lock files in parallel and one
of them fails to load, then all other loads were canceled. This
cancelation was remembered by the circuit breaker, such that locking
retries would fail.
2024-08-26 16:22:21 +02:00
Michael Eischer 36c4475ad9 rest: improve handling of HTTP2 goaway
The HTTP client can only retry HTTP2 requests after receiving a GOAWAY
response if it can rewind the body. As we use a custom data type,
explicitly provide an implementation of `GetBody`.
2024-08-26 15:44:17 +02:00
Michael Eischer b8f409723d make timeout for slow requests configurable 2024-08-26 14:14:43 +02:00
Michael Eischer 311b27ced8 restic: cleanup redundant code in test case 2024-08-25 23:18:55 +02:00
Michael Eischer 43b36ad2b0 restore: test timestamps for macOS resource forks are restored correctly 2024-08-25 23:18:55 +02:00
Michael Eischer 2e55209b34 restic: restore timestamps after extended attributes
restoring the xattr containing resource forks on macOS apparently
modifies the file modification timestamps. Thus, restore the timestamp
after xattrs.
2024-08-25 23:18:55 +02:00
Michael Eischer 5cca6e66be
Merge pull request #4981 from konidev20/fix-gh-4934-cleanup-removed-snaphots-from-cache
cache: clear snapshot files from cache during load index
2024-08-16 19:04:59 +00:00
Srigovind Nayak 88174cd0a4
cache: remove redundant index file cleanup
addressing code review comments
2024-08-17 00:21:49 +05:30
Srigovind Nayak b7d014b685
Revert "repository: removed redundant prepareCache method from Repository"
This reverts commit 720609f8ba.
2024-08-17 00:18:13 +05:30
Michael Eischer 74d3f92cc7
Merge pull request #4993 from MichaelEischer/fix-timeout-error
backend: return correct error on upload/request timeout
2024-08-15 22:07:37 +02:00
Michael Eischer 80f24584a5
Merge pull request #4998 from zmanda/ea_vss_fix
Fix extended attributes handling for VSS snapshots
2024-08-15 19:51:35 +00:00
Michael Eischer 8e00158c34
Merge pull request #5000 from deining/fix-typo
Fix typos
2024-08-15 19:42:14 +00:00
Michael Eischer 36b5580c1c
Merge pull request #4989 from plant99/progress-bar-for-restore-verify
restore: Add progress bar to 'restore --verify'
2024-08-15 19:34:05 +00:00
aneesh-n 19f487750e
Add test cases and handle volume GUID paths
Gracefully handle errors while checking for EA and add debug logs.
2024-08-11 19:25:58 -06:00
Shivashis Padhi f1407afd1f
restore: Add progress bar to 'restore --verify' 2024-08-11 22:25:21 +02:00
Andreas Deininger 4401265e36 Fix typos 2024-08-11 21:38:15 +02:00
Srigovind Nayak 5fd984ba6f
cache: add test for the automated cache clear to cache backend 2024-08-11 23:41:07 +05:30
Srigovind Nayak 720609f8ba
repository: removed redundant prepareCache method from Repository
* remove the prepareCache method from the Repository
* changed the signature of the SetIndex function to no longer return an error
2024-08-11 23:41:07 +05:30
Srigovind Nayak a23e7bfb82
cache: check for context cancellation before clearing cache 2024-08-11 23:41:07 +05:30
Srigovind Nayak f66624f5bf
cache: backend add List method and a cache clear functionality
* removes files which are no longer in the repository, including index files, snapshot files and pack files from the cache.

cache: fix ids set initialisation with NewIDSet()
2024-08-11 23:40:52 +05:30
aneesh-n 849c441455
Gracefully handle invalid prepared volume names 2024-08-11 01:48:25 -06:00
aneesh-n 1d392a36f9
Fix extended attributes handling for VSS snapshots 2024-08-11 01:23:47 -06:00
Michael Eischer 3f5e2160de
Merge pull request #4938 from MichaelEischer/bump-go-version
Bump go version to 1.21
2024-08-10 19:57:59 +02:00
Michael Eischer 283225f15f
Merge pull request #4994 from MichaelEischer/less-flaky-watchdog-test
backend: tweak timeouts to make watchdog timeout test less flaky
2024-08-10 19:35:38 +02:00
Michael Eischer 400ae55940 replace deprecated usages of math/rand 2024-08-10 19:34:49 +02:00
Michael Eischer 0b19f6cf5a Switch back to sha256 from the std library
The std library now also supports the sha assembly instructions on
ARM64. Thus, sha256-simd no longer provides a performance benefit.
2024-08-10 19:16:10 +02:00
Michael Eischer ad48751adb bump required go version to 1.21 2024-08-10 19:16:10 +02:00
Michael Eischer 86390b453d
Merge pull request #4980 from zmanda/unsupported-ea-handling
Skip ExtendedAttribute processing in Windows for volumes that do not support EA
2024-08-10 17:11:12 +00:00
Michael Eischer fa35e72214 backend: tweak timeouts to make watchdog timeout test less flaky 2024-08-10 19:08:03 +02:00
Michael Eischer 05571286b2
Merge pull request #4946 from mikix/json-errors
Improve error handling in --json mode
2024-08-10 17:04:56 +00:00
aneesh-n 18e9d71d7a
Fix review comments 2024-08-10 10:38:04 -06:00
Michael Eischer 853a686994 backend: return correct error on upload/request timeout 2024-08-10 18:06:24 +02:00
Michael Eischer a164789321
Merge pull request #4977 from MichaelEischer/warn-on-incomplete-metadata
backup: warn but store store item if extended metadata is incomplete
2024-08-10 17:59:25 +02:00
aneesh-n 71632a8197
Handle extended length paths 2024-08-05 16:03:43 -06:00
aneesh-n 85639f5159
Add handling for relative paths, vss paths, UNC paths 2024-08-04 13:19:13 -06:00
aneesh-n c13725b5d0
Check EA support only for volumes, files and dirs 2024-08-04 11:05:40 -06:00
aneesh-n 89712f6640
Formatted 2024-08-04 10:36:13 -06:00
aneesh-n 9dedba6dfc
Address review comments 2024-08-04 10:23:39 -06:00
aneesh-n 8c8a066c0e
Correct the bitmask for fileSupportsExtendedAttributes and add link 2024-08-03 18:06:47 -06:00
aneesh-n 56113a8da7
Skip EA processing for volumes that do not support EA 2024-08-03 16:03:30 -06:00
Michael Eischer 73c9780321 backup: store but warn if extended metadata for item is incomplete
Files were not included in the backup if the extended metadata for the
file could not be read. This is rather drastic. Instead settle on
returning a warning but still including the file in the backup.
2024-08-03 22:25:29 +02:00
Michael Terry 88f59fc2d6 json: switch backup and restore errors from string to struct types
This keeps backwards compatibility with the previous empty structs.
And maybe we'd want to put other fields into the inner struct later,
rather than the outer message.
2024-08-03 16:21:44 -04:00
Michael Eischer a48baf6f3a
Merge pull request #4976 from textaligncenter/backup-panic
fix panic in fs_reader
2024-08-03 19:56:15 +00:00
Michael Terry a376323331 restore: print JSON versions of errors in --json mode
Previously, they were printed as freeform text.

This also adds a ui.Terminal interface to make writing
tests easier and also adds a few tests.
2024-08-03 15:18:46 -04:00
Michael Eischer e622135e7e
Merge pull request #4974 from MichaelEischer/fix-cache-debug-log
cache: fix confusing debug log
2024-08-03 21:12:32 +02:00
textaligncenter d8ea178e69 fix panic in fs_reader 2024-08-03 19:08:04 +00:00
Michael Terry ad2585af67 backup: show actual error strings in --json mode
Previously, an error JSON fragment would look like:
{"message_type": "error", "error": {}}

This is because encoding/json cannot marshal an error interface.
Instead, we now call .Error() to get the string value.
2024-08-03 15:07:52 -04:00
Michael Terry f4bdfea1c9 backup: print scanner errors to stderr, not stdout 2024-08-03 15:07:52 -04:00
Michael Eischer d2f7c5a9c6
Merge pull request #4978 from konidev20/fix-gh-4949-repair-index-spurious-index
rewrite: skip saving empty indexes during MasterIndex.Rewrite
2024-08-03 18:53:57 +00:00
Srigovind Nayak 068d5b95c3
rewrite: skip saving empty indexes during MasterIndex.Rewrite 2024-08-03 23:34:59 +05:30
Michael Eischer d4db5a364e
Merge pull request #4958 from mikix/restore-errors
restore: clean up error handling when restoring metadata
2024-08-03 17:58:54 +00:00
Michael Eischer 4266dca1b6 cache: fix confusing debug log 2024-08-03 18:51:38 +02:00
Michael Eischer d407abb50f
Merge pull request #4960 from MichaelEischer/reduce-cancelation-delays
Add more checks for canceled contexts
2024-08-03 18:23:20 +02:00
Michael Eischer ae1cb889dd Add more checks for canceled contexts 2024-07-31 19:30:47 +02:00
Michael Terry 6a97833337 restore: clean up error handling when restoring metadata
- Fix a logic error that instead of reporting the *first*
  metadata-setting error that appears, we were instead reporting the
  *last* error (and only if the lchown call failed!).
- Don't show any errors when setting metadata for files in non-root
  mode (things like timestamps, attributes). Previously, only lchown
  errors were skipped. But other kinds of attribute errors make sense
  to skip as well. The code path happened to work correctly before
  because of the above logic error. But once that was fixed, this
  change needed to happen too.
2024-07-30 19:27:34 -04:00
Michael Eischer a9be986782 restorer: add minimal long path handling test 2024-07-29 21:11:47 +02:00
Michael Eischer 62c4a5e9a0 fs: fix long path handling for ResetPermissions, ClearAttributes and security descriptors 2024-07-29 20:10:58 +02:00
Michael Eischer 729e009485 replace all uses of target in relation to backup 2024-07-24 19:40:53 +02:00
Michael Eischer fa5ff0873a
Merge pull request #4925 from MichaelEischer/fix-restore-truncation
restore: Fix truncation of uptodate files
2024-07-21 12:28:12 +02:00
Michael Eischer 0dcac90bea restorer: don't track already uptodate blobs 2024-07-21 12:03:56 +02:00
Michael Eischer 10efa77103 restorer: add test for file truncation case 2024-07-21 12:03:56 +02:00
Michael Eischer 6f8e17a463 restorer: minor code cleanups 2024-07-21 12:03:56 +02:00
Michael Eischer 5d6ce59a8d restorer: also truncate files if their content is already uptodate
Files for which no blobs have to be restored, still have to be truncated
to the correct size. Take a file with content "foobar" that should be
replaced by restore with content "foo". The first three bytes are
already uptodate, such that no data has to be written. As file
truncation normally happens when writing data, a special case is
necessary.

This no blobs written special case is unified with the empty file
special case.
2024-07-21 12:03:56 +02:00
Michael Eischer 59c880fce3 restorer: only show size in text output for files 2024-07-20 12:46:28 +02:00
Alex Johnson 3bf2927006 Update snapshot summary on rewrite
Signed-off-by: Alex Johnson <hello@alex-johnson.net>
2024-07-16 12:06:50 -05:00
Michael Eischer dcfffd7779 restore: extend overwrite test for small files 2024-07-14 11:30:41 +02:00
Michael Eischer 44e3610b32 restore: progress bar total on windows 2024-07-14 11:21:11 +02:00
Michael Eischer 98cfb2c4c8 restore: test progress reporting for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer 26aa65e0d4 restore: add regression test for corrupt in-place restore of large file 2024-07-14 11:21:11 +02:00
Michael Eischer 2833b2f699 restore: fix progress bar for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer c77b2d5ca2 restore: avoid long cancelation delay for frequently used blobs 2024-07-14 11:21:11 +02:00
Michael Eischer f64191da9c restore: improve reporting of cancelation errors 2024-07-14 11:21:08 +02:00
Michael Eischer 2971a769da restore: fix corrupt restore of partially up to date files 2024-07-12 22:27:00 +02:00
Michael Eischer 40e5163114 restore: properly cancel file verification 2024-07-12 22:26:22 +02:00
Michael Eischer 31887ea9aa restore: fix hang on command cancelation 2024-07-12 22:24:35 +02:00
Michael Eischer 375c572c4f
Merge pull request #4884 from MichaelEischer/lock-and-not-exist-error-codes
Return different exit code if repo is locked or does not exist
2024-07-12 21:05:52 +02:00
Michael Eischer 8731667156 restore: do not delete target if it is a file
This is implicitly achieved by creating the target directory as the
first step, which will fail if a file already exist in its place.
2024-07-10 22:08:49 +02:00
Michael Eischer 94fdca08c4 return exit code 10 if repository does not exist 2024-07-10 21:46:26 +02:00
Michael Eischer d8dbc71deb
Merge pull request #4908 from MichaelEischer/improve-anonymous-s3
S3: Improve anonymous access
2024-07-10 20:19:23 +02:00
Michael Eischer f74e70cc36 s3: forbid anonymous authentication unless explicitly requested 2024-07-10 20:10:27 +02:00
Michael Eischer 4b364940aa s3: use http client with configured timeouts for s3 IAM communication
The default client has no timeouts configured opening network
connections. Thus, if 169.254.169.254 is inaccessible, then the client
would wait for until the operating system gives up, which will take
several minutes.
2024-07-07 11:32:40 +02:00
Michael Eischer a2a2401a68 s3: prevent repeated credential queries with anonymous authentication 2024-07-07 11:31:04 +02:00
Michael Eischer aedead2823 ui/termstatus: fix clearing status lines
To clear the status lines, they should be set to an empty array to
prevent future updates of those lines. Setting the status lines to an
array containing an empty string is wrong as this causes the output to
continuously add that empty status line after each message.
2024-07-06 11:27:35 +02:00
Michael Eischer 868219aad1 restore: test --dry-run plus --delete 2024-07-05 22:38:39 +02:00
Michael Eischer aa8e18cf32 restore: add deletions to progress output 2024-07-05 22:38:39 +02:00
Michael Eischer f4b15fdd96 restore: allow deleting a directory to replace it with a file
When the `--delete` option is specified, recursively delete directories
that should be replaced with a file.
2024-07-05 22:38:39 +02:00
Michael Eischer 168fc09d5f restore: use case insensitive file name comparison on windows 2024-07-05 22:38:39 +02:00
Michael Eischer 013a6156bd restore: remove unused parameter from SelectFilter 2024-07-05 22:38:39 +02:00
Michael Eischer ac44bdf6dd restore: add --delete option to remove files that are not in snapshot 2024-07-05 22:38:39 +02:00
Michael Eischer 144e2a451f restore: track expected filenames in a folder 2024-07-05 21:03:35 +02:00
Michael Eischer d762f4ee64 restore: simplfy selectFilter arguments 2024-07-05 21:03:35 +02:00
Michael Eischer a9a60f77ce restore: optimize memory usage 2024-07-05 21:03:35 +02:00
Michael Eischer 1369658a32 archiver: extract Readdirnames to fs package 2024-07-05 21:03:34 +02:00
Michael Eischer ae978d60cc restore: pass action enum to restore progress 2024-07-05 20:41:27 +02:00
Michael Eischer 83351f42e3 restore: add dry-run support 2024-07-05 20:41:27 +02:00
Michael Eischer c47bf33884 restore: print processed files at maximum verbosity 2024-07-05 20:41:27 +02:00
Michael Eischer c9a4a95848 check: suggest using `repair packs` to repair truncated pack files
Previously, that help message was only shown for running `check
--read-data`.
2024-07-05 20:04:25 +02:00
Viktor Szépe ac00229386 Fix typos 2024-07-03 20:02:06 +02:00
Michael Eischer 132965fd5c
Merge pull request #4864 from MichaelEischer/restore-delete
restore: improve file type mismatch handling
2024-06-26 21:27:57 +02:00
Michael Eischer 9817126341
Merge pull request #4863 from MichaelEischer/safer-repo-init2
repository: prevent initialization if a snapshot exists
2024-06-26 21:11:11 +02:00