MirrorSave
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,813 Commits 14 Branches 49 Tags 90 MiB
Commit Graph

2387 Commits

Author SHA1 Message Date
Michael Eischer c01b655229 backup: fix reading xattrs on Windows for root disks as regular user 
Unprivileged users cannot open the root disks with write permissions for
xattrs. Thus, only request read permissions for reading the xattrs.
 2024-06-26 20:13:50 +02:00
Michael Eischer 20a6c526e5 restic: also fix file path when reading windows xattrs 2024-06-14 22:40:25 +02:00
Michael Eischer b80aa7b1cc repository: prevent initialization if a snapshot exists 2024-06-14 20:37:01 +02:00
Michael Eischer deca7d08ac restorer: cleanup unexpected xattrs on windows 2024-06-14 20:17:06 +02:00
Michael Eischer 9572b7224f restorer: windows test fixes 2024-06-13 23:20:09 +02:00
Michael Eischer ca41c8fd11 restore: use fs function wrappers 
This ensures proper path handling on Windows.
 2024-06-13 22:40:37 +02:00
Michael Eischer 4d6042fe95 restore: remove unexpected xattrs from files 2024-06-13 22:21:00 +02:00
Michael Eischer ebbd4e26d7 restorer: allow directory to replace existing file 2024-06-13 21:57:48 +02:00
Michael Eischer ac729db3ce restorer: fix overwriting of special file types 
An attempt to replace an existing file with a hardlink previously ended
with a missing file.

Remove an existing file before trying to restore a special node. This
generalizes the existing behavior for symlinks to all special node
types.
 2024-06-13 21:56:18 +02:00
Michael Eischer c7902b7724 restorer: cleanup overwrite tests 2024-06-13 21:55:33 +02:00
Michael Eischer c598a751c2 restore: fine-grained sparse support for windows 2024-06-13 21:55:33 +02:00
Michael Eischer d265ec64f2 restore: correctly handle existing hardlinks 
With hardlinks there's no efficient way to detect which files are linked
with each other. Thus, just start from scratch when restore has to
modify a hardlinked file.
 2024-06-13 21:55:33 +02:00
Michael Eischer f19b69af25 restore: allow overwrite to replace empty directories and symlinks 
With an already existing file tree an old directory or symlink may exist
in a place where restore wants to create a new file. Thus, check for
unexpected file types and clean up if necessary.
 2024-06-13 21:55:33 +02:00
Michael Eischer f1c76a8286 restore: fix corrupted sparse files 2024-06-13 21:17:30 +02:00
Michael Eischer 1eccd6504b restore: test restore only changed parts functionality 2024-06-13 21:17:30 +02:00
Michael Eischer 5c3709e17a restore: add --overwrite=if-changed to skip files if their mtime&size matches 
--overwrite=always still checks the file content
 2024-06-13 21:17:30 +02:00
Michael Eischer a66658b4c9 restore: only restore changed file parts 
For files that should be overwritten, first check whether their content
matches the snapshot and only restore diverging parts.
 2024-06-13 21:17:30 +02:00
Michael Eischer 62e0e5e8d8 restore: cleanup options handling 2024-06-12 22:53:53 +02:00
Michael Eischer 8733cd21b9 restore: fix file encryption on windows 
The assumption that a file is not readonly is no longer valid.
 2024-06-12 22:53:53 +02:00
Michael Eischer 8aa3ab6ef8 restore: check file type in verify 2024-06-12 22:53:53 +02:00
Michael Eischer 7f7c995977 fix linter warnings 2024-06-12 22:36:52 +02:00
Michael Eischer ba53a2abb5 test overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer e47e08a688 restorer: separately track skipped files 2024-06-12 22:36:52 +02:00
Michael Eischer 64b7b6b975 restore/ui: refactor for extensibility 2024-06-12 22:36:52 +02:00
Michael Eischer 6a4ae9d6b1 restore: configurable overwrite behavior 2024-06-12 22:36:52 +02:00
Michael Eischer a23cb3a428 restore: reduce memory usage 2024-06-12 22:36:52 +02:00
Michael Eischer 2b50c2606c restorer: use options struct 2024-06-12 22:36:52 +02:00
Michael Eischer 0fcd89f892 restorer: remove special case for blobs with many occurrences 
Loading blobs by now is no longer prone to timeouts when processing
takes a long time.
 2024-06-12 22:36:52 +02:00
Michael Eischer fd2ff464a2 restorer: remove stale comment 2024-06-12 22:36:52 +02:00
Michael Eischer 607daeed4f restore: move nil pointer check into restoreui 2024-06-12 22:36:52 +02:00
Michael Eischer 30320a249a restore: let filerestorer also handle empty files 
This get's rid of the corresponding special cases.
 2024-06-12 22:36:52 +02:00
Michael Eischer c166ad7daf restore: factor out file creation helper 2024-06-12 22:36:52 +02:00
Michael Eischer da4512738a
Merge pull request #4853 from MichaelEischer/safer-repo-init 
init: double check that no repository exists yet
 2024-06-09 11:27:45 +02:00
Michael Eischer 8440b94159
Merge pull request #4807 from zmanda/windows-extendedattribs 
Back up and restore Extended Attributes on Windows NTFS
 2024-06-09 09:18:04 +00:00
Michael Eischer bab760369f repository: double check that there is not repository before init 
Apparently, calling `Stat` on the config file can be unreliable for some
backends.
 2024-06-09 00:05:32 +02:00
Michael Eischer 78485160fc
Merge pull request #4849 from fthoma/table-tcwidth 
Use character display width for table padding
 2024-06-07 19:53:14 +00:00
Michael Eischer 7cd324fe26 ui/table: avoid duplicate table cell width calculation 2024-06-07 21:45:40 +02:00
Michael Eischer edd3e214c2 ui/table: fix width calculation of multi-line column headers 2024-06-07 21:44:49 +02:00
Michael Eischer 3f878aa8e7
Merge pull request #4845 from greatroar/errors 
Fix error handling bug + clean up error messages
 2024-06-07 17:07:07 +00:00
Michael Eischer 112de8526d
Merge pull request #4852 from MichaelEischer/master 
crypto: cleanup ciphertext verification error
 2024-06-07 19:06:08 +02:00
Michael Eischer 0a70bbcea5
Merge pull request #4844 from MichaelEischer/improve-timeout-error 
backend: Improve timeout error message
 2024-06-07 19:05:39 +02:00
aneesh-n 4f053da06a
Fix test case by correcting cleanup order 2024-06-05 23:10:29 -06:00
aneesh-n 2101dfe448
Add missing return to fix failing test 2024-06-05 22:40:21 -06:00
aneesh-n 7a48c9ebd7
Fix review comments 2024-06-05 16:06:57 -06:00
Michael Eischer b82ecc6425 crypto: cleanup ciphertext verification error 2024-06-05 22:33:20 +02:00
Florian Thoma e9de9684f4
Use character display width for table padding 
Using len(...) for table cell padding produced wrong results for unicode
chracters leading to misaligned tables. Implementation changed to take
the actual terminal display width into consideration.
 2024-06-05 09:33:15 +02:00
Leo R. Lundgren b2bbbe805f azure: Improve error message in azure.Create() 2024-06-03 23:37:17 +02:00
Michael Eischer 660679c2f6
Merge pull request #4835 from MichaelEischer/fix-list-cancel 
backend/retry: do not log final error if context was canceled
 2024-06-01 19:17:30 +02:00
Michael Eischer db2398f35b backend: increase request progress timeout to 5 minutes 
Apparently, 2 minutes are too short in some cases and can result in
canceled List requests.
 2024-06-01 19:01:51 +02:00
Michael Eischer 6bf3d4859f backend: improve error on http request timeout 
Now yields a "request timeout" error instead of "context canceled".
 2024-06-01 18:52:39 +02:00
greatroar 4a874000b7 gs: Replace some errors.Wrap calls 
The first one in Create is already a WithStack error. The rest were
referencing code that hasn't existed for quite some time. Note that
errors from Google SDKs tends to start with "google:" or "googleapi:".

Also, use restic/internal/errors.
 2024-06-01 15:11:06 +02:00
Michael Eischer 38654a3bd7 backend/retry: do not log final error if context was canceled 
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
 2024-05-30 18:48:52 +02:00
Michael Eischer 2280fbfd2e
Merge pull request #4810 from konidev20/fix-gh-4768-add-custom-user-agent-for-http-client 
Allow custom User-Agent to be specified for outgoing requests
 2024-05-30 16:34:52 +00:00
Srigovind Nayak de7b418bbe http: allow custom `User-Agent` for outgoing HTTP requests 2024-05-30 15:38:06 +02:00
Michael Eischer 6869bdaaa8 backup: implement --skip-if-unchanged 2024-05-30 15:14:31 +02:00
Michael Eischer 7b4f81d964
Merge pull request #4829 from greatroar/termstatus 
termstatus: Do formatting in Message, not Terminal
 2024-05-30 11:57:44 +00:00
Michael Eischer 52b3fee42b
Merge pull request #4818 from MichaelEischer/move-code 
Move cache / index / hashing / pack packages
 2024-05-30 13:54:31 +02:00
greatroar 46f04bc5f0 termstatus: Do formatting in Message, not Terminal 2024-05-27 19:05:39 +02:00
Michael Eischer 1c6067d93d bloblru: variable name cleanup 2024-05-26 12:42:46 +02:00
Michael Eischer 21ce03cff2 bloblru: move defer outside critical section 2024-05-26 12:38:20 +02:00
Michael Eischer 0ad0b7ca7c bloblru: fix race condition that can compute value multiple times 2024-05-26 12:37:24 +02:00
Michael Eischer 7ea508e7b8
Merge pull request #4815 from greatroar/termstatus 
Termstatus refactoring
 2024-05-25 12:08:53 +00:00
Michael Eischer 496e57f956 hashing: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 5e0ea8fcfa pack: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 50ec408302 index: move to repository package 2024-05-25 13:13:03 +02:00
Michael Eischer 8e5d7d719c cache: move to backend package 2024-05-24 23:04:06 +02:00
Michael Eischer 3c7b7efdc9 repository: remove prune plan parts once they are no longer necessary 2024-05-24 22:18:14 +02:00
Michael Eischer 462b82a060 index: reduce size of compressed indexes 
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
 2024-05-24 22:18:14 +02:00
Michael Eischer 77873f5a9d repository: let prune control data structure of usedBlobs set 2024-05-24 22:18:14 +02:00
Michael Eischer 2033c02b09 index: replace CountedBlobSet with AssociatedSet 2024-05-24 22:18:14 +02:00
Michael Eischer 93098e9265 prune: hide implementation details of counted blob set 2024-05-24 21:42:56 +02:00
Michael Eischer 860b595a8b backend: increase watchdog test timeout for deflaking 2024-05-24 21:33:17 +02:00
Michael Eischer 027cc64737 repository: fix prune heuristic to allow resuming interrupted runs 
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
 2024-05-24 21:33:17 +02:00
Michael Eischer e52033a8bd index: slightly reduce Rewrite concurrency 
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
 2024-05-24 21:33:17 +02:00
Michael Eischer 57d69aa640 index: cleanup SaveIndex method 2024-05-24 21:33:17 +02:00
Michael Eischer 2ca1c37721 index: additional tests for new index save methods 2024-05-24 21:33:17 +02:00
Michael Eischer 5f7b48e65f index: replace Save() method with Rewrite and SaveFallback 
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.

However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
 2024-05-24 21:33:17 +02:00
Michael Eischer ad98fbf7dd restic: add IDSet.Clone() method 2024-05-24 21:33:17 +02:00
Michael Eischer 72482ce5bd index: misc cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer 68fa0e0305 prune: no longer disable automatic index updates 
this allows prune to resume an interrupted prune run.
 2024-05-24 21:33:17 +02:00
Michael Eischer 9aa0c90fb2 index: remove supersedes field 
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
 2024-05-24 21:33:17 +02:00
Michael Eischer 76e6719f2e repository: make CreateIndexFromPacks method private 2024-05-24 21:33:17 +02:00
Michael Eischer 04ad9f0c0c repository: remove Packer and SavePacker from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 550d1eeac3 repository: remove SaveIndex from interface 
The method is now only indirectly accessible via Prune or RepairIndex.
 2024-05-24 21:33:17 +02:00
Michael Eischer fb59e00614 index: rewrite MasterIndex load/save test to be independent of repository 2024-05-24 21:33:17 +02:00
Michael Eischer 447b486c20 index: deduplicate index loading of check and repository 2024-05-24 21:33:17 +02:00
Michael Eischer 6ca12c1b4a archiver: replace most uses of restic.Repository 2024-05-24 21:33:17 +02:00
Michael Eischer 864995271e repository: unwrap BlobHandle parameters of LookupBlob 
The method now uses the same parameters as LookupBlobSize.
 2024-05-24 21:33:17 +02:00
Michael Eischer 1266a4932f repository: fix parameter order of LookupBlobSize 
All methods should use blobType followed by ID.
 2024-05-24 21:33:17 +02:00
Michael Eischer e848ad651a restic: name parameters in restic interface 2024-05-24 21:33:17 +02:00
Michael Eischer 0bb0720348 test cleanups 2024-05-24 21:33:17 +02:00
Michael Eischer c01bcb1001 archiver: remove unused masterIndex from test 2024-05-24 21:33:17 +02:00
Michael Eischer 0aa5c53842 repository: replace HasBlob with LookupBlobSize 2024-05-24 21:33:17 +02:00
Michael Eischer 8f1e70cd9b repository: remove clearIndex and packSize from public interface 2024-05-24 21:33:17 +02:00
Michael Eischer 4df887406f repository: inline MasterIndex interface into Repository interface 2024-05-24 21:33:17 +02:00
Michael Eischer 3cc6827f09 forget: add feature flag for safe --keep-tags behavior 2024-05-24 20:45:33 +02:00
Michael Eischer 57f9739573 forget: Add --unsafe-allow-remove-all option 
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.

Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
 2024-05-24 20:45:33 +02:00
Michael Eischer c0e1f36830 forget: refuse deleting the last snapshot in a snapshot group 
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
 2024-05-24 20:45:33 +02:00
Michael Eischer d106ad6921 restic: regenerate snapshot keep policy golden test files 2024-05-24 20:45:33 +02:00
Michael Eischer e4a48085ae backend/retry: feature flag new retry behavior 2024-05-24 20:24:02 +02:00
Michael Eischer b1266867d2 repository: wait max 1 minutes for lock removal if context is canceled 
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
 2024-05-24 20:24:02 +02:00
Michael Eischer 98709a4372 retry: reduce total number of retries 
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions

The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.

Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
 2024-05-24 20:24:02 +02:00
Michael Eischer 512cd6ef07 retry: ensure that there's always at least one retry 
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
 2024-05-24 20:24:02 +02:00
Michael Eischer a60ee9b764 retry: limit retries based on elapsed time not count 
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
 2024-05-24 20:24:02 +02:00
Michael Eischer a3633cad9e retry: explicitly log failed requests 
This simplifies finding the request in the log output that cause an
operation to fail.
 2024-05-24 20:24:02 +02:00
greatroar 66d03c797e ui, termstatus: Move WrapStdio 
Saves some imports. ui still needs to import ui/termstatus from
message.go.
 2024-05-21 11:25:16 +02:00
greatroar 0b56214473 ui: Simplify stdio wrapper 
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
 2024-05-21 11:23:32 +02:00
greatroar 7f439a9c34 ui: Inline lineWriter into StdioWrapper 2024-05-21 11:00:09 +02:00
Michael Eischer c56ecec9aa azure: deduplicate cli and default credentials case 2024-05-18 22:15:54 +02:00
Maik Riechert 355f520936 Azure: add option to force use of CLI credential 2024-05-18 22:15:54 +02:00
Michael Eischer 223aa22cb0 replace some uses of restic.Repository with finegrained interfaces 2024-05-18 21:42:51 +02:00
Michael Eischer 291c9677de restic/repository: remove Backend() method 2024-05-18 21:42:51 +02:00
Michael Eischer 673496b091 repository: clean cache between CheckPack retries 
The cache cleanup pattern is also used in ListPack etc.
 2024-05-18 21:42:51 +02:00
Michael Eischer 3d2410ed50 Replace some repo.RemoveUnpacked usages 
These will eventually be blocked as they do not delete Snapshots.
 2024-05-18 21:42:51 +02:00
Michael Eischer d2c26e33f3 repository: remove further usages of repo.Backend() 2024-05-18 21:42:51 +02:00
Michael Eischer 8a425c2f0a remove usages of repo.Backend() from tests 2024-05-18 21:42:51 +02:00
Michael Eischer aa4647f773 repository: unexport PackBlobIterator 2024-05-18 21:42:51 +02:00
Michael Eischer 94e863885c check: move verification of individual pack file to repository 2024-05-18 21:42:50 +02:00
Michael Eischer e40943a75d restic: remove backend usage from lock test 2024-05-18 21:38:31 +02:00
Michael Eischer 67e2ba0d40 repository: Lock requires *repository.Repository 
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
 2024-05-18 21:38:31 +02:00
Michael Eischer d8b184b3d3 repository: convert test helper to return *repository.Repository 2024-05-18 21:38:31 +02:00
Michael Eischer a1ca5e15c4 migrations: add temporary hack for s3_layout 
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
 2024-05-18 21:38:31 +02:00
Michael Eischer 34d90aecf9 migrations: move logic of upgrade_repo_v2 to repository package 
The migration modifies repository internals and thus should live within
the repository package.
 2024-05-18 21:38:31 +02:00
Michael Eischer ab9077bc13 replace usages of backend.Remove() with repository.RemoveUnpacked() 
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
 2024-05-18 21:38:31 +02:00
Michael Eischer 0c1ba6d95d backend: remove unused Location method 2024-05-18 21:38:31 +02:00
Michael Eischer 74d90653e0 check: use ReadFull to load pack header in checkPack 
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
 2024-05-18 21:28:54 +02:00
Michael Eischer 8f8d872a68 fix compatibility with go 1.19 2024-05-18 21:28:54 +02:00
Michael Eischer ff0744b3af check: test checkPack retries 2024-05-18 21:28:54 +02:00
Michael Eischer 987c3b250c repository: test retries of ListPack 2024-05-18 21:28:54 +02:00
Michael Eischer bf16096771 repository: test LoadBlob retries 2024-05-18 21:28:54 +02:00
Michael Eischer 4f45668b7c repository: rework and extend LoadRaw tests 2024-05-18 21:28:54 +02:00
Michael Eischer ac805d6838 cache: cleanup debug logs 2024-05-18 21:28:54 +02:00
Michael Eischer 5214af88e2 cache: test forget behavior 2024-05-18 21:28:54 +02:00
Michael Eischer 3ff063e913 check: verify pack a second time if broken 2024-05-18 21:28:54 +02:00
Michael Eischer 385cee09dc repository: fix caching of tree packs in LoadBlobsFromPack 2024-05-18 21:28:54 +02:00
Michael Eischer e734746f75 cache: forget cached file at most once 
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.

By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
 2024-05-18 21:28:54 +02:00
Michael Eischer 97a307df1a cache: Always use cached file if it exists 
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
 2024-05-18 21:28:54 +02:00
Michael Eischer 433a6aad29 repository: remove redundant blob loading fallback from RepairPacks 
LoadBlobsFromPack already implements the same fallback behavior.
 2024-05-18 21:28:54 +02:00
Michael Eischer e401af07b2 check: fix error message formatting 2024-05-18 21:28:54 +02:00
Michael Eischer 7017adb7e9 repository: retry failed ListPack once 2024-05-18 21:28:54 +02:00
Michael Eischer e33ce7f408 repository: retry failed LoadBlob once 2024-05-18 21:28:54 +02:00
Michael Eischer 2ace242f36 repository: make reloading broken files explicit 2024-05-18 21:28:54 +02:00
Michael Eischer e9390352a7 cache: code cleanups 2024-05-18 21:26:00 +02:00
Michael Eischer 503c8140b1 repository: unify blob decoding code 2024-05-18 21:26:00 +02:00
Michael Eischer 6563f1d2ca repository: remove redundant debug log 2024-05-18 21:26:00 +02:00
Michael Eischer 021fb49559 repository: Implement repository.LoadUnpacked using LoadRaw 
Both functions were using a similar implementation.
 2024-05-18 21:26:00 +02:00
Michael Eischer 779c8d3527 debug/repair packs/upgrade repo v2: use repository.LoadRaw 
This replaces calling the low-level backend.Load() method.
 2024-05-18 21:26:00 +02:00
Michael Eischer 1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw 
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
 2024-05-18 21:26:00 +02:00
Michael Eischer 47232bf8b0 backend: move LimitReadCloser to util package 
The helper is only intended for usage by backend implementations.
 2024-05-18 21:26:00 +02:00
Michael Eischer 53d15bcd1b retry: add circuit breaker to load method 
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
 2024-05-18 19:59:26 +02:00
Michael Eischer 394c8ca3ed rest/rclone/s3/sftp/swift: move short file detection behind feature gate 
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
 2024-05-18 19:59:26 +02:00
Michael Eischer 6328b7e1f5 replace "too small" with "too short" in error messages 2024-05-18 19:59:26 +02:00
Michael Eischer aeb7eb245c retry: do not retry permanent errors 
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
 2024-05-18 19:59:26 +02:00
Michael Eischer bf8cc59889 Use generic backend-error-redesign feature flag instead of http-timeouts 
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
 2024-05-18 19:54:52 +02:00
Michael Eischer 4740528a0b backend: add tests for IsPermanentError 2024-05-18 19:54:52 +02:00
Michael Eischer 6a85df7297 backend: add IsPermanentError() method to interface 2024-05-18 19:54:52 +02:00
Michael Eischer cfc420664a mem: stricter handling of out of bounds requests 2024-05-18 19:54:52 +02:00
Michael Eischer d40f23e716 azure/b2/gs/s3/swift: adapt cloud backend 2024-05-18 19:54:51 +02:00
Michael Eischer e793c002ec local: stricter handling of short files 2024-05-18 19:54:21 +02:00
Michael Eischer b4895ebd76 rest: rework error reporting and report too short files 2024-05-18 19:54:21 +02:00
Michael Eischer eaa3f81d6b sftp: check for truncated files without an extra backend request 2024-05-18 19:54:21 +02:00
Michael Eischer c6d74458ee sftp: improve handling of too short files 2024-05-18 19:54:21 +02:00
aneesh-n 9d1b23964d
Update comment for licensing 2024-05-17 14:19:05 -06:00
aneesh-n 5cff6e084e
Add support for Windows EA in node 
Refactor Extended Attribute related functions in node files as windows apis get and set EA in bulk
 2024-05-17 14:18:20 -06:00
aneesh-n d6708505b9
Add helpers for Windows Extended Attributes 2024-05-17 14:15:03 -06:00
Michael Eischer 7ed560a201
Merge pull request #4796 from MichaelEischer/parallel-dump-load 
dump: Parallelize loading large files
 2024-05-14 22:35:44 +02:00
Michael Eischer 92221c2a6d
Merge pull request #4708 from zmanda/windows-securitydesc 
Back up and restore SecurityDescriptors on Windows
 2024-05-12 14:14:39 +00:00
Michael Eischer b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance 
Fix sftp upload performance
 2024-05-12 15:28:33 +02:00
Michael Eischer ebd01a4675 backend: add tests for watchdogRoundTripper 2024-05-09 23:46:17 +02:00
Michael Eischer 8778670232 backend: cancel stuck http requests 
requests that make no upload or download progress within a timeout are
canceled.
 2024-05-09 23:46:17 +02:00
Michael Eischer 0987c731ec backend: configure protocol-level connection health checks 
This should detect a connection that is stuck for more than 2 minutes.
 2024-05-09 23:46:17 +02:00
aneesh-n a4fd1b91e5
Fix review comments 
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
 2024-05-06 16:54:08 -06:00
Michael Eischer 4d55a62ada bloblru: add test for GetOrCompute 2024-05-05 12:00:25 +02:00
Michael Eischer 7cce667f92 fuse: switch to use bloblru.GetOrCompute 2024-05-05 11:38:42 +02:00
Michael Eischer bd03af2feb dump: add GetOrCompute to bloblru cache 2024-05-05 11:38:42 +02:00
Michael Eischer 45509eafc8 dump: load blobs of a file from repository in parallel 2024-05-05 11:38:42 +02:00
Michael Eischer ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling 
Rework repository.StreamPacks & better restorer error handling
 2024-05-01 16:37:41 +02:00
aneesh-n 672f6cd776
Fix review comments for privileges and security flags 2024-04-29 17:29:51 -06:00
aneesh-n 08c6945d61
Fix review comments 2024-04-29 16:21:38 -06:00
Aneesh N 3f76b902e5
Merge branch 'master' into windows-securitydesc 2024-04-29 14:40:34 -06:00
DRON-666 125dba23c5 Rearange code 2024-04-29 01:27:34 +03:00
DRON-666 7ee889bb0d Use S_FALSE and MaxInt 2024-04-29 01:25:25 +03:00
DRON-666 90b168eb6c isMountPointExcluded to isMountPointIncluded 2024-04-29 01:23:50 +03:00
DRON-666 24330c19a8 Use kebab case in option names 2024-04-29 01:21:33 +03:00
DRON-666 5703e5a652 Fix texts and comments 2024-04-29 01:18:46 +03:00
DRON-666 0a8f9c5d9c vss: Add tests for "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 bb0f93ef3d vss: Add "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 3bac1f0135 vss: Fix issues reported by linters 2024-04-28 22:45:21 +03:00
DRON-666 88c509e3e9 vss: Change `ErrorHandler` signature 
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
 2024-04-28 22:44:16 +03:00
DRON-666 9d3d915e2c vss: Add some tests 2024-04-28 22:44:16 +03:00
DRON-666 c4f67c0064 vss: Add volume filtering 
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
 2024-04-28 22:44:15 +03:00
DRON-666 7470e5356e vss: Add "timeout" option 
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
 2024-04-28 22:44:15 +03:00
DRON-666 78dbc5ec58 vss: Add initial support for extended options 2024-04-28 22:44:15 +03:00
Michael Eischer 935327d480 sftp: slightly increase write concurrency 
This should increase upload throughput for high latency links a bit.
 2024-04-28 11:50:09 +02:00
Michael Eischer 669a669603 sftp: Fix upload performance issue 
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
 2024-04-28 11:48:26 +02:00
Michael Eischer 940a3159b5 let index.Each() and pack.Size() return error on canceled context 
This forces a caller to actually check that the function did complete.
 2024-04-22 22:39:32 +02:00
Michael Eischer 31624aeffd Improve command shutdown on context cancellation 2024-04-22 22:31:38 +02:00
Michael Eischer b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune 
Refactor repair index / prune into the repository package
 2024-04-22 22:24:53 +02:00
Michael Eischer 2e6c43c695
Merge pull request #4761 from MichaelEischer/fix-cache-race 
cache: ignore ErrNotExist during cleanup of old files
 2024-04-22 21:46:06 +02:00
Michael Eischer f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob 
repository: Better error message if blob is larger than 4GB
 2024-04-22 21:45:06 +02:00
Michael Eischer 20d8eed400 repository: streamPack: separate requests for gap larger than 1MB 
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
 2024-04-22 21:21:23 +02:00
Michael Eischer cf700d8794 repository: streamPack: reuse zstd decoder 2024-04-22 21:21:23 +02:00
Michael Eischer 666a0b0bdb repository: streamPack: replace streaming with chunked download 
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.

Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.

To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
 2024-04-22 21:21:23 +02:00
Michael Eischer 621012dac0 repository: Add blob loading fallback to LoadBlobsFromPack 
Try to retrieve individual blobs via LoadBlob if streaming did not work.
 2024-04-21 21:35:55 +02:00
Michael Eischer 10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
coderwander a82ed71de7 Fix struct names 
Signed-off-by: coderwander <770732124@qq.com>
 2024-04-18 10:02:09 +08:00
Michael Eischer 09587e6c08 repository: duplicate a few blobs in prune tests 2024-04-14 13:57:19 +02:00
Michael Eischer defd7ae729 prune/repair index: reset in-memory index after command 
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
 2024-04-14 13:46:24 +02:00
Michael Eischer 038586dc9d repository: add minimal test for prune 2024-04-14 13:45:17 +02:00
Michael Eischer d8622c86eb prune: clean up internal interface 2024-04-14 13:45:15 +02:00
Michael Eischer 8d507c1372 repository: add basic test for RepairIndex 2024-04-14 13:45:15 +02:00
Michael Eischer 310db03c0e repair index: improve log output if index cannot be deleted 
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
 2024-04-14 13:45:13 +02:00
Michael Eischer 7d1b9cde34 repository: use normal Init method in tests 2024-04-14 13:45:11 +02:00
Michael Eischer b25fc2c89d repository: remove redundant flushes from tests 2024-04-14 13:45:10 +02:00
Michael Eischer c65459cd8a repository: speed up tests 2024-04-14 13:45:10 +02:00
Michael Eischer eda9f7beb4 ui/progress: add helper to print messages during tests 2024-04-14 13:45:08 +02:00
Michael Eischer 35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer 7ba5e95a82 check: allow tests to only verify pack&index integrity 2024-04-14 13:45:04 +02:00
Michael Eischer 4c9a10ca37 repair packs: deduplicate index rebuild 2024-04-14 13:45:02 +02:00
Michael Eischer 85e4021619 prune: move additional option checks to repository 2024-04-14 13:44:58 +02:00
Michael Eischer 55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout 
Deprecate s3legacy layout
 2024-04-11 22:09:34 +02:00
Michael Eischer fc3b548625 prune: move logic into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 866ddf5698 repair index: refactor code into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Michael Eischer bf054c09d2 backup: Ignore xattr.list permission error for parent directories 
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
 2024-04-10 20:46:15 +02:00
Michael Eischer 0747cf5319 cache: ignore ErrNotExist during cleanup of old files 
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
 2024-04-10 19:25:51 +02:00
Michael Eischer 49f98f25fc
Merge pull request #4742 from MichaelEischer/consistent-rtest-import 
Use consistent alias for interal/test package
 2024-04-03 21:47:32 +02:00
Michael Eischer 6cca1d5705
Merge pull request #4655 from ae-govau/unixsocket 
Enhancement: option to send HTTP over unix socket
 2024-04-03 19:29:21 +00:00
Michael Eischer ec2b79834a use consistent alias for interal/test package 2024-03-29 00:24:03 +01:00
Michael Eischer 5e98f1e2eb repository: fix test setup race conditions 2024-03-28 23:17:02 +01:00
Michael Eischer dc441c57a7 repository: unify repository initialization in tests 
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
 2024-03-28 23:17:02 +01:00
Michael Eischer 3ba1fa3cee repository: remove a few global variables 2024-03-28 23:17:02 +01:00
Michael Eischer 044e8bf821 repository: parallelize lock tests 2024-03-28 23:17:02 +01:00
Michael Eischer e8df50fa3c repository: remove global list of locks 2024-03-28 22:46:33 +01:00
Michael Eischer cbb5f89252 lock: move code to repository package 2024-03-28 22:46:33 +01:00
Michael Eischer 7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics 
Store snapshot statistics & print snapshot size
 2024-03-28 22:41:45 +01:00
Michael Eischer d705741571 backup: test that deviceID is only stored for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer a9b3d86c4f features: remove example feature 2024-03-28 19:12:07 +01:00
Michael Eischer a26d6ffa72 archiver: move deviceID handling behind feature flag 2024-03-28 19:12:07 +01:00
Michael Eischer 2ba21fe72b archiver: only store deviceID for hardlinks 
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
 2024-03-28 19:12:07 +01:00
Michael Eischer 8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index 
Deprecated legacy index format
 2024-03-28 18:24:47 +01:00
Adam Eijdenberg 6e775d3787 Enhancement: option to send HTTP over unix socket 
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
 2024-03-28 17:41:41 +01:00
avoidalone ac948fccda fix some typos 
Signed-off-by: avoidalone <wuguangdong@outlook.com>
 2024-03-11 14:35:12 +08:00
Michael Eischer f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer 1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Michael Eischer a9b64cd7ad features: print warning for stable/depreacted feature flags 2024-03-09 17:29:52 +01:00
Michael Eischer 70839155f2 features: add tests 2024-03-09 17:29:52 +01:00
Michael Eischer 5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer e71660cd1e backup: rename data_added_in_repo statistic to data_added_packed 2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia 062d408987
Clean up SecurityDescriptor helper 2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia 5764300022
Add changelog and fix lint error 2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia 90916f53de
Add test cases for security descriptors 2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia 70cf8e3788
Add support for backup/restore of security descriptors 2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia e3e59fef24
Fix CombineErrors and fillExtendedAttr error handling 2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia 09ce1b4e58
Create helper for SecurityDescriptor related functions 2024-02-24 13:16:25 -07:00
Michael Eischer a8f5684f68 archiver: test statistics in snapshot 2024-02-23 22:05:15 +01:00
Michael Eischer 681395955e archiver: test backup summary calculation 2024-02-23 21:46:39 +01:00
Michael Eischer 38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer 86897314d5 backup: expose data_added_in_repo in JSON output 
The value describes how much data was added after compression.
Previously, it was only available in the text output.
 2024-02-23 20:27:13 +01:00
Michael Eischer a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Michael Eischer 8b1a85711f archiver: unexport save/saveDir/saveTree methods 2024-02-23 20:24:21 +01:00
Aneesh Nireshwalia 4bbd25a37f
Add tests for generic attribute changes 2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia d4be734c73
Handle readonly empty files in windows 2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia eeb1aa5388
Add ability to report warnings to terminal 
Report warnings to terminal when unrecognized generic attributes are found in the repository.
 2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia 0962917974
Support windows metadata using generic attribs 
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
 2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia 62a8a599f1
Add optional messages for Equals helper 2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia 94de87d4b7
Add CombineErrors helper function 2024-02-22 16:57:00 -07:00
Michael Eischer 4c3218ef9f repository: include packID in StreamPack for decrypt/decompress errors 2024-02-17 19:38:01 +01:00
Michael Eischer 18b0bbbf42 repository: use fmt.Errorf in StreamPacks 2024-02-17 19:37:32 +01:00
Michael Eischer ed4a4f8748 check: exclude inaccessible files from the repair pack suggestion 2024-02-12 20:25:15 +01:00
Michael Eischer 4073299a7c check: fix missing error if blob is invalid 2024-02-12 20:20:13 +01:00
Michael Eischer 22a3cea1b3 checker: wrap all pack errors in ErrPackData 2024-02-12 20:19:32 +01:00
Michael Eischer 5b5d506472 backup: report files whose chunks failed to upload 2024-02-11 22:43:26 +01:00
Alexander Neumann c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Michael Eischer 5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer 86b38a0b17 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 17:01:05 +01:00
Michael Eischer c32e5e2abb pack: verify integrity of pack file header 2024-02-04 15:31:42 +01:00
Michael Eischer c97a271e89 repository: ask users to report corrupted data while saving blobs 2024-02-04 15:31:42 +01:00
Michael Eischer 193140525c repository: test verification of blobs/unpacked data 2024-02-04 15:31:42 +01:00
Michael Eischer 2dbb18128c repository: Allow skipping verification for tests 
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
 2024-02-03 18:22:47 +01:00
Michael Eischer 30a84e9003 backup: verify unpacked files before upload 2024-02-03 18:22:47 +01:00
Michael Eischer c01a0c6da7 backup: verify blobs before upload 
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
 2024-02-03 18:22:47 +01:00
Michael Eischer 16e3f79e8b repository: make repo.Options configurable for test repos 2024-02-03 18:22:47 +01:00
Michael Eischer bb92b487f7 repository: fix repack test 2024-02-03 18:22:47 +01:00
Michael Eischer e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu 
Ls ncdu
 2024-01-27 12:27:35 +00:00
Michael Eischer d4ed7c8858 walker: add tests for leaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer 2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer 9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Michael Eischer 246559e654 check: cleanup s3 legacy detection 2024-01-27 13:02:04 +01:00
Michael Eischer 1dfd854769 lock: switch to repository.List 2024-01-27 13:02:04 +01:00
Michael Eischer bfb56b78e1 replace some usages of restic.Repository with more specific interface 
This should eventually make it easier to test the code.
 2024-01-27 13:02:02 +01:00
Michael Eischer 3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs 
Refactor and test `repair packs`
 2024-01-27 13:00:51 +01:00
Michael Eischer 724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey 
repository: Introduce RemoveKey function
 2024-01-27 13:00:42 +01:00
Michael Eischer f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer 42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer 764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer 7c351bc53c repair pack: reenable auto index updates 
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
 2024-01-27 12:51:45 +01:00
Michael Eischer feeab84204 repair pack: extract the repair logic into the repository package 
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
 2024-01-27 12:51:45 +01:00
Michael Eischer cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00