MirrorSave
/
restic
mirror of https://github.com/restic/restic.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,937 Commits 14 Branches 49 Tags 90 MiB
Commit Graph

415 Commits

Author SHA1 Message Date
Jan Niehusmann 9267c25aa0 Omit "archived as %v" messages in quiet mode. 2016-12-03 10:28:49 +01:00
JP Mens 5996d671a0 Small typo in dry-run of remove snapshot 2016-12-02 17:33:05 +01:00
Alexander Neumann e5617b5fd1 Merge pull request #675 from restic/parent-check-hostname 
Use the hostname filter to find a parent snasphot
 2016-11-19 12:42:40 +01:00
Alexander Neumann 2828003d60 Test that existing files and dirs are restored 2016-11-15 21:41:41 +01:00
Alexander Neumann 16cef3b4c6 Use the hostname filter to find a parent snasphot 
Closes #674
 2016-11-15 21:04:51 +01:00
Alexander Neumann 699f39e3cf FindLatestSnapshot: Rename parameter to clarify meaning 2016-11-15 21:03:54 +01:00
Seb Patane 33b6a7381b Don't consider a pre-existing directory in the restore path to be a failure 
* When a directory already exists, CreateDirAt returns an error stating so
  * This means that the restoreMetadata step is skipped, so for directories which already exist no file permissions, owners, groups, etc will be restored on them
* Not returning the error if it's a "directory exists" error means the metadata will get restored
  * It also removes the superfluous "error for ...: mkdir ...: file exists" messages
* This makes the behaviour of directories consistent with that of files (which always have their content & metadata restored, regardless of whether they existed or not)
 2016-11-14 17:53:09 +10:00
Alexander Thaller b7b03dbd4a Added new flag to backup subcommand that reads the files to backup from a file 2016-11-12 15:45:32 +01:00
Alexander Neumann b56bde3f61 Remove fadvise 
This commit removes the use of FADV_DONTNEED, which also purges active
cached pages for other processes.
 2016-11-10 22:21:22 +01:00
Zlatko Čalušić d8f0e7cbd1 Fix REST backend HTTP keepalive 
This is subtle.  A combination od fast client disk (read: SSD) with lots
of files and fast network connection to restic-server would suddenly
start getting lots of "dial tcp: connect: cannot assign requested
address" errors during backup stage.  Further inspection revealed that
client machine was plagued with TCP sockets in TIME_WAIT state.  When
ephemeral port range was finally exhausted, no more sockets could be
opened, so restic would freak out.

To understand the magnitude of this problem, with ~18k ports and default
timeout of 60 seconds, it means more than 300 HTTP connections per
seconds were created and teared down.  Yeah, restic-server is that
fast. :)

As it turns out, this behavior was product of 2 subtle issues:

1) The body of HTTP response wasn't read completely with io.ReadFull()
   at the end of the Load() function.  This deactivated HTTP keepalive,
   so already open connections were not reused, but closed instead, and
   new ones opened for every new request.  io.Copy(ioutil.Discard,
   resp.Body) before resp.Body.Close() remedies this.

2) Even with the above fix, somehow having MaxIdleConnsPerHost at its
   default value of 2 wasn't enough to stop reconnecting.  It is hard to
   understand why this would be so detrimental, it could even be some
   subtle Go runtime bug.  Anyhow, setting this value to match the
   connection limit, as set by connLimit global variable, finally nails
   this ugly bug.

I fixed several other places where the response body wasn't read in
full (or at all).  For example, json.NewDecoder() is also known not to
read the whole body of response.

Unfortunately, this is not over yet. :( The check command is firing up
to 40 simultaneous connections to the restic-server.  Then, once again,
MaxIdleConnsPerHost is too low to support keepalive, and sockets in the
TIME_WAIT state pile up.  But, as this kind of concurrency absolutely
kill the poor disk on the server side, this is a completely different
bug then.
 2016-11-10 09:32:07 +01:00
Alexander Neumann 51322a1055 selectFunc: handle nil 2016-11-05 12:38:33 +01:00
Alexander Neumann c5bc802ff0 fs.DeviceID(): Return errors when fi is nil 2016-11-05 12:38:17 +01:00
Justin Clift ecc1f92787 Remove redundant check of error var e 
As per #649
 2016-10-25 18:10:53 +01:00
Alexander Neumann 1dd72693f9 forget: Remove unneeded index loading 2016-10-24 14:01:23 +02:00
Alexander Neumann fe1013e779 cmds/ls: Format timestamp 2016-10-19 22:11:37 +02:00
David 5f8a6cea6f don't print status info if running in the background 
clean

fix OS issues & format code

fix issues
 2016-10-15 18:12:19 +00:00
Alexander Neumann 250b36eeb1 Improve error message for 'forget' 
$ bin/restic forget /d 7 /w 4 /m 12
    argument "/d" is not a snapshot ID, ignoring
    argument "7" is not a snapshot ID, ignoring
    argument "/w" is not a snapshot ID, ignoring
    argument "4" is not a snapshot ID, ignoring
    argument "/m" is not a snapshot ID, ignoring
    cound not find a snapshot for ID "12", ignoring
 2016-10-10 20:55:02 +02:00
Daniel Örn ba8d960c8f using backtics instead of doublequotes 2016-10-05 08:26:32 +02:00
Daniel Örn 84421a7c68 structured file with gofmt 2016-10-05 07:30:46 +02:00
Daniel Örn 5c7325f44a Added long paths fix for samba network shares 2016-10-05 07:09:56 +02:00
Christian Kemper a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann 22f5fc5739 Improve help text for slice options 2016-09-29 20:39:55 +02:00
Alexander Neumann e994cacbfe Fix short-hand option clash 2016-09-29 20:37:45 +02:00
Alexander Neumann feed54caef Remove timing, simplify function matching 2016-09-28 20:10:40 +02:00
Alexander Neumann 4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann 2ae06a7a01 Rework debug log function 2016-09-28 19:56:03 +02:00
Alexander Neumann 25945718a1 Fix recursive call to debug.Log 2016-09-28 19:56:03 +02:00
Alexander Neumann 3601c39177 Add comments 2016-09-27 20:22:01 +02:00
Alexander Neumann 565d72ef36 Use cobra for all commands 2016-09-27 19:53:03 +02:00
Alexander Neumann a257a613d7 Fix debug log 2016-09-27 19:52:48 +02:00
Alexander Neumann 0a752b9fab test helpers: Always print stack trace 2016-09-27 19:50:26 +02:00
Alexander Neumann 04d6b5da2f Remove more unused bits 2016-09-21 20:45:18 +02:00
Alexander Neumann 1dfd3b8aa3 Remove unused bits and pieces 
Reported by https://github.com/dominikh/go-unused
 2016-09-21 20:22:32 +02:00
Alexander Neumann 0a9cbd47c7 Merge pull request #626 from rfjakob/master 
Add "-x", "--one-file-system" option
 2016-09-18 20:03:58 +02:00
Jakob Unterwurzacher 53701891a1 Add "-x", "--one-file-system" option 
Equivalent to rsync's "-x" option.

Notes to the naming:

"--exclude-other-filesystems"
is used by Duplicity,

"--one-file-system"
is used rsync and tar.

This latter should be more familiar to the user.
 2016-09-18 18:52:30 +02:00
Alexander Neumann 68b462d057 fuse: Add test for same timestamps 2016-09-18 18:30:25 +02:00
Alexander Neumann 649f789190 fuse: Fix test for timestamps with same second 2016-09-18 18:13:39 +02:00
Alexander Neumann 5494c1858e fuse: correctly handle snapshots 
The fuse code kept adding snapshots to the top-level dir "snapshots". In
addition, snapshots with the same timestamp (same second) were not added
correctly, they will now be suffixed by an incrementing counter, e.g.:

    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:44+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00-1

Closes #624
 2016-09-18 15:04:39 +02:00
Alexander Neumann b090c73bd4 Remove wrapper functions in errors package 
This way, our own errors package does not appear in the stack traces.
 2016-09-18 13:28:59 +02:00
Alexander Neumann 2b9a408ccc Return a fatal for location.Parse 2016-09-18 13:28:41 +02:00
Alexander Neumann 83c35bd6b5 Do not print stack trace when open repo failed 
Closes #622
 2016-09-18 13:24:46 +02:00
Alexander Neumann a9af896ddd Add verbose error when marshalling a node fails 
This code is introduced to watch for issue #529, in which two users
describe that restic failed encoding a time in a node to JSON with the
error message:

    panic: json: error calling MarshalJSON for type *restic.Node: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]

The error message now is:

    panic: Marshal: json: error calling MarshalJSON for type *restic.Node: node /home/fd0/shared/work/restic/restic/.git/hooks/applypatch-msg.sample has invalid ModTime year -1: -0001-01-02 03:04:05.000000006 +0053 LMT
 2016-09-17 10:43:04 +02:00
Alexander Neumann 64fe9ec048 Update github.com/jessevdk/go-flags 2016-09-15 22:29:49 +02:00
Alexander Neumann d4a2d70089 Retry umount for integration tests 2016-09-15 21:37:50 +02:00
Alexander Neumann 4ffca0f4b4 Improve integration tests for fuse 2016-09-15 21:17:20 +02:00
Alexander Neumann a0f3e94655 fuse: handle duplicate timestamps for snapshots 
This closes #606, which fails because several snapshots are created with
exactly the same timestamp, and the code checks that for each snapshot
there is a dir in the fuse mount. This fails for colliding timestamps,
so we now add a suffix "-1", "-2" etc for each duplicate timestamp.
 2016-09-15 21:15:49 +02:00
Alexander Neumann 6485a6cdc0 Simplify mount logic 2016-09-15 19:59:07 +02:00
Alexander Neumann 3975d76f23 Correct filenames for expire policy tests 2016-09-13 21:20:41 +02:00
Alexander Neumann bf6602bc1b Update golden file 2016-09-13 21:19:57 +02:00
Alexander Neumann 828267aaa3 Fix status for stdin archiver 2016-09-13 21:01:29 +02:00
Alexander Neumann a77c615909 Fix 'forget' command with tags 2016-09-13 20:56:18 +02:00
Alexander Neumann cfdf4c92f7 Add `--keep-tag` to `forget` command 2016-09-13 20:37:11 +02:00
Alexander Neumann 0f9fb37c78 Add tags to forget command 2016-09-13 20:20:55 +02:00
Alexander Neumann 673bce936e Add tags to 'backup' and 'snapshots' commands 2016-09-13 20:20:52 +02:00
Alexander Neumann 1f83635267 Add tags to snapshots and filter 2016-09-13 20:12:55 +02:00
Alexander Neumann ceb4a3ecc0 Merge pull request #613 from restic/read-password-from-file 
Read password from file
 2016-09-12 20:37:08 +02:00
Alexander Neumann 223dc78acb Improve statistics for `prune` 
Sample:

    counting files in repo
    building new index for repo
    [0:00] 100.00%  22 / 22 packs
    repository contains 22 packs (1377 blobs) with 90.610 MiB bytes
    processed 1377 blobs: 0 duplicate blobs, 0B duplicate
    load all snapshots
    find data that is still in use for 1 snapshots
    [0:00] 100.00%  1 / 1 snapshots
    found 409 of 1377 data blobs still in use, removing 968 blobs
    will delete 10 packs and rewrite 10 packs, this frees 64.232 MiB
    creating new index
    [0:00] 100.00%  7 / 7 packs
    saved new index as df467c6e
    done

Closes #581
 2016-09-12 14:26:47 +02:00
Alexander Neumann 65afeba19a Add option to read the password from a file 2016-09-12 14:09:22 +02:00
Alexander Neumann e443454c4b Add OS and Arch to 'version' output 2016-09-04 15:46:50 +02:00
Alexander Neumann b628bcee27 Remove redundant ParseID 2016-09-04 14:38:18 +02:00
Alexander Neumann dfc0cbf3a8 Use one test password 2016-09-04 14:30:14 +02:00
Alexander Neumann 512a92895f Rename WithTestEnvironment -> Env 2016-09-04 14:29:04 +02:00
Alexander Neumann 6ab425f130 Remove SetupRepo 2016-09-04 13:24:51 +02:00
Alexander Neumann f5b9ee53a3 Fix mock.Repository 2016-09-04 13:18:25 +02:00
Alexander Neumann ea073f58cf Correct comment 2016-09-04 13:08:09 +02:00
Alexander Neumann bef5c4acb8 Add mock.Repository, Rework SetupRepo 2016-09-04 12:52:43 +02:00
Alexander Neumann b5b3c0eaf8 Add repository.SaveTree 2016-09-03 21:10:25 +02:00
Alexander Neumann 1fb80bf0e2 Fix fuse mount 2016-09-03 21:10:25 +02:00
Alexander Neumann 436332d5f2 LoadDataBlob -> LoadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann fe8c12c798 Replace repolitoy.SaveAndEncrypt to SaveBlob() 2016-09-03 21:10:25 +02:00
Alexander Neumann 1cc59010f5 Remove LoadJSONPack, un-export loadBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann 878c1cd936 Add more comments 2016-09-03 21:10:25 +02:00
Alexander Neumann 5170c4898a Address hound comments 2016-09-03 21:10:25 +02:00
Alexander Neumann 2054e3c026 Fix tests 2016-09-03 21:10:25 +02:00
Alexander Neumann ffbe05af9b Rework crypto, use restic.Repository everywhere 2016-09-03 21:10:25 +02:00
Alexander Neumann 84f95a09d7 Introduce LoadTreeBlob and LoadDataBlob 2016-09-03 21:10:25 +02:00
Alexander Neumann 573410afab Fix archiver test 2016-09-03 21:10:25 +02:00
Alexander Neumann 619939ccd9 Reorder methods in interface Repository 2016-09-03 21:10:25 +02:00
Alexander Neumann 714a5d1dc4 Move tree walker to restic/walk 2016-09-03 21:10:25 +02:00
Alexander Neumann bc42dbdf87 Create package restic/errors 2016-09-03 21:10:24 +02:00
Alexander Neumann 765b5437bd Fix command 'dump' 2016-09-03 21:10:24 +02:00
Alexander Neumann 5d7b38cabf Remove sentinel errors 2016-09-03 21:10:24 +02:00
Alexander Neumann debf1fce54 Remove IDSize, TestRandomID -> NewRandomID 2016-09-03 21:10:24 +02:00
Alexander Neumann 0045f2fb61 Remove functions 2016-09-03 21:10:24 +02:00
Alexander Neumann 5764b55aee Rename Node.FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann 5e3a41dbd2 Rename struct member FileType -> Type 2016-09-03 21:10:24 +02:00
Alexander Neumann 88d0f24ce7 Reduce lock timeout to zero 2016-09-03 21:10:24 +02:00
Alexander Neumann eb6e3ba8b3 Fix imported package 2016-09-03 21:10:24 +02:00
Alexander Neumann 528c301891 Last fixes for integration tests 2016-09-03 21:10:24 +02:00
Alexander Neumann f7ae0cb78f Fix cmds/restic for new structure 2016-09-03 21:10:24 +02:00
Alexander Neumann 3695ba5882 Tests pass for restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann 4c95d2cfdc wip 2016-09-03 21:10:24 +02:00
Alexander Neumann cc6a8b6e15 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 51d8e6aa28 wip 2016-09-03 21:10:24 +02:00
Alexander Neumann f0600c1d5f wip 2016-09-03 21:10:24 +02:00
Alexander Neumann 90da66261a Copy ID from backend to restic 2016-09-03 21:10:24 +02:00
Alexander Neumann 82c2dafb23 Copy interfaces and basic types to package restic/ 2016-09-03 21:10:24 +02:00
Alexander Neumann bfdd26c541 Remove (unused) cache implementation 2016-09-03 21:10:24 +02:00
Alexander Neumann e699f6d1bd Update doc comment 2016-09-03 21:10:24 +02:00
Alexander Neumann 1aa52e5e1e Replace lowlevel syscall to restore symlink times 2016-08-30 21:45:16 +02:00