Commit Graph

341 Commits

Author SHA1 Message Date
Nick Craig-Wood f3c5745468 Add srcRemote and dstRemote parameters to DirMove #954 2017-02-16 17:42:37 +00:00
Nick Craig-Wood dac4bb22d3 mount: Make include and exclude filters apply to mount - fixes #1060 2017-02-15 23:28:53 +00:00
Nick Craig-Wood b52c80e85c sync: don't update mod times if --dry-run set - fixes #1100 2017-02-15 23:09:44 +00:00
Nick Craig-Wood f15c6b68b6 Re-add the async buffer on seek - fixes #1137 2017-02-15 22:54:21 +00:00
Nick Craig-Wood 3f778d70f7 Add sync.Pool to async reader 2017-02-15 22:37:58 +00:00
Nick Craig-Wood 7fa687b3e1 fs: Async buffer: use ReadFill to fill the chunks and increase to 1MB 2017-02-14 22:36:37 +00:00
Nick Craig-Wood 493da54113 Add --buffer-size parameter to control buffer size for copy 2017-02-14 22:36:37 +00:00
Nick Craig-Wood 541929258b check: Add --download flag to check all the data, not just hashes 2017-02-13 10:48:26 +00:00
Nick Craig-Wood 7047c67a5e sync: Fix log message containing <nil> 2017-02-13 09:23:21 +00:00
Nick Craig-Wood 01c747e7db Add cryptcheck command to check integrity of crypt remotes #1102 2017-02-12 16:30:18 +00:00
Nick Craig-Wood 666dae4229 Add --syslog flag to optionally log to syslog on capable platforms 2017-02-11 20:27:46 +00:00
Nick Craig-Wood ac1c041377 Redo log level flags
* -vv or --log-level DEBUG
  * -v or --log-level INFO
  * --log-level NOTICE (default)
  * -q --log-level ERROR

Replace Config.Verbose and Config.Quiet with Config.LogLevel

Fixes #739 Fixes #1108 Fixes #1000
2017-02-11 20:22:42 +00:00
Nick Craig-Wood 0366ea39c5 Reassign some logging levels 2017-02-11 17:56:05 +00:00
Nick Craig-Wood 80f53176d9 Rename log functions and factor into own file 2017-02-11 17:54:50 +00:00
Nick Craig-Wood 50e190ff54 cat: don't allocate buffers if not needed to reduce memory usage 2017-02-09 11:46:53 +00:00
Nick Craig-Wood dd20a297d6 cat: Fix go routine leak 2017-02-09 11:25:36 +00:00
Nick Craig-Wood c0ad29c06c Clarify logging and docs for --no-traverse incompatibilities - fixes #1059 2017-02-08 22:35:12 +00:00
Nick Craig-Wood d091d4a8bb rclone cat: add --head, --tail, --offset, --count and --discard
Fixes #819
2017-02-08 08:09:41 +00:00
Nick Craig-Wood 48cdedc97b Re-implement sync routine to work a directory at a time
Multiple directories (up to --checkers worth) are scanned at once.

This uses much less memory than the previous scheme - only the amount
of memory needed to hold an entire directory listing of objects.

For directory based remotes the speed is unchanged.

For bucket based remotes, instead of doing one API call to list the
whole bucket, it does multiple calls, one for each pseudo directory.
However these are done in parallel so in practice this seems to speed
up directory listings.

This replaces the existing sync method as it performs faster and uses
less memory.

The old sync method is available with the temporary --old-sync-method
flag.

Fixes #517
Fixes #439
Fixes #236
Fixes #1067
2017-02-04 10:30:25 +00:00
Nick Craig-Wood 7c6cd3a9e1 Make --delete-after the default and refactor --delete-{before,during,after} parsing 2017-02-04 10:30:25 +00:00
Nick Craig-Wood bcdd73369f Ignore --delete-before with --track-renames 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 86bec20b56 sync: factor accumulating the rename checks 2017-02-04 10:30:25 +00:00
Nick Craig-Wood c3b2b89473 Add ListDirSorted function to list a directory
* fix error return of readFilesFn also
2017-02-04 10:30:25 +00:00
Nick Craig-Wood 85f05c57d1 Clean empty directories between test runs 2017-02-04 10:30:25 +00:00
Nick Craig-Wood 726cb43be9 Complete SFTP remote #521
* Add unit tests
  * Fix up remote so it passes tests
  * Add docs
2017-02-04 10:29:46 +00:00
Nick Craig-Wood 288302c2cf Make fallback purge delete empty directories too.
This was implemented to make the SFTP unit tests pass.
2017-02-04 10:29:46 +00:00
Jack Schmidt b9a8315696 Basic SFTP support, Issue #521 2017-02-04 10:29:18 +00:00
Nick Craig-Wood 9d331ce04b Implement --ignore-checksum flag
Fixes #793 Fixes #863 Fixes #981
2017-02-03 08:11:10 +00:00
Nick Craig-Wood 2192805360 rclone config: when choosing from a list, allow the value to be entered 2017-01-29 15:51:26 +00:00
Nick Craig-Wood db0b93c0ad rclone config: allow rename and copy of remotes - fixes #641 2017-01-29 15:37:44 +00:00
Dario Giovannetti 9d36258923 Comply with XDG Base Directory specification
Fixes #868
2017-01-26 10:22:08 +00:00
Nick Craig-Wood b6848a3edb Fix race in Lister.Finished which was causing the tests to be unreliable 2017-01-19 20:11:17 +00:00
Nick Craig-Wood e2bf9eb8e9 Implement --suffix for use with --backup-dir only #98
This also makes sure we remove files we are about to override in the
--backup-dir properly.
2017-01-19 20:11:17 +00:00
Nick Craig-Wood 1fa258c2b4 Define a new Features() method for Fs
Optional interfaces are becoming more important in rclone,
--track-renames and --backup-dir both rely on them.

Up to this point rclone has used interface upgrades to define optional
behaviour on Fs objects.  However when one Fs object wraps another it
is very difficult for this scheme to work accurately.  rclone has
relied on specific error messages being returned when the interface
isn't supported - this is unsatisfactory because it means you have to
call the interface to see whether it is supported.

This change enables accurate detection of optional interfaces by use
of a Features struct as returned by an obligatory Fs.Features()
method.  The Features struct contains flags and function pointers
which can be tested against nil to see whether they can be used.

As a result crypt and hubic can accurately reflect the capabilities of
the underlying Fs they are wrapping.
2017-01-16 17:33:25 +00:00
ncw 3745c526f1 Implement --backup-dir - fixes #98
The parameter of backup-dir specifies a remote that all deleted or
overwritten files will be copied to.
2017-01-16 17:33:25 +00:00
Nick Craig-Wood c123c702ab Fix fs.Overlapping and factor fs.SameConfig 2017-01-14 09:55:53 +00:00
ncw 4aae7bcca6 Factor server side move detection 2017-01-14 09:55:53 +00:00
Brandur 3b1e0b66bb Return error on not found from ListFn
This changes `ListFn`'s implementation so that if it encounters a not
found error, instead of sending a fatal error to log, it coordinates the
return of the error between checker goroutines and sends it back to the
caller.

The main impetus here is that it allows an external program compiling
against rclone as a package to handle a not found, where it currently it
cannot.

This does change error output on a not found a little bit, we go from
this:

    2017/01/09 21:14:03 directory not found

To this:

    2017/01/09 21:13:44 Failed to ls: directory not found
2017-01-12 10:07:59 +00:00
Nick Craig-Wood 78c1f2839e Fix filters to add ** rules to directory rules
This fixes `--exclude ".*{,/**}"` to exclude all . files and
. directories.
2017-01-05 19:33:49 +00:00
Nick Craig-Wood de2d967abd Stop --track-renames hashing matching files - fixes #984
Also only hash files of the correct size.

This speeds it up a lot.
2017-01-05 17:58:01 +00:00
Marco Paganini 6611d92e21 Only start bandwidth ticker when necessary.
- Only start the token ticker when the timetable entry has more than one
  entry.
- This fixes the "Scheduled bandwidth change" log message when no
  bwlimit is specified.
- Fixes #987
2017-01-04 19:03:49 -08:00
Nick Craig-Wood f73ee5eade Make all config file variables be settable in the environment
These are set in the form RCLONE_CONFIG_remote_option where remote is
the uppercased remote name and option is the uppercased config file
option name.  Note that RCLONE_CONFIG_remote_TYPE must be set if
defining a new remote.

Fixes #616
2017-01-03 22:42:47 +00:00
Nick Craig-Wood 0d75d2585f Allow all options to be set from environment variables
The option names are munged changing - to _ making upper case and
prepending RCLONE_.  The values are as parsed by pflag.
2017-01-03 22:42:47 +00:00
Marco Paganini 3b0f944e23 Add time-based bandwidth limits.
- Change the --bwlimit command line parameter to accept both a limit (as
  before) or a full timetable (formatted as "hh:mm,limit
  hh:mm,limit...")
- The timetable is checked once a minute by a ticker function. A new
  tokenBucket is created every time a bandwidth change is necessary.
- This change is compatible with the SIGUSR2 change to toggle bandwidth
  limits.

This resolves #221.
2017-01-03 21:00:38 +00:00
Nick Craig-Wood f1221b510b Change --track-renames to use the length,hash pair stored in a map
This makes it much faster in the case of many files and use less
memory.

This also detects use of --no-traverse and disables it.
2017-01-03 20:37:06 +00:00
Nick Craig-Wood 274ab349f4 sync: Only allow --track-renames if have a common hash 2017-01-03 20:35:05 +00:00
Bjørn Erik Pedersen 47d3a450a4 sync: Track and perform server-side renames
This commits adds support for tracking of file renames if `track-renames` flag is set,
and it then performs server-side renames for remotes that support it, i.e.
remotes that implement either the `Mover` or the `Copier` interface.

Fixes #888
2017-01-03 20:35:05 +00:00
Nick Craig-Wood 5b8b379feb Version v1.35 2017-01-02 15:33:06 +00:00
Nick Craig-Wood 4dd5428b13 Fix rmdirs test and integration tests which depend on each other 2017-01-02 14:15:07 +00:00
Nick Craig-Wood 64ec220d5d Fix --no-update-modtime test on remotes which don't support hashes 2016-12-31 15:19:26 +00:00