MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,743 Commits 180 Branches 116 Tags 325 MiB
Commit Graph

707 Commits

Author SHA1 Message Date
Nick Craig-Wood 9f671c5dd0 fs: fix tests for *SepList 2018-08-21 10:58:59 +01:00
Alex Chen c6c74cb869 mountlib: fix mount --daemon not working with encrypted config - fixes #2473 
This passes the configKey to the child process as an Obscured temporary file with an environment variable to the
 2018-08-21 09:41:16 +01:00
Nick Craig-Wood 1b8f4b616c fs: move CommaSepList and SpaceSepList here from config 
fs can't import config so having them there means they are not usable
by rclone core.
 2018-08-20 17:52:05 +01:00
Fabian Möller f818df52b8 config: add List type 2018-08-20 17:38:51 +01:00
Nick Craig-Wood 7712a0e111 fs/asyncreader: skip some tests to work around race detector bug 
The race detector currently detects a race with len(chan) against
close(chan).

See: https://github.com/golang/go/issues/27070

Skip the tests which trip this bug under the race detector.
 2018-08-20 12:34:29 +01:00
Fabian Möller 6dc5aa7454 docs: clearify buffer-size is per transfer/filehandle 2018-08-17 18:11:40 +01:00
Fabian Möller 552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Nick Craig-Wood 6390dec7db fs/accounting: add --stats-one-line flag for single line stats 2018-08-17 17:58:00 +01:00
Nick Craig-Wood 80a3db34a8 fs/accounting: show the total progress of the sync in the stats #379 2018-08-17 17:58:00 +01:00
Nick Craig-Wood cb7a461287 sync: add a buffer for checks, uploads and renames #379 
--max-backlog controls the queue length.

Add statistics for the check/upload/rename queues.

This means that checking can complete before the uploads which will
give rclone the ability to show exactly what is outstanding.
 2018-08-17 17:58:00 +01:00
Andres Alvarez 990919f268 Add disclaimer about generated passwords being stored in an obscured format 2018-08-11 15:07:50 +01:00
Sebastian Bünger dd3e912731 fs/OpenOptions: Make FixRangeOption clamp range to filesize. 2018-08-10 11:14:34 +01:00
Nick Craig-Wood e979cd62c1 rc: fix formatting in docs 2018-08-07 21:05:21 +01:00
reddi1 6bfdabab6d rc: added core/stats to return the stats - fixes #2405 2018-08-07 20:56:40 +01:00
Andres Alvarez c1f5add049 Add tests for reveal functions 2018-08-07 20:51:50 +01:00
Oleg Kovalov 06c9f76cd2 all: fix go-critic linter suggestions 2018-08-06 21:14:03 +01:00
Fabian Möller 9815b09d90 fs: add multipliers for SizeSuffix 2018-07-29 18:17:05 +01:00
Fabian Möller 9c90b5e77c stats: use appropriate Lock func's 2018-07-22 11:33:19 +02:00
Nick Craig-Wood ab78eb13e4 sync: correct help for --delete-during and --delete-after 2018-07-18 19:30:14 +01:00
ishuah dcc74fa404 move: fix delete-empty-src-dirs flag to delete all empty dirs on move - fixes #2372 2018-07-17 10:34:34 +01:00
Nick Craig-Wood 4d7d240c12 config: Add advanced section to the config editor 2018-07-16 21:20:47 +01:00
Nick Craig-Wood d046402d80 config: Make sure Required values are entered 2018-07-16 21:20:47 +01:00
Nick Craig-Wood 9bdf465c10 config: make config wizard understand types and defaults 2018-07-16 21:20:47 +01:00
Nick Craig-Wood f3f48d7d49 Implement new backend config system 
This unifies the 3 methods of reading config

  * command line
  * environment variable
  * config file

And allows them all to be configured in all places.  This is done by
making the []fs.Option in the backend registration be the master
source of what the backend options are.

The backend changes are:

  * Use the new configmap.Mapper parameter
  * Use configstruct to parse it into an Options struct
  * Add all config to []fs.Option including defaults and help
  * Remove all uses of pflag
  * Remove all uses of config.FileGet
 2018-07-16 21:20:47 +01:00
Nick Craig-Wood 3c89406886 config: Make fs.ConfigFileGet return an exists flag 2018-07-16 08:50:52 +01:00
Nick Craig-Wood 85d09729f2 fs: factor OptionToEnv and ConfigToEnv into fs 2018-07-16 08:50:52 +01:00
Nick Craig-Wood b3bd2d1c9e config: add configstruct parser to parse maps into config structures 2018-07-16 08:50:52 +01:00
Nick Craig-Wood 4c586a9264 config: add configmap package to manage config in a generic way 2018-07-16 08:50:52 +01:00
Nick Craig-Wood 1c80e84f8a fs: Implement Scan method for SizeSuffix and Duration 2018-07-16 08:50:52 +01:00
Nick Craig-Wood dbb4b2c900 fs/config: don't print errors about --config if supplied - fixes #2397 
Before this change if the rclone was running in an environment which
couldn't find the HOME directory, it would print a warning about
supplying a --config flag even if the user had done so.
 2018-07-15 12:39:11 +01:00
Nick Craig-Wood a3d9a38f51 fs/fserrors: make sure Cause never returns nil 2018-07-13 10:31:40 +01:00
Nick Craig-Wood 144c1a04d4 fs: Fix parsing of paths under Windows - fixes #2353 
Before this copyto would parse windows paths incorrectly.

This change moves the parsing code into fspath and makes sure
fspath.Split calls fspath.Parse which does the parsing correctly for

This also renames fspath.RemoteParse to fspath.Parse for consistency
 2018-07-06 23:16:43 +01:00
Nick Craig-Wood 8c0335a176 build: fix for goimports format change 
See https://github.com/golang/go/issues/23709
 2018-07-03 22:33:15 +01:00
Nick Craig-Wood 947e10eb2b config: fix error reading password from piped input - fixes #1308 2018-06-28 11:54:15 +01:00
Nick Craig-Wood 935533e57f filter: raise --include and --exclude warning to ERROR so it appears without -v 2018-06-22 22:18:55 +01:00
Mateusz 56e1e82005 fs: added weekday schedule into --bwlimit - fixes #1822 2018-06-17 18:38:09 +01:00
Nick Craig-Wood 7110349547 Start v1.42-DEV development 2018-06-16 21:25:58 +01:00
Nick Craig-Wood a9adb43896 Version v1.42 2018-06-16 18:21:09 +01:00
Nick Craig-Wood 1c75581959 sync: fix TestCopyRedownload after ModifyWindow changes #2310 2018-06-10 17:34:00 +01:00
Filip Bartodziej ce109ed9c0 log: password prompt output fixed for unix - partially fixes #2220 2018-06-10 12:57:45 +01:00
Nick Craig-Wood ba43acb6aa sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310 2018-06-04 21:41:25 +01:00
ishuah d758e1908e copy: create (pseudo copy) empty source directories to destination - fixes #1837 2018-06-04 11:01:14 +01:00
ishuah 737aed8412 Ensure items in srcEmptyDirs are actually empty 2018-06-04 11:01:14 +01:00
Stefan 4009fb67c8
fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319, #2328 2018-06-03 20:45:34 +02:00
Nick Craig-Wood 3ef938ebde lsf: add --absolute flag to add a leading / onto path names 2018-06-03 10:42:34 +01:00
Kasper Byrdal Nielsen aad75e6720 check: Add one-way argument 
--one-way argument will check that all files on source matches the files on detination,
but not the other way. For example files present on destination but not on source will not
trigger an error.

Fixes: #1526
 2018-05-31 09:42:16 +01:00
Nick Craig-Wood 3a2248aa5f rc: add core/gc to run a garbage collection on demand 2018-05-24 15:10:28 +01:00
Nick Craig-Wood 573ef4c8ee rc: enable go profiling by default on the --rc port 
This means you can use the pprof tool on a running rclone, eg

    go tool pprof http://localhost:5572/debug/pprof/heap
 2018-05-24 15:10:28 +01:00
Nick Craig-Wood 512f4b4487 Update error checking on fmt.Fprint* after errcheck update 
Now we need to check or ignore errors on fmt.Fprint* explicitly -
previously errcheck just ignored them for us.
 2018-05-22 09:41:13 +01:00
Nick Craig-Wood 8e625e0bc3 config: add ConfirmWithDefault to change the default on AutoConfig 2018-05-16 09:09:41 +01:00
Nick Craig-Wood e56be0dfd8 lsf: Add --csv flag for compliant CSV output 2018-05-13 12:18:21 +01:00
Nick Craig-Wood 2a32e2d838 operations: turn ListFormatted into a Format method on ListFormat 2018-05-13 12:17:55 +01:00
Nick Craig-Wood f77efc7649 lsf: Add 'm' format specifier to show the MimeType 2018-05-13 12:17:55 +01:00
Nick Craig-Wood aadbcce486 fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry 2018-05-13 12:17:55 +01:00
Nick Craig-Wood 909c3a92d6 lsf: implement 'i' format for showing object ID - fixes #1476 2018-05-13 12:17:55 +01:00
Nick Craig-Wood 826975c341 fs: add Optional ID() method to Object and implement it in backends 
ID() shows the internal ID of the Object if available.
 2018-05-13 12:17:55 +01:00
Nick Craig-Wood e80ae4e09c build: remove unused struct fields spotted by structcheck 2018-05-05 17:32:41 +01:00
Nick Craig-Wood 1320e84bc2 build: remove unused code spotted by the deadcode linter 2018-05-05 17:32:41 +01:00
Nick Craig-Wood cb5bd47e61 build: fix errors spotted by ineffassign linter 
These were mostly caused by shadowing err and a good fraction of them
will have caused errors not to be propagated properly.
 2018-05-05 17:32:41 +01:00
Nick Craig-Wood f95835d613 fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors. 
Before this change fatal errors which were wrapped in a system error (eg a
URLError) were not recognised as fatal errors.
 2018-05-05 12:58:28 +01:00
Nick Craig-Wood be79b47a7a sync: log when we abandon the sync due to a fatal error 2018-05-05 12:58:28 +01:00
Nick Craig-Wood be22735609 fs/accounting: fix deadlock on GetBytes 
A deadlock could occur since we have now put a mutex on GetBytes from
StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu)
- GetBytes (s.mu).

Fix this by giving stringSet its own locking and excluding the call
which caused the deadlock from the mutex in StatsInfo.String.
 2018-05-05 12:58:28 +01:00
Nick Craig-Wood 1b1b3c13cd sync: add a test for aborting on max upload 2018-05-05 12:58:28 +01:00
Nick Craig-Wood 5c128272fd Implement --max-transfer flag to quit transferring at a limit #1655 2018-05-05 12:58:28 +01:00
Nick Craig-Wood d178233e74 sync,march: check the cancel context on every channel send and receive 
This fixes a deadlock on sync when all the copying channels receive a
Fatal Error.
 2018-05-05 12:58:28 +01:00
Fabian Möller 9fdf273614 fs: improve ChunkedReader 
- make Close permanent and return errors afterwards
- use RangeSeek from the wrapped reader if present
- add a limit to chunk growth
- correct RangeSeek interface behavior
- add tests
 2018-05-05 12:49:42 +01:00
Nick Craig-Wood 55674c0bfc Start v1.41-DEV development 2018-04-28 12:37:55 +01:00
Nick Craig-Wood e4c380b2a8 Version v1.41 2018-04-28 11:46:27 +01:00
Nick Craig-Wood 74cbdea0ef Revert "copy: create (pseudo copy) empty source directories to destination" 
Unfortunately this commit attempts to create every directory rather
than just the empty ones, so will need re-working.

Removing this feature for the 1.41 release

This reverts commit 0daced29db.
 2018-04-28 10:02:32 +01:00
ishuah 0daced29db copy: create (pseudo copy) empty source directories to destination - fixes #1837 2018-04-27 16:15:32 +01:00
Richard Yang a81ec00a8c dedupe: Add dedupe largest functionality - fixes #2269 2018-04-26 16:21:07 +01:00
Nick Craig-Wood 9f04ce282e rc: fix setting bwlimit to unlimited 2018-04-26 12:21:29 +01:00
Nick Craig-Wood 764440068e filter: fix --min-age and --max-age together check 
Somehow in the code reorganisation of
11da2a6c9b the check for --min-age and
--max-age got switched around.  This commit fixes that and means you
can use --min-age and --max-age together.
 2018-04-26 09:17:22 +01:00
Nick Craig-Wood a703216286 filter: take double negatives out of filter flag help 2018-04-26 09:17:13 +01:00
Nick Craig-Wood 9ab2521ef2 rc: autogenerate and tidy the docs and commands 
* Rename rc/pid -> core/pid
  * Sort the output of `rc list`
  * Make a script to autogenerate the docs
  * Tidy docs
 2018-04-23 20:57:17 +01:00
Nick Craig-Wood 21a10e58c9 rc: implement core/memstats to print internal memory usage info 2018-04-23 20:49:36 +01:00
Nick Craig-Wood 24980d7123 config: fix typo in error message #2268 2018-04-21 22:49:30 +01:00
Nick Craig-Wood bcdb7719c6 fs/hash: install QuickXorHash as a supported rclone hash type #2262 2018-04-20 21:02:57 +01:00
Nick Craig-Wood c51d97c752 hashsum: make generic tool for any hash to produce md5sum like output 2018-04-20 21:02:37 +01:00
Nick Craig-Wood ef3bcec76c fs: Extend SizeSuffix to include TB and PB for rclone about 2018-04-17 21:53:42 +01:00
Nick Craig-Wood 1ac6dacf0f about: complete other providers and re-work internals 
* Implement about for:
    * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox
  * Implement `--json` and `---full` flag for `rclone about`
  * change About interface to return a Usage structure
  * Remove operations.About as it is too thin an interface
  * Implement Integration test

Relates to #1138 and #1564
 2018-04-17 21:53:27 +01:00
a-roussos 94e277d759 about: add new command 'about' to get quota info from a remote 
Implemented for drive only.

Relates to #1138 and #1564.
 2018-04-17 21:50:14 +01:00
Nick Craig-Wood 8c3740c2c5 config: Improve the Provider matching to have a negated match #2140 
This makes it easier to make classes of provider in the config.
 2018-04-13 16:06:37 +01:00
Giri Badanahatti acd5d4377e config,s3: hierarchical configuration support #2140 
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
 2018-04-13 16:05:35 +01:00
Peter Baumgartner 1db68571fd s3,swift: Add --use-server-modtime 
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
 2018-04-13 13:32:17 +01:00
Nick Craig-Wood d97fe3b824 fs/operations: make dedupe work with mega 
* factor into its own files
  * remove assumptions about having a given hash type
  * make tests work if the remote has no hash
 2018-04-13 13:23:55 +01:00
Nick Craig-Wood dcf8334673 fs: add --dump goroutines and --dump openfiles 
These are developer flags useful for tracking down resource leaks.
 2018-04-11 20:55:58 +01:00
Nick Craig-Wood 37be78705d fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost 
Before this change mega (which uses a different host per download)
would open too many sockets.
 2018-04-11 20:51:28 +01:00
Michael P. Dubner 92b266d361 rc: new call rc/pid - closes #2211 2018-04-09 13:33:04 +01:00
Nick Craig-Wood cbec59146a lsf: make sure we use localtime in tests - fixes Box integration tests 
This problem was introduced with eca99b33c0.  It seems Box is the only
remote which converts time zones, so if you give it a GMT time zone,
it returns a PST time zone which represents the same instant.
 2018-04-09 11:46:49 +01:00
Nick Craig-Wood be54fd8f70 Remove builds conditional on go1.7 since that is now guaranteed #2154 
Old fallback code was deleted and the go1.7 style code inlined where
appropriate.
 2018-04-07 11:42:55 +01:00
Nick Craig-Wood e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood 80588a5a6b Replace "golang.org/x/net/context" with "context" for go1.7+ #2154 2018-04-07 11:42:08 +01:00
Nick Craig-Wood 67023f0040 Require go1.7 for compilation #2154 
* Update the travis tests to exclude go1.6
  * Update the compile check to require go1.7+
  * Update misc go1.6 workarounds marked in the source
 2018-04-06 20:18:14 +01:00
Matt Holt e13f65b953 serve restic: Print actual listener address 2018-04-04 14:56:26 +01:00
Nick Craig-Wood eca99b33c0 lsd,lsf: make sure all times we output are in local time - fixes #2183 
Previous to this change times from lsd/lsf were output in whatever
timezone they were in whereas times from lsl were converted to
localtime.
 2018-04-01 15:40:04 +01:00
Stefan a8267d1628
link: allow creating public link to files and folders - closes #1562 2018-03-29 09:10:19 +02:00
Mateusz Pabian 91b068ad3a sync: implement --ignore-errors - fixes #642 2018-03-23 22:01:10 +00:00
Nick Craig-Wood 940df88eb2 Start v1.40-DEV development 2018-03-19 14:20:48 +00:00
Nick Craig-Wood 26f1c55987 Version v1.40 2018-03-19 10:06:13 +00:00