MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,348 Commits 181 Branches 116 Tags 326 MiB
Commit Graph

81 Commits

Author SHA1 Message Date
albertony ec117593f1 Fix lint issues reported by staticcheck 
Used staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-13 21:13:50 +02:00
Nick Craig-Wood e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
albertony e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
Nick Craig-Wood f4c5f1f185 box: retry operation_blocked_temporary errors #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood 825f7826f5 box: add --box-owned-by to only show items owned by the login passed #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood 34140b2f57 box: delete items in parallel in cleanup using --checkers threads #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood e18ae5da09 box: factor directory listing and cleanup listing into one function #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood b61912b4c8 box: add --box-list-chunk to control listing chunk size #5545 2021-10-15 15:28:54 +01:00
Nick Craig-Wood bfecf5301b box: when doing cleanup delete as much as possible - fixes #5545 
Before this change the cleanup routine exited on the first deletion
error.

This change counts any errors on deletion and exits when the iteration
is complete with an error showing the number of deletion failures.
Deletion failures will be logged.
 2021-10-15 15:28:54 +01:00
Nick Craig-Wood 308323e9c4 box: make listings of heavily used directories more reliable #5545 
Before this change we uses limit/offset paging for directories in the
main directory listing routine and in the trash cleanup listing.

This switches to the new scheme of limit/marker which is more reliable
on a directory which is continuously changing. It has the disadvantage
that it doesn't tell us the total number of items available, however
that wasn't information rclone uses.
 2021-10-15 15:28:54 +01:00
Nick Craig-Wood 3fbaa4c0b0 backends: make NewObject return fs.ErrorIsDir if possible 
This changes the interface to NewObject so that if NewObject is called
on a directory then it should return fs.ErrorIsDir if possible without
doing any extra work, otherwise fs.ErrorObjectNotFound.

Tested on integration test server with:

go run integration-test.go -tests backend -run TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir -branch fix-stat -maxtries 1
 2021-10-14 17:15:50 +01:00
Nick Craig-Wood 94dbfa4ea6 fs: change Config callback into state based callback #3455 
This is a very large change which turns the post Config function in
backends into a state based call and response system so that
alternative user interfaces can be added.

The existing config logic has been converted, but it is quite
complicated and folloup commits will likely be needed to fix it!

Follow up commits will add a command line and API based way of using
this configuration system.
 2021-05-14 14:07:44 +01:00
Nick Craig-Wood b78c9a65fa backends: remove log.Fatal and replace with error returns #5234 
This changes the Config interface so that it returns an error.
 2021-04-27 18:18:08 +01:00
Nick Craig-Wood ef3c350686 box: return errors instead of calling log.Fatal with them #5234 2021-04-27 18:18:08 +01:00
albertony 2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
Nick Craig-Wood d27c35ee4a box: use upload preflight check to avoid listings in file uploads 
Before this change, rclone checked to see if an object existed before
doing an upload by listing the destination directory. This was very
inefficient, especially with large directories.

After this change rclone uses the pre upload check API call which
checks to see if it is OK to upload an object, and also returns the ID
of an existing object which saves rclone having to do a directory
listing.
 2021-04-25 11:45:44 +01:00
Nick Craig-Wood 1378bfee63 box: fix transfers getting stuck on token expiry after API change 
Box recently changed their API, changing the case of returned API items

> On May 10th, 2021, as part of our continued infrastructure upgrade,
> Box's API response headers will standardize to return in a case
> insensitive manner, in line with industry best practices and our API
> documentation. Applications that are using these headers, such as
> "location" and "retry-after", will need to verify that their
> applications are checking for these headers in a case-insensitive
> fashion.

Rclone was reading the raw headers from the `http.Header` and not
using the `Get` accessor method which meant that it was sensitive to
case changes.

This fixes the problem by using the `Get` accessor method.

See: https://forum.rclone.org/t/box-backend-incompatible-with-box-api-changes-being-deployed/22972
 2021-03-24 09:45:17 +00:00
Nick Craig-Wood 4013bc4a4c Fix excessive retries missing --max-duration timeout - fixes #4504 
This change checks the context whenever rclone might retry, and
doesn't retry if the current context has an error.

This fixes the pathological behaviour of `--max-duration` refusing to
exit because all the context deadline exceeded errors were being
retried.

This unfortunately meant changing the shouldRetry logic in every
backend and doing a lot of context propagation.

See: https://forum.rclone.org/t/add-flag-to-exit-immediately-when-max-duration-reached/22723
 2021-03-13 09:25:44 +00:00
Nick Craig-Wood acb9e17eb3 box: fix finding directories in a case insentive way #4830 2021-01-26 14:48:33 +00:00
Nick Craig-Wood 4acd68188b box: fix NewObject for files that differ in case #4830 2020-12-07 17:38:22 +00:00
Nick Craig-Wood 9d574c0d63 fshttp: read config from ctx not passed in ConfigInfo #4685 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood f7efce594b config: add context.Context #3257 #4685 
This add config to the Config callback in the backends and the related
config functions.
 2020-11-09 18:05:54 +00:00
Nick Craig-Wood 8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Tim Gallant 30eb094f28 oauthutil: adds SharedOptions for OAuth backends 
1. adds SharedOptions data structure to oauthutil
2. adds config.ConfigToken option to oauthutil.SharedOptions
3. updates the backends that have oauth functionality

Fixes #2849
 2020-08-07 16:32:01 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
David 8bf265c775 box: allow authentication with access token - fixes #4114 2020-07-28 11:43:44 +01:00
Nick Craig-Wood a41a294e1d box: cancel in progress multipart uploads and copies on rclone exit #4300 2020-06-25 15:22:53 +01:00
Nick Craig-Wood 2121c0fa23 dircache: factor DirMove code out of backends into dircache 
Before this change there was lots of duplicated code in all the
dircache using backends to support DirMove.

This change factors this code into the dircache library.
 2020-06-25 09:41:36 +01:00
Nick Craig-Wood a8652e2252 dircache: simplify interface, fix corner cases and apply to backends 
Dircache was changed to:

- Remove special cases for the root directory
- Remove Fatal errors
- Call FindRoot on behalf of the user wherever possible
- Bring up to modern Go standards

Backends were changed to:

- Remove calls to FindRoot
- Change calls to FindRootAndPath to FindPath
- Don't make special cases for the root

This fixes several corner cases, for example removing a non existent
directory if FindRoot hasn't been called.
 2020-06-25 09:41:36 +01:00
buengese b6b8958fb4 box: implement CleanUp - fixes #4326 2020-06-18 23:39:59 +02:00
Roman Kredentser 55ad1354b6 link: Add --expire and --unlink flags 
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
 2020-06-05 14:51:05 +01:00
Nick Craig-Wood 973e3d6a7b backends: make sure backends expand ~ and environment vars in file names they use 
See: https://forum.rclone.org/t/relative-path-in-rclone-config-service-account-json/16693
 2020-06-03 17:39:08 +01:00
Nick Craig-Wood 49ba4eeb86 oauthutil: tidy interface to Config to add Options struct 
The interface was getting so that a new function was needed for every
Config variant. Adding an Options struct fixes this.
 2020-05-26 11:27:01 +01:00
Nick Craig-Wood c08617c70f box: Calculate Free amount in About call 2020-05-25 16:47:34 +01:00
Nick Craig-Wood 9db8ecbc32 box: implement About to read size used - fixes #4264 2020-05-23 18:46:44 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
David 0c0ed2fe04 box: Remove unnecessary iat from jws claims 2020-04-23 17:52:14 +01:00
Tim Gallant 01cb256b84 box: pass options to rest.Opts for uploadPart 2020-04-23 11:07:21 +01:00
Tim Gallant 0b0163dde2 box: pass options to rest.Opts for upload 2020-04-23 11:07:21 +01:00
Sunil Patra 39319b4858 @Sunil-P 
box: Added support for interchangeable root folder for Box backend - #3422
 2020-04-22 17:00:13 +01:00
David Bramwell 8a3c4c6a7b
box: add token renew function for jwt auth - Fixes #4901 2020-04-22 16:53:03 +01:00
Nick Craig-Wood bafe7d5a73 backends: move encoding definitions from fs/encodings 2020-01-16 14:40:36 +00:00
Nick Craig-Wood 3c620d521d backend: adjust backends to have encoding parameter 
Fixes #3761
Fixes #3836
Fixes #3841
 2020-01-16 14:40:36 +00:00
Fabian Möller 05e4f10436 box: use lib/encoder 2019-09-30 14:05:49 +01:00
David 4788545b05 box: add options to get access token via JWT auth 2019-09-20 17:15:16 +01:00
Nick Craig-Wood 58a531a203 rest: add context propagation to rest library #3257 
This fixes up the calling and propagates the contexts for the backends
which use lib/rest.
 2019-09-09 23:27:07 +01:00