MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,368 Commits 180 Branches 116 Tags 386 MiB
Commit Graph

81 Commits

Author SHA1 Message Date
Nick Craig-Wood bf4a16ae30 crypt: support metadata 2022-06-29 17:30:37 +01:00
Nick Craig-Wood 866c873daa backend: allow wrapping backend tests to run in make quicktest 2022-06-29 17:30:37 +01:00
albertony fdd2f8e6d2 Error strings should not be capitalized 
Reported by staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-23 23:26:02 +02:00
albertony 8697f0bd26 about: improved error message 2022-05-13 12:08:10 +01:00
Nick Craig-Wood b91c349cd5 local: fix hash invalidation which caused errors with local crypt mount 
Before this fix if a file was updated, but to the same length and
timestamp then the local backend would return the wrong (cached)
hashes for the object.

This happens regularly on a crypted local disk mount when the VFS
thinks files have been changed but actually their contents are
identical to that written previously. This is because when files are
uploaded their nonce changes so the contents of the file changes but
the timestamp and size remain the same because the file didn't
actually change.

This causes errors like this:

    ERROR: file: Failed to copy: corrupted on transfer: md5 crypted
    hash differ "X" vs "Y"

This turned out to be because the local backend wasn't clearing its
cache of hashes when the file was updated.

This fix clears the hash cache for Update and Remove.

It also puts a src and destination in the crypt message to make future
debugging easier.

Fixes #4031
 2021-11-24 12:09:34 +00:00
Sinan Tan 8c61a09be2 crypt: add test cases and documentation for base64 and base32768 filename encoding #5801 2021-11-15 17:57:02 +00:00
Max Sum c217145cae crypt: add base64 and base32768 filename encoding options #5801 2021-11-15 17:57:02 +00:00
Nick Craig-Wood e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
Atílio Antônio c08d48a50d
docs: improve grammar and fix typos (#5361) 
This alters some comments in source files, but is interested mainly in documentation files and help messages.
 2021-11-04 12:50:43 +01:00
albertony e2f47ecdeb docs: punctuation cleanup 
See #5538
 2021-10-20 22:56:19 +02:00
Nick Craig-Wood 1cb31e8cc7 crypt: fix uploads with --crypt-no-data-encryption 
Before this change, when uploading to a crypt, the ObjectInfo
accidentally used the encrypted size, not the unencrypted size when
--crypt-no-data-encryption was set.

Fixes #5498
 2021-10-12 17:12:41 +01:00
Ivan Andreev 7190c058a7 crypt: return wrapped object even with no-data-encryption 
In presence of no_data_encryption the Crypt's Put method used to over-optimize
and returned base object. This patch makes it return Crypt-wrapped object now.

Fixes #5498
 2021-10-12 00:41:12 +03:00
Jonta d8d621c175
docs: grammar/readability (#5633) 2021-09-30 15:34:00 +02:00
Dominik Mydlil 3fe2aaf96c crypt: support timestamped filenames from --b2-versions 
With the file version format standardized in lib/version, `crypt` can
now treat the version strings separately from the encrypted/decrypted
file names. This allows --b2-versions to work with `crypt`.

Fixes #1627

Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com>
 2021-04-12 15:59:18 +01:00
albertony 12cd322643
crypt: log hash ok on upload 2021-03-23 18:36:51 +01:00
Vesnyx 4ea7c7aa47
crypt: add option to not encrypt data #1077 (#2981) 
Co-authored-by: Ivan Andreev <ivandeex@gmail.com>
 2021-02-17 03:40:37 +03:00
Nick Craig-Wood 47aada16a0 fs: add Shutdown optional method for backends 2020-11-27 17:35:01 +00:00
Nick Craig-Wood 8b96933e58 fs: Add context to fs.Features.Fill & fs.Features.Mask #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood d846210978 fs: Add context to NewFs #3257 #4685 
This adds a context.Context parameter to NewFs and related calls.

This is necessary as part of reading config from the context -
backends need to be able to read the global config.
 2020-11-09 18:05:54 +00:00
Josh Soref e4a87f772f docs: spelling: e.g. 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Dan Hipschman 70f92fd6b3 crypt: small simplification, no functionality change 2020-10-12 17:20:39 +01:00
Nick Craig-Wood 70c8566cb8 fs: Pin created backends until parents are finalized 
This attempts to solve the backend lifecycle problem by

- Pinning backends mentioned on the command line into the cache
  indefinitely

- Unpinning backends when the containing structure (VFS, wrapping
  backend) is destroyed

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood 0d066bdf46 alias,cache,chunker,crypt: make any created backends be cached to fix rc problems 
Before this change, when the above backends created a new backend they
didn't put it into the backend cache.

This meant that rc commands acting on those backends did not work.

This was fixed by making sure the backends use the backend cache.

See: https://forum.rclone.org/t/rclone-rc-backend-command-not-working-as-expected/18834
 2020-09-01 18:21:03 +01:00
Nick Craig-Wood 3f0d54daae crypt: fix purge bug introduced by refactor #1891 
In this commit

a2afa9aadd fs: Add directory to optional Purge interface

We failed to encrypt the directory name so the Purge failed.

This was spotted by the integration tests.
 2020-09-01 15:16:14 +01:00
Nick Craig-Wood a2afa9aadd fs: Add directory to optional Purge interface - fixes #1891 
- add a directory to the optional Purge interface
- fix up all the backends
- add an additional integration test to test for the feature
- use the new feature in operations.Purge

Many of the backends had been prepared in advance for this so the
change was trivial for them.
 2020-07-31 17:43:17 +01:00
Nick Craig-Wood 61ff7306ae crypt: add --crypt-server-side-across-configs flag 
This can be used for changing filename encryption mode without
re-uploading data.

See: https://forum.rclone.org/t/revert-filename-encryption-method/17454/
 2020-06-27 11:40:15 +01:00
Roman Kredentser 55ad1354b6 link: Add --expire and --unlink flags 
This adds expire and unlink fields to the PublicLink interface.

This fixes up the affected backends and removes unlink parameters
where they are present.
 2020-06-05 14:51:05 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood 52b7337d28 crypt: change backend encode/decode to output a plain list 
This commit changes the output of the rclone backend encode crypt: and
decode commands to output a plain list of decoded or encoded file
names.

This makes the command much more useful for command line scripting.
 2020-05-13 18:11:45 +01:00
Anagh Kumar Baranwal 856c2b565f crypt: Added decode/encode commands to replicate functionality of cryptdecode 
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
 2020-05-04 16:23:33 +00:00
Nick Craig-Wood 1648c1a0f3 crypt: calculate hashes for uploads from local disk 
Before this change crypt would not calculate hashes for files it was
uploading. This is because, in the general case, they have to be
downloaded, encrypted and hashed which is too resource intensive.

However this causes backends which need the hash first before
uploading (eg s3/b2 when uploading chunked files) not to have a hash
of the file. This causes cryptcheck to complain about missing hashes
on large files uploaded via s3/b2.

This change calculates hashes for the upload if the upload is coming
from a local filesystem. It does this by encrypting and hashing the
local file re-using the code used by cryptcheck. For a local disk this
is not a lot more intensive than calculating the hash.

See: https://forum.rclone.org/t/strange-output-for-cryptcheck/15437
Fixes: #2809
 2020-04-22 11:33:48 +01:00
Nick Craig-Wood 44b1a591a8 crypt: get rid of the unused Cipher interface as it obfuscated the code 2020-04-22 11:33:48 +01:00
Nick Craig-Wood cd3c699f28 lib/readers: factor ErrorReader from multiple sources 2020-04-19 15:18:49 +01:00
Nick Craig-Wood 7a5a74cecb crypt: clarify that directory_name_encryption depends on filename_encryption 
See: https://forum.rclone.org/t/directory-name-encryption-is-set-to-always-false-when-choosing-filename-encryption-off/14600
 2020-02-28 16:26:45 +00:00
buengese 9c858c3228 crypt: correctly handle trailing dot 2020-01-22 01:40:04 +01:00
Thomas Eales 42de601fa6 crypt: reorder the filename encryption options 
This brings the default `standard` to the top of the list to replace `off`
 2020-01-12 14:23:35 +00:00
Nick Craig-Wood 3f7af64316 config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
Fabian Möller 6c0a749a42 crypt: remove checkValidString 
Remove the usage of checkValidString in decryptSegment to allow all
paths which can be created by encryptSegment to be decryptable.
 2019-09-30 14:05:49 +01:00
Nick Craig-Wood ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood 3ebcb555f4 fs: add optional features UserInfo and Disconnect 2019-08-12 13:49:23 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Nick Craig-Wood 99bc013c0a crypt: remove stray debug in ChangeNotify 2019-05-12 16:50:03 +01:00
Nick Craig-Wood 7c4fe3eb75 local: define OpenWriterAt interface and test and implement it #2252 
This will enable multipart downloads in future commits
 2019-05-11 23:35:19 +01:00
Nick Craig-Wood eeab7a0a43 crypt: Implement Optional methods SetTier, GetTier - fixes #2895 
This implements optional methods on Object
- ID
- SetTier
- GetTier

And declares that it will not implement MimeType for the FsCheck test.
 2019-04-16 13:33:10 +01:00
Nick Craig-Wood ac8d1db8d3 crypt: support PublicLink (rclone link) of underlying backend - fixes #3042 2019-04-16 13:33:10 +01:00
Nick Craig-Wood cdf12b1fc8 crypt: Fix wrapping of ChangeNotify to decrypt directories properly 
Also change the way it is added to make the FsCheckWrap test pass
 2019-04-16 13:33:10 +01:00