MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,390 Commits 180 Branches 116 Tags 382 MiB
Commit Graph

36 Commits

Author SHA1 Message Date
Nick Craig-Wood a07d376fb1 vfs: reduce memory usage by re-ordering commonly used structures 2022-08-04 10:19:47 +01:00
Nick Craig-Wood e749bc58f4 vfs: reduce memory use by embedding sync.Cond 2022-08-04 10:19:47 +01:00
albertony ec117593f1 Fix lint issues reported by staticcheck 
Used staticcheck 2022.1.2 (v0.3.2)

See: staticcheck.io
 2022-06-13 21:13:50 +02:00
Nick Craig-Wood e43b5ce5e5 Remove github.com/pkg/errors and replace with std library version 
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.

This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
 2021-11-07 11:53:30 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
Nick Craig-Wood 421585dd72 accounting: add context to Account and propagate changes #3257 
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
 2020-07-28 16:41:17 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Nick Craig-Wood 0f9267d5fc vfs: factor waiting code from read and writes into common function 2020-05-18 18:09:23 +01:00
Nick Craig-Wood 3de9bd9d04 vfs: fix hang in read wait code - Fixes #4039 
Before this fix, rclone would sometimes hang in vfs.readAt().

This was due to a race condition causing rclone to miss the timeout
signal.

This was fixed by a small amount of extra locking.

This very likely also fixes a number of "failed to wait for
in-sequence read" errors.
 2020-05-18 18:09:23 +01:00
Nick Craig-Wood 1a8c5708c5 vfs: ignore file not found errors from Hash in Read.Release 
There is nothing we can do about this at this point and this error can
happen when moving files so we ignore it to clean the logs up.
 2020-05-04 12:18:28 +01:00
Nick Craig-Wood 238f26cc90 vfs: stop reading File members from outside file.go 
This also fixes locking for ReadFileHandle and WriteFileHandle
accessing File members
 2020-04-19 15:16:43 +01:00
Nick Craig-Wood 393b94bb70 vfs: add --vfs-read-wait and --vfs-write-wait flags 
--vfs-read-wait duration    Time to wait for in-sequence read before seeking. (default 5ms)
    --vfs-write-wait duration   Time to wait for in-sequence write before giving error. (default 1s)

See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
 2020-02-27 16:12:33 +00:00
Nick Craig-Wood 5beeac7959 vfs: make ReadAt for non cached files work better with non-sequential reads 
This makes ReadAt for non cached files wait a short time (up to 5mS)
if it gets an out of order read (which would normally cause a seek and
which take a long time) to see if the gap will be filled with an in
order read.

This makes mount2 based on go-fuse work more efficiently and enables
async reading in normal mount.

A similar change was done for WriteAt in af030f74f5
 2019-11-11 16:20:27 +00:00
Nick Craig-Wood be5392f448 vfs: only calculate one hash for reads 
This speeds up mounting on the local backend enormously.
 2019-11-11 16:20:27 +00:00
Nick Craig-Wood ba121eddf0 vfs: make objects of unknown size readable through the VFS 
These objects (eg Google Docs) appear with 0 length in the VFS.

Before this change, these only read 0 bytes.

After this change, even though the size appears to be 0, the objects
can be read to the end.  If the objects are read to the end then the
size on the handle will be updated.
 2019-09-14 13:09:07 +01:00
Nick Craig-Wood 57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Aleksandar Jankovic 8243ff8bc8 accounting: isolate stats to groups 
Introduce stats groups that will isolate accounting for logically
different transferring operations. That way multiple accounting
operations can be done in parallel without interfering with each other
stats.

Using groups is optional. There is dedicated global stats that will be
used by default if no group is specified. This is operating mode for CLI
usage which is just fire and forget operation.

For running rclone as rc http server each request will create it's own
group. Also there is an option to specify your own group.
 2019-07-28 14:48:19 +01:00
Aleksandar Jankovic be0464f5f1 accounting: change stats interface 
This is done to make clear ownership over accounting object and prepare
for removing global stats object.

Stats elapsed time calculation has been altered to account for actual
transfer time instead of stats creation time.
 2019-07-28 14:48:19 +01:00
Aleksandar Jankovic f78cd1e043 Add context propagation to rclone 
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions

Context propagation is needed for stopping transfers and passing other
request-scoped values.
 2019-06-19 11:59:46 +01:00
Fabian Möller 552eb8e06b vfs: try to seek buffer on read only files 2018-08-17 18:10:28 +01:00
Fabian Möller bd3ad1ac3e vfs: add option to read source files in chunks 2018-05-05 12:49:42 +01:00
Nick Craig-Wood e5be471ce0 Use io.SeekStart/End/Current constants now for go1.7+ #2154 2018-04-07 11:42:36 +01:00
Nick Craig-Wood 7fb53a031c vfs: don't cache the object in read and read/write handles 
This should help with inconsistent reads when the source object changes.
 2018-03-04 11:20:22 +00:00
Nick Craig-Wood 44276db454 vfs: make -ve sized files appear as 0 size. #320 
This means that Google docs will no longer appear as huge files in
`rclone mount`.  They will not be downloadable, though sometimes
trying twice will work.
 2018-01-31 16:22:05 +00:00
Nick Craig-Wood 11da2a6c9b Break the fs package up into smaller parts. 
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.

The new code layout is documented in CONTRIBUTING.
 2018-01-15 17:51:14 +00:00
Nick Craig-Wood 76b151984c vfs: cache the size of the object in the read handle 2017-11-20 17:57:13 +00:00
Nick Craig-Wood 60945d0a37 vfs: remove misleading comment 2017-11-20 12:46:23 +00:00
Nick Craig-Wood c31556c6d1 vfs: Make sure all public methods are locked in Read and Write Handle 2017-11-04 10:24:10 +00:00
Nick Craig-Wood 2083ac6e2a vfs: add ECLOSED and tidy errors 2017-11-04 10:24:10 +00:00
Nick Craig-Wood 5634659ea3 mount,vfs: unify Read and Write handles in preparation for ReadWrite handles 2017-11-04 10:24:10 +00:00
Nick Craig-Wood e18122e88b vfs: add tests and subsequent fixes 
* Tests for VFS layer
  * Small fixes found during testing
  * Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
  * Fix nil object bugs on File
 2017-11-04 10:24:10 +00:00
Nick Craig-Wood a5dc62f6c1 vfs: Make file handles compatible with OS 
* Implement directory handles
  * Unify OpenFile
  * Add all the methods to match *os.File
  * Add StatParent and Rename methods to VFS
 2017-11-03 13:00:00 +00:00
Nick Craig-Wood e8883e9fdb vfs: factor flags into vfsflags and remove global variables 2017-11-03 12:59:59 +00:00
Nick Craig-Wood c1aaff220d Factor new vfs module out of cmd/mountlib 
This is an OS style file system abstraction with directory caching
used in mount, cmount, serve webdav and serve http.
 2017-11-03 12:59:59 +00:00