MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,458 Commits 180 Branches 116 Tags 325 MiB
Commit Graph

126 Commits

Author SHA1 Message Date
albertony 2925e1384c Use binary prefixes for size and rate units 
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
 2021-04-27 02:25:52 +03:00
albertony 2ec0c8d45f stats: correct spelling of data rate units 2021-04-27 02:25:52 +03:00
albertony 98579608ec docs: cleanup spelling of size and rate units 2021-04-27 02:25:52 +03:00
Nick Craig-Wood 268a7ff7b8 rc: add a full set of stats to core/stats 
This patch adds the missing stats to the output of core/stats

- totalChecks
- totalTransfers
- totalBytes
- eta

This now includes enough information to rebuild the normal stats
output from rclone including percentage completions and ETAs.

Fixes #5116
 2021-03-22 10:10:36 +00:00
albertony 97fc3b9046
rc: avoid +Inf value for speed in core/stats (#5134) 
Fixes #5132
 2021-03-18 10:02:30 +01:00
Nick Craig-Wood a7fd65bf2d config: move account initialisation out into accounting 
Before this change the initialisation for the accounting package was
done in the config package for some strange historical reason.
 2021-03-11 17:29:26 +00:00
Nick Craig-Wood 2aebeb6061 accounting: fix --bwlimit when up or down is off - fixes #5019 
Before this change the core bandwidth limit was limited to upload or
download value if the other value was off.

This fix only applies a core bandwidth limit when both values are set.
 2021-02-13 12:45:12 +00:00
Nick Craig-Wood cdd99a6f39 fs/accounting: fix occasionally failing test on macOS 2021-02-07 14:59:53 +00:00
Nick Craig-Wood 0d8bcc08da Revert "fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother" #4987 
This reverts commit 463a18aa07.

In practice this caused Windows to have a max bandwidth limit of 8
MiB/s. This is because of the limited resolution (1ms) of the Windows
timer.

This needs fixing in a different way.

See: https://forum.rclone.org/t/bug-report-for-v1-54-0-beta-5133/22015
 2021-02-01 22:23:50 +00:00
Nick Craig-Wood 463a18aa07 fs/accounting: make edge bandwidth limiters have smaller bursts to make smoother 
This change decreases the edge limiter burst size which dramatically
increases the smoothness of the bandwidth limiting.

The core bandwidth limiter remains with a large burst so it isn't
affected by double rate limiting on the edge limiters.

See: #4395
See: https://forum.rclone.org/t/bwlimit-is-not-really-smooth/20947
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 0a932dc1f2 Add --bwlimit for upload and download #1873 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 8856e0e559 fshttp: Obey bwlimit in http Transport - fixes #4395 
This change uses the bwlimit code to apply limits to the receive and
transmit data functions in the HTTP Transport.

This means that all HTTP transactions will have limiting applied -
this includes listings for example.

For HTTP based transorts this makes the limiting in Accounting
redundant and possibly counter productive
 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 3b6df71838 accounting: refactor bwlimit code to allow for multiple slots 2021-01-30 17:39:30 +00:00
Nick Craig-Wood 50344e7792 accounting: factor --tpslimit code into accounting from fshttp 2021-01-08 10:29:57 +00:00
Nick Craig-Wood e972f2c98a log: make it easier to add parameters to JSON logging 2021-01-05 19:44:25 +00:00
Ingo Weiss 1d40bc1901 fs: Accumulate stats when using --dry-run 
Fixes #4624
 2020-12-29 21:11:12 +00:00
Maciej Zimnoch 70e8b11805
accounting: fix data race in Transferred() 
startedTransfers is accessed by multiple threads, and it wasn't
protected by the mutex call in Transferred() func.

Fixes #4799
 2020-11-26 22:54:38 +00:00
Nick Craig-Wood 2e21c58e6a fs: deglobalise the config #4685 
This is done by making fs.Config private and attaching it to the
context instead.

The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
 2020-11-26 16:40:12 +00:00
Nick Craig-Wood 1fb6ad700f accounting: add context.Context #3257 #4685 2020-11-09 18:05:54 +00:00
Nick Craig-Wood af55a74bd2 stats: add counter for deleted directories - fixes #4676 2020-11-03 11:47:00 +00:00
Josh Soref bbe7eb35f1 docs: spelling: server-side 
Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-28 18:16:23 +00:00
Josh Soref d0888edc0a Spelling fixes 
Fix spelling of: above, already, anonymous, associated,
authentication, bandwidth, because, between, blocks, calculate,
candidates, cautious, changelog, cleaner, clipboard, command,
completely, concurrently, considered, constructs, corrupt, current,
daemon, dependencies, deprecated, directory, dispatcher, download,
eligible, ellipsis, encrypter, endpoint, entrieslist, essentially,
existing writers, existing, expires, filesystem, flushing, frequently,
hierarchy, however, implementation, implements, inaccurate,
individually, insensitive, longer, maximum, metadata, modified,
multipart, namedirfirst, nextcloud, obscured, opened, optional,
owncloud, pacific, passphrase, password, permanently, persimmon,
positive, potato, protocol, quota, receiving, recommends, referring,
requires, revisited, satisfied, satisfies, satisfy, semver,
serialized, session, storage, strategies, stringlist, successful,
supported, surprise, temporarily, temporary, transactions, unneeded,
update, uploads, wrapped

Signed-off-by: Josh Soref <jsoref@users.noreply.github.com>
 2020-10-14 15:21:31 +01:00
LaSombra aac2406e19
cmd: add --progress-terminal-title to print ETA to terminal title 
Adds a flag, --progress-terminal-title, that when used with --progress,
will print the string `ETA: %s` to the terminal title.

This also adds WriteTerminalTitle to lib/terminal
 2020-10-06 16:34:26 +01:00
Nick Craig-Wood 433b73a5a8 accounting: stabilize display order of transfers on Windows 
Before this change we sorted transfers in the stats list solely on
time started. However if --check-first was in use then lots of
transfers could be started in the same millisecond. Because Windows
time resolution is only 1mS this caused the entries to sort equal and
bounce around in the list.

This change fixes the sort so that if the time is equal it uses the
name which should stabilize the order.

Fixes #4599
 2020-09-24 19:10:37 +01:00
Nick Craig-Wood 5c49096e11 acounting: fix incorrect speed and transferTime in core/stats 
Before this change the code which summed up the existing transfers
over all the stats groups forgot to add the old transfer time and old
transfers in.

This meant that the speed and elapsedTime got increasingly inaccurate
over time due to the transfers being culled from the list but their
time not being accounted for.

This change adds the old transfers into the sum which fixes the
problem.

This was only a problem over the rc.

Fixes #4569
 2020-09-15 12:01:18 +01:00
Nick Craig-Wood 75de30cfa8 sync: fix --cutoff-mode soft & cautious so it doesn't end the transfer early 
Before ths fix --cutoff-mode soft and cautious would emit a Fatal
error which stopped the sync immediately.

This fix introduces a new error which is checked in the sync error
processing which stops the sync gracefully.

Fixes #4576
 2020-09-09 12:53:21 +01:00
Nick Craig-Wood 05bea46c3e accounting: remove new line from end of --stats-one-line display 2020-09-08 16:10:52 +01:00
Nick Craig-Wood f5439ddc54 accounting: fix deadlock in stats printing 
The deadlock was caused in transfermap.go by calling mu.RLock() in one
function then calling it again in a sub function. Normally this is
fine, however this leaves a window where mu.Lock() can be called. When
mu.Lock() is called it doesn't allow the second mu.RLock() and
deadlocks.

    Thead 1                    Thread 2
    String():mu.RLock()
                               del():mu.Lock()
    sortedSlice():mu.RLock()                     - DEADLOCK

Lesson learnt: don't try using locks recursively ever!

This patch fixes the problem by removing the second mu.RLock(). This
was done by factoring the code that was calling it into the
transfermap.go file so all the locking can be seen at once which was
ultimately the cause of the problem - the code which used the locks
was too far away from the rest of the code using the lock.

This problem was introduced in:

bfa5715017 fs/accounting: sort transfers by start time

Which hasn't been released in a stable version yet
 2020-08-05 17:13:00 +01:00
Nick Craig-Wood 122a47fba6 accounting: Allow transfers to be canceled with context #3257 
This makes all transfers cancelable even if the backend doesn't
support context as all transfers are done using the Accounting
framework.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 421585dd72 accounting: add context to Account and propagate changes #3257 
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
 2020-07-28 16:41:17 +01:00
Nick Craig-Wood 26001d520a fs: add --bwlimit-file flag to limit speeds of individual file transfers 2020-07-28 11:46:24 +01:00
Chaitanya Bankanhal 8913679d88
accounting: Fix elapsed time not show actual time since beginning 
This fixes the elapsed time display in the statistics output in the rc and in the log messages.
 2020-07-26 11:59:50 +01:00
Nick Craig-Wood bcd362fcd5 accounting: fix documentation for speed/speedAvg 
Fix the documentation for the very confusingly names speed and
speedAvg stats items.

See: https://github.com/rclone/rclone-webui-react/issues/99
 2020-07-17 15:35:15 +01:00
Max Sum bfa5715017 fs/accounting: sort transfers by start time 2020-07-07 15:42:06 +01:00
Max Sum e2183ad661 fs/accounting: use transferMap instead of stringSet 2020-07-07 15:42:06 +01:00
Nick Craig-Wood 05ddef117a accounting: add HasBuffer method to Account 2020-06-30 12:03:39 +01:00
Nick Craig-Wood b58bb03e95 test: Don't run unreliable tests on CI #4171 2020-06-15 21:34:37 +01:00
Gary Kim 763944f673
rcd: fix incorrect prometheus metrics - fixes #4341 
This was caused by using the stats group from the context passed in by the rcd
rather than the global stats group.

Signed-off-by: Gary Kim <gary@garykim.dev>
 2020-06-14 10:09:24 +01:00
Nick Craig-Wood 3c14a893fb asyncreader: Make StopBuffer as well as Abandon and fix confusion in callers 2020-06-09 18:05:12 +01:00
Martin Michlmayr 041b201abd doc: fix typos throughout docs and code 2020-05-25 11:23:58 +01:00
Martin Michlmayr 4aee962233 doc: fix typos throughout docs and code 2020-05-20 15:54:51 +01:00
Martin Michlmayr fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood 177195aeeb accounting: fix race clearing stats 
This race was introduced by

10a6a92e52 accounting: reset bytes read during copy retry
 2020-05-12 17:02:32 +01:00
Nick Craig-Wood cb5979a468 accounting: factor stats into its own structure 
This makes it very obvious which mutex to take for accessing the
values.
 2020-05-12 17:02:32 +01:00
Ankur Gupta 10a6a92e52 accounting: reset bytes read during copy retry - fixes #4178 
During a copy/sync command, if an operation fails due to a network
issue and is retried, the underlying io.Reader is re-initialised,
but the stats for bytes already read are not reset, leading to incorrect
stats. THis was fixed by resetting the bytes read when an Account is
re-initialized.
 2020-05-10 17:58:22 +00:00
Nick Craig-Wood 8bf6ab2c52 accounting: fix race condition in tests 2020-04-24 12:32:09 +01:00
Nick Craig-Wood f5455d865b accounting: check for max transfer in WriteTo 
Before this change the max transfer tests were failing for remotes
which were using WriterTo.
 2020-04-23 11:13:13 +01:00
Nick Craig-Wood fdada79ebf accounting: support WriterTo for less memory copying 
This should reduce memory copying when the async buffer is in use and
improve speeds.
 2020-04-19 15:18:49 +01:00
Nick Craig-Wood 424554bc85 fs: generalise machinery for putting extra values when using --use-json-log 2020-04-11 18:16:21 +01:00
reddi 12a208a880 fs: expand stats output for json log 2020-04-11 18:16:21 +01:00