Before this change only serve http was Shutting down its server which
was causing other servers such as serve restic to leave behind their
unix sockets.
This change moves the finalisation to lib/http so all servers have it
and removes it from serve http.
Fixes#6648
Before this change, we started the http listener even if --stdio was
supplied.
This also moves the log message so the user won't see the serving via
HTTP message unless they are really using that.
Fixes#6646
In the lib/http refactor
52443c2444 restic: refactor to use lib/http
We forgot to serve the data and wait for the server to finish. This is
not tested in the unit tests as it is part of the command line
handler.
Fixes#6644Fixes#6647
The webdav library was confused by the Path manipulation done by
lib/http when stripping the prefix.
This patch adds the prefix back before calling it.
Fixes#6650
- add support for unix sockets (which skip the auth).
- add support for multiple listeners
- collapse unnecessary internal structure of lib/http so it can all be
imported together
- moves files in sub directories of lib/http into the main lib/http
directory and reworks the code that uses them.
See: https://forum.rclone.org/t/wip-rc-rcd-over-unix-socket/33619Fixes: #6605
Before this change if we copied files of unknown size, then they lost
their metadata.
This was particularly noticeable using --s3-decompress.
This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.
Fixes#6546
The current default AnnounceInterval is too short, causing the
multicast domain to be flooded with NOTIFY announcements,
which may prevent other dlna devices from sleeping.
This change allows users to set the announcement interval,
and it's default value also increased to 12 minutes.
Even within the interval, rclone can still passively respond to
M-SEARCH requests from other devices.
Verify the http service listening address and the SSDP server
announcement address to prevent accidental listening of IPv6 addresses
that do not support dlna yet and may be globally accessible.
Unlistened addresses on the interface will also be filtered out of the
SSDP announcement to avoid misleading other services in the multicast domain.
Changes in github.com/anacrolix/dms changed upnp.ServiceURN to include a
namespace identifier. This identifier was previously hardcoded, but is
now parsed out of the URN. The old SOAP action header parsing logic was
duplicated in rclone and did not handle this field. Resulting responses
included a URN with an empty namespace identifier, breaking clients.
Before this change, rclone serve sftp operating with a new rclone
after the md5sum/sha1sum detection was reworked to just run a plain
`md5sum`/`sha1sum` command in
3ea82032e7 sftp: support md5/sha1 with rsync.net #3254
Failed to signal to the remote that md5sum/sha1sum wasn't supported as
in
71e172a139 serve/sftp: support empty "md5sum" and "sha1sum" commands
We unconditionally return good hashes even if the remote being served
doesn't support the hash type in question.
This fix checks the hash type is supported and returns an error
MD5 hash not supported
When the backend is first contacted this will cause the sftp backend
to detect that the hash type isn't available.
Unfortunately this may have cached the wrong state so editing or
remaking the config may be necessary to fix it.
strings.ReplaceAll(s, old, new) is a wrapper function for
strings.Replace(s, old, new, -1). But strings.ReplaceAll is more
readable and removes the hardcoded -1.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
In this commit
f4c40bf79d mount: add --devname to set the device name sent to FUSE for mount display
The --devname parameter was added. However it was soon noticed that
attempting to mount via the rc gave this error:
mount helper error: fusermount: unknown option 'fsname'
mount FAILED: fusermount: exit status 1
This was because the DeviceName (and VolumeName) parameter was never
being initialised when the mount was called via the rc.
The fix for this was to refactor the rc interface so it called the
same Mount method as the command line mount which initialised the
DeviceName and VolumeName parameters properly.
This also fixes the cmd/mount tests which were breaking in the same
way but since they aren't normally run on the CI we didn't notice.
Fixes#6044
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This is possible now that we no longer support go1.12 and brings
rclone into line with standard practices in the Go world.
This also removes errors.New and errors.Errorf from lib/errors and
prefers the stdlib errors package over lib/errors.
Before this fix, rclone only generated an RSA server key when the user
didn't supply a key.
However the RSA server key is being deprecated as it is now insecure.
This patch generates an ECDSA server key too which will be used in
preference over the RSA key, but the RSA key will carry on working.
Fixes#5671
This replaces built-in os.MkdirAll with a patched version that stops the recursion
when reaching the volume part of the path. The original version would continue recursion,
and for extended length paths end up with \\? as the top-level directory, and the error
message would then be something like:
mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
Includes adding support for additional size input suffix Mi and MiB, treated equivalent to M.
Extends binary suffix output with letter i, e.g. Ki and Mi.
Centralizes creation of bit/byte unit strings.
Restructuring of config code in v1.55 resulted in config
file being loaded early at process startup. If configuration
file is encrypted this means user will need to supply the password,
even when running commands that does not use config.
This also lead to an issue where mount with --deamon failed to
decrypt the config file when it had to prompt user for passord.
Fixes#5236Fixes#5228
If you are using rclone a library you can decide to use the rclone
config file system or not by calling
configfile.LoadConfig(ctx)
If you don't you will need to set `config.Data` to an implementation
of `config.Storage`.
Other changes
- change interface of config.FileGet to remove unused default
- remove MustValue from config.Storage interface
- change GetValue to return string or bool like elsewhere in rclone
- implement a default config file system which panics with helpful error
- implement getWithDefault to replace the removed MustValue
- don't embed goconfig.ConfigFile so we can change the methods
Before this change, if one connection was authenticating this would
block any others from authenticating.
This was due to ssh.NewServerConn not being called in a go routine
after the Accept call.
This is fixed by running the ssh authentication in a go routine.
Thanks to @FiloSottile for advice on how to fix this.
See: https://github.com/golang/go/issues/43521
When using `--baseurl` before this patch, if a request was made to the
base URL without a trailing / then rclone would return a 404 error.
Unfortunately GVFS / Nautilus makes the request without the /
regardless of what the user put in.
This patch redirects the request to the base URL with a /. So if the
user was using `--baseurl rclone` then a request to
http://localhost/rclone would be redirected with a 308 response to
http://localhost/rclone/Fixes#4814
This is done by making fs.Config private and attaching it to the
context instead.
The Config should be obtained with fs.GetConfig and fs.AddConfig
should be used to get a new mutable config that can be changed.
This caches all the objects returned from the List call. This makes
opening them much quicker so speeds up prune and restores. It also
uses fewer transactions. It can be disabled with
`--cache-objects=false`.
This was discovered when using the B2 backend when the budget was
being blown on list object calls which can avoided with a bit of
caching.
For typical 1 million file backup for a latop or server this will only
use a small amount more memory.
This adds a context.Context parameter to NewFs and related calls.
This is necessary as part of reading config from the context -
backends need to be able to read the global config.
This is preparation for getting the Accounting to check the context,
buf first we need to get it in place. Since this is one of those
changes that makes lots of noise, this is in a seperate commit.
This patch enables rclone to be used as a library from within restic
- exposes NewServer
- exposes Server
- implements http.RoundTripper
Co-authored-by: Jack Deng <jackdeng@gmail.com>
This uses the refactored goftp library which doesn't include the minio
driver. This reduces the binary size by 1.5MB
See: https://gitea.com/goftp/server/pulls/120
- Use Driver.CheckPasswd instead of server.CheckPasswd
- Make server.CheckPasswd return an error
- Remove awful findID to find parent function hack
- Remove Driver.Init as it is no longer called
- Fix backwards incompatible PublicIp -> PublicIP change
See: https://gitea.com/goftp/server/issues/117
Otherwise, we get en dashes in the man page, making args more difficult
to copy/paste to a command line.
Before:
Use –addr to specify ...
After:
Use --addr to specify ...
This includes a new directory listing template which was originally
from the Caddy project (used with permission and copyright attribution).
This is used whenever we serve directory listings so `rclone serve
http`, `rclone serve webdav` and `rclone rcd --rc-serve`
This also modifies the tests so they work with the original template which
is easier to debug.
Basically, solving #3541 with a different approach - bringing in
the upstream upnpav module, and changing ChildCount from int to a
*int to avoid childCount="0" in the XML output when that value is
simply unknown.
Current approach is leading to some recursion issues and according
to the DLNA spec it shouldn't be necessary, anyway.
Unfortunately bcrypt only hashes the first 72 bytes of a given input
which meant that using it on ssh keys which are longer than 72 bytes
was incorrect.
This swaps over to using sha256 which should be adequate for the
purpose of protecting in memory passwords where the unencrypted
password is likely in memory too.
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
Before this change the race tests were taking too long. The bcrypt
function went from about 20ms to 1s under the race detector and this
is called for every transaction on webdav.
This change reduces the bcrypt strength so it takes 1ms non race so
the race tests pass and still has adequate security for in memory only
storage.
Before this change the sftp handler returned a nil error for unknown
operations which meant the server crashed when one was encountered.
In particular the "Readlink" operations was causing problems.
After this change the handler returns ErrSshFxOpUnsupported which
signals to the remote end that we don't support that operation.
See: https://forum.rclone.org/t/rclone-serve-sftp-not-working-in-windows/12209
Seems to be some corner cases that are not being handled, so taking a different
approach that should be a little more robust.
Also, changing resources to be served under a subpath: We've been serving
media at /res?path=%2Fdir%2Ffilename.mp4; change that to be just /r/dir/filename.mp4.
It's cleaner, easier to reason about, and a necessary first step towards just
serving the resources via httplib anyway.
Allows for filename.srt, filename.en.srt, etc., to be automatically associated with video.mp4 (or whatever) when playing over dlna.
This is the "modern" method, which I've verified to work on VLC and in LG webOS 2. There is a vendor specific mechanism for Samsung that I havn't been able to get working on my F series.
Also made some minor corrections to logging and container IDs.
Add a minimal number of mime types to augment go's built in types
for environments which don't have access to a mime.types file (eg
Termux on android)
Fixes#3475
Before this fix serve dlna was only using the built in database of
mime types to look up the mime types of files. On Android (and
possibly other systems) this is very small.
The symptoms of this problem was serve dlna only listing images and
not videos.
After this fix we use the backend's idea of the mime type if possible
which will be more accurate.
Fixes#3475
piyushgarg
Nick Craig-Wood
Nolan Woods
Tom Mombourquette
asdffdsazqqq
albertony
Robert Newson
YanceyChiew
Josh Soref
Simon Bos
Joram Schrijver
Eng Zer Jun
Nil Alexandrov
Alain Nussbaumer
thomae
Atílio Antônio
Ivan Andreev
Thomas Stachl
Cnly
Antoine GIRARD
Tom
Deepak Sah
Josh Soref
Jack
Tim Burke
Matteo Pietro Dazzi
Martin Michlmayr
Maxime Suret
calisro
Michael G
Dan Walters
Lars Lehtonen
Paul Tinsley
Ankur Gupta
Xiaoxing Ye
Fionera
Chaitanya