MirrorSave
/
rclone
mirror of https://github.com/rclone/rclone.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
802 Commits 180 Branches 116 Tags 386 MiB
Commit Graph

69 Commits

Author SHA1 Message Date
Nick Craig-Wood 945f49ab5e Make ContentType be preserved for cloud -> cloud copies - fixes #733 2016-10-03 20:02:04 +01:00
Asko Tamm 61665ddd10 s3: add support for setting storage class in config and command line 2016-09-13 19:49:44 +01:00
Nick Craig-Wood 0cb9bb3b54 Redo http Transport code 
* Insert User-Agent in Transport - fixes #199
  * Update timeouts to use Context
  * Modernise transport
 2016-09-12 17:50:19 +01:00
Nick Craig-Wood e6a0521ca2 Make it possible to test Fs multiple times and use this with crypt 
We test both the filename encryption modes for crypt.
 2016-08-23 17:45:37 +01:00
Radek Šenfeld 2003ba356b User-configurable Amazon S3 ACL 
fixes #413
 2016-08-23 12:25:08 +01:00
Nick Craig-Wood 8c2fc6daf8 s3: Add instructions on how to use rclone with minio 2016-07-11 12:12:28 +01:00
Nick Craig-Wood 2a1d4b7563 s3: Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions - fixes #567 2016-07-06 11:14:59 +01:00
Nick Craig-Wood ab43005422 Make NewObject return an error 
* make it return an error
  * make a canonical error fs.ErrorNotFound
  * make a test for it
  * remove logs/debugs of error
 2016-06-28 08:51:57 +01:00
Nick Craig-Wood b1f131964e Rename NewFsObject to NewObject 2016-06-28 08:51:57 +01:00
Nick Craig-Wood 1a87b69376 Get rid of LimitedFs - FIXME needs docs on copying single files 
If remote:path points to a file make NewFs return a sentinel error
fs.ErrorIsFile and an Fs which points to the parent.

Use this to remove the LimitedFs and just add this file to the
--files-from list.

This means that server side operations can be used also.

Fixes #518
Fixes #545
 2016-06-28 08:51:43 +01:00
Nick Craig-Wood a67c7461ee s3: skip SetModTime for objects > 5GB - fixes #534 2016-06-19 17:26:44 +01:00
Justin R. Wilson 6e35a3b3ce Add AES256 server-side encryption for s3 - Fixes #491 
Add a configuration key and support for AES256 server-side encryption.
 2016-06-18 14:28:38 +01:00
Nick Craig-Wood 2cfb3834f2 Log errors with %v 2016-06-18 09:36:47 +01:00
Nick Craig-Wood df1092ef33 Change Fs.Put so that it must cope with existing files 
This should fix duplicate files on drive and 409 errors on
amazonclouddrive however it will slow down the upload slightly as
another roundtrip will be needed.

None of the other Fses needed adjusting.

Fixes #483
 2016-06-13 19:29:10 +01:00
Nick Craig-Wood 4c5b2833b3 Convert to using github.com/pkg/errors everywhere 2016-06-13 17:43:03 +01:00
Nick Craig-Wood c2d0e86431 Add more tests for List() and fix resulting problems 2016-05-07 14:50:35 +01:00
Nick Craig-Wood 68ec6a9f5b Add a directory parameter to Fs.List() 2016-05-06 16:52:34 +01:00
Nick Craig-Wood 753b0717be Refactor the List and ListDir interface 
Gives more accurate error propagation, control of depth of recursion
and short circuit recursion where possible.

Most of the the heavy lifting is done in the "fs" package, making file
system implementations a bit simpler.

This commit contains some code originally by Klaus Post.

Fixes #316
 2016-05-06 16:52:34 +01:00
Nick Craig-Wood 414b35ea56 Change the interface of SetModTime to return an error - #348 2016-03-22 15:56:44 +00:00
Nick Craig-Wood a1323eb204 s3: Fix uploading files bigger than 50GB - fixes #386 2016-03-10 16:48:55 +00:00
Nick Craig-Wood 88cca8a6eb Simplify literals (after running gofmt -s over the code) 2016-02-29 16:57:23 +00:00
Jakub Gedeon 694d390710 s3: Check if directory exists during Mkdir 
If you dont have privs to create a bucket in S3 but it exists, don't
fail with an auth error, but detect that the mkdir was not needed and
return successfully.
 2016-02-27 13:24:46 +00:00
Nick Craig-Wood e08e35984c Add help to remote chooser in rclone config - fixes #43 2016-02-21 13:40:16 +00:00
klauspost ef06371c93 Create separate interface for object information. 
Take out read-only information about a Fs in a separate struct to limit access.

See discussion at #282.
 2016-02-19 13:31:09 +00:00
Nick Craig-Wood ce4c1d4f35 s3: Fix empty checks in auth 2016-02-09 17:19:33 +00:00
Brian Stengaard ce05ef7110 Add IAM role and Env credentials 
This will make the s3 provider authentaction logic

  - Configured credentials if both key and secret available
  - Anonymous if key and secret missing and env_auth not set
  - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool)
    - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables
    - IAM role credentials as fallback
 2016-02-09 16:32:36 +00:00
Nick Craig-Wood cae19df058 s3: URL escape CopySource 
This fixes metadata update and copy for files with `+` in

Fixes #315
 2016-01-27 17:39:33 +00:00
Nick Craig-Wood ccba859812 Test all available hashes for each remote 2016-01-23 09:10:36 +00:00
klauspost 78c3a5ccfa Add support for multiple hash types. 
Add support for multiple hash types with negotiation of common hash types for comparison.

Manually rebased version of #277 (see discussion there)
 2016-01-11 13:39:33 +01:00
Joseph Spurrier fe0289f2f5 s3: Fix corrupting Content-Type on mod time update 
This fixes an issue where updating the modification time resets the
content-type to the S3 default of binary/octet-stream which breaks
static websites that expect an html file to have a content-type of
text/html.
 2016-01-02 11:47:52 +00:00
Nick Craig-Wood a795d93bc3 swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172 2015-11-07 15:32:40 +00:00
Nick Craig-Wood b257de4aba Be more constistent with naming in remotes 
* External objects are called Fs and Object
  * Object.fs always points to the Fs
 2015-11-07 11:14:46 +00:00
klauspost 345c98ed62 Update to AWS SDK 0.10.0 
Tested with S3 and Dreamhost

Here is a link to the release notes:

http://aws.amazon.com/releasenotes/5476699172355228
 2015-11-02 19:52:11 +00:00
Nick Craig-Wood 9f1daabb2c s3: allow anonymous access to public repositories - fixes #154 
When setting up the remote, leave both the access key and secret key
blank.
 2015-10-03 16:04:21 +01:00
Nick Craig-Wood e9c915e6fe Fix golint warnings 2015-09-22 18:47:16 +01:00
Nick Craig-Wood 2ed158aba3 Fixes from go vet and errcheck 2015-09-22 07:31:12 +01:00
Nick Craig-Wood e2717a031e Implement Mover and DirMover interfaces fixes #115 
* unit tests
  * local
  * drive
  * dropbox
 2015-09-01 21:49:13 +01:00
Nick Craig-Wood 92745aa950 Add Root() to Fs interface 2015-09-01 21:49:13 +01:00
Nick Craig-Wood d298b578ab s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114 2015-08-28 08:47:41 +01:00
Nick Craig-Wood a96b522958 Implement server side copies if possible - fixes #99 
Add optional fs.Copier interface

Implemented for
  * swift
  * s3
  * drive
  * dropbox
  * google cloud storage
 2015-08-23 21:18:38 +01:00
Nick Craig-Wood fedf81c2b7 Add Name() to Fs interface to return name as passed to NewFs 2015-08-23 13:36:38 +01:00
Nick Craig-Wood 337110b7a0 s3: remove verbose debug about invalid md5sums for multipart upload 2015-08-16 18:14:22 +01:00
Nick Craig-Wood 6a8de87116 s3: make v2 signatures work for ceph 2015-08-16 15:51:38 +01:00
Nick Craig-Wood 0da6f24221 s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101 2015-08-16 15:51:04 +01:00
Leonid Shalupov 3fcff32524 do not print stats in quiet mode - fixes #70 
...unless had some errors or stats interval requested.

Add fs.ErrorLog to differentiate between Logs which should be
suppressed and errors which shouldn't.
 2015-08-10 11:17:54 +01:00
Nick Craig-Wood 1d254a3674 Implement --timeout and --conntimeout - fixes #54 
NB dropbox still to do
 2015-05-10 11:29:55 +01:00
Nick Craig-Wood f88d171afd s3: ignore etags from multipart uploads which aren't md5sums - fixes #56 2015-05-10 11:29:55 +01:00
Nick Craig-Wood 46d39ebaf7 Factor Mime Type guessing into fs.MimeType() 2015-03-02 09:21:15 +00:00
Nick Craig-Wood 85f8552c4d Tidy logging 2015-02-27 15:22:05 +00:00
Nick Craig-Wood 0faed16899 s3: list an unlimited number of items - fixes #22 2015-02-10 17:58:29 +00:00