Go to file
Gilbert Gilb's 536ebefff4
feat(backends/s3): add warmup support before repacks and restores (#5173)
* feat(backends/s3): add warmup support before repacks and restores

This commit introduces basic support for transitioning pack files stored
in cold storage to hot storage on S3 and S3-compatible providers.

To prevent unexpected behavior for existing users, the feature is gated
behind new flags:

- `s3.enable-restore`: opt-in flag (defaults to false)
- `s3.restore-days`: number of days for the restored objects to remain
  in hot storage (defaults to `7`)
- `s3.restore-timeout`: maximum time to wait for a single restoration
  (default to `1 day`)
- `s3.restore-tier`: retrieval tier at which the restore will be
  processed. (default to `Standard`)

As restoration times can be lengthy, this implementation preemptively
restores selected packs to prevent incessant restore-delays during
downloads. This is slightly sub-optimal as we could process packs
out-of-order (as soon as they're transitioned), but this would really
add too much complexity for a marginal gain in speed.

To maintain simplicity and prevent resources exhautions with lots of
packs, no new concurrency mechanisms or goroutines were added. This just
hooks gracefully into the existing routines.

**Limitations:**

- Tests against the backend were not written due to the lack of cold
  storage class support in MinIO. Testing was done manually on
  Scaleway's S3-compatible object storage. If necessary, we could
  explore testing with LocalStack or mocks, though this requires further
  discussion.
- Currently, this feature only warms up before restores and repacks
  (prune/copy), as those are the two main use-cases I came across.
  Support for other commands may be added in future iterations, as long
  as affected packs can be calculated in advance.
- The feature is gated behind a new alpha `s3-restore` feature flag to
  make it explicit that the feature is still wet behind the ears.
- There is no explicit user notification for ongoing pack restorations.
  While I think it is not necessary because of the opt-in flag, showing
  some notice may improve usability (but would probably require major
  refactoring in the progress bar which I didn't want to start). Another
  possibility would be to add a flag to send restores requests and fail
  early.

See https://github.com/restic/restic/issues/3202

* ui: warn user when files are warming up from cold storage

* refactor: remove the PacksWarmer struct

It's easier to handle multiple handles in the backend directly, and it
may open the door to reducing the number of requests made to the backend
in the future.
2025-02-01 18:26:27 +00:00
.github CI: bump golangci-lint to v1.63.4 2025-01-28 19:55:45 +01:00
changelog feat(backends/s3): add warmup support before repacks and restores (#5173) 2025-02-01 18:26:27 +00:00
cmd/restic feat(backends/s3): add warmup support before repacks and restores (#5173) 2025-02-01 18:26:27 +00:00
contrib bump cobra and add completions for fish 2021-05-21 13:47:52 +10:00
doc feat(backends/s3): add warmup support before repacks and restores (#5173) 2025-02-01 18:26:27 +00:00
docker bump dockerfile to go 1.23 2025-01-18 18:27:43 +01:00
helpers build dragonflybsd binaries 2024-11-30 15:47:39 +01:00
internal feat(backends/s3): add warmup support before repacks and restores (#5173) 2025-02-01 18:26:27 +00:00
.gitattributes internal/repository: Fix LoadBlob + fuzz test 2022-06-06 17:02:28 +02:00
.gitignore feat: support AWS assume role 2024-01-06 21:19:58 +01:00
.golangci.yml CI: disable shadow builtin rule 2025-01-28 19:51:14 +01:00
.readthedocs.yaml doc: add HTMLZip format to .readthedocs.yaml 2023-11-02 00:16:47 +03:00
CHANGELOG.md Generate CHANGELOG.md for 0.17.3 2024-11-08 20:36:25 +01:00
CONTRIBUTING.md CONTRIBUTING.md: Clarify handling of small bugfixes 2024-01-26 23:51:54 +01:00
GOVERNANCE.md Document project governance 2018-05-28 22:29:06 +02:00
LICENSE Update LICENSE 2017-11-02 11:39:49 +01:00
Makefile Remove vendor from build scripts 2020-03-01 11:30:02 +01:00
README.md Minor README.md cleanups 2024-03-09 23:56:16 +01:00
VERSION Set development version for 0.17.3 2024-11-08 20:36:48 +01:00
build.go bump required go version to 1.21 2024-08-10 19:16:10 +02:00
doc.go gofmt all files 2022-08-19 19:12:26 +02:00
go.mod Merge pull request #5225 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.0 2025-02-01 17:03:33 +01:00
go.sum Merge pull request #5225 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob-1.6.0 2025-02-01 17:03:33 +01:00

README.md

Documentation Build Status Go Report Card

Introduction

restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).

For detailed usage and installation instructions check out the documentation.

You can ask questions in our Discourse forum.

Quick start

Once you've installed restic, start off with creating a repository for your backups:

$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.

and add some data:

$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00%  54.732 MiB/s  1.582 GiB / 1.582 GiB  2580 / 2580 items  0 errors  ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved

Next you can either use restic restore to restore files or use restic mount to mount the repository via fuse and browse the files from previous snapshots.

For more options check out the online documentation.

Backends

Saving a backup on the same machine is nice but not a real backup strategy. Therefore, restic supports the following backends for storing backups natively:

Design Principles

Restic is a program that does backups right and was designed with the following principles in mind:

  • Easy: Doing backups should be a frictionless process, otherwise you might be tempted to skip it. Restic should be easy to configure and use, so that, in the event of a data loss, you can just restore it. Likewise, restoring data should not be complicated.

  • Fast: Backing up your data with restic should only be limited by your network or hard disk bandwidth so that you can backup your files every day. Nobody does backups if it takes too much time. Restoring backups should only transfer data that is needed for the files that are to be restored, so that this process is also fast.

  • Verifiable: Much more important than backup is restore, so restic enables you to easily verify that all data can be restored.

  • Secure: Restic uses cryptography to guarantee confidentiality and integrity of your data. The location the backup data is stored is assumed not to be a trusted environment (e.g. a shared space where others like system administrators are able to access your backups). Restic is built to secure your data against such attackers.

  • Efficient: With the growth of data, additional snapshots should only take the storage of the actual increment. Even more, duplicate data should be de-duplicated before it is actually written to the storage back end to save precious backup space.

Reproducible Builds

The binaries released with each restic version starting at 0.6.1 are reproducible, which means that you can reproduce a byte identical version from the source code for that release. Instructions on how to do that are contained in the builder repository.

News

You can follow the restic project on Mastodon @resticbackup or subscribe to the project blog.

License

Restic is licensed under BSD 2-Clause License. You can find the complete text in LICENSE.

Sponsorship

Backend integration tests for Google Cloud Storage and Microsoft Azure Blob Storage are sponsored by AppsCode!

Sponsored by AppsCode