## Files - CDN Origin Server for Cloud Storage [![crates.io](https://meritbadge.herokuapp.com/files)](https://crates.io/crates/files) ### Purpose The purpose of this service is to be your CDN Origin Server for Cloud Storage for any JSON client including JavaScript front-ends like React with fetch. Files supports [aws s3](https://aws.amazon.com/s3/), [vultr](https://www.vultr.com/products/object-storage/), [wasabi](https://wasabi.com/), [yandex](https://cloud.yandex.com/en/services/storage), and [digital ocean](https://www.digitalocean.com/products/spaces/). Files is built to be simple and blazing fast with JWT verification, bucket management, and object management. To use this service you need to have a running [broker](https://crates.io/crates/broker) server. ### Features * Very performant with almost no CPU and memory usage * Supports bucket and object management * Supports cloud storage providers: [aws s3](https://aws.amazon.com/s3/), [vultr](https://www.vultr.com/products/object-storage/), [wasabi](https://wasabi.com/), [yandex](https://cloud.yandex.com/en/services/storage), and [digital ocean](https://www.digitalocean.com/products/spaces/) * Supports being an origin server for [Bunny.net](https://bunny.net/), [CDN77](https://www.cdn77.com/), [Cloudflare](https://www.cloudflare.com/cdn/), or other CDN providers * Under 1000 lines of code * Supports CORS * Multi-tenant * Supports JWT authentication * Supports JWT caching with expiry checking to minimize verify API calls * Supports SSL - full end-to-end encryption * JSON API * Auto-provision and renews SSL cert via LetsEncrypt or use your own SSL cert * Uses user authorization scoping * CLI Application - [zn](https://crates.io/crates/zn) * Built on [broker](https://crates.io/crates/broker) ### Use 1. create a user on [broker](https://crates.io/crates/broker) with the following scopes - for full permissions `files:full` or granular permissions `files:provider, files:get_object, files:put_object, files:delete_object, files:create_bucket, files:list_bucket, or files:delete_bucket` 2. login to broker and get a JWT 3. attach the JWT as an Authorization: Bearer {token} to the following JSON API endpoints 4. Add your credentials for your provider(s) - the creds are just for the specific user that creates them * Valid providers are `aws`, `wasabi`, `yandex`, `vultr`, and `do` * Valid regions are: | Provider | Region | |--- | --- | | wasabi | wa-us-east-1 | | wasabi | wa-us-east-2 | | wasabi | wa-us-west-1 | | wasabi | wa-eu-central-1 | | aws | us-east-1 | | aws | us-east-2 | | aws | us-west-1 | | aws | us-west-2 | | aws | ca-central-1 | | aws | ap-south-1 | | aws | ap-northeast-1 | | aws | ap-northeast-2 | | aws | ap-northeast-3 | | aws | cn-north-1 | | aws | cn-northwest-1 | | aws | eu-north-1 | | aws | eu-central-1 | | aws | eu-west-1 | | aws | eu-west-2 | | aws | eu-west-3 | | aws | me-south-1 | | aws | sa-east-1 | | do | nyc3 | | do | ams3 | | do | spg1 | | do | fra1 | | vultr | ewr1 | | yandex | ru-central1 | #### Add/Update Provider ```html POST /provider ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "name": "aws", "access_key": "AKIAIOSFODNN7EXAMPLE", "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" } ``` will return: `200` or `500` or `400` or `401` #### Get URL Object ```html GET /files/{region}/{bucket}/{object_path} ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test" } ``` will return: `200` or `500` or `400` or `401` 200 - will return a file stream with the proper content-type #### Get Object ```html POST /get ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test" } ``` will return: `200` or `500` or `400` or `401` 200 - will return a file stream with the proper content-type #### Put Object ```html POST /put ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test", "file": "dGhpc2lzYXN0cmluZw==" } ``` - `file` is a base64 encoded binary file - this is doable with the FileReader class in the browser will return: the provider's status code (e.g. `200`, `400`) - note: this action will overwrite a file if the file is the same name #### Delete Object ```html POST /del ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "object_path": "/test.pdf", "region": "wa-us-east-1", "bucket": "test" } ``` will return: the provider's status code (e.g. `204`, `409`) #### Create Bucket ```html POST /create_bucket ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "region": "wa-us-east-1", "bucket": "test" } ``` will return: `200` with no body response or the provider's status code (e.g. `409`) and the provider's error ```json { "error": "provider's response as an xml string" } ``` - note: if the bucket already exists the request will return 200 #### Delete Bucket ```html POST /delete_bucket ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "region": "wa-us-east-1", "bucket": "test" } ``` will return: the provider's status code (e.g. `204`, `409`) - note: all objects in the bucket have to be deleted before the bucket can be deleted (based on provider's rules) #### List Bucket ```html POST /list_bucket ``` - authenticated endpoint (Authorization: Bearer {jwt}) example: ```json { "region": "wa-us-east-1", "bucket": "test", "prefix": "/", "delimiter": "/" } ``` - `delimiter` is an optional field will return: `200`, `400`, `500`, `401` 200 - will return an array of objects ```json { "objects": [{ "key": "/test.pdf", "e_tag": "33a64df551425fcc55e4d42a148795d9f25f89d4", "storage_class": "STANDARD", "size": 100, "last_modified": "2021-04-14T22:34:04.000Z" }] } ``` #### Health Check ```html GET or HEAD / ``` - public endpoint will return: `200` ### Install ``` cargo install files ``` - the `origin` can be passed in as a flag - default `*` - the `port` can be passed in as a flag - default `9999` - can only be set for unsecure connections - the `secure` flag for https can be true or false - default `false` - the `auto_cert` flag for an autorenewing LetsEncrypt SSL cert can be true or false - requires a resolvable domain - default `true` - the `key_path` flag when `auto_cert` is `false` to set the SSL key path for your own cert - default `certs/private_key.pem` - the `cert_path` flag when `auto_cert` is `false` to set the SSL cert path for your own cert - default `certs/chain.pem` - the `certs` flag is the storage path of LetsEncrypt certs - default `certs` - the `db` flag is the path where the embedded database will be saved - default `db` - the `domain` flag is the domain name (e.g. api.broker.com) of the domain you want to register with LetsEncrypt - must be fully resolvable - the `broker` flag is the broker domain/ip/port of the broker server - default `http://localhost:8080` - production example: `./files --secure="true" --domain="files.broker.com" --broker="https://broker.broker.com"` ### Service There is an example `systemctl` service for Ubuntu called `files.service` in the code ### TechStack * [Tide](https://crates.io/crates/tide) * [RocksDB](https://crates.io/crates/rocksdb) ### Inspiration * [Broker](https://crates.io/crates/broker)