본문 바로가기
Tech/Linux

rclone으로 Oracle Object Storage 사용하기

by 타이호 2020. 5. 18.

rclone이라는 오픈소스는 다양한 스토리지 EndPoint를 연결해서 로컬 디스크 처럼 마운트 할 수 있는 기능을 제공한다. 많은 사람들이 구글 드라이브를 연동해서 쓰지만, 지원하는 스토리지 중 오라클 클라우드의 Object Storage가 있어 서로 다른 VM에서 동일한 Object Storage를 연동하는 것을 테스트 해본다.

 

1. rclone

www.rclone.org

위의 지원 가능 Storage에서 Oracle Cloud Object Storage를 선택하면 Swift항목으로 넘어가는데, 이 것은 Oracle Cloud Object Storage가 Openstack의 swift와 동일하게 사용할 수 있어서 swift api를 사용하면 되는 듯 하다.

하지만 Oracle Object Storage는 AWS S3와 호환되기 때문에 여기에서는 S3 Compatible API를 사용해서 적용해본다.

 

2. Object Storage 생성

새로운 bucket을 하나 생성한다.

 

 

3. rclone 설치

www.rclone.org/downloads/

위 링크에서 해당되는 플랫폼의 압축파일을 다운로드 한다. 테스트는 ubuntu 20.04에서 진행하기 때문에 Linux AMD64로 다운한다.

$ wget https://downloads.rclone.org/v1.51.0/rclone-v1.51.0-linux-amd64.zip
$ unzip rclone-v1.51.0-linux-amd64.zip 
Archive:  rclone-v1.51.0-linux-amd64.zip
   creating: rclone-v1.51.0-linux-amd64/
  inflating: rclone-v1.51.0-linux-amd64/rclone.1  
  inflating: rclone-v1.51.0-linux-amd64/README.txt  
  inflating: rclone-v1.51.0-linux-amd64/README.html  
 extracting: rclone-v1.51.0-linux-amd64/git-log.txt  
  inflating: rclone-v1.51.0-linux-amd64/rclone
$ cd rclone-v1.51.0-linux-amd64/
$ sudo mv rclone /usr/local/bin/.
$ rclone
Usage:
  rclone [flags]
  rclone [command]

Available Commands:
  about           Get quota information from the remote.
  authorize       Remote authorization.
  cachestats      Print cache stats for a remote
  cat             Concatenates any files and sends them to stdout.
  check           Checks the files in the source and destination match.
  cleanup         Clean up the remote if possible
  config          Enter an interactive configuration session.
  copy            Copy files from source to dest, skipping already copied
  copyto          Copy files from source to dest, skipping already copied
  copyurl         Copy url content to dest.
  cryptcheck      Cryptcheck checks the integrity of a crypted remote.
  cryptdecode     Cryptdecode returns unencrypted file names.
  dbhashsum       Produces a Dropbox hash file for all the objects in the path.
  dedupe          Interactively find duplicate files and delete/rename them.
  delete          Remove the contents of path.
  deletefile      Remove a single file from remote.
  genautocomplete Output completion script for a given shell.
  gendocs         Output markdown docs for rclone to the directory supplied.
  hashsum         Produces an hashsum file for all the objects in the path.
  help            Show help for rclone commands, flags and backends.
  link            Generate public link to file/folder.
  listremotes     List all the remotes in the config file.
  ls              List the objects in the path with size and path.
  lsd             List all directories/containers/buckets in the path.
  lsf             List directories and objects in remote:path formatted for parsing
  lsjson          List directories and objects in the path in JSON format.
  lsl             List the objects in path with modification time, size and path.
  md5sum          Produces an md5sum file for all the objects in the path.
  mkdir           Make the path if it doesn't already exist.
  mount           Mount the remote as file system on a mountpoint.
  move            Move files from source to dest.
  moveto          Move file or directory from source to dest.
  ncdu            Explore a remote with a text based user interface.
  obscure         Obscure password for use in the rclone.conf
  purge           Remove the path and all of its contents.
  rc              Run a command against a running rclone.
  rcat            Copies standard input to file on remote.
  rcd             Run rclone listening to remote control commands only.
  rmdir           Remove the path if empty.
  rmdirs          Remove empty directories under the path.
  serve           Serve a remote over a protocol.
  settier         Changes storage class/tier of objects in remote.
  sha1sum         Produces an sha1sum file for all the objects in the path.
  size            Prints the total size and number of objects in remote:path.
  sync            Make source and dest identical, modifying destination only.
  touch           Create new file or change file modification time.
  tree            List the contents of the remote in a tree like fashion.
  version         Show the version number.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

 

4. rclone 설정

rclone에서 S3를 지원하기 때문에 Oracle Cloud에서 S3형태로 Access할 수 있는 access_key와 secret)key를 생성한다.

  • Generate Secret Key

  • 생성되고 나면 secret key가 보이고 해당 키는 이후에는 볼 수 없기 때문에 복사해 둔다. 

  • access key는 생성되고 나면 화면에서 확인 할 수 있다.

  • rclone.conf는 ~/.config/rclone/rclone.conf 로 생성한다.
## 설정 파일 템플릿
[oci-dest]
type = s3
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_ACCESS_SECRET_KEY
region = YOUR_REGION_IDENTIFIER
endpoint = https://YOUR_NAMESPACE.compat.objectstorage.YOUR_REGION_IDENTIFIER.oraclecloud.com

## 예제
[oci-dest]
type = s3
env_auth = false
access_key_id = c8a2b587e1e29bfb8b4....
secret_access_key = yrukorkZWVdD7pIY...
region = ap-seoul-1
endpoint = https://<namespace>.compat.objectstorage.ap-seoul-1.oraclecloud.com

5. Test

rclone명령어를 통해 access할 수 있다.

  • bucket조회
$ rclone lsd oci-dest:
          -1 2020-05-12 02:40:19        -1 alex-bucket
          -1 2020-01-14 02:48:29        -1 oss-object-storage
          -1 2020-05-13 02:28:35        -1 shared-bucket
  • 파일 복사
$ rclone copy ./rclone-v1.51.0-linux-amd64.zip oci-dest:/shared-bucket/.
$ rclone ls oci-dest:/shared-bucket
 11913756 rclone-v1.51.0-linux-amd64.zip

  • mount oracle object storage
## 일반적인 마운트
rclone mount oci-dest:shared-bucket /home/ubuntu/oracle --daemon

## 옵션 적용
rclone mount oci-dest:shared-bucket /home/ubuntu/oracle  --allow-non-empty --buffer-size 32M --use-mmap --dir-cache-time 72h --drive-chunk-size 16M  --timeout 1h  --vfs-cache-mode minimal --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 1G --cache-db-purge --daemon
  • unmount
fusermount -uz <마운트 포인트>
  • 스토리지 write 테스트
$ dd if=/dev/zero bs=1G count=1 of=write_1GB_test
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.2844 s, 70.3 MB/s

댓글0