반응형
rclone이라는 오픈소스는 다양한 스토리지 EndPoint를 연결해서 로컬 디스크 처럼 마운트 할 수 있는 기능을 제공한다. 많은 사람들이 구글 드라이브를 연동해서 쓰지만, 지원하는 스토리지 중 오라클 클라우드의 Object Storage가 있어 서로 다른 VM에서 동일한 Object Storage를 연동하는 것을 테스트 해본다.
1. rclone
위의 지원 가능 Storage에서 Oracle Cloud Object Storage를 선택하면 Swift항목으로 넘어가는데, 이 것은 Oracle Cloud Object Storage가 Openstack의 swift와 동일하게 사용할 수 있어서 swift api를 사용하면 되는 듯 하다.
하지만 Oracle Object Storage는 AWS S3와 호환되기 때문에 여기에서는 S3 Compatible API를 사용해서 적용해본다.
2. Object Storage 생성
새로운 bucket을 하나 생성한다.
3. rclone 설치
위 링크에서 해당되는 플랫폼의 압축파일을 다운로드 한다. 테스트는 ubuntu 20.04에서 진행하기 때문에 Linux AMD64로 다운한다.
$ wget https://downloads.rclone.org/v1.51.0/rclone-v1.51.0-linux-amd64.zip
$ unzip rclone-v1.51.0-linux-amd64.zip
Archive: rclone-v1.51.0-linux-amd64.zip
creating: rclone-v1.51.0-linux-amd64/
inflating: rclone-v1.51.0-linux-amd64/rclone.1
inflating: rclone-v1.51.0-linux-amd64/README.txt
inflating: rclone-v1.51.0-linux-amd64/README.html
extracting: rclone-v1.51.0-linux-amd64/git-log.txt
inflating: rclone-v1.51.0-linux-amd64/rclone
$ cd rclone-v1.51.0-linux-amd64/
$ sudo mv rclone /usr/local/bin/.
$ rclone
Usage:
rclone [flags]
rclone [command]
Available Commands:
about Get quota information from the remote.
authorize Remote authorization.
cachestats Print cache stats for a remote
cat Concatenates any files and sends them to stdout.
check Checks the files in the source and destination match.
cleanup Clean up the remote if possible
config Enter an interactive configuration session.
copy Copy files from source to dest, skipping already copied
copyto Copy files from source to dest, skipping already copied
copyurl Copy url content to dest.
cryptcheck Cryptcheck checks the integrity of a crypted remote.
cryptdecode Cryptdecode returns unencrypted file names.
dbhashsum Produces a Dropbox hash file for all the objects in the path.
dedupe Interactively find duplicate files and delete/rename them.
delete Remove the contents of path.
deletefile Remove a single file from remote.
genautocomplete Output completion script for a given shell.
gendocs Output markdown docs for rclone to the directory supplied.
hashsum Produces an hashsum file for all the objects in the path.
help Show help for rclone commands, flags and backends.
link Generate public link to file/folder.
listremotes List all the remotes in the config file.
ls List the objects in the path with size and path.
lsd List all directories/containers/buckets in the path.
lsf List directories and objects in remote:path formatted for parsing
lsjson List directories and objects in the path in JSON format.
lsl List the objects in path with modification time, size and path.
md5sum Produces an md5sum file for all the objects in the path.
mkdir Make the path if it doesn't already exist.
mount Mount the remote as file system on a mountpoint.
move Move files from source to dest.
moveto Move file or directory from source to dest.
ncdu Explore a remote with a text based user interface.
obscure Obscure password for use in the rclone.conf
purge Remove the path and all of its contents.
rc Run a command against a running rclone.
rcat Copies standard input to file on remote.
rcd Run rclone listening to remote control commands only.
rmdir Remove the path if empty.
rmdirs Remove empty directories under the path.
serve Serve a remote over a protocol.
settier Changes storage class/tier of objects in remote.
sha1sum Produces an sha1sum file for all the objects in the path.
size Prints the total size and number of objects in remote:path.
sync Make source and dest identical, modifying destination only.
touch Create new file or change file modification time.
tree List the contents of the remote in a tree like fashion.
version Show the version number.
Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.4. rclone 설정
rclone에서 S3를 지원하기 때문에 Oracle Cloud에서 S3형태로 Access할 수 있는 access_key와 secret)key를 생성한다.
- Generate Secret Key
- 생성되고 나면 secret key가 보이고 해당 키는 이후에는 볼 수 없기 때문에 복사해 둔다.
- access key는 생성되고 나면 화면에서 확인 할 수 있다.
- rclone.conf는 ~/.config/rclone/rclone.conf 로 생성한다.
## 설정 파일 템플릿
[oci-dest]
type = s3
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_ACCESS_SECRET_KEY
region = YOUR_REGION_IDENTIFIER
endpoint = https://YOUR_NAMESPACE.compat.objectstorage.YOUR_REGION_IDENTIFIER.oraclecloud.com
## 예제
[oci-dest]
type = s3
env_auth = false
access_key_id = c8a2b587e1e29bfb8b4....
secret_access_key = yrukorkZWVdD7pIY...
region = ap-seoul-1
endpoint = https://<namespace>.compat.objectstorage.ap-seoul-1.oraclecloud.com5. Test
rclone명령어를 통해 access할 수 있다.
- bucket조회
$ rclone lsd oci-dest:
-1 2020-05-12 02:40:19 -1 alex-bucket
-1 2020-01-14 02:48:29 -1 oss-object-storage
-1 2020-05-13 02:28:35 -1 shared-bucket- 파일 복사
$ rclone copy ./rclone-v1.51.0-linux-amd64.zip oci-dest:/shared-bucket/.
$ rclone ls oci-dest:/shared-bucket
11913756 rclone-v1.51.0-linux-amd64.zip
- mount oracle object storage
## 일반적인 마운트
rclone mount oci-dest:shared-bucket /home/ubuntu/oracle --daemon
## 옵션 적용
rclone mount oci-dest:shared-bucket /home/ubuntu/oracle --allow-non-empty --buffer-size 32M --use-mmap --dir-cache-time 72h --drive-chunk-size 16M --timeout 1h --vfs-cache-mode minimal --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit 1G --cache-db-purge --daemon- unmount
fusermount -uz <마운트 포인트>- 스토리지 write 테스트
$ dd if=/dev/zero bs=1G count=1 of=write_1GB_test
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.2844 s, 70.3 MB/s반응형
