{"id":368,"date":"2021-08-03T23:35:55","date_gmt":"2021-08-03T23:35:55","guid":{"rendered":"https:\/\/mri.sbollmann.net\/?p=368"},"modified":"2021-08-03T23:59:48","modified_gmt":"2021-08-03T23:59:48","slug":"cloudstor-and-high-performance-computing-systems","status":"publish","type":"post","link":"https:\/\/mri.sbollmann.net\/index.php\/2021\/08\/03\/cloudstor-and-high-performance-computing-systems\/","title":{"rendered":"CloudStor and High Performance Computing Systems"},"content":{"rendered":"\n

When using a High Performance Computing System you also might have run into the problem of how to get data in and out of this system. <\/p>\n\n\n\n

We are very lucky in Australia to have CloudStor<\/a>, which is Aarnets hosted OwnCloud version with 1TB of storage available for every researcher and we can use it to move data between different computing systems.<\/p>\n\n\n\n

The only problem is that on an HPC you can\u2019t open a browser and you also can\u2019t install the owncloud client \u2013 so what other options exist?<\/p>\n\n\n\n

There is a wonderful tool called rclone<\/a>, which interfaces with all cloud storage providers out there \ud83d\ude42<\/p>\n\n\n\n

You can find a detailed description how to connect rclone to CloudStor<\/a>, so let’s go through this step by step and set it up together:<\/p>\n\n\n\n

Connect to your favorite HPC and install rclone on there. The official install instructions for rclone are a bit misleading as we will not be able to run as sudo or use their install script without sudo or install rpm or apt packages – so if you try any of the official instructions on an HPC you will see errors similar to this:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Let’s install it in our home directory \ud83d\ude42<\/p>\n\n\n\n

cd\nmkdir tools\ncd tools\nwget https:\/\/downloads.rclone.org\/rclone-current-linux-amd64.zip \nunzip rclone-current-linux-amd64.zip \nrm rclone-current-linux-amd64.zip \ncd rclone-*\necho \"export PATH=\\$PATH:$PWD\" >> ~\/.bashrc\nsource ~\/.bashrc\nrclone config<\/code><\/pre>\n\n\n\n
Create a new remote: n<\/p>\n\n\n\n
Provide a name for the remote: CloudStor<\/p>\n\n\n\n
For the “Storage” option choose: webdav<\/p>\n\n\n\n
As “url” set: https:\/\/cloudstor.aarnet.edu.au\/plus\/remote.php\/webdav\/<\/p>\n\n\n\n
As “vendor” set OwnCloud: 2<\/p>\n\n\n\n
Set your CloudStor username after generating an access token<\/a>: <\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Choose to type in your own password: y<\/p>\n\n\n\n
Enter the Password \/ Token from the CloudStor App passwords page and confirm it again:<\/p>\n\n\n\n
Leave blank the bearer_token: <hit Enter><\/p>\n\n\n\n
No advanced config necessary:  <hit Enter> <\/p>\n\n\n\n
accept the configuration:  <hit Enter> <\/p>\n\n\n\n
Quit the config: q<\/p>\n\n\n\n
<\/p>\n\n\n\n
Now we can download <\/strong>data to the HPC easily:<\/p>\n\n\n\n
rclone copy --progress --transfers 8 CloudStor:\/raw-data-for-science-paper .<\/code><\/pre>\n\n\n\n
or upload <\/strong>data to CloudStor:<\/p>\n\n\n\n
rclone copy --progress --transfers 8 . CloudStor:\/hpc-data-processed<\/code><\/pre>\n\n\n\n
If you need to upload lots of data, there is a very nice wrapper script from AARNET that tweaks a few settings and checks if all files have been transferred: https:\/\/github.com\/AARNet\/copyToCloudstor<\/a><\/strong><\/p>\n\n\n\n
git clone https:\/\/github.com\/AARNet\/copyToCloudstor \ncd copyToCloudstor\n.\/copyToCloudstor.sh ~\/scratch\/data\/ CloudStor:\/all-my-important-HPC-data-for-nature-paper\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
When using a High Performance Computing System you also might have run into the problem of how to get data in and out of this system. We are very lucky in Australia to have CloudStor, which is Aarnets hosted OwnCloud version with 1TB of storage available for every researcher and […]<\/p>\n","protected":false},"author":1,"featured_media":369,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,28,31,26],"tags":[],"class_list":["post-368","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloudstor","category-data-management","category-high-performance-computing","category-tutorials"],"jetpack_featured_media_url":"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/08\/image-5.png","_links":{"self":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/comments?post=368"}],"version-history":[{"count":8,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/368\/revisions"}],"predecessor-version":[{"id":380,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/368\/revisions\/380"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/media\/369"}],"wp:attachment":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/media?parent=368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/categories?post=368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/tags?post=368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}