{"id":284,"date":"2021-01-10T02:22:17","date_gmt":"2021-01-10T02:22:17","guid":{"rendered":"https:\/\/mri.sbollmann.net\/?p=284"},"modified":"2021-08-03T06:05:05","modified_gmt":"2021-08-03T06:05:05","slug":"connect-aarnet-cloudstor-with-osf","status":"publish","type":"post","link":"https:\/\/mri.sbollmann.net\/index.php\/2021\/01\/10\/connect-aarnet-cloudstor-with-osf\/","title":{"rendered":"Connect AARNET Cloudstor with OSF"},"content":{"rendered":"\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"Connect Aarnet Cloudstor with OSF\" width=\"750\" height=\"422\" src=\"https:\/\/www.youtube.com\/embed\/GtyXKJ_Kbj0?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p><a rel=\"noreferrer noopener\" href=\"https:\/\/cloudstor.aarnet.edu.au\/\" target=\"_blank\">Cloudstor <\/a>is a wonderful storage service providing 1TB of owncloud storage for researchers in Australia. It&#8217;s like dropbox, but your data is in Australia, very fast to access from aarnet institutions and they provide a <a rel=\"noreferrer noopener\" href=\"https:\/\/news.aarnet.edu.au\/data-analysis-in-cloudstor-swan-is-ready-for-researchers\/\" target=\"_blank\">jupyter notebook service SWAN<\/a> that can directly access this data \ud83d\ude42 <\/p>\n\n\n\n<p>Wouldn&#8217;t it be nice to make your research data publicly available from this? But, how inconvenient are the links for public access (<a rel=\"noreferrer noopener\" href=\"https:\/\/cloudstor.aarnet.edu.au\/plus\/s\/V7puKkVut34J5hP\" target=\"_blank\">https:\/\/cloudstor.aarnet.edu.au\/plus\/s\/V7puKkVut34J5hP<\/a>) if you want to share data in a research paper? Sharing files like this doesn&#8217;t give users a <a rel=\"noreferrer noopener\" href=\"https:\/\/ardc.edu.au\/collaborations\/fair-principles\/\" data-type=\"URL\" data-id=\"https:\/\/ardc.edu.au\/collaborations\/fair-principles\/\" target=\"_blank\">FAIR <\/a>experience. And what if Cloudstor goes away one day? Then you have a non-working link in a paper &#8230; not great! And have you tried downloading multiples files from the command line from Cloudstor (e.g. in a jupyter notebook)? There is no easy way of programmatically interacting with your storage. <\/p>\n\n\n\n<p>These and many other problems <a rel=\"noreferrer noopener\" href=\"https:\/\/osf.io\/\" target=\"_blank\">OSF.io<\/a> can solve for us! OSF is a free, open platform to support research and enable collaboration (+a longterm fund to preserve data for 50 years+). It connects to multiple cloud services (Amazon S3, Box, Dropbox, GitHub, Google Drive, ownCloud, OneDrive, GitLab, Bitbucket, Dataverse, figshare: <a rel=\"noreferrer noopener\" href=\"https:\/\/docs.google.com\/spreadsheets\/d\/1TlxeshcWd1HpKDoI2jF-m8iolcXMsbehpb-o1IZ9Ez4\/edit#gid=0\" target=\"_blank\">Feature Comparison Chart &#8211; Storage Add-ons &#8211; Google Sheets<\/a>) and combines them nicely and provides a findable and accessible interface to all of them. OSF also offers a command line interface to interact with all these in a consistent fashion (for an example how to use this in a jupyter notebook <a rel=\"noreferrer noopener\" href=\"https:\/\/mri.sbollmann.net\/index.php\/2020\/05\/27\/google-colab-osf\/\" data-type=\"post\" data-id=\"1\" target=\"_blank\">see here<\/a>)! OSF also has a snapshot feature (under Registration), that preserves your data on theses services in time (great for our paper in case Cloudstor stops working). <\/p>\n\n\n\n<p>Let&#8217;s see how to connect our aarnet cloudstor account with OSF \ud83d\ude42<\/p>\n\n\n\n<p>First we have to <a rel=\"noreferrer noopener\" href=\"https:\/\/osf.io\/register\/\" target=\"_blank\">create an OSF account<\/a>. Make sure to create a &#8220;pure&#8221; OSF account and don&#8217;t login via ORCID or through your institution as I had problems with authenticating from the command line client otherwise!<\/p>\n\n\n\n<p>Click on your Account icon and go to settings:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"226\" height=\"213\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image.png\" alt=\"\" class=\"wp-image-285\"\/><\/figure>\n\n\n\n<p> Then navigate to Configure add-on accounts:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"401\" height=\"338\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-1.png\" alt=\"\" class=\"wp-image-286\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-1.png 401w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-1-300x253.png 300w\" sizes=\"auto, (max-width: 401px) 100vw, 401px\" \/><\/figure>\n\n\n\n<p>AARNET Cloudstor is based on ownCloud so we have to Connect to this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"910\" height=\"75\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-2.png\" alt=\"\" class=\"wp-image-287\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-2.png 910w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-2-300x25.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-2-768x63.png 768w\" sizes=\"auto, (max-width: 910px) 100vw, 910px\" \/><\/figure>\n\n\n\n<p>To get the login credentials we first need to create an App password in Cloudstor. For this go to your cloudstor settings -&gt; Security App passwords : <a rel=\"noreferrer noopener\" href=\"https:\/\/cloudstor.aarnet.edu.au\/plus\/settings\/personal?sectionid=security\" target=\"_blank\">Settings &#8211; CloudStor (aarnet.edu.au)<\/a><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"235\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-4-1024x235.png\" alt=\"\" class=\"wp-image-289\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-4-1024x235.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-4-300x69.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-4-768x177.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-4.png 1118w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>now create a new app password for OSF:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"996\" height=\"176\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-6.png\" alt=\"\" class=\"wp-image-291\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-6.png 996w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-6-300x53.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-6-768x136.png 768w\" sizes=\"auto, (max-width: 996px) 100vw, 996px\" \/><\/figure>\n\n\n\n<p>Copy this username and Password\/Token to OSF and add your Cloudstor URL https:\/\/cloudstor.aarnet.edu.au\/plus\/ and hit save.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"898\" height=\"362\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-3.png\" alt=\"\" class=\"wp-image-288\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-3.png 898w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-3-300x121.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-3-768x310.png 768w\" sizes=\"auto, (max-width: 898px) 100vw, 898px\" \/><\/figure>\n\n\n\n<p>Now we can use this storage in an OSF project. Let&#8217;s create a new project and test this. In your OSF project navigate to Add-ons -&gt; Select Add-ons:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"336\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7-1024x336.png\" alt=\"\" class=\"wp-image-292\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7-1024x336.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7-300x98.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7-768x252.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7-1536x504.png 1536w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-7.png 1573w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Now enable ownCloud and confirm that OSF can access Files in Cloudstor:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"598\" height=\"778\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-8.png\" alt=\"\" class=\"wp-image-293\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-8.png 598w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-8-231x300.png 231w\" sizes=\"auto, (max-width: 598px) 100vw, 598px\" \/><\/figure>\n\n\n\n<p>After this we have to Import the Account from Profile:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"91\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-9-1024x91.png\" alt=\"\" class=\"wp-image-294\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-9-1024x91.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-9-300x27.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-9-768x69.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-9.png 1153w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>and we should see our Cloudstor content. We can now select a subfolder we want to use in OSF:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"408\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-11-1024x408.png\" alt=\"\" class=\"wp-image-296\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-11-1024x408.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-11-300x120.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-11-768x306.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-11.png 1201w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>If you get an error try to select an empty folder, because I had sometimes problems when there where already files in there (it doesn&#8217;t always happen and for me this bug only appears when I create files in the owncloud webinterface &#8211; but uploaded files usually work). Now we can use OSF to upload files to Cloudstor:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"54\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12-1024x54.png\" alt=\"\" class=\"wp-image-297\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12-1024x54.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12-300x16.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12-768x41.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12-1536x82.png 1536w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-12.png 1543w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p> and they automagically appear in cloudstor:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"150\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-13-1024x150.png\" alt=\"\" class=\"wp-image-298\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-13-1024x150.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-13-300x44.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-13-768x112.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-13.png 1442w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Now we can create a DOI to our project and cite this in our paper (e.g. http:\/\/doi.org\/10.17605\/OSF.IO\/Y5CQ9). How good is that? The DOI will always point to this project and if cloudstor one day goes away we can simply move our data to the next storage provider (or use the snapshotting feature of OSF). <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"542\" height=\"115\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-15.png\" alt=\"\" class=\"wp-image-300\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-15.png 542w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-15-300x64.png 300w\" sizes=\"auto, (max-width: 542px) 100vw, 542px\" \/><\/figure>\n\n\n\n<p>Even better: We can register our study and OSF will preserve an exact snapshot of all files and give us a DOI to this immutable preserved dataset \ud83d\ude42 (this snapshot will also include files from external storages backends!)<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"461\" src=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-1024x461.png\" alt=\"\" class=\"wp-image-301\" srcset=\"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-1024x461.png 1024w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-300x135.png 300w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-768x346.png 768w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-1536x692.png 1536w, https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-16-2048x922.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>I hope this was helpful \ud83d\ude42 Let me know in the comments if you run into any problems.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Cloudstor is a wonderful storage service providing 1TB of owncloud storage for researchers in Australia. It&#8217;s like dropbox, but your data is in Australia, very fast to access from aarnet institutions and they provide a jupyter notebook service SWAN that can directly access this data \ud83d\ude42 Wouldn&#8217;t it be nice [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":293,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,2,28,27,7,3],"tags":[],"class_list":["post-284","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloudstor","category-code","category-data-management","category-osf","category-publishing","category-reproducibility"],"jetpack_featured_media_url":"https:\/\/mri.sbollmann.net\/wp-content\/uploads\/2021\/01\/image-8.png","_links":{"self":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/284","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/comments?post=284"}],"version-history":[{"count":2,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/284\/revisions"}],"predecessor-version":[{"id":309,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/posts\/284\/revisions\/309"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/media\/293"}],"wp:attachment":[{"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/media?parent=284"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/categories?post=284"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mri.sbollmann.net\/index.php\/wp-json\/wp\/v2\/tags?post=284"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}