Cheap file storage online?? I think AWS S3 immediately jumps to mind for most people :P
I’m going to be using it to store my training images for GAN. I’ve never set up one before, so I was pleasantly surprised to find that it wasn’t too painful at all.
For people who want to start your own, read on!
So I began here, with the in-house AWS tutorial - it’s pretty good!
Once your bucket is up and running, how do you upload your files via code (not the console)?
In comes boto3!
Use the quickstart guide here:
pip install awscli, AND install AWS CLI (AWS command line). I wanted to see if there were other options without using AWS CLI, but… nope. so just install it haha.
aws configure, input your access key id and access key, and for people who selected Singapore as their region, input
ap-southeast-1 (you can find the relevant names for other regions here if you need).
There was a next field (I can’t remember!!) but just press enter to skip it if you don’t need it (that’s what I did) and you’re done with the configuration!!
Get yourself into jupyter notebook, and here we go:
import boto3 s3 = boto3.resource('s3') # tells boto which aws resource you are using for bucket in s3.buckets.all(): # prints out all the bucket names in the s3 print(bucket.name) s3.Bucket('picture_bucket').upload_file('/mypics/picture.jpg','picture.jpg') # uploads the file into the bucket from the given file path, and saves it as the name specified.
DONE DONE DONEEEEE
now to get that spider going…