Getting started with AWS S3

29 Dec 2017


Cheap file storage online?? I think AWS S3 immediately jumps to mind for most people :P

I’m going to be using it to store my training images for GAN. I’ve never set up one before, so I was pleasantly surprised to find that it wasn’t too painful at all.

For people who want to start your own, read on!

So I began here, with the in-house AWS tutorial - it’s pretty good!

Once your bucket is up and running, how do you upload your files via code (not the console)?

In comes boto3!

Use the quickstart guide here:

Do your pip install awscli, AND install AWS CLI (AWS command line). I wanted to see if there were other options without using AWS CLI, but… nope. so just install it haha.

Type in aws configure, input your access key id and access key, and for people who selected Singapore as their region, input ap-southeast-1 (you can find the relevant names for other regions here if you need).

There was a next field (I can’t remember!!) but just press enter to skip it if you don’t need it (that’s what I did) and you’re done with the configuration!!

Get yourself into jupyter notebook, and here we go:

import boto3

s3 = boto3.resource('s3') # tells boto which aws resource you are using

for bucket in s3.buckets.all(): # prints out all the bucket names in the s3

s3.Bucket('picture_bucket').upload_file('/mypics/picture.jpg','picture.jpg') # uploads the file into the bucket from the given file path, and saves it as the name specified.


now to get that spider going…