AWS S3- Simple Storage Service
Amazon S3 is one of the main building blocks of AWS.
It’s advertised as ”infinitely scaling” storage.
It’s widely popular and deserves its own section.
Many websites use AWS S3 as a backbone.
Many AWS services uses AWS S3 as an integration as well.
AWS S3 Overview — Buckets
Amazon S3 allows people to store objects (files) in “buckets” (directories).
Buckets must have a globally unique name.
Buckets are defined at the region level.
Files can be 0 bytes to 5TB.
Unlimited storage, Files are stored in buckets.
When you upload a file to S3, will recieve a HTTP code 200 if upload successful.
Naming convention:
• No uppercase
• No underscore
• 3–63 characters long
• Not an IP
• Must start with lowercase letter or number
S3 provides developers & IT teams with secure, durable, highly scalable object storage, easy to use with simple web services interfaces to store & retrieve any amount of data.
AWS S3 Overview — Objects
• Objects (files) have a Key. The key is the FULL path:
• <my_bucket>/my_file.txt
• <my_bucket>/my_folder1/another_folder/my_file.txt
• There’s no concept of “directories” within buckets (although the UI will trick you to think otherwise).
• Just keys with very long names that contain slashes (“/”)
• Object Values are the content of the body:
• Max Size is 5TB
• If uploading more than 5GB, must use “multi-part upload”
• Metadata (list of text key / value pairs — system or user metadata).
• Tags (Unicode key / value pair — up to 10) — useful for security / lifecycle.
• Version ID (if versioning is enabled).
AWS S3 -Versioning
• You can version your files in AWS S3.
• It is enabled at the bucket level.
• Same key overwrite will increment the “version”: 1, 2, 3….
• It is best practice to version your buckets.
• Protect against unintended deletes (ability to restore a version).
• Easy roll back to previous version.
• Any file that is not versioned prior to enabling versioning will have version “null”.
S3 Encryption for Objects
• There are 4 methods of encrypting objects in S3.
• SSE-S3: encrypts S3 objects using keys handled & managed by AWS.
• SSE-KMS: leverage AWS Key Management Service to manage encryption keys.
• SSE-C: when you want to manage your own encryption keys.
• Client Side Encryption.
SSE-S3
• SSE-S3: encryption using keys handled & managed by AWS S3
• Object is encrypted server side
• AES-256 encryption type
• Must set header: “x-amz-server-side-encryption”: “AES256”
SSE-KMS
• SSE-KMS: encryption using keys handled & managed by KMS
• KMS Advantages: user control + audit trail
• Object is encrypted server side
• Must set header: “x-amz-server-side-encryption”: ”aws:kms”
SSE-C
• SSE-C: server-side encryption using data keys fully managed by the customer outside of AWS
• Amazon S3 does not store the encryption key you provide
• HTTPS must be used
• Encryption key must provided in HTTP headers, for every HTTP request made
Client Side Encryption
• Client library such as the Amazon S3 Encryption Client
• Clients must encrypt data themselves before sending to S3
• Clients must decrypt data themselves when retrieving from S3
• Customer fully manages the keys and encryption cycle
S3- Storage Tiers/Classes:
Encryption in transit (SSL)
AWS S3 exposes:
• HTTP endpoint: non encrypted
• HTTPS endpoint: encryption in flight
• You’re free to use the endpoint you want, but HTTPS is recommended
• HTTPS is mandatory for SSE-C
• Encryption in flight is also called SSL / TLS
S3 Security
User based
• IAM policies — which API calls should be allowed for a specific user from IAM console
Resource Based
• Bucket Policies — bucket wide rules from the S3 console — allows cross account
• Object Access Control List (ACL) — finer grain
• Bucket Access Control List (ACL) — less common
S3 Bucket Policies
JSON based policies
• Resources: buckets and objects
• Actions: Set of API to Allow or Deny
• Effect: Allow / Deny
• Principal: The account or user to apply the policy to
Use S3 bucket for policy to:
• Grant public access to the bucket
• Force objects to be encrypted at upload
• Grant access to another account (Cross Account)
S3 Security:
Networking:
• Supports VPC Endpoints (for instances in VPC without www internet)
Logging and Audit:
• S3 access logs can be stored in other S3 bucket
• API calls can be logged in AWS CloudTrail
User Security:
• MFA (multi factor authentication) can be required in versioned buckets to
delete objects
• Signed URLs: URLs that are valid only for a limited time (ex: premium video
service for logged in users)
S3 Websites
• S3 can host static websites and have them accessible on the www
• The website URL will be:
• <bucket-name>.s3-website-<AWS-region>.amazonaws.com
OR
• <bucket-name>.s3-website.<AWS-region>.amazonaws.com
• If you get a 403 (Forbidden) error, make sure the bucket policy allows public reads!
S3 CORS
• If you request data from another S3 bucket, you need to enable CORS
• Cross Origin Resource Sharing allows you to limit the number of websites that can request your files in S3 (and limit your costs)
AWS S3 — Consistency Model
• Read after write consistency for PUTS of new objects
• As soon as an object is written, we can retrieve it
ex: (PUT 200 -> GET 200)
• This is true, except if we did a GET before to see if the object existed
ex: (GET 404 -> PUT 200 -> GET 404) — eventually consistent
• Eventual Consistency for DELETES and PUTS of existing objects
• If we read an object after updating, we might get the older version
ex: (PUT 200 -> PUT 200 -> GET 200 (might be older version))
• If we delete an object, we might still be able to retrieve it for a short time
ex: (DELETE 200 -> GET 200)
S3- Life Cycle Management Lab:
Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.
To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇
