Abstract

This whitepaper is intended for solutions architects and developers who are building solutions that will be deployed on Amazon Web Services (AWS). It provides architectural patterns on how we can build a stateless automation to copy S3 objects between AWS account and how to design systems that are secure, reliable, high performing, and cost efficient.

Introduction

   Amazon Simple Storage Service (Amazon S3) is object storage with a simple web service interface to store and retrieve any amount of data from anywhere on the web. It is designed to deliver 99.999999999% durability, and scale past trillions of objects worldwide. It is simple to move large volumes of data into or out of Amazon S3 with Amazon's cloud data migration options. Once data is stored in S3, it can be automatically tiered into lower cost, longer-term cloud storage classes like S3 Standard - Infrequent Access and Amazon Glacier for archiving.

We will be further explaining, how we can perform a copy of the objects(folder/file) uploaded to S3 bucket from one AWS account to another account.

Scenario: We have multiple AWS accounts with consolidated billing. Linked(Source) account running EC2 instances with different TimeZone upload the logs of applications to S3 for backup. The application runs daily log rotation and uploads the data to S3. The payee master(Destination) account has some log analysis application which needs the application data from all the linked(Source) account in a single S3 bucket.

Problem:  As the log rotation depends on the EC2 instance Timezone, we cannot schedule a script to sync/copy the data on a specific time between S3 Buckets.

Solution Walkthrough:

  1. When an object is uploaded to Source S3 bucket, SNS event notification associated with an S3 bucket will notify the SNS topic in source account.
  2. The SNS topic which has a lambda function subscribed to it will run the Lambda function.
  3. The Lambda function will assume the Destination Account IAM Role and copy the object from Source Bucket to Destination bucket.

Note: The S3 bucket event will have the source S3 bucket name and its object.

Solution flow diagram:

AWS Resource in Source Account:

  • IAM Role
  • S3 Bucket
  • Lambda function
  • SNS Notification

AWS Resource in Destination Account:

  • IAM Role
  • S3 Bucket

Configuration in Source AWS Account:

  1. Create an IAM role, this will be used for creating the Cloudwatch log and running Lambda function. The Role also should assume the Role of Destination IAM. Attach below AWS policy and Trust relationship for Lambda service.

 

  1. Attach Cloud watch log policy with CreateLogGroup, CreateLogStream and PutLogEvents. This policy will be used by Lambda to upload the lambda output to CloudWatch logs.

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"logs:CreateLogGroup",

"logs:CreateLogStream",

"logs:PutLogEvents"

],

"Resource": "arn:aws:logs:*:*:*"

}

]

}

 

       b. Create an inline policy to Assume the role of Destination IAM user.

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "Stmt1489133353000",

"Effect": "Allow",

"Action": [

"sts:AssumeRole"

],

"Resource": [

"arn:aws:iam::<Destination AWS Account Number>:role/<Destination Role>"

]

}

]

}

        c. Trust the Lambda in IAM Role.

 

 

2. Create S3 Bucket in Source Account, to which the logs will be uploaded.

Add below Bucket Access policy to the IAM Role created in Destination account.

{

"Version": "2008-10-17",

"Id": "Policy1398367354624",

"Statement": [

{

"Sid": "CrossAccount",

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam:: <Destination AWS Account Number>:role/<Destination Role>"

},

"Action": "s3:*",

"Resource": "arn:aws:s3:::<Source Bucket>/*"

}

]

}

 

3. Create the Lambda Function

Lambda function will assume the Role of Destination IAM Role and copy the S3 object from Source bucket to Destination.

 

  1. In the Lambda console, choose Create a Lambda function.
  2. Directly move to configure function.
  3. For Name, enter a function name. The function name should match the name of the S3 Destination Bucket.
  4. Enter a description that notes the source bucket and destination bucket used.
  5. For Runtime, choose Python 2.7.
  6. For Code entry type, choose Edit code inline.
  7. Paste the following into the code editor:

 

import urllib

import boto3

import ast

import json

print('Loading function')

 

def lambda_handler(event, context):

s3 = boto3.client('s3')

sns_message = ast.literal_eval(event['Records'][0]['Sns']['Message'])

target_bucket = context.function_name

source_bucket = str(sns_message['Records'][0]['s3']['bucket']['name'])

key = str(urllib.unquote_plus(sns_message['Records'][0]['s3']['object']['key']).decode('utf8'))

copy_source = {'Bucket':source_bucket, 'Key':key}

print "Copying %s from bucket %s to bucket %s ..." % (key, source_bucket, target_bucket)

sts_client = boto3.client('sts')

assumedRoleObject = sts_client.assume_role(

RoleArn="arn:aws:iam::<Destination Account ID>:role/<Destination Role>",

RoleSessionName="AssumeRoleSession1"

)

credentials = assumedRoleObject['Credentials']

s3 = boto3.client(

's3',

aws_access_key_id = credentials['AccessKeyId'],

aws_secret_access_key = credentials['SecretAccessKey'],

aws_session_token = credentials['SessionToken'],

)

s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)

 

    h. Select the Existing Role option and select the IAM Role created in above Step

 

4. Create SNS Topic.

The SNS topic will be used by S3 bucket. When an object is uploaded to S3 bucket, it will invoke SNS Topic. SNS is subscribed with Lambda function which will trigger the Lambda function created in the previous step.

  1. Create SNS topic in Source Account.
  2. In the SNS topic options, select Edit topic policy

In the Popup window, select the Advanced view TAB as below screenshot and update the policy provided below.

 

 

{

"Version": "2008-10-17",

"Id": "<default_policy_ID>",

"Statement": [

{

"Sid": "<default_statement_ID>",

"Effect": "Allow",

"Principal": {

"AWS": "*"

},

"Action": "SNS:Publish",

"Resource": "arn:aws:sns:us-east-1:<Source Account ID>:<Source SNS Topic Name>",

"Condition": {

"ArnLike": {

"AWS:SourceArn": "arn:aws:s3:::<Source S3 Bucket>"

}

}

}

]

}

 

c. Create the subscription for the SNS Topic with protocol select Lambda and lambda function created in the previous step as Endpoint. This will run the Lambda function when SNS Topic is invoked.

 

 

5. Create the Notification Event for Source S3 Bucket.

  1. In the S3 bucket property option select the Event option
  2. Add Notification to Event
  3. Provide the name for the Notification Event
  4. Select ObjectCreate(All) and Prefix for the object which we want to upload to Destination Bucket. This will make any object uploaded to <Bucket Name>/<Prefix> will be uploaded to destination
  5. Provide the Suffix for your log files
  6. Select SNS as Send to and SNS topic name
  7. Save the Event Notification

Note: By providing Prefix and Suffix we can clearly define the objects we want to upload from Source to Destination S3 bucket.

 

 

Configuration in Destination AWS Account:

  1. Create IAM Role, which will be used by Lambda to Copy the objects.
  1. Attach S3 Bucket Role to IAM

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"s3:GetObject"

],

"Resource": [

"arn:aws:s3:::<Source Bucket>",

"arn:aws:s3:::<Source Bucket>/*"

]

},

{

"Sid": "Stmt1489073558000",

"Effect": "Allow",

"Action": [

"s3:PutObject"

],

"Resource": [

"arn:aws:s3:::<Destination Bucket>",

"arn:aws:s3:::<Destination Bucket>/*"

  •  
  •  
  •  
  •  

b. Add the Trust Relationship Source Account IAM Role for Lambda service

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::<Source Account>:role/<Source Role>",

"Service": "lambda.amazonaws.com"

},

"Action": "sts:AssumeRole"

}

]

}

Configuration Validation:

  1. Upload an object to the source bucket.
  2. Verify that the object was copied successfully to the destination buckets.
  3. Optional: view the CloudWatch logs entry for the Lambda function execution. For a successful execution, this should look like the following screenshot.

 

Known Issue and Limitation:

  1. If the file size is greater than 5GB, replace the S3 copy command in the lambda function.

From: s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)

To     : s3.copy(copy_source, context.function_name, key)

 

       2. As of now, the LAMBDA has a timeout value of 5 minutes. If the files size is huge the lambda function used in the document will not copy the data.

Conclusion

The solution is simple and can be used for multiple use cases such as cross account-cross region replications, centralized auditing for logs, and centralized backup/achieve locations.