View Categories

Configuring a Sync to S3 Gateway

Easily push files to Amazon S3 using our Sync feature for Hosted~FTP~ Enterprise T2 and above accounts. By following rules you setup in the Sync process, you can have specified user directories sync towards your S3 server to: upload, move, delete, or upload and move files automatically. This article will cover how to setup a Sync to S3 process.

How to setup Amazon S3 with Key-based Access #

Step Instructions Reference
1 In the Amazon S3 console, create the bucket to sync from (if it does not exist yet). Click here to review the Amazon documentation for creating an S3 bucket. Click here to review the Amazon documentation for creating an S3 bucket
2 In the Amazon SQS console, confirm that you are in the same region as the S3 bucket and then create a new queue. Fill in the following:
  • Type: Choose ‘Standard’.
  • Name: Enter the name for your queue.
  • Access policy: Paste the contents of “Sync an AWS S3 bucket to HostedFTP Using Key-Based Access – Queue Access Policy Template.json” and search/replace the following missing information:
    • The region that the queue is getting created in.
    • The AWS account number containing the queue.
    • The name entered in step b above.
    • The name of the S3 bucket from step 1.
 Click here to review the Amazon documentation for creating an SQS queue.
3 In the Amazon S3 console, click on the bucket from step 1 and then enable event notifications to the SQS queue created in step 2.
  • In the ‘General configuration’ section:
    • Enter an ‘Event name’.
    • Optionally enter a ‘Prefix’ and/or ‘Suffix’.
  • In the ‘Event types’ section:
    • Check ‘All object create events’.
  • In the ‘Destination’ section:
    • Select the ‘SQS queue’ option.
    • In the ‘SQS queue’ dropdown, select the name of the queue created in step 2.
 Click here to review the Amazon documentation for enabling S3 event notifications
4 In the Amazon IAM console, create an IAM user that has permission to access the S3 bucket and the SQS queue.
  • User name: Enter the name of the user
  • Access type: Programmatic access. Download an access key / secret access key during the user creation.
  • Permissions: Attach existing policies directly. Paste the contents of “Sync an AWS S3 bucket to HostedFTP Using Key-Based Access – IAM Policy Template.json” and search/replace the following information:
    • The region that the queue is getting created in.
    • The AWS account number containing the queue.
    • The name entered in step 2 above.
    • The name of the S3 bucket from step 1.
 Click here to review the Amazon documentation for creating an IAM user.

How to setup Amazon S3 with Cross-account access #

Step Instructions Reference
1 In the Amazon S3 console, create the bucket to sync from (if it does not exist yet). Click here to review the Amazon documentation for creating an S3 bucket. Click here to review the Amazon documentation for creating an S3 bucket
2 In the Amazon SQS console, confirm that you are in the same region as the S3 bucket and then create a new queue. Fill in the following:
  • Type: Choose ‘Standard’.
  • Name: Enter the name for your queue.
  • Access policy: Paste the contents of “Sync an AWS S3 bucket to HostedFTP Using Key-Based Access – Queue Access Policy Template.json” and search/replace the following missing information:
    • The region that the queue is getting created in.
    • The AWS account number containing the queue.
    • The name entered in step b above.
    • The name of the S3 bucket from step 1.
    • The AWS account number that is running the Hosted~FTP~ software.
  • If the SQS queue is encrypted with a KMS customer master key (CMK), then paste the contents of “Sync an AWS S3 bucket to HostedFTP Using Cross-Account Access – CMK Access Policy Template.json” into the existing queue access policy and search/replace the following missing information:
    • The AWS account number that is running the Hosted~FTP~ software.
 Click here to review the Amazon documentation for creating an SQS queue.
3 In the Amazon S3 console, click on the bucket from step 1 and then enable event notifications to the SQS queue created in step 2.
  • In the ‘General configuration’ section:
    • Enter an ‘Event name’.
    • Optionally enter a ‘Prefix’ and/or ‘Suffix’.
  • In the ‘Event types’ section:
    • Check ‘All object create events’.
  • In the ‘Destination’ section:
    • Select the ‘SQS queue’ option.
    • In the ‘SQS queue’ dropdown, select the name of the queue created in step 2.
 Click here to review the Amazon documentation for enabling S3 event notifications
4 In the Amazon S3 console, add a bucket policy to the bucket from step 1.
  • Paste the contents of Sync an “AWS S3 bucket to HostedFTP Using Cross-Account Access – S3 Bucket Policy Template.json” (if there is an existing policy, then the new statement should just be appended to the list of what is already in place). Search/replace the following missing information:
    • The AWS account number that is running the Hosted~FTP~ software.
    • The name of the bucket
 Click here to review the Amazon documentation for adding a bucket policy to an S3 bucket.

#

How to setup Sync to S3 in Hosted~FTP~ #

1. Logging in as the administrator, go to your Sync tab.

width=

2. The Sync tab shows you a list of your existing Sync processes for you to easily manage. Click on the Add button to setup and create a new Sync process. width=

3. See table below for details on setting up each field

Name Name of the Sync process
Actions Select what you want the Sync process to do: Upload, Move, Delete, Upload + Move. For this use case, please select Upload
Source The source of where the data is located to sync. For this use case, please select Hosted~FTP~
User+ Use a * to include all users, or place specific usernames separated by a comma (i.e. username1, username2)
User- Place usernames in this field to exclude them from the sync process
Path+ Use a * to include all paths, or place specific paths separated by a comma (i.e. //Folder1/SubFolder1/.*, //Folder1/SubFolder2/.*) Note: /.* is required to pull all data from the target path Example in Path+ (does not include subfolders) to filter for a specific file extension: //prod/Folder1/Subfolder1/[^/]*.csv Example in Path+ (folders with spaces) to include all files and subfolders underneath target folder: //prod/Folders1/Subfolders1/.* Example in Path+ (does include subfolders) to filter for a specific file extension: //prod/Folder1/Subfolder1/.*.csv Example in Path+ to include all files and subfolders underneath target folder: //prod/Folder1/Subfolder1/.*
Path- Place paths in this field to exclude them from the sync process
Target The source of where you will sync the data into. For this use case, please select AWS S3
Access Key Access key provided by your AWS S3
Secret Access Key The secret access key for the AWS S3
Bucket Name Input field for the password or private key
Folder The designated folder to apply the sync process.
Javascript Additional Javascript commands that can be included. (Reach out to our support team for assistance)

4. Manage the Options if required:

width=

Options: To have the sync process run fully, all options must be enabled.
Active Turns the sync process on or off after being created
Accept Accepts additions to the process
Process Disabling this option queues up the sync but does not process. For the sync to process, this must be enabled

5. Click Save to create the sync process. If all options are enabled (by default), the sync will run in the background every time uploaded data meets the rules.

Scroll to Top