How to Move Files from One AWS S3 Bucket to Another

 AWS S3 is an acronym for Amazon Web Services' Simple Storage Service, which is one of the most popular AWS services. Amazon S3 was first announced in 2006, and it has since become a popular platform for object-based storage. In S3, data is saved as objects in buckets, which provide storage space for up to 5 terabytes per item. It's a low-cost storage service that's built to last 99.999999999 percent of the time.

S3 also allows you to manage data across many websites, mobile apps, backup and restore big data analytics, and a variety of other applications.If you use S3 to store your data, you may have run into a situation where you need to copy files between two distinct AWS S3 buckets inside the same account, or you may need to do so in the future.

In this post, I'll teach you how to copy files from one S3 bucket to another without writing any scripts or writing any code. The good news is that everything will be controlled from the console. So, without further ado, let's get started on our topic.

Note that this will have no effect on files in the source bucket, so it's just a copy command from one area to another.

The First Step is to Define Your Buckets.

I have two separate buckets that are already pre-created in the S3 interface. I've already established them for demonstration purposes; one is called aws-simplified-source-bucket, and the other is called aws-simplified-destination-bucket. These are the two buckets where I want to move my files from the source to the destination.

So, if we go inside the aforementioned source bucket, we'll see that I've put two little JSON files in a folder for demonstrative purposes.

If we simply go to our destination bucket, we should see that it is empty, which is exactly what we want because we want to move files from the source to the destination.

As I previously stated, this will be done within the same AWS account, and if you need to do it across accounts, there's a nice blog article that you can see here.

Step two: synchronisation of data

We'll utilise AWS Data Sync in the next step; this is a new service from AWS that allows you to easily sync data from a source bucket to a destination bucket. You can find the utility by searching for Data Sync or AWS Data Sync in the search field. You can use this to automate data acceleration from on-premises to the cloud or vice versa, as well as between AWS Storage services.

The wizard can be seen below, indicating which option we wish to use for data transfer between on-premise storage and AWS storage services. Because we'll be transferring between two AWS S3 buckets in this demonstration, I'll select "Between AWS storage services" and click Get Started.

Setting up Configurations (Step 3)

We'll need a slew of options for where our files are stored in the next stage. It entails four stages in order to get everything set up for you.

Configuring the Source Location (Step 3.1)

We must configure the source location in the first sub-step of step 3 here. So we'd like to choose a new location, and then we'll need to set up a few more factors. To configure the source location, look at the screenshots below.

Configuring the Destination Location (Step 3.2)

You must configure the target location in this stage. This is going to be quite similar to what we accomplished in the last sub-step. To configure the destination location, look at the screenshots below.

We can now proceed to the next step by clicking on the next button. You'll see a successful pop-up intent in the Green colour as soon as Data sync is configured.

Substep 3.3: Task Configuration

Now we must configure settings for what is known as a task, which will be the real object that will conduct the transfer. Let's set up the task.

Demo Transfer is the name of the task. Task

Verify Data: Only the data that has been transferred should be verified ( Here on clicking markdown, we have a bunch of different options here, but I am going with this, and this one does only verify the data that gets transferred to destination )

Set Bandwidth: Make use of what you have. (Again, there are numerous alternatives here; I chose this because I want maximum speed; however, if you have any limitations, such as a throttle or limited bandwidth, you may choose appropriately.) You can also enable queuing, which is a useful feature.

In the next section, we have a Data transfer configuration with two options: transfer only modified data or transfer all data. I'm going with the option of only transferring data that has changed because it's cost-effective and suitable for the majority of individuals. You can still use the other if you have a special use case where you want to run a series of processes every time a file is uploaded. Be cautious because it will have a larger financial impact.

Then you have the option to keep the deleted files. You also have the option to overwrite files; see the screenshot below for more information.

We also have some additional basic options for filtering setup down here, as well as a scheduling option if you want this to be a regular job. So, we're going to do it manually here. Then you have some task logging configuration, which we don't care about. I'm going to move on to the next option in the bottom right corner.

Step 4: Creating a Task

In this step, we'll go over our configurations again, and once you're satisfied that everything is in order, click Create a Task at the bottom.

After a few seconds, we can see a data sync console with our job status visible and an option to start in the upper right corner; if you click on the drop-down, you have two options: start with defaults or start with overriding. You can see an example of this in the screenshot below.

We'll select "start with defaults" and click it. A green pop intent will appear, indicating that execution has begun, and the task status has now changed to running. After a few minutes, we can see that the task status has changed to " success," and that's all.

Thank you for taking the time to read this.