How to Download Large Files From Your Server to Amazon S3 Directly

We discussed how to split huge files into smaller parts in Windows and subsequently upload them to your Linux server via SSH, then join them into a big file in the cloud yesterday. If you want to keep a second backup of the large file in the cloud, Dropbox is a good option, unfortunately, Dropbox does not allow users to calculate and view the MD5 or SHA1 checksum of your uploaded files, so perhaps it’s not guaranteed that the file in the cloud is identical to the one in your local computer.

Alternatively, Amazon S3 may help you, it’s the abbreviation of Amazon Simple Storage Service, which is a cheap and reliable online storage, you only pay for the storage and bandwidth you used, and the price is very competitive, furthermore, it will show you the MD5 checksum( called “ETag” ) of the selected file in the properties panel, allowing you to copy the string to clipboard and perform a comparison easily.

How to Download Large Files From Your Server to Amazon S3 Directly

There are many Amazon S3 clients presently, but most of them are designed to upload files from your local hard disk to the cloud, some of them are even not free. If you have paid for Amazon S3 and your host, there is no reason to pay for another service.

There are many tips and tutorials on installing applications or writing scripts to download files from the internet to Amazon S3, or login to Amazon EC2 with putty and perform file operations, they are too geeky for me and will cost too much time and effort, I do not prefer these manners.

Finally, I found this php script which can be used to upload files to Amazon S3 easily, you can let it download files from your server with a tiny modification.

First you should open the file named “page.php” with a text editor, find the “CHANGE THIS” string and change it to your Access Key ID, and change the “CHANGE THIS TOO” string to your Secret Access Key, which can be found in the email sent to you by Amazon corporation.

Second, you should replace all the “yourbucket” string with a new one, which will be used to name the “Bucket” in AWS Management Console, you can change it to any string you like, for example, diggfreeware_backup_2011_0709.

Third, you should change the string $_FILES['theFile']['name'] and $_FILES['theFile']['tmp_name'] to the URL of the file on your server, they should look like this:

and

Then upload the three files to your server and visit page.php via any browser such as FireFox, choose any file( it will not be uploaded ) and hit the upload button, the file “1GB.dat”( which is defined in the $fileName variable in the page.php file, as seen above ) will be uploaded to Amazon S3 in a few minutes. Because the bandwidth of your server is bigger than the bandwidth of your local computer, the transmit speed is greatly “enlarged”.

However, the upload process will fail if your file is bigger than 2GB, because many php functions can not handle the file larger than 2GB, you will see a failure message before the upload process begins.

So you need to split the file into small ones which are under 2GB. You can split the file “2G.dat” into every 1164967936 bytes with the command ( in putty ) below:

The size of my original file is 2,329,935,872 bytes( 2222MB ), it’s splitted into two 1111MB size files and uploaded to Amazon S3 from my server without any problem.


Comments are closed.