4

Options to efficiently synchronize 1 million files with remote servers?

view full story
linux-howto

http://serverfault.com – At a company I work for we have such a thing called "playlists" which are small files ~100-300 bytes each. There's about a million of them. About 100,000 of them get changed every hour. These playlists need to be uploaded to 10 other remote servers on different continents every hour and it needs to happen quick in under 2 mins ideally. It's very important that files that are deleted on the master are also deleted on all the replicas. We currently use Linux for our infrastructure. I was thinking about trying rsync with the -W option to copy whole files without comparing contents. I haven't tri (HowTos)

taskli's picture
Submitted by taskli on 06/15/2012 – Made popular on 06/15/2012

Stories similar to Options to efficiently synchronize 1 million files with remote servers?

I am building a web application where my users will be able to upload files.

I'm using rsync from Cygwin to do a large scale data transfer from an aging HP MSA 1000 to a new DAS attached to a different server. I have a daemon running on the remote server in read only mode and a local copy writing the files to disk.

One of my servers is an image repository with over a million files spread across about 300 directories. Each file averages only a couple hundred kilobytes.

We have a central repository of files in say server-1 at /srv/www. Then we have N number of servers as well.

I'm using

rsync --delete -Rav remote:/path/to/maildir/cur /backup/

but I'm finding lots of files on remote that are not getting copied by rsync, while others are copied. Rsync does not throw any errors, it's like it just doesn't see some files.

By large file tree I mean about 200k files, and growing all the time. A relatively small number of files are being changed in any given hour though.

By bidirectional I mean that changes may occur on either server and need to be pushed to the other, so rsync doesn't seem appropriate.

By distant I mean that the servers are both in data centers, but geographically remote from each other.

Hello!

I got the following trouble: Tangerine only scans a part of my music collection in the selected folder, about 1000 of 5000 files are missing.

I work in a company that has a slow large WAN covering many different locations on all continents.

Releases of a particular internal application consist of DLLs and data files that can cumulatively top 100 MB, and there is talk of the application growing.

I have a script which will upload files from a specific directory (with rsync) to the server, but in the mean time another script will pull files from the server (also with rsync and with the --delete flag).

This means, that when I add a lot of files to it and rsync is in the process of uploading files and the download script is also being run, some files (which are not yet uploaded to the server

I try to synchronize files between two machines using rsync, but it seems to copy all the files regardless file modification time. Two consecutive call of the command produces the same results, so it is clearly broken.

The command in question is:

rsync -r local/path user@host:/remote/path -v

The local file system is ext4 while the remote is ext3.