1

secure synchronization of large amount of data

view story
linux-howto

http://serverfault.com – I need to automatically mirror a large amount (terabytes) of files in two unix machines over a slow link (1 Mbps). This needs to be done frequently, but the data doesn't change too much (delta transmission doesn't saturate the link). The usual solution would be rsync, but there's an additional requirement: it's undesirable, from a security standpoint, that either the source or destination machines have (keyless) ssh keys to each other, or any kind of filesystem access. All communication between the two machines should thus be initialized (and mediated) through a third machine. I've asked (HowTos)

u85500's picture
Submitted by u85500 on 11/18/2012

Stories similar to secure synchronization of large amount of data

I have two Linux servers that have a large amount of data (1TB+) that need to be synchronized over a slow connection (100 KB/s).

A lot of the data overlaps, but are in different locations.

I would like some sort of rsync / unison tool where I can mirror the servers.

It would need to be more intelligent and identify if the file exists at the destination (possibly in another location with the sam

I'm looking for a simple on-site (i.e.

My company has to image a large amount of machines by the end of the year. Each of the machines will have hardware RAID 1 and running CentOS 6. Could someone recommend a method to automate the OS installation? I have a little mini desktop, and I can buy a switch, to serve as an installation server.

I have several Entropy Keys with egd in front of them and then all load balanced by haproxy. I then have many client machines using the haproxy service IP as a network source of entropy. I have no idea how much entropy they are requesting.

The Entropy Keys can produce a limited amount of usable entropy. The specs say about 30 kilobit/sec minimum.

Summary of My Need

We put a large amount of files on a filesystem for analysis at a later time. We can't control how many files we're going to have, and this one box needs access to all of them.

Unchangeable Limitations

I can't change the inode limit. It's ext4, and it's the default 4 billionish There will always be a lot of files.

I wonder what the best way to backup files VIA network

I have Solaris machines with oracle 10i with VERITAS cluster , machines are connected to EMC storage

/data/oracle directory is mounted on the EMC storage

What I want is to backup /data/oracle directory ( 70G ) on some backup machine VIA network ( include soft links files ) , while the Reliability of copying files is very very important

I

I have a linux server whic is used as web/file/svn/sql server. It is accessed by 100 machines for storing their code files on this server(svn/code/mysql).

The OS is on a 80 GB HDD. The data is on another 500 GB HDD.

I have 4 machines set up. 2 are running Ubuntu 12.04 LTS, 1 is running Windows 2003 server, and 1 is dual booting WinXP and Win7. 1 of the Ubuntu machines is configured to share out 2 disk volumes ("Data" and "DiskImages". My windows machines can see these shares. My other Ubuntu machine cannot.

We had a similar requirement when copying large amounts of data from USB drives to our servers (not MP3 players). With the file copy not happening in alphabetical order we found it difficult to tell how far the copying process had progressed.