lsyncd – Live Syncing (Mirror) Daemon

On 2010年10月14日, in linux, by netoearth

Lsyncd uses rsync to synchronize local directories with a remote machine running rsyncd. Lsyncd watches multiple directories trees through inotify. The first step after adding the watches is to rsync all directories with the remote host, and then sync single file by collecting the inotify events. So lsyncd is a light-weight live mirror solution that should be easy to install and use while blending well with your system. See lsyncd –help for detailed command line options.

download:https://code.google.com/p/lsyncd/

Ladies and gentlemen, start your  servers!

First, you need 2 severs; one being the server and the other the client. (you could do this on one host if you wanted to see how it works for a proof of concept)

Install OpenSSH server

First you’ll need to install OpenSSH Server on the remote system:
apt-get install openssh-server

Configure SSH for Passwordless Logins

You’ll need to configure passwordless logins between the two hosts you want to use, this is how rsync will pass the files back and forth.  I’ve previously written a HOWTO on this topic, so we’ll crib from there.

First, generate a key:

ssh-keygen -t rsa

UPDATE: actually, it’s easier to do it this way

ssh-keygen -N '' -f ~/.ssh/id_dsa

(Enter)

You shouldn’t have a key stored there yet, but if you do it will prompt you now; make sure you overwrite it.

Enter passphrase (empty for no passphrase):

(Enter)

Enter same passphrase again:

(Enter)

We’re not using passphrases so logins can be automated, this should only be done for scripts or applications that need this functionality, it’s not for logging into servers lazily, and it should not be done as root!

Now, replace REMOTE_SERVER with the hostname or IP that you’re going to call when you SSH to it, and copy the key over to the server:

cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2'

UPDATE: now you can use ssh-copy-id for this instead (hat tip briealeida)

ssh-copy-id REMOTE_SERVER

Set the permissions to a sane level:

ssh REMOTE_SERVER 'chmod 700 .ssh'

Lastly, give it a go to see if it worked:

ssh REMOTE_SERVER

You should be dropped to a prompt on the remote server. If not you may need to redo your .ssh directory, so on both servers:

`mv ~/.ssh ~/.ssh-old`

and goto 10

Install rsync and lsyncd

Next up is to install rsync and lsyncd.  First, rsync is simple, and could already be installed (you don’t need to run it as a server, just the client), make sure you have it with:

apt-get install rsync

Next is lsyncd.  There is no official Debian package yet, but it’s simple to build from source and install.  First off, if you don’t have build essentials you’ll need them, as well as libxml2-dev to build the lsyncd source.  Installing those is as simple as:

apt-get install libxml2-dev build-essential

Now we’ll get the lsyncd code (you can check for a newer version at http://lsyncd.googlecode.com) and build that:

wget http://lsyncd.googlecode.com/files/lsyncd-1.26.tar.gz
tar -zxf lsyncd-1.26.tar.gz
cd lsyncd-1.26
./configure
make; make install

This install does not install the configuration file, so we’ll do that manually now:

cp lsyncd.conf.xml /etc/

Configure lsyncd

Next up, we’ll edit the configuration file now located in /etc  The file is a simple, well documented XML file, and mine ended up like so – just be sure to change the source and target hosts and paths to work with your systems:

<lsyncd version="1.25">
    <settings>
        <logfile filename="/var/log/lsyncd"/>
        <!--Specify the rsync (or other) binary to call-->
        <binary filename="/usr/bin/rsync"/>
        <!--uncomment to create a file containing pid of the daemon-->
        <!--pidfile      filename="/tmp/pid"/-->
        <!--this specifies the arguments handled to the rsync (or other) binary.
        option is the default literal. only '%r' will be replaced with r when recursive
	operation is wanted, d when not. exclude file will be replaced with -exclude-from FILE
	source will be the source path to sync from destination will be the
        destination path to sync to -->
        <callopts>
            <option text="-lt%r"/>
            <option text="--delete"/>
            <exclude -file/>
        <source />
        <destination />
        </callopts>
    </settings>
    <directory>
        <source path="/var/www/sync_test"/>
        <target path="desthost::module/"/>
        <!--  	or it can also be an absolute path for localhost
        <target path="/absolute/path/to/target"> 	-->
    </directory>
</lsyncd>

Launch lsyncd in debug for testing

We’re ready to give it a go, may as well run it in debug for fun and to learn how lsyncd does what it does:

lsyncd --conf /etc/lsyncd.conf.xml --debug

Watch for errors, if none are found, continue.

Add files and watch them sync

Now we just need to copy some files into this directory on the source box:

/var/www/sync_test

And again, watch for any errors on the screen, if these come back as a failed connection it’ll be an SSH/key issue, common, and not too difficult to solve. From here add some directories and watch how they’re queued up, and then take a look at them on the remote box: from this point out it “just works”. Now give it more to do by adding files and directories, and then the logging for errors while they sync. As it stands the system uses the source system as the preferred environment, so any files that change, or are added or removed, will be processed on the remote system. This is analogous to how Dropbox works, you can use multiple sources (your laptop, your desktop, etc) and their server serves as the remote system, keeping all the clients in line.

Tagged with:  

Comments are closed.