ipsure logo
Logo and Language
Login icon Language selection icon
Hello, guest
*NIX BACKUP Active category menu left background Active category menu right background Hands-On blog header image Right block of Hands-On blog header image Final menu block of Hands-On blog header image
MS TIP PKI PROJECTS Active category menu left background Active category menu right background CMS Türkçe HANDS-ON SERVICES IT BUSINESS CONTACT ABOUT REFERENCES TERMS RSS
Home page Hands-On Services IT Business Contact About References Terms of Use RSS

19/01/2010

Automatic Backup And Synchronization Solution with Rsync on FreeBSD PART 1

Filed under: backup, projects — Tags: , , , , — Sezgin Bayrak @ 12:01

It is not that easy as it is thought to introduce a stable and first of all a tidy strategy to constitute a backup solution that can be easily managed and monitored for our both individual applications as well as the high scale corporate infrastructures. That’s why we still see joyless system administrators running from one server to another with USB hard disks in their hands waiting for the mass data transfers to be completed even in professional data centers and environments. I’m immensely aware that for most of the companies, the time and the energy sacrificed for this burdensome operation is preferred as a cost that have to be tolerated rather than to invest on expensive commercial backup and recovery solutions. On the other hand, when the value and the importance of the data that must be backed up are all taken into consideration, the risks are obvious that the legacy backup models which are performed manually in only specific periods entertain.

In this topic we will be investigating my project over performing an automatic backup and synchronization of clients and workstations in your local network, servers that are resided in your data center, machines located inside your intranet and extranet or even at your remote offices, by a central server that is running FreeBSD on it. Through especially the principals of rsync operation and relevant algorithms, this solution which is leaning on the transfer of only the data that is modified can be used as an healthy, stable and secure solution between the nodes in situations that the backup and synchronization may cause serious problems for particularly large sized data where long distances and limited bandwidth are the major cases. Our solution isn’t dependent on OS type of the nodes that will be backed up and can be used for backing up any servers and stations including Microsoft, unix and linux. I think that the comprehensibility and the applicability would be fine and the mismatch problems would be minimized as the unix shell programming techniques are being used predominantly in the solution. When we implement the application, we will be receiving html notification e-mails regarding all backup processes as you can see in the screenshot below.

rsync email notification

Notification e-mail sent by Rsync server

As a beginning, we have to select the server and also the relevant disk sizes that will achieve the storage, initialize UFS partition and the slices according to our project in accordance with the number of the nodes and the size of the data that will be backed up inside our infrastructure. Even not mandatory, when the importance of the machine where our backups are going to be stored is taken into consideration, a suitable disk redundancy will be reasonable. In configuration, RAID1 for its minimal investment requirement can be preferred by the small infrastructures while RAID5 for larger ones will do the job providing enough amount of security with least loss of size. As a result, the strategy of determination of the servers and relevant storage media will completely be differing upon the infrastructure that this solution will be implemented.

At least in our sample case, thinking that we will be storing our backups inside /usr partition, I’d like to remind you that this partition must have enough size. But if you will not allocate a new dedicated machine for this project and will use an existing one that you don’t have a chance to configure the disk slices from the beginning, you may adapt another partition (e.g., /var) with a suitable amount of space to this implementation. But you’ll need to realign the scripts and predefined variables properly in that direction.

Additional applications and services which this project depends and that can be simply installed by using FreeBSD ports path are Bash and Mutt. If these packages are already installed in your system, we can continue with the implementation. If not, install them after updating your ports tree.

If you don’t have the ports tree installed;

# portsnap fetch
# portsnap extract

If the ports tree is present but not up-to-date;

# portsnap fetch
# portsnap update

Subsequently we can continue with the package installations;

# cd /usr/ports/shells/bash
# make all install clean
# cd /usr/ports/mail/mutt
# make all install clean

Please download the rbs-project-v2.0.tar.gz file to a suitable location in your server by using the link below. This tarball includes the install.sh that will perform the installation together with the scripts, templates and the configuration file inside. install.sh script will perform the automatic installation of sample structure that constitutes the skeleton of the implementation. You may need to make minor arrengements and modifications on relevant paths and scripts according to your infrastructure in order to start benefiting from this application truly.

downloadrbs-project-v2.0.tar.gz

MD5 = 526c1307221509050fe66ab610c40325
SHA256 = a2e12fc15d2936efc0eda49822c57195a0744582dae40c23f2c34f26dddca146

Decompress the file and enter the installation directory;

# tar -xvf rbs-project-v2.0.tar.gz
# cd rbs-project-v2.0

Edit the variables.inc configuration file placed inside the root of installation directory following the instructions that will be mentioned below before running the install.sh script which is going to perform plenty of time saving tasks for you such as; creating common “backup” user group that Rsync, FTP service and also applications like Samba shall use together for accessing backups and for the recovery purposes, creating necessary folders and sub directories, placing templates to the related locations by parsing the parameters corresponding to variables you specify there inside the configuration file.

# variables.inc
# Sezgin Bayrak, sbayrak@ipsure.com
# http://www.ipsure.com (2010)

RSYNC_SERVER                    hostname.yourdomain.com

<BEGIN>

HOST:                           host1
DOMAIN:                         yourdomain.com
IP:                             192.x.x.1
FOLDER_NAME:                    HOSTFolders
DB_DIR_NAME:                    AUTOCURRENT
FOLDER_MAX_DAY:                 10
DB_DIR_MAX_DAY:                 5
IMAGE_PATH:                     http://web.server.com/rsync/img
FOLDER_OK_IMAGE:                host1-ok.jpg
FOLDER_FAILURE_IMAGE:           host1-fail.jpg
DB_DIR_OK_IMAGE:                host1db-ok.jpg
DB_DIR_FAILURE_IMAGE:           host1db-fail.jpg
MAIL_ADDRESS:                   mailbox@yourdomain.com

<END>

<BEGIN>

HOST:                           host2
DOMAIN:                         yourdomain.com
IP:                             212.x.x.1
FOLDER_NAME:                    AppFolder
DB_DIR_NAME:                    DBFolder
FOLDER_MAX_DAY:                 21
DB_DIR_MAX_DAY:                 7
IMAGE_PATH:                     http://web.server.com/rsync/img
FOLDER_OK_IMAGE:                host2-ok.jpg
FOLDER_FAILURE_IMAGE:           host2-fail.jpg
DB_DIR_OK_IMAGE:                host2db-ok.jpg
DB_DIR_FAILURE_IMAGE:           host2db-fail.jpg
MAIL_ADDRESS:                   rsyncadmin@yourdomain.com

<END>

Configure the variables of the templates inside “variables.inc” severally and one under the other for every single host you want to backup to this server.

During configuration, without changing the variable names (e.g., IP:) in the first column, just edit the statements (e.g., 192.x.x.1) across them. As these variable names are called inside the scripts and they are carrying distinctive marks, if changed, conflictions and incompatibility issues will occur and the application may not run as expected.

After executing install.sh script, make sure that relevant management and storage folders are created under /usr/backup-sys/agents/, /usr/backup-sys/mail/ and /usr/backups/ paths with exactly the same HOST names defined as the first variables of the configuration lines in variables.inc for the source HOST machines that backup and synchronization will take place;

/usr/backup-sys/agents/HOST
/usr/backup-sys/mail/HOST
/usr/backups/HOST

I’d like to draw your attention to that /usr/backups/HOST path(s) are going to be the path(s) which you’ll define for storage inside rsyncd.conf that we’ll touch upon a little later.

For example;

/usr/backup-sys/agents/ns1
/usr/backup-sys/mail/ns1
/usr/backups/ns1

FOLDER_NAME: It represents the subfolder beneath the HOST folder at rsync server in which the dispersed files and folders at source will be stored after synchronization. In summary, it’s the area where the cumulative backup of files and folders at source are going to be kept on server.

DB_DIR_NAME: It represents the subfolder beneath the HOST folder at rsync server in which the database backups processed at source will be stored after their transfer. In short, it’s where the MS SQL, MySQL backup files are going to be kept on server.

Following the execution of install.sh, make sure that those are also created for every host in accordance with your configuration;

/usr/backups/HOST/FOLDER_NAME                   # For storage
/usr/backups/HOST/DB_DIR_NAME                   # Foe storage
/usr/backup-sys/mail/HOST/FOLDER_NAME           # For e-mail management scripts
/usr/backup-sys/mail/HOST/DB_DIR_NAME           # For e-mail management scripts

For example;

/usr/backups/websrv-01/AppFolder
/usr/backups/websrv-01/AUTOCURRENT
/usr/backup-sys/mail/websrv-01/AppFolder
/usr/backup-sys/mail/websrv-01/AUTOCURRENT

FOLDER_MAX_DAY variable predicates utmost how many days the compressed version of the synchronized folder will remain on system, DB_DIR_MAX_DAY variable however predicates utmost how many days the compressed version of the transferred database backup will be kept on system. You can calibrate this duration by particularizing the severity of the files that are backed up, mostly their sizes, disk capacity of your rsync server and the demands for keeping historical data.

Considering your web server configuration IMAGE_PATH implies the absolute URL by where the images and icons that will be used inside the automatic notification e-mails are going to be called. FOLDER_OK_IMAGE | FOLDER_FAILURE_IMAGE | DB_DIR_OK_IMAGE | DB_DIR_FAILURE_IMAGE variables represents the visuals that symbolize the states of success/failure/partially ok. situations per hosts and folders in form of caption bars as you can see in the sample notification e-mail output above. Rearrange these as you like for every host you plan to back up. (Images are placed under rbs-project-v2.0/images folder for you in order to be used as templates. Change and use them in your related web path.)

For example;

http://www.ipsure.com/rsyncmail/img             # Don not add "/" to the end of /img

Don’t forget to set the access rights for the images you placed under /img folder;

(Typically 644)

-rw-r--r--  1 ftpuser  ftp   23439 Nov 20 02:01 mikrodbfail.jpg
-rw-r--r--  1 ftpuser  ftp   23931 Nov 20 02:01 mikrodbok.jpg
-rw-r--r--  1 ftpuser  ftp   23269 Nov 20 02:01 mikrofail.jpg
-rw-r--r--  1 ftpuser  ftp   24073 Nov 20 02:01 mikrook.jpg

MAIL_ADDRESS variable defines to whom the notification e-mails will be sent for the related host. If you want to deliver these e-mails to more than one authorized person, you can set this e-mail address as a distribution list in your mail server.

If you fulfilled the configuration inside the variable.inc for every host that you want to back up to this server then we can execute install.sh script which is located at the same path.

# ./install.sh

RSYNC INSTALLATION

Now by using up-to-date FreeBSD ports tree, let’s perform rsync installation;

# cd /usr/ports/net/rsync/
# make

We go ahead by selecting SSH, FLAGS and ACL options. During the installation, rsync pkg. (rsync-3.0.6.tar.gz) and its patches (rsync-patches-3.0.6.tar.gz) will be added to our system.

# make install clean

After the completion of installation, we add the line below to our /etc/rc.conf file in order to set the rsyncd start at system boot time;

# echo 'rsyncd_enable="YES"' >> /etc/rc.conf

We’re going to edit rsync configuration file as you see below. We’re setting up general configuration lines in terms of rsync’s standart operation, within GLOBAL PARAMETERS section at the upper portion of the existing rsyncd.conf file. While designating the log path with “log file” parameter where the general transfer logs will be kept regarding all modules, we’re also enabling chroot feature by “use chroot = yes” parameter,  indicating valid user and group for chrooting wih uid and gid values. With the help of the charset = utf-8 parameter, we’re enforcing the usage of UTF-8 character set in order not to face any character corruptions in the names of the files that are in transition for especially between windows – unix systems. In MUDULE PARAMETERS section, we’re defining every module (machines) separately by their host names that are authorized to send backup files to this server. Subsequently, applying host based access restrictions by IP address for security reasons, indicating that chroot mechanism will be in place for related module for providing extra protection against possible vulnerabilities in application and also specifying /./ (dot-dir) in module’s path to mark the point that the chroot will occur. This will allow the application to run in a chroot completely independent from system’s root “/” path for the top of the transfer hierarchy. In order to explicate the matters for the client (in other words the sending side) we will be discussing the details relative to a Microsoft Windows server [mikrodb] module which has various files and folders with a MSSQL database to be backed up and an Apache web server [websrv-01] module inside the sample configuration file below. In practice you can actualize a backup implementation in any scale you wish by editing the following names and IP addresses according to your own network.

# vi /usr/local/etc/rsyncd.conf
# rsyncd.conf - Example file, see rsyncd.conf(5)

# GLOBAL PARAMETERS

# Set this if you want to stop rsync daemon with rc.d scripts
pid file = /var/run/rsyncd.pid
log file = /usr/backups/transfer.log
lock file = /var/run/rsyncd.lock
charset = utf-8

# Edit this file before running rsync daemon!!

uid = root
gid = backup
use chroot = yes
max connections = 15    #Max. concurrent connection limit to rsync server
syslog facility = local5

# MODULE PARAMETERS

[companydc]
 path = /usr/backups/./companydc
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=192.168.1.10
 hosts deny=*

[mikrodb]
 path = /usr/backups/./mikrodb
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=212.x.x.145
 hosts deny=*

[webdb]
 path = /usr/backups/./webdb
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=212.x.x.132
 hosts deny=*

[websrv-01]
 path = /usr/backups/./websrv-01
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=212.x.x.129,212.x.x.145
 hosts deny=*

[mail]
 path = /usr/backups/./mail
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=212.x.x.136
 hosts deny=*

[ux]
 path = /usr/backups/./ux
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=212.x.x.131
 hosts deny=*

[ns1]
 path = /usr/backups/./ns1
 use chroot = yes
 read only=no
 transfer logging=yes
 hosts allow=88.x.x.28
 hosts deny=*

For varying requirements, you can inspect all other parameters you may use in this configuration file at this page: http://www.samba.org/ftp/rsync/rsyncd.conf.html

Now let’s initiate our rsync daemon;

# rsync --daemon

Let’s check it out;

# ps aux | grep rsync
root    66415  0.0  0.0  3128  1348  ??  Ss    8:49PM   0:00.00 rsync –daemon

You can use the command lines below for restarting the rsync service when needed or for standart start/stop manipulations;

# /usr/local/etc/rc.d/rsyncd restart
# /usr/local/etc/rc.d/rsyncd stop
# /usr/local/etc/rc.d/rsyncd start
# netstat -a | grep rsync
tcp4       0      0 *.rsync         *.*         LISTEN
tcp6       0      0 *.rsync         *.*         LISTEN

As it’s seen above our server has started to listen on TCP 873 which is the rsync port. If the servers and the clients you want to backup are resided inside a different DMZ or firewalled diverse networks than our FreeBSD server lives in, do not forget to edit your firewall rule base to allow TCP 873 port from these nodes to our rsync server. Besides, taking into consideration that real ethernet IP(s) may not be seen by the rsync server because of the various NAT rules that might be set to different firewall interfaces for those machines which are inside different networks, you have to set the module IP addresses to relevant NAT IP(s).

CRON SETTINGS

We have to adjust required settings to automate our scripts with cron for regular reporting and management of the backups. First we’ll examine the existing cron settings for root;

# crontab -l
crontab: no crontab for root

So there hasn’t been any crontab installed for root yet. In order to compose one, we have a tmp  file created by using crontab -e command and start to type via toggling “ESC”, “i” keys just like in “vi” editor. As can be seen in the example below, we’re first creating the line which triggers rotate-transfer-logs.sh script to run at 00:00 every night that will compress and store the current transfer.log file under /usr/backups/ while forming a new one and removing the compressed versions older than 7 days. Then we’re entering necessary definitions individually for every hosts indicated inside rsyncd.conf and variables.inc.

SHELL=/usr/local/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/etc

Notice the above lines. These are the lines that indicates in which shell (bash) cron will run and which environmental variables it will use. If they’re provided wrong or missing, it will cause our scripts not to run automatically even they return fine results when they’re executed directly in command line interface. Furthermore notice that in every line regarding the hosts, cron has been granted to change directory to related path of the host before the script is executed. It’s because the certain variables inside relevant scripts are taking references from the top level directories for especially path determination and verification purposes.

# crontab -e
SHELL=/usr/local/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/etc
#minute hour    mday    month   wday    command
# Daily rotation for rsync transfer.log
0       0       *       *       *       /usr/backup-sys/server/rotate-transfer-logs.sh
# Backup management and reporting for folders and database of host: mikrodb
30      4       *       *       *       cd '/usr/backup-sys/agents/mikrodb' && /usr/backup-sys/agents/mikrodb/mikrodb-control.sh
0       5       *       *       *       cd '/usr/backup-sys/agents/mikrodb' && /usr/backup-sys/agents/mikrodb/mikrodb-control-db.sh
# Backup management and reporting for folders and database of host: websrv-01
30      5       *       *       *       cd '/usr/backup-sys/agents/websrv-01' && /usr/backup-sys/agents/mikrodb/websrv-01-control.sh
0       6       *       *       *       cd '/usr/backup-sys/agents/websrv-01' && /usr/backup-sys/agents/mikrodb/websrv-01-control-db.sh

After you finished configuration, save file and quit with :wq!

/tmp/crontab.9WEzi5dlRx: 9 lines, 662 characters.
crontab: installing new crontab

Consequently we installed a new crontab for root user.

SETTINGS FOR HOST (CLIENT) SIDE

Under this subject heading, we’ll be discussing over the settings that have to be followed through on the host (client) machines which will be sending the backups. If we divide it into two parts as Microsoft and *NIX, things that have to be done are given below in detail for both OS types.

Microsoft Servers and Clients:

By using the links below, download cwRsync_3.1.0_Installer.zip and UTF8-cwrsync-dll.rar files into your server or client.

download

.

cwRsync_3.1.0_Installer.zip

UTF8-cwrsync-dll.rar

.

Execute cwRsync_3.1.0_Installer.exe file after decompressing cwRsync_3.1.0_Installer.zip and complete the installation within C:\Program Files by following the short instructions. Then decompress UTF8-cwrsync-dll.rar file and overwrite the original cygwin1.dll file under C:\Program Files\cwRsync\bin\ with the new cygwin1.dll inside UTF8-cwrsync-dll folder. Create an empty text file by name rsync.log inside C:\Program Files\cwRsync\bin\ . This log will be transferred to rsync server along with the data backed up. After completing these steps your Microsoft server or client will be ready to communicate with your rsync server.

Let’s assume that we want to backup an application folder called F:\AppFolder and its all contents found at your web server (websrv-01.mydomain.com). And let the IP address of rsync server be 212.x.x.1 which you have installed and configured above. To actualize this scenario, we have to create a batch file named ms-rsync.bat and define backup processes with particular rsync command and parameters;

@ECHO OFF
REM *********************************************
REM MS-RSYNC.BAT; Batch file to synch F:\AppFolder and logs to Rsync server.
REM *********************************************
@echo Processing MAIN RSYNC Transfer ...
del C:\Progra~1\cwRsync\bin\rsync.log
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" --itemize-changes --delete --stats /cygdrive/f/AppFolder 212.x.x.1::websrv-01 >C:/Progra~1/cwRsync/bin/rsync.log
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" /cygdrive/c/Progra~1/cwRsync/bin/rsync.log 212.x.x.1::websrv-01
@echo Done !
exit

Let’s analyse the commands and relevant parameters in order to master the subject:

I think you’ve noticed that while we used C:\Progra~1\cwRsync\bin\rsync command inside MSDOS script, we eleminated the space between “Program Files” statement in the form of “Progra~1” on purpose. Don’t forget this in your implementations. Otherwise the batch may fail.

-rltpED parameters set;
-r: (–recursive) recurse into directories while copying
-l: (–links) copy symlinks as symlinks without changing
-t: (–times) preserve modification times of the items
-p: (–perms) preserve permissions of the items
-E: (–executability) preserve executability of the items
-D: (–devices –specials) preserve device and special files

“–chmod=u=rwx,g=rx,o=” parameters set;
–chmod: This parameter directs rsync to apply one or more comma-separated “chmod” strings to the permission of the files in the transfer. In summary, we can rearrange the permission bits of every file that are going to be transfered to rsync server according to our implementation intents.
In this context;
u=rwx (user bits) assigns read, write and execute rights to root user
g=rx (grup bits) assigns only read and execute rights to “backup” group we created while installation
o= (others bits) drops any access rights for any user and groups except root and backup, as it has been left empty

Thereby when we transfer our backups to rsync server, nobody will be able to access, inspect, change or restore backups with any method (ftp, NetBIOS, etc.) except root and backup group. For this reason we had specified particularly root:backup ownerships both inside rsyncd.conf and for /usr/backups storage area path. By all means, we’ve decided this model as our implementation is a backup unit for just synchronizing the backups. If we wanted to implement a NAS and enable access for whole world, we’d prefer to use rwxrw-rw- (766) bits rather than rwxr-x— (750).

–itemize-changes: Requests an itemized list of the changes that have been made to each file, including the attribute changes. Through this parameter, we can see at notification e-mails which files are modified in which way and because of this which files are put into synchronization. Consequently as we gain an opportunity to follow the files one by one which are modified at source, we also gain an extremely beneficial monitoring chance in terms of security.

–delete: Deletes the items at receiver side (rsync server) that do not exist anymore at source and so are deprived of validity during the synchronization.

–stats: It generates a verbose set of statistics regarding the file transfers allowing you to feel how effective rsync’s delta-transfer algorithm is for your data.

/cygdrive/f/ is the way how cwRsync expresses Microsoft Windows disk and slices in unix style. (=> F:\)

AppFolder is the application folder at client side that will backed up in our example.

212.x.x.1::websrv-01 tells AppFolder and its contents will be sent to “websrv-01″ storage area defined inside the rsyncd.conf  (path = /usr/backups/./websrv-01) on Rsync server with 212.x.x.1 IP address.

With >C:/Progra~1/cwRsync/bin/rsync.log expression we’re having logs generated regarding the transfer being processed. Also we deliver these logs to websrv-01 storage area at the following lines.

Anyway you can now submit your ms-rsync.bat cmd script file into Scheduled Tasks for automation, considering the running times of the management and reporting scripts defined inside the cron at the opposite side (Rsync server). Needless to say that it will be reasonable you schedule it to a suitable time before the control scripts at rsync side starts to run. While estimating this, don’t forget to consider the size of your backups and the amount of time needed to transfer them.

Regarding your various needs and the experimental designs, you can find out all other parameters you can use with rsync command at http://www.samba.org/ftp/rsync/rsync.html

REMINDER: While designing your backup strategies and determining the folders to be synchronized, it is not an effective or actually a needed decision to make to include the CACHE folders which are generally benifited by miscellaneous application development techniques for fast content delivery. That kind of folders will probably contain thousands of small temporary objects that will consume time and resources while copying if you include them in your plan.

Now let’s punctuate the rsync usage on Microsoft clients section, diversifying it with a different example;

@ECHO OFF
REM *********************************************
REM MS-RSYNC.BAT; Batch file to synch Mikro accounting folders, MSSQL FULL|DIFF|TRN DB backups and logs to Rsync server.
REM *********************************************
C:\Progra~1\cwRsync\bin\rsync -rav --stats --delete /cygdrive/f/Mikro/v12xx/COMPANY-A-09/PARAMS /cygdrive/f/BACKUP/MIKROFolders/COMPANY-A-09/
C:\Progra~1\cwRsync\bin\rsync -rav --stats --delete /cygdrive/f/Mikro/v12xx/COMPANY-A-09/SYSTEM /cygdrive/f/BACKUP/MIKROFolders/COMPANY-A-09/
C:\Progra~1\cwRsync\bin\rsync -rav --stats --delete /cygdrive/f/Mikro/v12xx/COMPANY-B-09/PARAMS /cygdrive/f/BACKUP/MIKROFolders/COMPANY-B-09/
C:\Progra~1\cwRsync\bin\rsync -rav --stats --delete /cygdrive/f/Mikro/v12xx/COMPANY-B-09/SYSTEM /cygdrive/f/BACKUP/MIKROFolders/COMPANY-B-09/
C:\Progra~1\cwRsync\bin\rsync -rav --stats --delete /cygdrive/c/MikroXML /cygdrive/f/BACKUP/MIKROFolders/
del C:\Progra~1\cwRsync\bin\rsync.log
del C:\Progra~1\cwRsync\bin\rsyncdb.log
@echo Processing MAIN RESYNC Transfer ...
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" --itemize-changes --delete --stats /cygdrive/f/BACKUP/MIKROFolders 212.x.x.1::mikrodb >C:/Progra~1/cwRsync/bin/rsync.log
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" --itemize-changes --delete --stats /cygdrive/f/BACKUP/AUTOCURRENT 212.x.x.1::mikrodb >C:/Progra~1/cwRsync/bin/rsyncdb.log
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" /cygdrive/c/Progra~1/cwRsync/bin/rsync.log 212.x.x.1::mikrodb
C:\Progra~1\cwRsync\bin\rsync -rltpED "--chmod=u=rwx,g=rx,o=" /cygdrive/c/Progra~1/cwRsync/bin/rsyncdb.log 212.x.x.1::mikrodb
@echo Done !
exit

In the example above, “Mikro” and “MikroXML” are the main folders which we want to backup. First we’re conflating these different folders from local C:\ and F:\ directories in the same local F:\BACKUP\MIKROFolders\ directory before rsync synchronization starts. We only use -rav (-a: –archive) parameters as we perform copying process just between local paths. (Rsync’s delta-transfer algorithm is not used and the whole file is sent as it is in local transfers) Then we truncate the logs regarding prior rsync synchronization. Finally, we transfer all the logs and MIKROFolders together with \BACKUP\AUTOCURRENT folder which contains daily full, differential and transaction log database backups generated by a Maintenance Plan on MS SQL.

MS SQL Maintenance Plan folder
You have to program Scheduled Tasks to run ms-rsync.bat more than once in a day in order to accurately transfer especially DIFF and TRN backups that shall be processed within short periods during the day.

*NIX Servers:

Just like we did on Rsync Server, we also have to intall rsync (rsync-3.0.6.tar.gz) on unix and linux systems which will send the backups. But we don’t need to run it as –daemon and make it run at system boot times as we did on server. We’ll transfer our backups simply by using the “rsync” command at sender system. Here are the examples;

#!/usr/local/bin/bash
# rsync.sh;  synchronizes hosted web folders under /usr/local/www/apache22/data path to Rsync server.
#
# WEB FOLDERS
rsync -rav --itemize-changes --stats --delete /usr/local/www/apache22/data/WEBSITE.COM 212.x.x.1::hostingsrv > /space/rsync-website.com.log
rsync -rav --itemize-changes --stats --delete /usr/local/www/apache22/data/WEBSITE.NET 212.x.x.1::hostingsrv > /space/rsync-website.net.log
# LOGS
rsync -rav /space/rsync-website.com.log 212.x.x.1::hostingsrv
rsync -rav /space/rsync-website.net.log 212.x.x.1::hostingsrv
exit

As can be seen above, It shall be sufficient to use -rav parameters for the transfers between same OS types. We mustn’t forget to # touch and create related rsync-*.log files under our working directory (e.g, /space).

Now on the hosting server in above example, let’s transfer today’s MySQL backup files that are created as defined at my Backing Up MySQL Databases article;

#!/usr/local/bin/bash
# rsyncdb.sh; transfers all MySQL DB backups to Rsync server.
#
# MYSQL DATABASES
rsync -ravx --stats /space/backup/*-backup-`date +%Y.%m.%d`*.sql.gz 212.x.x.1::hostingsrv
exit
# crontab -e
SHELL=/usr/local/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/etc
#minute hour    mday    month   wday    command
# Synch Web folders daily (/w logs) to Rsync server
50      0       *       *       *       /space/rsync.sh
# Transfer daily MySQL database backups to Rsync server
20      1       *       *       *       /space/rsyncdb.sh

In PART-2 we’ll be inspecting the contents of the control scripts and emphasizing what we can do further more for customization inside these scripts.

Related Posts with Thumbnails
Subscribe to our RSS feeds Email Subscription via FeedBurner RSS Subscription via FeedBurner

  No related posts.

2 Comments »

Trackbacks

Reader Comments

There are currently no reader comments available at this time.

RSS feed for comments RSS feed for comments on this post. TrackBack URL

Leave a comment