Archive

Archive for the ‘Windows XP’ Category

Free, Secure and Efficient Windows Backups with rsync, ssh, and Cygwin

August 9th, 2012 No comments

Back in early 2010 I needed a free yet secure and efficient backup system for a small office with windows clients. After a lot of research I decided to use Cygwin to run ssh and rsync.

Most files don’t change on a daily basis, but we have to back up those stupidly large outlook PST files that are constantly updated. I try to strip out attachments every so often to keep them under 4 gigabytes but even at that size, backing them up over a wireless-G network can take far too long. That means I need a backup scheme that will only transfer the changed parts of files (deltas), and is efficient enough to compute those deltas on the client side. That leaves rsync or an rsync-based product as pretty much the only option.

I surveyed many free and commercial solutions and a lot of them didn’t do file deltas. Anything that did had other issues I didn’t like, or was just way too expensive. I am also not a believer in full system images. Let me digress to discuss that briefly…

My primary goal is to readily recover from catastrophic loss. Dead hard drive or lost/stolen laptop, office burned to the ground. -Recovery of that excel sheet from last week or last month is a distant secondary concern (to me). Making backups available for users to restore their own files is another issue that I avoid by not doing it. I haven’t found it to be necessary, but I certainly could provide a simple to use web-based view of their archives if it really came down to it.

In the event of major failure or loss, I’m probably reinstalling a new OS on new hardware and reinstalling apps (due to license restrictions, burned up hardware, downtime opportunity, hacker intrusion, whatever). Hard drive clone images are therefore not my priority. Documenting setup of new workstations and servers is a better use of my time than maintaining massive clone sets I’ll probably never use.

I’m sure many would disagree with me but this blog is less about justification and more about how it’s done. If you didn’t already want to use ssh and rsync under Cygwin, then you wouldn’t be here. I had a hard time collecting and understanding most of the instructions I could find. I have been running these backups for probably two years now and I have some decent documentation I wanted to share. Keep in mind, this blog post is concerned with just one part of my total backup solution; there’s plenty of other things going on that I’m not going to cover in any detail.

How it Works:

My workstations back themselves up nightly to a folder on the same hard drive using a traditional backup program. I use it because it’s reliable, sends emails, copies open files (via volume shadow service), makes some registry backups, and schedules reliably. Unfortunately it does not make registry backups or do smart file handling for users other than the one running the program. That’s an area of improvement I could use for a couple of shared workstations. I haven’t yet found any good 3rd party scripts to make selected registry backups or I might have replaced it by now.

Vanilla rsync doesn’t copy open files on Windows. I have seen a hack for rsync that is 3-5 years old and abandoned. I could probably hack it back in if I had the time, but it still wouldn’t give me the rest of the handy features described above.

So the way I do this is I have my windows backup program run simultaneously every evening on every workstation to mirror selected folders to a backup folder and toss in various registry backups and junk I want kept safe. Later that night, I have an otherwise idle Linux box connect to each workstation one after the other (to keep wireless bandwidth under control). It clones each workstation’s backup directory using rsync. The clone operation runs pretty fast, with pretty much just the outlook PST file deltas taking any significant time. There’s no issues with open files because nothing else touches the backup directory. My workstations normally sleep so I send out WAKE-ON-LAN packets to anything wired and for wireless machines they have a scheduled task wake them up.

I install Cygwin on each workstation and configure it to run an ssh daemon. The workstation ssh daemon is configured to only allow authorized host keys and IP addresses, so it should be about as secure as a windows box can get. It also ensures that any ssh connection can only execute rsync, and rsync can only access the backup folder read-only.

My Linux server runs a script that executes rsync (locally, as a client, on the server -if that makes any sense). The rsync command line specifies the ssh connection to the workstation. The workstation’s ssh daemon responds, we authenticate, and we have a secure communications tunnel. The first command sent via ssh is to run rsync as a server (on the workstation). Then our linx-side rsync client authenticates with the workstation’s rsync server, and we request the “mirror” configuration (see rsync.conf, “mirror” is just a made up name I liked). This encrypted ssh-rsync ‘tunnel’ then facilitates copying of the backup directory, including its registry dumps. The rsync protocol is smart and only copies deltas.. among other smart things it does. The rsync in server mode on the workstation does a lot of the processing to determine the deltas and compresses the data it needs to send. This keeps the copy time low on wireless/DSL links when handling massive outlook .pst files.

These are my step-by step directions for setting up new workstations. I doubt anyone would copy it exactly, but it should provide a good end-to-end example anyone can use to get started. Keep in mind that most of this stuff comes from somewhere else on the net. I have merely collected the relevant bits and tied some things together. I have no desire to take credit for any of it, but at the same time I don’t really care to go and dig up all of my original sources to give them credit either. I’m sure you can google as well as I can.

Installing Cygwin on Windows Workstations:
First off, add a new “Backups” Administrative User – assign a new random password and record it somewhere. We’ll never log in directly but we will ssh in with a private key. Run the reg file (below) to hide this user from Windows’ stupid Welcome screen.

Run the cygwin setup program as the Backups user so all of the file permissions will be nice and uniform.

run-as

Create folder c:\cygwin
Create folder c:\cygwin\packages
Download cygwin’s setup.exe and place it into c:\cygwin
Install Cygwin (launch setup.exe)
Root Directory = c:\cygwin
Local Package Directory = c:\cygwin\packages
Choose a mirror site >
Click on ‘View’ so that it shows: Full (all Packages)
Install some additional packages:

YOU CLICK THE ITEM WHERE IT SAYS SKIP AND IT CHANGES TO A VERSION NUMBER
THEN YOU SEARCH FOR THE NEXT ITEM AND DO THE SAME – it remembers all the ones you selected


links
nano
openssh
rdiff-backup
rsnapshot
rsync
screen
shutdown
unzip
wget
zip

Select all of the above files and let Cygwin finish installing.

While that’s working away, add port 12345 to the windows firewall using the advanced rule dialog (port 12345 is a dummy example use a smarter port number)
I name it such: sshd backups 12345 TCP

Starting Cygwin

It is essential that you make sure it runs as the Backups user WITH administrator privileges.
‘run-as’ does not automatically give such privileges, depending on your UAC settings.

set admin token

run Cygwin as the Backups administrator user

run-as

# change the “Backups” user’s home dir “/etc” – rsync (when run from sshd) will look for its config file there

nano --nowrap /etc/passwd

Now make certain that your “Backups” windows user has the appropriate rights to run sshd and rsync:

editrights -a SeTcbPrivilege -u Backups
editrights -a SeAssignPrimaryTokenPrivilege -u Backups
editrights -a SeCreateTokenPrivilege -u Backups
editrights -a SeIncreaseQuotaPrivilege -u Backups
editrights -a SeServiceLogonRight -u Backups
editrights -u Backups -l

editrights -r SeDenyInteractiveLogonRight -u Backups
editrights -r SeDenyNetworkLogonRight -u Backups
editrights -r SeDenyRemoteInteractiveLogonRight -u Backups

Install the ssh server:

ssh-host-config

say yes to privilege separation

(manually answer Yes to questions except)
CYGWIN ntsec
(tty is deprecated, FYI)

The sshd windows service should be installed and configured to be run as ‘Backups’ user

When the script says “This script plans to use cyg_server, Do you want to use a different name? Answer Yes, and type Backups as the user name.
Sure you can use cyg_server if you want to be like everyone else, but I have my “Backups” user running other jobs and cyg_server is not intuitive.


$ ssh-host-config

*** Info: Generating /etc/ssh_host_key
*** Info: Generating /etc/ssh_host_rsa_key
*** Info: Generating /etc/ssh_host_dsa_key
*** Info: Generating /etc/ssh_host_ecdsa_key

*** Query: Overwrite existing /etc/ssh_config file? (yes/no) yes
*** Info: Creating default /etc/ssh_config file
*** Query: Overwrite existing /etc/sshd_config file? (yes/no) yes
*** Info: Creating default /etc/sshd_config file

*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) yes
*** Info: Updating /etc/sshd_config file

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec
*** Info: On Windows Server 2003, Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: You appear to be running Windows XP 64bit, Windows 2003 Server,
*** Info: or later.  On these systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd).

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: No privileged account could be found.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) yes
*** Query: Enter the new user name: Backups
*** Query: Reenter: Backups

*** Query: Please enter the password for user 'Backups':
*** Query: Reenter:

*** Info: The sshd service has been installed under the 'Backups'
*** Info: account.  To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'.  Otherwise, it will start automatically
*** Info: after the next reboot.

*** Info: Host configuration finished. Have fun!

When this is done, a new host key is created. Make sure you double check the permissions on the newly generated host key. Don’t leave your private host key exposed more than necessary.

cd /etc ; chmod og-wrx ssh_host_key ssh_host_dsa_key ssh_host_rsa_key ssh_host_ecdsa_key

The ssh daemon is set up, but it is not yet configured exactly as I want it.

The sshd config file just created needs to be replaced. The next operation copies over a new config file. Double-check it and it’s file permissions.

Copy config file templates

Now it’s safe to copy over the default config files to C:\cygwin\etc

cp /cygdrive/c/Install/CYGWIN/{rsyncd*,ssh*,validate-rsync} /etc/.

Make certain that your files are owned by the “Backups” windows user with appropriate rights:

cd /etc
chown Backups:None rsyncd* ssh*config ssh_authorized_keys ssh_known_hosts validate-rsync
chmod 644 rsyncd.conf ssh*config ssh_authorized_keys ssh_known_hosts
chmod 755 validate-rsync
chmod 600 rsyncd.secrets
chmod 644 /etc/passwd /etc/group

chmod 755 /var
touch /var/log/sshd.log
chmod 664 /var/log/sshd.log

Here’s what these files do and what’s in them:

Hide_Backups_Account.reg : Hides the “Backups” user from the Welcome screen. Just run it.
rsyncd.conf : rsync config specifies the rsync profile(s) and user allowed to access it
rsyncd.secrets : rsync user:password list
sshd_config : configures sshd to pretty much allow only public key authentication
ssh_authorized_keys : lists the client ip address(es), specific command they can run, and public key for the ssh login
ssh_known_hosts : lists host keys. you can only authorize from a known host – preventing MITM attacks
validate-rsync : a shell script that make sure a valid rsync command is being requested.
This is the only command ssh allows to be executed (see authorized_keys)
It basically make sure that the command begins with "rsync --server " so a malicious client can’t run wild on your windows box.

Hide_Backups_Account.reg:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\SpecialAccounts\UserList]
"Backups"=dword:00000000

rsyncd.conf:

gid = users
read only = true
use chroot = false
transfer logging = false
log file = /var/log/rsyncd.log
log format = %h %o %f %l %b
auth users = rsyncuser
secrets file = /etc/rsyncd.secrets
strict modes = false
list = false
[mirror]
path = /cygdrive/c/Backups/Nightly_LOCAL

rsyncd.secrets:

rsyncuser:plaintextpassword

sshd_config:

Port 12345
HostKey /etc/ssh_host_rsa_key
HostKey /etc/ssh_host_dsa_key
HostKey /etc/ssh_host_ecdsa_key

# be real careful with this don't use on a server, just dumb windoze clients you will never ssh into
PermitRootLogin=forced-commands-only

SyslogFacility AUTH
LogLevel INFO

StrictModes no
UsePrivilegeSeparation yes

IgnoreRhosts yes
IgnoreUserKnownHosts yes

# For this to work you will also need host keys in /etc/ssh_known_hosts
#RhostsAuthentication no
RhostsRSAAuthentication no

HostbasedAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no

RSAAuthentication no
PubkeyAuthentication yes
AuthorizedKeysFile /etc/ssh_authorized_keys

ssh_authorized_keys:

from="8.8.8.8,8.8.4.4",command="/etc/validate-rsync" ssh-dss {long public key}

ssh_known_hosts:

a.yourdomain.com,8.8.8.8 ssh-rsa {long public key}
b.yourdomain.com,8.8.4.4 ssh-rsa {long public key}
{repeated for more hosts}

validate-rsync:

#!/bin/sh

case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
*\|*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac

Customize The Config Files

With the above files in place, continue on in Cygwin (as the Backups administrator user)

# configure a new random password for the “rsyncuser” user specified in rsync.conf
# you will need this password for your backup server later

nano rsyncd.secrets

from
rsyncuser:plaintextpass
to
rsyncuser:new_random_plaintext_password

# specify the path that rsync clients will be allowed to mirror

nano rsyncd.conf

path = /cygdrive/c/Backups/Nightly_LOCAL

Now everything should be in place. Test the ssh server.

TESTING SSH

ALWAYS initially run sshd in debug mode and make sure to click ALLOW on the retarded windows firewall popup.. otherwise you may become upset and bitter towards ms.
(Maybe you should add sshd.exe to the firewall manually and avoid such tragedy)

To debug sshd, remember to always shift-right-click and “run-as” on the Cygwin icon as “Backups” user.
(otherwise you may get nasty file permission issues. in particular, ssh expects /var/empty/ to be owned by the user running sshd)

/usr/sbin/sshd -D -ddd

The debug output should tell you if there is an immediate problem.

Then from some other computer just telnet and type in gibberish+ENTER to make sure you have your sshd up and running on the correct port.
telnet the_workstation 12345

If the telnet worked and says something like “SSH-2.0-OpenSSH_5.9” then you are in luck. Type in some gibberish and hit enter, ssh should respond like so:

“Protocol mismatch.
Connection to host lost.”

This confirms ssh is running, the port is correct, and it’s not blocked by some stupid firewall.
(sshd stops running after a connection in debug mode, so keep that in mind and pay attention to your cygwin window)

If that’s all a go, make sure to leave sshd running in debug mode. Don’t forget it exits after every connection!

/usr/sbin/sshd -D -ddd

Backup Server Setup

Now let’s move on to the unix server. It will be calling the workstations and initiating the rsync process.

I use a fairly simple shell script which does a dry-run and then runs a real rsync. I’m going to skip that and just show my rsync command line.

(remove --dry-run from the command line when you are certain it’s all set up correctly)

/usr/bin/rsync --dry-run --itemize-changes --verbose --inplace --progress --max-delete=50 --stats --compress --numeric-ids --archive --delete-after --password-file=/r/backup.pass --exclude-from '/r/excludes-all.txt' -e "ssh -p 12345 -l Backups -i /r/keys/myhost-key" rsyncuser@1.2.3.4::mirror /BACKUPS/workstations/1.2.3.4/mirror

Many options on the command line are just things I like to have. Your backups may call for different settings.

--dry-run
goes thru all the motions but only pretends to sync – very handy

--itemize-changes
lists all files changed/removed/etc

--verbose
even more chatty output

--inplace
modifies files in place, rather than writing to a new file and replacing it afterwards
this can be handy for large files or when the rsync session is broken prematurely,
as the next rsync will simply continue where it left off

--progress
shows the progress of each file being copied/changed

--max-delete=50
suppose you mess up the configuration or the workstation directory is empty by mistake the last thing you want is your backup to obediently delete thousands of files in a heartbeat
in my situation, nobody ever deletes much, so 50 files per rsync is fine as it will eventually catch up

--stats
shows summary stats at the end of the sync

--compress
this is pretty much essential

--numeric-ids
I think this means to assign user and group ids by number rather than name.
I prefer to have intact file ownerships. The server doesn’t have those users or groups on it.

--archive
I would guess this keeps the file permissions identical

--delete-after
Waits to delete files at the end of the operation.
There’s some tricky situations that make this option quite handy you should read up on it.

--password-file=/r/backup.pass
This is the rsync user’s password. The user is specified in the rsyncd.conf file.
The username and password are listed in rsyncd.secrets
I don’t just put it on the command line since it would show in the process list.

--exclude-from '/r/excludes-all.txt'
list of files to exclude from the transfer
this uses what I would call a ‘non-obvious’ syntax so read up on it


-e "ssh -p 12345 -l Backups -i /r/keys/myuser-key"

this is what tells rsync to establish the ssh tunnel first
-p = port number obviously
-l = login (the ssh login not rsync login)
-i = the private key file associated with the “Backups” user


rsyncuser@1.2.3.4::mirror

the rsync user name
the IP address or host name of the workstation to be backed up
mirror = the rsync profile (defined in rsyncd.conf on the workstation)


/BACKUPS/workstations/1.2.3.4/mirror

this is the local target path for the copied data

So try out the command line and hopefully you have the right keys and passwords and you get a good summary of what rsync would do. If that’s all a go, then reboot your workstation and try the telnet again. This time sshd should have been started automatically and not in any debug mode. Keep it simple, try a simple telnet like we did before, make sure it responds as above. Then try rsync from your server. Might save you some debugging grief.

If you have trouble with sshd when booting normally, you want to check your file permissions and that the service is installed right and running as the correct user. If necessary, stop the sshd service and try it from debug mode again, watching the output in the cygwin window for hints.

You can test an ssh connection itself rather than running it via rsync.


ssh -p 12345 -l Backups -i /r/keys/myhost-key 1.2.3.4


You should get back the word “Rejected” – this would come from the validate-rsync script executing on the workstation.
That script makes sure that the worst a malicious client can do is run rsync.


ssh -p 12345 -l Backups -i /r/keys/myhost-key 1.2.3.4 "rsync --server --version"


This should return the version of rsync running on the client, along with a list of it’s capabilities.
If you get this far, then there’s no trouble with your ssh connection or keys, so look for rsync authentication or configuration problems.

Once it’s set up it’s pretty much bullet proof. I may revise this blog from time to time with minor corrections but this is what I’ve been using for a couple of years now and I have no plans to make any major changes.

With these workstation backups in a central location, there are many options for making further differential-based archives and encrypted offsite storage. I may blog about rdiff-backup and its ilk later on.

Categories: Backups, Linux, Windows 7, Windows XP Tags: , ,