The rsync utility may be one of the best kept secrets on any platform.
Certainly, its use as a backup tool is well known, but perhaps a bit less well
known is it’s ability to replicate a complete partition’s contents with
permissions and file dates intact (also important for backups). I have used
rsync to copy one or more partitions from one machine to another using a
portable USB hard drive or over a network in order to install the OS on a new
machine or to get the contents of a dying drive backed up and then copied over
to a new drive. This article will describe these procedures from a Linux/Unix
user’s perspective.
From here on I assume you have a good working knowledge of a Linux based system and are familiar with drive partitions and how a typical Linux system finds and mounts these partitions into the file system. In other words, this is less a tutorial and more of a collection of notes for “those who know what they are doing”. I will not be held responsible for any data loss. If in doubt, do a backup first, etc.
As the computers I use are all Intel processors and since I use the
distribution’s Linux kernels, this process does work without too many issues
to fix up. If a customized kernel has been built, it may not be portable to
the new machine. Another issue is having the correct Xorg video driver for
the target computer, assuming X is used. If your /etc/fstab uses UUIDs to
specify the partitions for the mount points, they will most assuredly have
changed and fstab will need to be edited or the system will not be able to
mount the partitions on the new drive. If fstab uses Linux device names you
may get lucky and not have to edit anything if you laid out the partitions on
the new drive the same as on the old one.
This process uses a “live” Linux CD and an intermediate disk drive of sufficient capacity to hold the data or both machines connected via a network. I originally used the excellent KNOPPIX CD-ROM for this task but have recently been using an older version (3.6) of the Parted Magic CD-ROM. The Parted Magic CD boots up quickly and loads completely in RAM making it a very responsive system. The Parted Magic system only has a “root” user meaning one has full access/control of the machine (be careful!). Using a live CD means that the installed OS does not have open files or other issues that might prevent a faithful copy.
While I originally used the network to transfer the files, even on a 100 Mpbs network and using compression, it takes some time to move several GiB from one machine to the other. Recently I bought a 500 GiB USB hard drive for another purpose but have so much capacity that I chose to use it as the intermediate storage when I wanted to replicate an Xubuntu installation on identical computers. Performance is respectable and only one machine needs to be setup and booted at a time.
Single computer using a portable drive
This is the more simple method, so I’ll describe it first.
Boot the computer with the live CD. On some machines pressing some key, such as F12, while the BIOS splash screen is shown will be needed.
Mount the partitions. Use Gparted, if needed, to create partition(s) on the
target drive (the source drive should be left untouched). Note the mount
points—/media/sda1 for the source drive, /media/sdb1 for the target
drive, etc. (some CDs may use /mnt instead of /media for the mount
directory).
Open a terminal and issue the following command (replace the ? with the partition number noted above):
rsync -avHxz /media/sda?[/] /media/sdb?
Note that I put the trailing / in square brackets on the source path. This
means the / is optional but its presence changes the place the data will
appear on the target drive. Without the / the data will be stored in a
directory named sda1, i.e. under /media/sdb1/sda1 would be found the
contents of /media/sda1 such as boot/, bin/, dev/ etc. With the /
appended to the source path the root of the target partition will contain
the individual files and directories exactly as the root of the source drive.
If you have a single large partition on the target drive, omitting the/ will
allow placing the contents of each source partition in its own directory.
The rsync options I use are as follows:
- -a archive mode
- -v verbose
- -H preserve hard links
- -x one files system—don’t cross file system boundaries
- -z compress file data during transfer
For complete details consult the rsync manual page. Compression may be omitted when transferring locally from one drive to another as it takes time to perform the compression. Over a network link compression is almost always to your advantage.
Once rsync completes, you can use the following command as a
quick check:
df -h
Which will show how much disk space is used in each partition. The ‘Used’ columns should match.
Repeat the steps of mounting and using rsync for each partition you wish to
copy.
Shutdown the source machine and boot the target machine.
Create and format the partitions on the target machine’s drive as needed.
Repeat the steps above on the target machine except that the portable drive becomes the source and the target machine’s internal drive is the target.
If you used the trailing / to copy the source machine’s contents to the
target drive, use it again to build the directory structure into the root
directory of the target partition. If you did not use the trailing / but
instead have a directory named sda1, for example, on the portable drive’s
partition, then add that directory name to the source path with a trailing
/:
rsync -avHxz /media/sdb?/sda?/ /media/sda?
Note the difference in the paths!
Repeat for all the partitions you wish to copy.
Two computers on a network
The steps are essentially the same for the actual rsync command, but a bit
of additional preparation must be done first as SSH (Secure SHell) is used to
establish the connection between the computers.
Boot each machine with a live CD. Using the same distribution/version will probably make life a bit easier. This example is from my notes when using KNOPPIX.
Target machine:
Open a terminal and give the root user a password. Something like admin
will do. On some CDs you will be root already as noted by the # at the
end of your terminal’s command prompt. On others you may have to use the su
or sudo -i command to become root.
Mount the target partition. Make certain it is mounted rw (read/write) as some live CDs mount partitions as ro (read only) by default.
Start the rsync network daemon:
rsync --daemon
Finally, the Secure Shell daemon is started. On Debian and derivatives—Ubuntu mixes, KNOPPIX, etc.—use the startup script:
/etc/init.d/ssh start
Source machine:
As on the target machine, gain root privileges.
Mount the source partition. It is preferred that this partition be mounted read only, ro.
Invoke rsync over the network using SSH:
rsync -avHxz /media/sda?[/] 192.168.0.2:/media/sda?
The rsync command looks much the same as in the simple example except there is the addition of a network address and a colon for the target drive. The network address is self explanatory and the colon tells rsync to use the remote shell, in this case SSH, for the transport.
When prompted, enter the password given to the target machine. Answer
yes when prompted to accept the SSH key.
The transfer will now commence. Head for dinner or something as it will take a while on a large partition.
Final Tweaks
As mentioned above, some adjustments will need to be made if the partition will be used as a live system.
/etc/fstab
This is the File System TABle and tells the kernel where each partition is to be mounted. While Linux device names have been traditionally used, later versions have chosen to use UUIDs that are long hexadecimal strings almost guaranteed to be unique to a given partition. The other advantage of a UUID is that no matter the underlying hardware type, the kernel can find the partition.
To learn the UUIDs on your system:
ls -l /dev/disk/by-uuid
From which you’ll get an output similar to this:
total 0
lrwxrwxrwx 1 root root 10 Nov 3 07:05 0f2802fa-8820-437d-84be-3b2d8ae74814 -> ../../sda1
lrwxrwxrwx 1 root root 10 Nov 7 02:56 1e462d1b-6e63-4526-bfbb-6c162f17fa7a -> ../../sdc2
lrwxrwxrwx 1 root root 10 Nov 3 07:05 4e29a94f-e2fd-4bd2-92d2-e0aa472def45 -> ../../sda5
lrwxrwxrwx 1 root root 10 Nov 3 07:05 5f1e3b38-3f3c-45d1-adb8-fa26225dd0a1 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Nov 7 02:56 77e88bae-d103-4105-ab42-17414fb2c23d -> ../../sdc5
lrwxrwxrwx 1 root root 10 Nov 3 07:05 9c49ad4f-910e-406f-9d17-ad576576a083 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Nov 7 02:56 BC8019918019536E -> ../../sdc1
lrwxrwxrwx 1 root root 10 Nov 7 02:56 cb33422c-a07f-4332-9f14-0742436c7f0b -> ../../sdc7
lrwxrwxrwx 1 root root 10 Nov 7 02:56 ed6bbb2f-9f2b-4aca-935c-2279fbff458c -> ../../sdc6
lrwxrwxrwx 1 root root 10 Nov 3 07:05 fb9274c0-1cce-407b-a15c-4bbf7e2d8fb4 -> ../../sda2
As I prefer to work in an X Window terminal, I simply copy the UUID I need for
a given partition (/, home, including the swap partition!) and paste it
into the editor where I am editing fstab.
initramfs
To get the system running, modern Linux systems use an initial RAM file system that gets the system up and running and then the kernel mounts the real filesystem and runs from there. This initial RAM disk image needs to be rebuilt after the system is copied to the new partition for it to boot.
If you’re going to be moving the data to another type—from EIDE to SCSI/SATA or to a USB drive, the initramfs will likely need additional kernel modules added. A symptom of this is that booting from the new drive takes a long time and the boot loader drops into a busybox shell. Often times a list of needed kernel modules is printed to the screen. Copy these down as they’ll be needed later.
The initramfs is built upon kernel installation when the system is running normally from its installation. As we’re still running from the live CD, how can we update the initramfs? The answer is to use a chroot. The chroot is setup using the commands from this Ubuntu page on live CD customization:
mount --bind /dev/ /media/sda?/dev
chroot /media/sda? # use the target partition
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devpts none /dev/pts
You are now in the chroot. Any commands will be run from the target partition although the active kernel is from the live CD which may cause a harmless warning later when initramfs is updated.
Debian makes it easy to specify additional modules for the initramfs by
including them in the /etc/initramfs-tools/modules configuration file. When I
wanted to continue to use my Debian installation from my USB hard drive after
the internal drive crashed and I was waiting for the new one, I had to add the
following modules to initramfs:
ehci-hcd
uhci-hcd
ohci-hcd
usbhid
As the boot loader message included all four files, I just included them all.
Now update the initramfs (back up the original /boot/initrd.img-<kernel version> first):
update-initramfs -u -k 3.2.0-4-686-pae
Note that your kernel version may be quite different!
The boot loader
Finally, the boot loader needs to be configured. If another Linux installation already exists on the drive, this can be as easy as booting into it and running:
update-grub
Where by default grub will probe other partitions and add the newly copied installation automatically.
If you’ve copied the installation to a new drive, then grub will need to be installed. While still in the chroot run:
grub-install /dev/sda # or sdb as appropriate
update-grub
Go ahead and reboot from the newly copied installation.
Summary
Perhaps other GUI programs exist for this task. Still, rsync is probably
included on any Linux live CD which makes it an excellent tool for this task.
Comments
Comment by Nate Bargmann on 2013-01-14 18:40:11 -0600
As another twist, let’s add another architecture to the mix!
Picking up a used Thinkpad T410 with an Intel Core i5 processor set, I found that running Debian amd64 (x86_64) much better than the i386 version. As I did the install on another drive first I moved everything over to the main drive as above. When I got to the chroot part I received an error of a Bash incompatible executable file format.
Oops!
So, I brought the Debian Live CD Project’s amd64 Rescue CD to the fight! Once installed to a USB flash drive using Unetbootin things were off and running.
The one caveat is that once booted the rescue image logs in automatically as a
normal user at a Bash shell prompt. How to execute root commands? The su
command wanted a password so I tried sudo for the mount and chroot
commands and it works without need of a password! After the chroot command
is run the UID is set to 0 and root commands may be run directly. To log off
the rescue image I used the ‘Ctl-Alt-Del’ key combination and the
image commenced its shutdown.
On booting I was greeted with the Grub menu I expected and the copied system was off and running.
Comment by Nate Bargmann on 2020-02-29 21:43:37 -0600
Changing out the hard disk with the boot partition is a pain. On my Lenovo M73 I had to disable secure boot in the BIOS and then follow the instructions at:
https://wiki.debian.org/GrubEFIReinstall
I chose to use the Refind USB image and boot the kernel directly. Then follow the steps in the Wiki to restore EFI and then secure boot.
The next morning I decided to update the BIOS and had to do the whole dance all over again with the added fun having to set the BIOS to allow booting in legacy mode to update the BIOS! There there were several steps of trial and error working my way back to an EFI/secure boot system by retracing the steps in the Wiki link above including having to restore the key in the BIOS NVRAM to permit signing and loading the Virtual Box kernel modules.
Whew!