Refering to the myth, the author of the article writes:

One of the reasons behind this idea is that the positioning of a hard disk drive’s head is not precise enough to ensure that the data is overwritten with new information from the exact same byte.

A study, published on December 2008, claims that tests performed on both last and older generation hard drives have shown that recovering even a single byte of data after a complete overwrite is practically impossible.

Security researchers from Heise Security, who have reviewed the paper presented at last year’s edition of the International Conference on Information Systems Security (ICISS), explain that a single byte of data can be recovered with a 56 percent probability, but only if the head is positioned precisely eight times, which in itself has a probability of occurring of only 0.97%.

Since I was one of those who believed the statement about the multiple overwrites, I found the article very interesting. I haven’t read the study itself though.

Related Articles

• Partition Misalignment Slows Down 4096-Byte Sector Hard Disks
• How to create a single file of Sphinx based documentation
• Choosing a format for data backups – tar vs cpio
• Shred changes default number of passes to 3
• The Complete Fedora Kernel Headers

To be able to correct things I had to boot using a Fedora 9 kernel in order to make any changes to the files and get rid of this issue. What was needed was to stop using device names in fstab, but use UUIDs instead. To get the UUID for a particular block device, for example /dev/sda1, I ran as root:

# blkid /dev/sda1
/dev/sda1: LABEL="/boot" UUID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" TYPE="ext3"

Now replace the device node name in /etc/fstab to read:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    /boot    ext3    defaults    1 2

Of course, I could have used the label of that partition, but I guess UUIDs are more unique and therefore preferred.

Related Articles

• A different approach…HAL
• Hard disk upgrade on an old motherboard
• UDEV
• Encrypt devices using dm-crypt and LUKS
• CloneZilla instead of Partimage

$ ps -ef | grep httpd | grep -v grep
root      1926     1  0 Dec03 ?        00:00:55 /usr/sbin/httpd.worker
apache    2608  1926  0 14:31 ?        00:00:06 /usr/sbin/httpd.worker
apache   22192  1926  0 01:05 ?        00:00:02 /usr/sbin/httpd.worker

So, in my case the child processes are run by user “apache“. This could also be determined by the user and group directives inside Apache’s configuration file, /etc/httpd/conf/httpd.conf:

User apache
Group apache

So, in order to make a directory writable by the webserver we have to set the directory’s owner or group to Apache’s owner or group and enable the write permission for it. Usually, we set the directory to belong to the Apache group (apache or www-data or whatever user is used to launch the child processes) and enable the write permission for the group.

chgrp apache /path/to/mydir
chmod g+w /path/to/mydir

In many cases, usually in shared hosting environments, it is not possible to change the ownership of files and directories. In those cases you could just set the write permission for everyone (others):

chmod o+w /path/to/mydir

Which method is more secure depends on how /path/to/mydir is accessed.

If it is accessed through the web server with an HTTP request it does not really matter which of the above methods has been used in order to make /path/to/mydir writable by the web server, because, in any case, the web server will be able to write to /path/to/mydir.

If the directory is accessed by other means, for instance by another local program which is run by an untrusted local user, then, obviously, the first method is more secure.

I guess this explains how to make a directory or file writable by the web server process.

Related Articles

• Redmine deployment delayed
• .htaccess Cheat Sheet
• Issues with the feeds are now resolved
• File and Directory diff in color in Midnight Commander
• Script for Apache Error Report

Related Articles

• User management from the command line
• Using a CUPS printer from command line
• Creative PC-CAM 750 on Fedora 10
• Creative Commons v3.0 Licenses Launched
• Blanking a rewritable CD/DVD in GNOME

vsftpd Configuration

Assuming vsftpd has already been installed in the standard location, the directory /etc/vsftpd/, which contains its configuration files, should exist. You can edit vsftpd’s default configuration file (/etc/vsftpd/vsftpd.conf), but in this example, we will create a new configuration file from scratch.

Create a new configuration file named /etc/vsftpd/vsftpd-anon.conf and open it in your favourite text editor and write down the directives that follow:

Set the server to run in standalone mode. This means that vsftpd will run into the background and handle the incoming requests on its own. The alternative method (listen=NO) would require you to set up a xinetd service. This would not be a bad idea, but for the sake of this example, it would be a waste of time.

listen=YES

The following directives prevent local users from logging in and enables anonymous access respectively.

local_enable=NO
anonymous_enable=YES

The following directive disables write access to the ftp server’s filesystem. This is a global switch, so noone will be able to upload or modify any files on your ftp site.

write_enable=NO

Sets the root directory for anonymous connections. By default, this is /var/ftp/.

anon_root=/var/ftp

The following configuration directives are optional and can be safely omitted.

Limit the rate at which anonymous users can retrieve files.

anon_max_rate=2048000

Enable logging information about user logins an file transfers. The log file is located at /var/log/vsftpd.log.

xferlog_enable=YES

Set the interface and port the service will listen on. By default, vsftpd will bind to all local network interfaces on port 21, which is the standard port of the File Transfer Protocol. Note that listen_address accepts only numeric IP addresses (no hostnames).

listen_address=192.168.0.100
listen_port=21

The entire vsftpd-anon.conf file

#
# Sample anonymous FTP server configuration
#
# Mandatory directives
#
listen=YES
local_enable=NO
anonymous_enable=YES
write_enable=NO
anon_root=/var/ftp
#
# Optional directives
#
anon_max_rate=2048000
xferlog_enable=YES
listen_address=192.168.0.100
listen_port=21

Start or Stop the FTP server

Assuming you have created the supplementary vsftpd-anon.conf configuration file, run as user root:

vsftpd /etc/vsftpd/vsftpd-anon.conf

To stop the service run:

killall vsftpd

Alternatively, you can send the SIGTERM signal to a specific vsftpd process.

On the other hand, if you had edited vsftpd’s default configuration file, you could start/stop the service using the /etc/init.d/vsftpd initscript.

Sharing files and directories

An FTP server without any files is like having a swimming pool without any water in it. In order to make some files and directories available through your FTP service you have two options:

1. Copy or move the files or directories inside the anon_root directory.
2. Create bind mounts of the directories you want to share in the anon_root directory.

You may wonder why you cannot just create some symbolic links inside anon_root pointing to the directories you want to share. Even if you created those symlinks and connected to the service using an FTP client, you would notice that you are not permitted to reach the linked location. This happens because anonymous users are restricted (chrooted) to anon_root and, therefore, no location outside this directory is accessible using symlinks.

Bind mounts are the solution to this problem. When bind-mounting, you mount a directory (A) to another directory (B) on the same or different filesystem, so that the contents of directory A appear as contents of directory B. It’s like a symlink, but at a lower level of the filesystem and that’s why you can reach locations outside the chroot jail.

In our scenario, the installation tree of a Linux distribution is shared through the FTP service. It is assumed that the installation medium has been inserted into the drive and either the system or you have mounted it, for example, to the directory /media/CentOS/. We want the contents of the DVD to be accessible through the FTP server, so we need to bind-mount the DVD contents to a directory inside anon_root. As user ‘root‘ issue the following command:

mount --bind /media/CentOS /var/ftp/pub

Now, connecting to the FTP service you will notice that the contents of the pub/ directory is the CentOS installation tree.

It is quite obvious that, despite the fact that vsftpd does not support the creation of a virtual filesystem (mainly a virtual directory structure) internally, one can be easily implemented with bind-mounts.

Do not forget the firewall

When we run a server temporarily on the desktop computer, we tend to forget to open the necessary ports on the filewall. In the case of vsftpd, you should open port 21 or the port number you have assigned to the listen_port configuration directive. Please consult the documentation of your firewall management application about how to perform this action.

Further Reading

• All the supported configuration directives for vsftpd.

Related Articles

• When it comes to error messages…
• Caching Nameserver using dnsmasq
• Setup the SSH server to use keys for authentication
• Set up the VNC Server in Fedora
• Making a directory writable by the webserver

Creation of the TAR archive and the MD5 sums file

In the following example it is assumed that the files to backup reside in the myfiles/ subdirectory, the name of the tar archive will be mybackup.tar and the name of the file containing the md5sums will be mybackup.md5.

$ tar -cvpf mybackup.tar myfiles/ \
    | xargs -I '{}' sh -c "test -f '{}' && md5sum '{}'" \
    | tee mybackup.md5

Some notes:

• You can use any tar switch for the creation of the archive except -C. If you need to change to another directory, do it using cd or else no md5 sums will be recorded.
• Make sure that you include the -v (–verbose) switch when invoking tar, as the paths need to be printed to stdout in order to be processed by xargs.
• In the xargs statement, the -I ‘{}’ part indicates that the '{}' string will be replaced by the path that is passed to xargs through the pipe.
• The sh -c “test -f ‘{}’ && md5sum ‘{}’” does two things: tests if the path ('{}') is a file and calculates the md5 sum for it.
• In the last part, tee is used in order to print the md5sum to the stdout and also to the mybackup.md5 file.

When this operation ends, you will end up with two files: mybackup.tar and mybackup.md5.

Special thanks to:

* Anvil for the suggestion to use bash -c "...test goes here..." stuff.
* Giorgos Keramidas for the improvement he suggested, so that the md5 sum calculation is not limited to regular files only:

sh -c "test -d '{}' || md5sum '{}'"

VeriTAR will verify the md5 sums of regular files only, so either test you use when creating the TAR archive, it is still fine.

VeriTAR – Tar archive verification

VeriTAR [Veri(fy)TAR] is a command-line utility that verifies the md5 sums of files within a tar archive. Due to the tar (‘ustar‘) format limitations the md5 sums are retrieved from a separate file and are checked against the md5 sums of the files within the tar archive. The process takes place without actually exctracting the files.

It works with corrupted tar archives. The program carries on to the next file within the archive skipping the damaged parts. At the moment, this relies
on Python’s tarfile module internal functions.

VeriTAR is written in Python.

Works with compressed TAR archives (gzip or bz2).

• VeriTAR Development Website and Bug Tracking
• Downloads

Provided that you have used the method above (or any other method) in order to create a file with the md5 sums together with the tar archive, you can easily verify the contents of the archive with veritar.

$ veritar mybackup.tar mybackup.md5

Please not that veritar’s output and command line switched need some work, but for now it does the job.

Veritar is released under the Apache License version 2.

It is completely unsupported, but you can still get community support at our software forums. This is also the place where you can inform me about any bugs.

Known issues

1. Multi-volume tar archives are not supported at the moment
2. Tar archives in which the metadata of the first archived file has been corrupted cannot be processed due to a limitation in the tarfile Python module at the time of writing
3. Although the checksum of any algorithm, md5, sha1, crc(crc32), could be used, the current alpha version is not very flexible.
4. It may crash on damaged archives on older python versions.

Related Articles

• Verify a burned CD/DVD image on Linux
• Choosing a format for data backups – tar vs cpio
• How to extract RPM or DEB packages
• Error when using old run/bin installers under Linux
• Cheap Biometrics – Use Keystroke Dynamics to Identify and Verify Users

• Tests using tar:
  1. Random 1-byte corruption.
  2. Partial corruption of one of the archived files metadata.
• Tests with cpio:
  1. Random 1-byte corruption.
  2. Total corruption of one of the archived files metadata. (same result with partial header corruption)

Information about the two formats was found at the following web pages:

• CPIO specification (New ASCII format with CRC added)
• TAR specification (USTAR format)

The following tests assume the directory and file structure outlined below:

WORKING_DIR/
          bak/
               1.pdf
               2.pdf
               3.pdf

Before continuing I would like to thank the folks at the Linux-Greek-Users mailing list for their advice and ideas. I had initially posted the following material in the LGU list.

TAR Tests

Testing corruption of tar archives.

Random 1-byte corruption of the tar archive

In this test one random byte of the archive was replaced by a zero (0).

$ md5sum bak/*
11875e4e35a40686d81a37aa448aac2e  bak/1.pdf
30c63be455dbada1ffc985c5465d0723  bak/2.pdf
096dc1c77a2a0f4d9f953abd7264843f  bak/3.pdf

$ tar -cvf bak.tar bak/
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf

$ tar -dvf bak.tar bak/
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf

$ python -c 'f=open("bak.tar","r+"); f.seek(12334); f.write("0"); f.close()'

$ tar -dvf bak.tar bak/
bak/
bak/2.pdf
bak/3.pdf
bak/3.pdf: Contents differ
bak/1.pdf

$ mkdir out

$ tar -xvf bak.tar -C out/
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf

$ md5sum out/bak/*
11875e4e35a40686d81a37aa448aac2e  out/bak/1.pdf
30c63be455dbada1ffc985c5465d0723  out/bak/2.pdf
2d0b2aa54047d6e97b45fbb43f8f1bdc  out/bak/3.pdf

Conclusion: The md5 sums of the original 3.pdf and the extracted 3.pdf differ. The rest of the files has been extracted accurately.

Partial corruption of one of the archived files metadata

In this test, 200 bytes of the total 500 bytes of metadata of the 2nd archived file are destroyed. Note that the 1st archived file is the directory bak/

$ md5sum bak/*
b0ec395ca8cb79f2ce98397ec0e00981  bak/1.pdf
fbe2f3f799579251682ee6de0e4d828d  bak/2.pdf
afb18f2dbbb43673c641691b458dbcce  bak/3.pdf

$ tar -cvf bak.tar bak/
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf

$ tar -dvf bak.tar bak/
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf

In USTAR format, metadata occupy 500 bytes. The tar magic string starts at position 257 after the metadata start position. In this test, as it was already mentioned, 200 bytes of data are destroyed (range 200->400):

$ python
Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> magic = "ustar  \x00"
>>> f = open("bak.tar", "rb+")
>>> magic2_pos = f.read().find(magic, 258)
>>> meta2_start = magic2_pos - 57
>>> f.seek(meta2_start)
>>> f.write("0"*200)
>>> f.close()
>>>

$ tar -dvf bak.tar bak/
bak/
tar: Skipping to next header
bak/3.pdf
bak/1.pdf
tar: Error exit delayed from previous errors

$ mkdir out

$ tar -xvf bak.tar -C out/
bak/
tar: Skipping to next header
bak/3.pdf
bak/1.pdf
tar: Error exit delayed from previous errors

$ md5sum out/bak/*
b0ec395ca8cb79f2ce98397ec0e00981  out/bak/1.pdf
afb18f2dbbb43673c641691b458dbcce  out/bak/3.pdf

Conclusion: Although one of the archived files metadata has been destroyed, tar has managed to successfully extract the rest of the files, regardless of the fact that they were after the corrupted part of the archive. The success of the extraction is confirmed by comparing the extracted files’ md5 sums with the chewcksums of the original files.

CPIO Tests

Testing corruption of cpio archives.

Random 1-byte corruption of the cpio archive

In this test one random byte of the archive was replaced by a zero (0).

$ md5sum bak/*
11875e4e35a40686d81a37aa448aac2e  bak/1.pdf
30c63be455dbada1ffc985c5465d0723  bak/2.pdf
096dc1c77a2a0f4d9f953abd7264843f  bak/3.pdf

$ find bak/ | cpio -v -o -H crc > bak.cpio
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf
25919 blocks

$ cpio -vi --only-verify-crc < bak.cpio
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf
25919 blocks

$ python -c 'f=open("bak.tar","r+"); f.seek(12334); f.write("0"); f.close()'

$ cpio -v -i --only-verify-crc < bak.cpio
bak/
bak/2.pdf
cpio: bak/3.pdf: checksum error (0x2b7dbd48, should be 0x2b7dbda8)
bak/3.pdf
bak/1.pdf
25919 blocks

$ mkdir out2

$ cd out2/

$ cpio -vid < ../bak.cpio
bak
bak/2.pdf
cpio: bak/3.pdf: checksum error (0x2b7dbd48, should be 0x2b7dbda8)
bak/3.pdf
bak/1.pdf
25919 blocks

$ cd ..

$ md5sum out2/bak/*
11875e4e35a40686d81a37aa448aac2e  out2/bak/1.pdf
30c63be455dbada1ffc985c5465d0723  out2/bak/2.pdf
cd9ea8e6298a42f44b59322b31e55958  out2/bak/3.pdf

Conclusion: The md5 sums of the original 3.pdf and the extracted 3.pdf differ. The rest of the files has been extracted accurately.

Corruption of one the archived files metadata

In this test the metadata of one of the archived files is destroyed.

$ md5sum bak/*
11875e4e35a40686d81a37aa448aac2e  bak/1.pdf
30c63be455dbada1ffc985c5465d0723  bak/2.pdf
096dc1c77a2a0f4d9f953abd7264843f  bak/3.pdf

$ find bak/ | cpio -v -o -H crc > bak.cpio
bak/
bak/2.pdf
bak/3.pdf
bak/1.pdf
25919 blocks

$ python
Python 2.5.1 (r251:54863, Oct 30 2007, 13:54:11)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> magic = "070702"
>>> f = open("bak.cpio", "r+")
>>> magic2_pos = f.read().find(magic, 1)
>>> f.seek(magic2_pos)
>>> metadata_length = magic2_pos + 6 + 13*8 + 4  # 4: μέρος του pathname
>>> f.write("0"*metadata_length)
>>> f.close()
>>>

$ cpio -v -i --only-verify-crc < bak.cpio
bak/
cpio: premature end of file

$ mkdir out3

$ cd out3

$ cpio -vid < ../bak.cpio
bak
cpio: premature end of file

Conclusion: Neither verification nor extraction. cpio (at least Fedora's version) does not have the ability to skip to a healthy header and the operation ends prematurely. The use of a recovery tool in order to recover the healthy files within the archives is mandatory.

Conclusion

Here follow the pros and cons (this is not a complete list) of each format:

CPIO
+ per-file CRC checksum. The backed up data on the DVD can be verified in-place without the need of any 3rd party software.
+ No limit for pathnames.

- when the cpio archive gets partially corrupted, as it can happen on a DVD, then the cpio program cannot skip the damaged files and move on to the next healthy archived file. The use of recovery software is needed.
- you have to use the find command's tests in order to include/exclude files in/from the archive.
- It cannot save extended attributes.

TAR
+ Even if some part of the archive gets corrupted, the tar program can skip to the next healthy archived file and extract it. This is very important as it eliminates the need of the 3rd party recovery software.
+ File and directory inclusions/exclusions are possible with command-line options and with file/dir lists read from a file.
+ It can save extended attributes, but 3rd party software may not be able to read the archive correctly.

- No CRC checksum is saved, so checking the data in-place requires two things: to have kept the checksums of the archived files and to have an external program that can check those checksums against the archived data. If this is not possible, then keeping the data on the hard drive in addition to the backup is needed in order to compare them using tar's -d switch.
- The maximum length of a pathname in the USTAR format is 156 bytes.

It is obvious that both of the two formats and/or programs are incomplete. The pros of one are the cons of the other. This was rather a surprise.

My final choice was the tar format because I consider the fact that it does not need a 3rd party program to extract the data from a damaged archive a great advantage. I have also created an utility, Veritar, that can verify the md5 sums of the files inside a tar archive with the md5sums that have been kept in a separate file during the creation of the archive. More information in my upcoming post about tar crc/md5 verification....

Related Articles

• VeriTAR – Verify checksums of files within a TAR archive
• How to extract RPM or DEB packages
• Effective data wiping with a single complete overwrite
• More Data Recovery Tools
• The importance of regular data backups

• GParted-CloneZilla Live CD – This project by LarryT combines the gparted hard disk partitioning tool with CloneZilla.
• Clonezilla-SysRescCD – This is another excellent combination of tools made by Spiros Georgaras. This liveCD combines the system repair and data recovery collection of tools of the SysRescCD with CloneZilla.

Related Articles

• Partition images with Partimage and Partimaged
• The importance of regular data backups
• Filesystem Backup Again
• More Data Recovery Tools
• How to recover lost files

Some raw utility listings can be found here and here.

Below are some of the editor’s (that’s me) picks. Be advised, I have not tested any of these utilities. Some Digital Forensics Tool Testing Images can be used in order to test data recovery software and compare their level of effectiveness.

All of these programs can operate on images generated with utilities like dd or directly on the hard disk partition.

• Foremost – This is a very popular file carver. The headers and footers of files to be recovered can be defined in a configuration file.
• Scalpel – This program has derived from Foremost 0.69 and, as it is stated in their home page, it is less resource hungry than Foremost; therefore, it can used in very low-end machines.
• Magic Rescue – This program uses the “magic bytes” in the file contents in order to recognize file types. The program uses its default “recipes” in order to recover files, but the user can define custom recipes.
• The Sleuth Kit and the Autopsy Forensic Browser are a set of command line tools and a graphical interface respectively, which can be used to investigate a hard disk. Actually, this is not for everyday data recovery, but rather applies to forensic analysis or to any other serious examination of the partition itself or of the linear representation of the file activity, etc.

The recommended operating system for most of the above utilities is Linux, but some of them can run under Windows too. By checking the above lists of software, you will be surprised by the number of open-source software that is available for data recovery or partition analysis.

Related Articles

• The importance of regular data backups
• How to recover lost files
• Choosing a format for data backups – tar vs cpio
• Effective data wiping with a single complete overwrite
• CloneZilla instead of Partimage

Related Articles

• Verify a burned CD/DVD image on Linux
• Creative Commons v3.0 Licenses Launched
• More Data Recovery Tools
• Minimal OpenSolaris Installation – OVF Image
• GRUB background image