Disk Fragmentation and System Performance

When addressing system performance issues, a key element that is often overlooked is Disk Fragmentation.  Even on a brand new system with plenty of RAM and high-end processors, the performance of the hard disk may be a bottleneck causing system performance issues.  It takes time to load large data files into memory - issues become particularly noticeable when dealing with movies, video clips, database files or .ISO image files which may easily be several gigabytes in size.  On a freshly formatted disk, these files load fairly quickly.  Over time, however you may start to notice performance degradation - caused by disk fragmentation.
We touched on disk fragmentation when we were discussing the Page File a couple of months ago, but we never really got into the nuts and bolts of it.  To understand disk fragmentation though, you need to understand the basic structure of hard disks.  When you format a hard disk, the formatting process divides the disk into sectors, each of which contains space for 512 bytes of data.  The file system then combines groups of sectors into clusters.  A cluster is the smallest unit of space available for holding a single file - or part of a file.  On NTFS disks, the cluster sizes are determined based on the drive size as shown below (this information is also available in Microsoft KB 314878).  When formatting disks it is possible to change the cluster size, however this may cause additional performance issues.

Drive Size (Logical Volume)	Cluster Size	Sectors
512MB or less	512 bytes	1
513MB - 1,024MB (1GB)	1,024 bytes (1kb)	2
1,025MB - 2,048MB (2GB)	2,048 bytes (2kb)	4
2,049MB +	4,096 bytes (4kb)	8

Using the information above, if you were to take a 100MB video file, the file would be divided into roughly 25,000 pieces.  If you save this 100MB file onto a freshly formatted disk, the information would be written in contiguous clusters.  Since all of the clusters holding the data for this file are physically adjacent to each other, the mechanical components of the hard disk work very efficiently, pulling the data in one operation.  In addition, the hard disk's cache and the Windows disk cache can anticipate data requests and fetch data from nearby clusters.  This data can then be retrieved by an application from cached memory which is faster than retrieving the information from the disk itself. 
Seems pretty straightforward, right?  The problem is that the hard disks don't stay neatly organized for very long.  Whenever you add data to an existing file, the file system has to allocate more clusters for storage.  Typically, these clusters wind up being in a different physical location on the disk.  As you delete files, you create gaps in the arrangement of the contiguously stored files.  As you save new files (and this is especially true for large files), the file system uses up all of these bits of free space - resulting in the new files being scattered all over the disk in noncontiguous pieces.  And thus we end up with fragmented disks and system performance issues because the disk heads have to spend time moving from cluster to cluster before they can read or write the data.
Enter Disk Defragmenter.  This utility physically rearranges the files so that they are stored (as much as possible) in physically contiguous clusters.  In addition to the consolidation of files and folders, the Defragmenter utility also consolidates free space - meaning that it is less likely for new files to be fragmented when you save them.  For operating systems prior to Windows Vista, you had to manually run the utility or schedule automatic defragmentation via a scheduled task.  On Windows Vista, Disk Defragmenter runs as a low-priority background task that is automatically run on a weekly basis without requiring user intervention.  On Windows Server 2008, which uses the same Disk Defragmenter, the automatic defragmentation is not enabled by default.  Also, the color-coded display that was part of earlier versions of the utility has been retired (believe it or not, more than a few people have asked about that!).  Aside from the GUI version of the tool, you can also use a command-line version that enables some more granular control over the process.  The utility name is DEFRAG.EXE and does require administrative privileges to run.  The basic operation of the utility involves passing it a driver letter, for example: defrag.exe c: would perform a defragmentation of the C: drive.  You can also specify other options through the use of command-line switches:

• -c: Defragments all volumes on the system.  You can use this switch without needing to specify a drive letter or mount point
• -a: Perform an analysis of the selected drive and provides a summary output (shown below):

• -r: Performs a partial defragmentation by consolidating only file fragments that are less than 64MB in size.  This is the default setting
• -w: Performs a full defragmentation by consolidating all file fragments regardless of size
• -f: Force defragmentation of the volume even if the amount of free space is lower than normally required.  When running this, be aware that it can result in slow system performance while the defragmentation is occurring
• -v: Displays verbose reports.  When used in combination with the -a switch, only the analysis report is displayed.  When used alone, both the analysis and defragmentation reports are shown.
• -i: Runs the defragmentation in the background and only if the system is idle
• -b: Optimizes boot files and applications, but leaves the rest of the drive untouched

So now that we've covered what disk fragmentation is and how to address it, there are some caveats.  You must have at least 15 percent free space on the disk volume before Disk Defragmenter can completely defragment the volume.  If you have less free space, then a partial defragmentation will occur (unless you force the defragmentation with the -f switch).  Also, you cannot defragment a volume that has been marked by the OS as possibly containing errors.  This is where you would need to use the CHKDSK.EXE utility to ensure that there are no underlying disk issues.  Some other things to look out for:

• Empty the Recycle Bin before defragmenting.  Disk Defragmenter does not defragment the Recycle Bin
• As we discussed in our Page File post, if you want to defragment the page file, you need to zero it out first and then defragment the disk
• By default, fragments that are greater than 64MB in size are ignored by Disk Defragmenter.  Fragments of this size (which already contain at least 16,000 contiguous clusters) have a negligible impact on performance
• Disk Defragmenter will not defragment files that are in use.  For best results, shut down all running programs, or log off and log back in as an administrative account before defragmenting the disk

//

Types of fragmentation[edit]

File system fragmentation may occur on several levels:

• Fragmentation within individual files and their metadata.
• Free space fragmentation, making it increasingly difficult to lay out new files contiguously.
• The decrease of locality of reference between separate, but related files.

File fragmentation[edit]

Individual file fragmentation occurs when a single file has been broken into multiple pieces (called extents on extent-based file systems). While disk file systems attempt to keep individual files contiguous, this is not often possible without significant performance penalties. File system check and defragmentation tools typically only account for file fragmentation in their "fragmentation percentage" statistic.

Free space fragmentation[edit]

Free (unallocated) space fragmentation occurs when there are several unused areas of the file system where new files or metadata can be written to. Unwanted free space fragmentation is generally caused by deletion or truncation of files, but file systems may also intentionally insert fragments ("bubbles") of free space in order to facilitate extending nearby files (see preventing fragmentation below).

File scattering

File segmentation, also called related-file fragmentation, or application-level (file) fragmentation, refers to the lack of locality of reference (within the storing medium) between related files (see file sequence for more detail). Unlike the previous two types of fragmentation, file scattering is a much more vague concept, as it heavily depends on the access pattern of specific applications. This also makes objectively measuring or estimating it very difficult. However, arguably, it is the most critical type of fragmentation, as studies have found that the most frequently accessed files tend to be small compared to available disk throughput per second.


To avoid related file fragmentation and improve locality of reference (in this case called file contiguity), assumptions about the operation of applications have to be made. A very frequent assumption made is that it is worthwhile to keep smaller files within a single directory together, and lay them out in the natural file system order. While it is often a reasonable assumption, it does not always hold. For example, an application might read several different files, perhaps in different directories, in exactly the same order they were written. Thus, a file system that simply orders all writes successively, might work faster for the given application.


//