March 2006 | Main | May 2006

April 20, 2006

Virtual Server Defrag Heating Up

Virtual server defragmentation is getting even more attention with the media coverage of our recent partnership initiative with Microsoft. In case anybody missed it, Microsoft's recent announcement mentioned us as one of their key partners involved in pushing forward virtualization technology. See articles in Redmond Magazine, InformationWeek, and CRN.

Many IT professionals are currently using Diskeeper to defragment Virtual Server 2005 and VMWare with much success, however there is still work to be done to ensure that the most thorough defragmentation job can be achieved on the host and virtual levels.

These technical challenges are why Microsoft has tapped us yet again to push forward defragmentation technology on the Windows platform.

Server virtualization has the effect of making the mechanical disk drive an even bigger bottleneck. As I've said before on this blog, automatic defragmentation should play a key part in your virtualization strategy.

—Paul

Posted by paul.shomo at 09:47 PM | Comments (0) | TrackBack

April 11, 2006

Yet even more on Free Space Defrag....

As I suggested in an earlier blog, we have had a handful of comments on Paul's free space blogs. Given the comments, and the relative ambiguity of this whole discussion, I feel I need to clarify Diskeeper's viewpoints on the matter a bit further. I apologize in advance if this blog seems to originate from the Department of Redundancy Redundancy. Also, I apparently have some as yet undiagnosed affliction that causes me to be exceedingly long-winded. I seem to suffer from the inability to write anything less than a novel about any subject; so a second apology for the blog length.

To business...

Is it possible that some comments originated from Diskeeper Corporation that said free space consolidation is not important, or did not grant it the proper value? Sure it's possible. Personally I have not seen them, but that might just be me, or I may interpret them differently. What I would argue against is the misuse of a comment: i.e. where a statement is taken out of perspective (a frequent topic of late). A case in point would be misinterpreting Microsoft saying free space defragmentation is important, to meaning that free space needs to be "all in one pool". Microsoft correctly, even if with some ambiguity, values free space consolidation and states in the apparently somewhat disputed report that "Ideally, this space would be available in a few contiguous portions of the disk."

So exactly how many is "a few" - that is up for debate and personal interpretation. Yes, one could interpret this to say the best application of "a few" is equal to "one". However, Microsoft does not implicitly say "one".

I would disagree, from a technical perspective, that one chunk of free space is EVER better, because given two free space fragments: even on a static volume (more on volume data flux later) the absolute worst result is two fragments for a newly created file. Getting free space into one chuck can easily mean moving half of all the data on a volume. Even thousand of free space fragments can be perfectly fine in certain environments (e.g. where the volume writes and deletes millions of small files -files at least large enough to expand out of the MFT that is).

Also, keep in mind that if you only have a file in two or three or half-dozen fragments you do have fragmentation, but you don't have a fragmentation problem. Fragmentation is a concern because is affects performance and reliability. If it didn't have negative side effects neither you nor I would be here writing, reading or caring in general about this. And remember that fragmentation was actually invented as a solution to file systems that were unable to write to a volume if the operation could not be executed contiguously. All production had to stop and files shuffled around off-line before data storage could resume! Let us not forget the sins of our past.

Returning to the Microsoft statement, I agree with defining "a few" as a "handful", and always have. OK, so how many is "a handful"? A good way to understand how Diskeeper defines this is in the Diskeeper Performance display. This is an accurate, albeit general depiction of whether or not the fragmentation of a file is affecting its read performance. Data is accessed in various methods (e.g. 8K OLTP, 64K Stream. Read, Video Seq., etc), and this technology does not portend to know them all. It uses a standard read-file methodology.

Let me state for the record that "Diskeeper Corporation has always believed in the importance of free space defragmentation." It is not a change of heart or a newly discovered conviction.

Here's a bit more technical perspective on free space and re-fragmentation:

It is well known that NTFS is, for the most part, a relational database (deviates from the formal structure with the directory's hierarchical implementation), that uses a transaction processing model. As such, NTFS metadata (which is stored on the volume) and other dynamic files are modified constantly as data changes atomically on the volume. Add-in the fact there are many, many other user/program interactive dynamic changes (print spools, application data, temporary files, etc) and you end up with a constantly changing stream of data on a volume. Yes, the dynamism of some volumes is far greater than others. A database's transaction volume, a clustered system's quorum disk, a system's boot volume, a printer's spool file, the volume on which a user profile is stored; are all going to be in greater flux than say a disk volume on which data is archived. And yes it is good practice, where possible and practical, to separate dynamic data from static data by partitioning. And example would be separation of SQL database logging activities from the data (i.e. MDF and LDF) files. While on this aside, a separate physical disk for the SQL logs is ideal.

Do check out the whitepaper Paul referred in an earlier blog on free space. You can also view a live visual demo of the way file writes occur in modern NTFS in the More Info/FAQ section of the Diskeeper Multimedia Tour.

Applying Diskeeper's Defragmentation Methods...

Where do we suggest using the "Comprehensive Method" and where the "Recommended"? In most cases, Recommended is the way to go. For dynamic volumes it is the best bet to keep fragmentation down without overusing resources to consolidate free space, that isn't likely to actually benefit. It is a specially designed "automatic" algorithm built from years of experience.

For the less common volume that hosts very, very large files and doesn't have the high data fluctuation associated with certain programs or system purposes, use the more aggressive "Comprehensive Method". We have received rave reviews for our enhancements in comprehensive defragmentation, but it may not be at that level for everyone. If it is not meeting your needs, please contact our Tech Support staff and provide them the details of your problem. That information is escalated to developers (and myself) whose job it is to continually improve the product.

In either case we strongly recommend running regular AUTOMATIC defragmentation. With Diskeeper's "do what needs doing, don't overkill" approach, it avoids "the cure becoming worse than the disease syndrome" (taking more resources than it returns).

As the product evolves look forward to Diskeeper becoming smarter and recognizing the environment it is in and learning-adaptive algorithms to fit the circumstances.

OK, we goofed...

As we promised we will post critical comments (see the Comments in Paul's earlier blog entries), so long as they are constructive, and I feel that all the responses we have had on the recent free space discussion have been that. I very much appreciate the comments we have received, and the opportunity to respond. In fact, we have yet to receive a negative for the sole purpose of being negative (am I tempting someone here :-) ). I think that speaks highly of those who follow this blog, and I hope that we learn from your comments and improve and clarify our position on the technical matters which our products are designed to solve.

-Michael

Posted by Michael at 08:35 PM | Comments (3)

My last entry

Regarding my last entry:

Someone pointed out (correctly) that the article from Microsoft was not in fact new. I had picked it up on my google alerts and had some difficulty verifying the date it was written. I offer my apologies for the mistake.

I also want to point out that some of the commenters on my entry are confused about free space defragmentation. All 3rd party defragmenters defragment free space. Everyone agrees that it needs to be defragmented.

The only question is to what degree. Some in the industry have promoted free space "consolidation" or the putting of all the free space into one-pool as an attempt at product differentiation. We disagree with this approach of putting it into one-pool, and always have.

Has Diskeeper improved over the years at free space defragmentation? Yes. Should we suddenly switch to "consolidate" free space into one-pool? No.

The point of my last post was to point out that the one-pool philosophy is incorrect. Going the extra mile to put all of the free space into one-pool is a waste of resources as it only temporarily creates a pretty disk map. In the article I referenced, Microsoft also endorses a different approach when they state that free space should be in, "a few contiguous portions of the disk."

I will let the Diskeeper Product Manager Michael elaborate further.

Posted by paul.shomo at 07:07 PM | Comments (0)