May 07, 2008
Defragmentation, Diskeeper, and Disk Encryption
Back in the 1990's, Diskeeper Corporation (a Microsoft Gold Partner) and Microsoft co-wrote Windows kernel-level code, and exposed an API (Application Programming Interfaces), prior to release of NT4 for third party software providers. This "MoveFile" API as it is called, has allowed defragmentation programs to operate as user-mode applications ever since, through Windows Server 2008, Windows 7, etc... (and yes, Diskeeper 2008 works on Windows 7). The benefit of having integrated defrag based file movement in Windows is that all file movement in an online defragmentation operation by a product such as Diskeeper, is entirely handled by the operating system. From this viewpoint a defragmenter is logic that instructs the file system, via that API, to put file fragments that it discovers back together again into an available and contiguous free space on a given Windows volume. The process has been 100% safe for data since introduction in NT4.
More recently, the NTFS file system added and fully supports disk/file encryption as evidenced by EFS (Encrypting File System), and Microsoft products such as Windows Bit Locker.
It is fairly well known that Windows also ships with a native defragmenter (a product initially provided by Diskeeper for the Windows 2000 platform). It then stands to reason that Microsoft has a vested interest in maintaining continuing compatibility with defragmentation and, minimally, their own drive/file encryption solutions, and this is certainly the case.
Over the years our vast install base (10 million+) has brought to our attention rare and temporary incompatibilities that other products have had with the Microsoft MoveFile API.
Disk Encryption is becoming and increasingly more popular security measure undertaken in corporate enterprise - and even home use. Limitations in Windows provided tools, often drives IT professionals to seek out advanced third party disk encryption solutions (much as they do with disk defragmentation). In past years, as this technology grew from infancy in the Windows arena, a few disk encryption applications had temporary issues with the MoveFile API.
In every case to date, where it has ever even been an issue, the manufacturer of that Encryption software program has recognized this issue and corrected it, or offered workarounds. Those workarounds may require specific functionality in a third party defragmenter such as Diskeeper's File Exclusion feature, as the native defragmenter that ships with Windows lacks this. In those cases, depending on the technology implemented by the Encryption software vendor, a specific file (called a boot loader file) should not be moved.
In fact, with Diskeeper's technology and market leadership, we have worked closely with engineers and support staff from Encryption manufactures to ensure their products work perfectly for our customers.
For any current or future Diskeeper Corporation customers looking to employ a disk encryption program, we recommend checking the Encryption ISV's (Independent Software Vendors) Knowledge Base, Help files, or directly contacting the support staff to determine if any special measures are necessary. Based on our experience we also recommend choosing a mature and established vendor in this arena - sound advice for any company looking to deploy a large volume of licenses of any application. As always, you should make sure you are using the most recent version for any encryption solution, such as Guardian Edge EPHD 7.2, Utimaco (SafeGuard Easy) 4.20.x - 4.40.2 with their hotfix SGEflt.sys, or PointSec 6.3.1.
In summary, there are no known issues with running both disk encryption software and disk defragmentation software. Combining Diskeeper and data encryption software with provide users with a secure and optimized computing experience.
Posted by Michael at 12:52 PM | Comments (2)
April 29, 2008
Diskeeper has repositioned the bar...again.
We regularly get feedback from users on how Diskeeper, Sitekeeper or Undelete has helped them out. We have a section on our website where these are posted, so I won't make a habit of posting them here, but I really thought this story epitomized what we designed InvisiTasking to do:
"Being a long term user (since the offering of Diskeeper as the ONLY product able to defrag Server NTFS partitions back in NT4) I have had significant experience with Diskeeper. While the driving factor for our purchase of Diskeeper was the name recognition and reliability of the product, much to my surprise (and total glee!) Diskeeper is as revolutionary today, as it was in my past. With the addition of automation (instead of scheduled defrags, it now does it on the fly) Diskeeper has repositioned the bar...again.
The overall stability of the servers has stabilized - across 8 servers, I have a mean realization of 5% improvement - this is covering the spectrum from an SQL server realizing only 3% improvement to a File server realizing over 14% improvement in file handling, speed of requests being filled. The biggest measure has been that we have not had to maintain the monthly allocation of 4 hours to server maintenance. This 4 hours is the only acceptable downtime for our production servers, to be rebooted, disk-cleaned and defragged - not to include that this is the only time for hardware upgrades, major server software upgrades, etc. There were times prior to my installation of Diskeeper, that the maintenance window would come and go, with only 80% of the defrags completed, 60% of the servers rebooted - and a laundry list of application updates, which would have to be rescheduled to occur during production time.
After the first 2 months of allowing Diskeeper to do its job, I no longer had to manually defrag - it was done! I now get every action item scheduled for that maintenance window completed, with time to spare! If NOTHING else, anyone who maintains an aggressive server maintenance program will reap big rewards in the time saved over this process alone... "
Paul Parker
Houston, TX
Posted by Michael at 06:19 PM | Comments (0)
February 11, 2008
New Testing Paper on DK2008 Enhancements
If you are interested in reading up on the technical improvements that Diskeeper 2008 offers over previous versions, we finally got around to broadly publishing a white paper on the subject.
You can find it here, in an PDF version of our normally printed Business Customer newsletter - Newskeeper.
The article is titled Diskeeper 2008, Under the Hood.
Posted by Michael at 09:49 PM | Comments (0)
October 05, 2006
Diskeeper Hugs a Tree
Due to a record pace for company software volume license sales in 2006 (thanks to our customers volume sales have been up 4 quarters in a row), Diskeeper Corporation is returning the favor to the environment. The next Diskeeper version will be using chlorine-free, recycled paper for its retail packaged Diskeeper products. And, it's a practice we plan to continue for the future.
http://www.marketwire.com/mw/release_html_b1?release_id=168621
The huge revenue boost also allows us to concentrate on the expansion of our R&D teams - we actually have two independently operating Research teams in addition to our product-specific development groups. It's the Research program teams that have produced some of the current defragmentation pioneering technologies in Diskeeper 9 & 10, as well as the industry-wide ground breaking new technology that is at the core of the new upcoming release. A recent key acquisition to the team is Rick Cadruvi, the original co-creator (with our CEO) of Diskeeper for OpenVMS (back in the 80's). Rick was well known as one of the premier industry experts on OpenVMS operating system design. With many years of in-depth technical expertise on Windows, he re-joins us as a technology leader, and is already quite busy inventing great new technologies.
Look forward as we (the staff of Diskeeper Corporation) return our thanks for purchasing Diskeeper Corporation products in the form of revolutionary and technology leading new products to make your computers run faster and more efficiently.
Posted by Michael at 12:54 AM | Comments (1)
September 11, 2006
Industry Experts on Diskeeper
Paul Thurrott has been covering Vista builds for some time in his reviews and articles, often testing Diskeeper 10 as part of the review.
As he noted in his latest review (http://www.winsupersite.com/reviews/winvista_rc1_03.asp) there are ways to get Diskeeper installed on Vista RC1.
I completely agree with his comment that it's not a good idea to run Diskeeper on RC1; that is, until we officially offer a product.
Those who have tested the current Diskeeper betas for Vista have noted that some features are disabled or reduced in functionality. That is because we haven't fully tested them. I don't anticipate anything breaking, but some of our advanced features (e.g. FragShield) do some relatively fancy things (on Win2k and XP) that may not carry over well to Vista without proper adjustment on our part. For the patient enthusiast, we'll have a new RC1-compatible beta soon, and a "full" product when Vista is officially released.
----
Having written a paper on the subject earlier this year, I was interviewed by Processor magazine on server virtualization a couple of months ago. While a few points I discussed were slightly muddied in translation (e.g. "CPU searching for files" is technically incorrect wording - but at least conveys the message) the interview covered some points system administrators/computer power users need to look out for when building these systems.
Virtualization is a popular buzz word these days, but it is important to use this technology in the right places and under the right circumstances. That concept, everyone interviewed for the article agreed upon. There was, however, some debate on the importance of defragmentation.
Alessandro Perilli, well-known virtualization expert and host of virtualization.info (the leading independent site for everything you ever wanted to know about virtualization) added his comments on the subject as well. http://www.virtualization.info/2006/08/is-defragmentation-real-benefit-for.html
You can also check out Michael Otey's (Windows IT Pro) earlier "top 10" in February 2006, which counted defragmentation among the key priorities for VM performance. http://www.windowsitpro.com/Article/ArticleID/48726/48726.html?Ad=1
The simple fact behind these expert's viewpoint is that the disk is the weak link in a modern, general purpose, computer, slowing everything else down (where disk access is required). Alleviate the bottleneck and the whole system works faster.
Posted by Michael at 09:31 PM | Comments (0)
May 18, 2006
Defragmenting databases
In order to squeeze optimal performance out of a database, there are two "layers" that need defragmentation; the database file(s) themselves, and the internal records/indexes within those files.
Diskeeper, of course safely and automatically defragments files and free space (yes even live databases). Advanced Diskeeper features like I/O Smart ensure that defragmentation can occur during production without impact to the system.
Diskeeper does not offer, nor perform, internal file defragmentation. I've authored several white papers that discuss the internal defragmentation/compaction tools Microsoft offers with their Exchange and SQL products. It is those tools, development by the application vendors, that have the understanding required to address internal fragmentation. You can read up on those white papers in the Knowledge Center.
The problem with these free provided tools is they are cumbersome to use and integrate with IT operations. Fortunately there are vendors that have dramatically improved upon the basic scripting tools and offer powerful features so you can avoid the long and often complex coding normally required to operate them.
I'd like to introduce you to a couple of great products I'm familiar with and personally recommend.
Exchange Maintained is a robust program that provides scheduling for all the necessary actions to properly defragment and compact Exchange datastores. You can find out more about this great product at (http://www.fixthisbox.com/). With valuable features like backup, error checking and integrity verifications, this tool has it all.
Norb Technologies (http://www.norbtechnologies.com/) offers the intuitive and powerful Visual Defrag for Microsoft SQL Server 2005, 2000 and 7.0 with a flexible scheduler and detailed visual reports so you can target defragmentation where it's needed most, a SQL administrator can get helpful reports and keep on top of database performance.
Both vendors offer product tours for their products and are very reasonably priced. Check with them for trial versions.
Posted by Michael at 12:57 AM | Comments (2)
March 24, 2006
Apples to Apples
Following up on Paul's recent posting on free space consolidation, I felt the following blog has some relevance. Please note that Paul's post has received quite a bit of feedback and I will respond to that in the next few days, time permitting, because it is a very involved subject.
Just over three years ago Gartner, a leading IT Analyst group published a report on defragmentation, and specifically the impact of defragmentation methodologies on free space fragmentation. Gartner enjoys a well-earned reputation for quality and accuracy of its reports, and this typically is the case. However, they do on rare occasion make mistakes. I will play devil's advocate and argue a particular little misstep in this report. Keep in mind that the information about the built-in defragmenter is correct; however, the broader conclusion derived from data applies flawed logic.
In this defragmentation report Gartner states "The Windows 2000 Server built-in defragmentation tool is a multi-pass defragmenter that must be run over and over to defragment the disk, especially when defragmenting very large disks with heavy fragmentation and limited free space."
This is all true. You have probably seen this for yourself.
It continues, "As such, multi-pass defragmenters characteristically fragment remaining free space on the disk, which accelerates fragmentation later."
This is where Gartner goes astray. Very plainly stated, Diskeeper is an automatic multi-pass (I always think of Milla Jovavich in Fifth Element when I say that) defragmenter that has always consolidated free space. However the purpose of this blog is not to argue philosophic viewpoints on multi-pass versus single-pass. That's a whole separate lengthy discussion. I will only address the leap in logic.
For the sake of this discussion let's assume that fragmented free space actually does intrinsically accelerate future fragmentation. Again that is another lengthy separate subject, and one that is not black and white, which I will discuss in detail in a few days.
For kicks and giggles, I'll describe the logic in an analogy; embellishing on the story for mild entertainment purposes (well at least it was entertaining for me to write) and then toss in an entirely redundant quiz.
Apples to Apples:
Joe is hungry and, being a health conscious individual, wants a healthy snack. He goes to the local market to get a bite to eat. He passes by the potato chips and candy bars. Noting a green sourball in the candy section his mouth goes into a panic frenzy remembering how much he hates sour food. He quickly recoups and makes his way to the fruit stand. A juicy red apple, glistening with moisture from the recently run fruit-sprinklers, beckons him. He buys and eats the apple, savoring the sweet taste, while thinking; "I love red apples".
Why, pray-tell, does he not think to say "I love apples"? Because Joe is wise and knowledgeable and knows that not all apples are red. Many are green, and they typically have a slightly bitter taste. Joe doesn't like bitter or sour food, (he accidentally ate a crab apple from his backyard when he was 8 years old) and would never think to buy a green apple. In fact, Joe is such a learned individual he knows apples can come in yellow as well, and for that matter, with a little genetic engineering, could probably come in pink polka-dots!
An apple, by commonly agreed definition, has certain similar characteristics - that's why we call them all apples. But, Joe knows that color is just an attribute of the apple, just as is taste (sour/sweet) or size (large/small) or whether it is natural or engineered that distinguishes different types of apples. He knows that all apples are not the same.
-----
Here's a multiple-guess question as might appear on a Logic/IQ test:
Fact1: Sue is a girl
Fact2: Sue has red hair
Based on this data we can conclude:
A. All girls have red hair
B. Some girls have red hair
C. Stop beating this dead horse, I get the point, so finish the blog already
D. Both B and C, with A being the wrong answer (hint: pick this one)
Posted by Michael at 01:55 AM | Comments (0)
June 08, 2005
Diskeeper and email servers
Diskeeper works safely and effectively with email servers, whether they are Microsoft, Exchange,
Lotus, Domino, QUALCOMM, Eudora, or others. Diskeeper is uniquely designed to run in the background while these applications are operational and available to users. There is no need to stop or shutdown these applications or services to defragment.
There are two types of volume-centric fragmentation with which Diskeeper is immediately concerned: file fragmentation and free space fragmentation. File fragmentation concerns computer files that are not contiguous, but rather are broken into scattered parts. Free space fragmentation describes a condition in which unused space on a disk is scattered into many small parts rather than a small
number of larger spaces. File fragmentation causes problems with accessing data stored in computer files, while free space fragmentation causes problems creating new data files or extending (adding to) old ones.
Taken together, the two types of fragmentation are commonly referred to as disk or volume fragmentation.
Typically email application databases such as Microsoft Exchange and Lotus Domino are made up of a large container file that is preallocated in size at the point of creation. As the database increases beyond the initial assessment the file becomes fragmented.
Over a period of time, any popular email application server will also experience "internal" fragmentation of its database. This is where a record is removed, but the space it occupied within the database is still there and is either reused for a new record or must be skipped
over.
Let's say you have 250,000 records represented in an email server database. If an individual record (e.g. a deleted email) is removed, the location is simply marked as deleted. In the course of doing business hundreds, perhaps thousands of records are added and deleted. It doesn't take long for the internal organization of a database file, its indexes, and other related files to quickly become quite
disorganized. The speed of locating a particular record or segment of information is directly related to the amount time spent skipping over these holes or internal fragments.
The tools for Microsoft Exchange (ESE and EDB Utilities) deal with this internal record fragmentation by rearranging the internal records/indexes on the fly when possible, and at times requiring a whole new copy of the database to be created and each record copied to the new file. Even if this copy is done to a freshly formatted volume or a defragmented volume with a free space chunk large enough to contain the entire database, it's quite likely that this new copy will become fragmented. It is strongly recommended to run Diskeeper after performing email maintenance that rewrites the database. Otherwise, it is possible to actually worsen mail server performance due to additional required disk I/O.
The benefit of defragmenting an email server environment is no different than defragmenting any other system. It simply takes less time and system resources to access a contiguous file than one broken into many individual pieces. This improves not only response time but also reliability of the system. Thorough database maintenance requires a combination of Diskeeper (disk defragmentation), and the email server utilities (internal record/index defragmentation), to achieve optimum performance and response time.
Posted by Michael at 07:35 AM | Comments (0)
May 13, 2005
New white paper available
Executive Software has released a new white paper on the increase in disk fragmentation resulting from the global increases in volume size and quantity of files stored, in both personal and corporate computing environments.
The paper addresses the explosions in data volume resulting from new technology and dropping costs, the resulting increase in disk fragmentation and reduction in overall performance. While consistent defragmentation is becoming crucial to performance and stability, the volume explosion is making manual defragmentation of every workstation, laptop and server increasingly impossible.
Posted by Michael at 07:13 AM | Comments (0)