DSMCHECK (c) - Disc Space Preview Tool 2001.11.15 REQUIREMENTS ------------ The DSMCHECK utility requires that the target operating system be MPE/iX release 5.0 or greater. INSTALLATION ------------ The DSMCHECK installation kit is delivered as a self-extracting file, containing several MPE NMPRG files that have been downloaded as PC files in WRQ Reflection format. (If you do not use Reflection, please contact your Sales Representative for a DAT tape.) Once the extract file has been copied to your PC, execute it (Start Menu RUN command or double-click in Explorer). The DSMCHECK fileset will be extracted to directory named "dsmcheck", which will be built just below the root-level directory (for example: C:\dsmcheck). Then run REFLECTION, log on to your HP3000, change to the desired destination Group and open the REFLECTION file transfer window. For 'Transfer Type', chose 'Labels', then upload the fileset (currently 8 programs). The program file DSMCHECK must reside in a group which has IA, BA, PH and PM capabilities. Furthermore, it may be desirable to assign a lockword or adjust group/account access to as to restrict user access. DISCLAIMER ---------- YOU ARE GRANTED THE RIGHT TO USE AND TO MAKE AN UNLIMITED NUMBER OF COPIES OF DSMCHECK. YOU MAY NOT DISASSEMBLE, DECOMPOSE, REVERSE ENGINEER, OR ALTER ANY OF THE PROGRAM IN ANY MANNER. THIS SOFTWARE IS PROVIDED AS SHAREWARE AND CANNOT BE SOLD, BUNDLED WITH ANY OTHER SOFTWARE OR DISTRIBUTED BY ITSELF WITHOUT THE EXPRESS WRITTEN AUTHORIZATION OF SOLUTIONSOFT SYSTEMS, INC. THE AUTHOR AND DISTRIBUTOR OF THIS PROGRAM ACCEPTS NO RESPONSIBILITY FOR DAMAGES RESULTING FROM THE USE OF THIS PRODUCT AND MAKES NO WARRANTY OR REPRESENTATION, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. THIS SOFTWARE IS PROVIDED "AS IS", AND YOU, ITS USER, ASSUME ALL RISKS WHEN USING IT. OVERVIEW -------- The Disk Space Preview Tool is a product that is made available at no cost. It is intended to be used as a tool to provide a preview of the condition of a disk space environment. The output from the preview tool provides information for evaluating- 1)free space fragmentation, 2)free space wasted beyond the EOF, 3)fragmentation of files and 4)file distribution imbalance. This evaluation will enable a system manager to determine if he would like to go further and consider the Disk Space Manager tool that is used to correct these problem conditions. A description of the Disk Space Manager is the last page of this document. For information regarding either this Disk Space Preview Tool or the Disk Space Manager contact your SolutionSoft representative at 888-884-7337 or 408-346-1400. QUICK START GUIDE ----------------- For users who do not have time or do not want to read the usage detail, just follow these instructions and send the DSMOUT file to SolutionSoft for a follow-up explanation. :RUN DSMCHECK :LS 1 :LF @.@.@ 1 40960 :EXIT USAGE DETAIL ------------ To run the Disk Space Preview Tool just execute the following command: :DSMCHECK (C) SolutionSoft Systems, Inc., 1997-2001. All Rights Reserved. C.06.02 (8/06/2001) DSMCHECK> To begin with, the list of commands that are available with this tool can be listed with the HELP command: DSMCHECK>HELP (DO) Do [redo_stack_index] (EX) Exit (HE) Help (LF) ListFile file_set | ^file_name [extents [size]] (LM) ListMode [SECTOR | KBYTE | PAGE] (LR) ListRedo (LS) ListSet ldev (RE) Redo [redo_stack_index] : mpe_command_line In the next few pages the use of the ListSet and ListMode commands are demonstrated. The default unit of measure is SECTORS and this is what is used in these examples. This unit of measure can be changed with the ListMode command. In the ListSet command the volume set (a volume set consists of one or more volumes/ldevs) is designated by any ldev # that is a member of the volume set being designated. DSMCHECK>LS 1 -------------------------------------------------------------------------- LDEV : 2 -- (MPEXL_SYSTEM_VOLUME_SET:MEMBER2) LARGEST FREE AREA: 92944 TOTAL FREE SPACE: 1004736 0 BLOCK(S) OF 1- 9 CONTIG. SECTORS = 0 FREE SECTORS. 0% 405 BLOCK(S) OF 10- 99 CONTIG. SECTORS = 18144 FREE SECTORS. 2% 363 BLOCK(S) OF 100- 999 CONTIG. SECTORS = 86880 FREE SECTORS. 9% 58 BLOCK(S) OF 1000- 9999 CONTIG. SECTORS = 245152 FREE SECTORS. 24% 28 BLOCK(S) OF 10000- 99999 CONTIG. SECTORS = 654560 FREE SECTORS. 65% 0 BLOCK(S) OF 100000-AND UP CONTIG. SECTORS = 0 FREE SECTORS. 0% -------------------------------------------------------------------------- LDEV : 1 -- (MPEXL_SYSTEM_VOLUME_SET:MEMBER1) LARGEST FREE AREA: 338944 TOTAL FREE SPACE: 1163104 0 BLOCK(S) OF 1- 9 CONTIG. SECTORS = 0 FREE SECTORS. 0% 330 BLOCK(S) OF 10- 99 CONTIG. SECTORS = 15008 FREE SECTORS. 1% 266 BLOCK(S) OF 100- 999 CONTIG. SECTORS = 60944 FREE SECTORS. 5% 83 BLOCK(S) OF 1000- 9999 CONTIG. SECTORS = 231200 FREE SECTORS. 20% 3 BLOCK(S) OF 10000- 99999 CONTIG. SECTORS = 47504 FREE SECTORS. 4% 4 BLOCK(S) OF 100000-AND UP CONTIG. SECTORS = 808448 FREE SECTORS. 70% -------------------------------------------------------------------------- TOTALS FOR MPEXL_SYSTEM_VOLUME_SET (IN SECTORS): LARGEST FREE AREA: 338944 TOTAL FREE SPACE: 2167840 0 BLOCK(S) OF 1- 9 CONTIG. SECTORS = 0 FREE SECTORS. 0% 735 BLOCK(S) OF 10- 99 CONTIG. SECTORS = 33152 FREE SECTORS. 2% 629 BLOCK(S) OF 100- 999 CONTIG. SECTORS = 147824 FREE SECTORS. 7% 141 BLOCK(S) OF 1000- 9999 CONTIG. SECTORS = 476352 FREE SECTORS. 22% 31 BLOCK(S) OF 10000- 99999 CONTIG. SECTORS = 702064 FREE SECTORS. 32% 4 BLOCK(S) OF 100000-AND UP CONTIG. SECTORS = 808448 FREE SECTORS. 37% -------------------------------------------------------------------------- TOTALS FOR MPEXL_SYSTEM_VOLUME_SET (IN SECTORS): FREE FREE PERMANENT TRANSIENT (IN SECTORS) 122416 2045424 14134608 358608 (TOTAL) Time: 10:50:11:00 started for ldev 2 Time: 10:50:17:00 100% completed for ldev 2 Time: 10:50:17:00 started for ldev 1 Time: 10:50:47:20 100% completed for ldev 1 -------------------------------------------------------------------------- TOTALS FOR MPEXL_SYSTEM_VOLUME_SET (IN SECTORS): TOTAL FILES SCANNED: 27850 TOTAL FILES QUALIFIED TO BE TRUNCATED: 488 TOTAL SECTORS QUALIFIED TO BE TRUNCATED: 2613504 The histograms that you see above are identical to those provided when running DISKFREE A. The level of fragmentation is indicated by the percentage of free space that is in the categories less than 999 contiguous sectors as well as looking at the number of blocks. The total line, as shown above, headed by "FREE FREE PERMANENT TRANSIENT" gives the amount of free space that is less than 256 sectors under the first FREE heading. This size of contiguous free space is virtually useless. Free space sizes greater than 256 up to 2048 sectors are somewhat useable but not when creating larger files. The second condition that can be analyzed from this command is the amount of free space that can be returned if the space beyond the EOF in flat files is released. The FILE LIMIT is not affected when this free space is returned. This means that if the file is appended to later the needed space is automatically allocated at that time. In this example, 2,613,504 sectors of free space can be gained. With 256 bytes per sector this is 256 X 2,613,504 = 669 Mb of disk space that can be changed from PERM space to FREE space with no negative effects. The TruncateSet (TS) command of Disk Space Manager is used to regain this free space. After this, the Disk Space Manager command, DefragSet (DS), would be used to move all free space into nice big pieces of contiguous free space. This ListFile command can be used to examine files and print out fragmentation and distribution information for those files that meet the selection criteria of number of extents and/or file size. The following example will examine all files and print fragmentation and distribution information for files that have at least 50 extents while disregarding file size. DSMCHECK>LF @.@.@ 50 FILE: DBCBPMSG.TRUST.ALGEN72 LDEV # of Extents Min Size Max Size Disk Space % of File 1 30 2048 2048 61440 38.13 2 52 256 2048 99696 61.87 TOTAL: 82 256 2048 161136 SECTORS FILE: O4114.OUT.HPTARG LDEV # of Extents Min Size Max Size Disk Space % of File 1 49 608 2048 98912 35.96 2 89 256 2048 176128 64.04 TOTAL: 138 256 2048 275040 SECTORS FILE: MAIN.SPDATA.ROBE LDEV # of Extents Min Size Max Size Disk Space % of File 1 18 16 1712 14640 30.89 2 33 16 1712 32752 69.11 TOTAL: 51 16 1712 47392 SECTORS FILE: NLP.PUB.NED LDEV # of Extents Min Size Max Size Disk Space % of File 1 60 256 2048 116736 100.00 TOTAL: 60 256 2048 116736 SECTORS FILE: PLP.PUB.NED LDEV # of Extents Min Size Max Size Disk Space % of File 1 63 256 2048 122880 100.00 TOTAL: 63 256 2048 122880 SECTORS FILE: BIGS.UMB1.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 158 256 2048 316016 100.00 TOTAL: 158 256 2048 316016 SECTORS FILE: LOOP2601.EMP.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 74 128 2048 143488 100.00 TOTAL: 74 128 2048 143488 SECTORS FILE: SET01.TRUST.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 222 256 2048 446816 100.00 TOTAL: 222 256 2048 446816 SECTORS FILE: SET02.TRUST.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 213 256 2048 428592 100.00 TOTAL: 213 256 2048 428592 SECTORS The next example will examine all files and print fragmentation and distribution information for files that have at least 1 extent (which is all files, but this parameter is required) and greater than 440000 sectors: DSMCHECK>LF @.@.@ 1 440000 FILE: CPYDB302.SED.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 1 800016 800016 800016 100.00 TOTAL: 1 800016 800016 800016 SECTORS FILE: PDB02.SED.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 1 800016 800016 800016 100.00 TOTAL: 1 800016 800016 800016 SECTORS FILE: TRXDB302.SED.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 1 800016 800016 800016 100.00 TOTAL: 1 800016 800016 800016 SECTORS FILE: SET01.TEMP2.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 1 446816 446816 446816 100.00 TOTAL: 1 446816 446816 446816 SECTORS FILE: SET01.TRUST.ESTBASE LDEV # of Extents Min Size Max Size Disk Space % of File 3 222 256 2048 446816 100.00 TOTAL: 222 256 2048 446816 SECTORS There are several things to consider when evaluating file fragmentation and distribution. Before these are listed it should be pointed out that all of these considerations are optimized for all files through the execution of one intelligent command, ManageSet (MS), which is part of the Disk Space Manager. This command can be run on-line, real-time with total safety (this means without doing a backup first or running any integrity analysis tools before or after). ManageSet runs with incredible speed and an amazing low resource utilization. See the last page of this document for a description of the Disk Space Manager and the ManageSet command. Regarding file fragmentation: For files that are less than 512k in size, they should be in 1 extent on one drive because they can be read in one I/O if they reside in contiguous disk space. For files larger than 512k in size, they should be spread across all disk drives in their volume set with 2 extents per drive. More extents for a file creates the need for more I/O activity on that file. Regarding file distribution: Larger files benefit greatly in access time if they have more disk heads available to do the accessing. This is why it is best to have these files spread evenly across all disk drives in the volume set. The more even the spread the greater the chance that the I/O load associated with the file will be divided among all drives. This greatly decreases the likelihood of I/O disk head contention. Another level of consideration is that not only should the allocated file space be spread evenly across all drives, but the file records with actual data in them should be spread as evenly across all drives as well. Consider an IMAGE database file set. This file will necessarily be fully allocated but may only be half full of actual data records. A spread of 1 extent per drive across a four drive system, such as done by an UNLOAD/RELOAD, would yield a distribution of data only on the first two drives and nothing on drives 3 and 4. The ManageSet defaults to 2 extents per drive to provide for a more even spread of the actual data. Another consideration is that an UNLOAD/RELOAD always starts with the same ldev when it reloads the files. This means that all of the partially full files will have their data toward the same drives. This increases the likelihood of I/O bottlenecks on these drives. Again, ManageSet helps to balance the data as well as the file by starting a file's spread on a different ldev than that used to start the previous file's spread. This is done in a round robin fashion. MPE/iX is designed to not place non-system file data on ldev 1. No new extents will be allocated on ldev 1 until the free space is all used up on the other drives on the system volume set. This can be of concern in a 4 drive or less system volume set that is used for application data. If ldev 1 is larger than about 1.5 Gb, the chances are that performance would be helped in a 4 drive system by utilizing all 4 drives instead of just 3 to access application data. With the ManageSet command a parameter called USEONE can be applied to force the use of ldev 1. Of course, this forcing is done only while maintaining good performance guidelines for minimum available free space on ldev 1. Finally, when adding new disk drives, if permanent space is not balanced out across the new drives, there will be significant performance degradation. This may be immediate and therefore highly noticeable or gradual and less obvious, but it will nevertheless be real. With the Disk Space Manager's PopulateSet (PS) command the files can be spread evenly across all drives including the new drives. This eliminates the necessity of performing an UNLOAD/RELOAD to accomplish disk drive balancing. C O N C L U S I O N S ===================== WHY DISK SPACE MANAGER? ----------------------- * PERFORMANCE can be severely impacted by fragmentation of free space and fragmentation of files on the Spectrum Series of HP3000's. * PERFORMANCE can be severely impacted due to poor I/O balancing among available disk drives. * DISK SPACE that should be available free space is wasted in many files beyond the end-of-file. HOW DISK SPACE MANAGER IS CONSTRUCTED IS AS IMPORTANT AS WHAT IT DOES: ---------------------------------------------------------------------- FAST AND SAFE * Disk Space Manager is built on top of the MPE/iX Transaction Manager to assure maintenance of system integrity. * The first time through average free space defragmentation time is about five minutes per gigabyte. * The first time through average truncation time is about two minutes per gigabyte. * Running routines regularly results in much faster times. CONVENIENT * Roughly 95% of full defragmentation can be accomplished on a system while users process as normal. Successive iterations process in just seconds and usually accomplish full defragmentation while users continue to process as normal. * Any Disk Space Manager process can be terminated at any time without losing what has already been accomplished. * Low system resource utilization means other users on the system will not notice that Disk Space Manager is running. WHAT DOES DISK SPACE MANAGER DO? -------------------------------- TRUNCATION - Free space is returned from the unused portion of files that MPE has left behind. This is often 5 to 15% of total disk space. Truncation does not alter the EOF or the file-limit or even the date the file was last modified. DEFRAG - Unusable and inefficient blocks of free space are consolidated into larger efficient blocks. SPREAD - I/O performance is enhanced when large files are distributed across multiple disk drives. This option reduces file fragmentation to exactly what is desired all the way to one extent per drive. COMBINE - Small files and program files can be combined onto one disk drive and one extent to improve system performance. INTELLIGENT MACRO COMMANDS: POPULATESET - New drives are incorporated into a volume set through balancing out the I/O load without having to reload the system. MANAGESET - Truncate, Defrag, Spread, and Combine are performed automatically according to an intelligent algorithm that yields a managed volume set that will out-perform an unload and reload. ManageSet can be performed while the system is in normal use. Because ManageSet is doing a great amount of spreading and combining, users may notice an impact on response time. If users notice an unacceptable slow down of their response times - no problem - just terminate the process and resume later. All that has been accomplished remains accomplished. When you restart ManageSet, all files that meet the desired spread and combine criteria will be skipped. No time, effort, or resources are wasted. MANAGEFILE - Allows ManageSet functionality to be focused on a designated file set. This is critical when a very active database can be made available for only a few minutes.