Monday, 27 June 2011

AIX JFS error


How to resolve issues with local JFS file systems that have run out of available space.
 
Resolving full filesystems:
 
To resolve an out-of-filesystem-space issue, complete these steps: 
·         Determine what filesystems are full. 
·         Determine where space is allocated within the source filesystem.
·         Take the required steps to resolve the out-of-space condition. 
 
Once the above steps have been completed, the situation should be resolved,
or the reason for the problem should be understood.
 
If these steps do not resolve the issue, filesystem corruption MAY be involved.Unmount the filesystem and run a full fsck against it to verify that no corruption problems exist.
 
Determining what filesystems are full ?
 
The df command is used to get filesystem status information. The relevant 
field to consider is %Used.
 
%Used = percentage of total filesystem space currently allocated
 
Example: 
 
   Filesystem   1024-blocks      Free %Used    Iused %Iused Mounted on
   /dev/hd4           12288        68   99%     1823    23% /
   /dev/hd2          409600     20436   96%    16181    16% /usr
   /dev/hd9var         8192      6088   26%      163     8% /var
   /dev/hd3           12288     11340    8%       87     3% /tmp
   /dev/hd1           57344     13872   76%     1459    11% /home
In this example, most of the available free space in the root filesystem / 
is allocated. At this point, we have determined free-space problems on the / filesystem. The next issue is to determine what kind of space problem exists. 
Keep in mind that the mounts are hierarchical and have a bottom-up precedent. 
In other words, a filesystem mounted below a second filesystem cannot access 
data in the second filesystem. For example, if you have a mount entry called 
/myfilesystem/mydata immediately followed by a mount entry called /myfilesystem,then the /myfilesystem mount point cannot access the /myfilesystem/mydata filesystem and any data that resides there. 
 
Determining where space is allocated within each filesystem?
 
There are two commands generally used to determine how and where filesystem
allocation is placed: df and du . 
 
df uses the space in a filesystem that is currently unallocated to determine
the space that is used in a filesystem. For instance, if you have a filesystem that consists of 8192 512-byte blocks, and 4096 of those blocks are currently not allocated to anything, then the total space being used by the filesystem would be 4096 512-byte blocks. 
 
Allocated Storage = Total Storage - Unallocated Storage
 
df is inherently the most reliable command to report filesystem usage, because df reports information based on the filesystem as a whole.
 
du is a file-oriented command. It reports the space allocated to a specified 
file or directory. du must have a destination parameter, and is not isolated 
to a filesystem. For instance, running du / would give allocation information 
for all files in / . This would include all files in the / filesystem and any 
other filesystem mounted under / , such as /tmp, /var, and /usr . You could use the -x option of du to keep the operations within the filesystem, but there are cases where the results of using this option may be incomplete. 
 
du will only report space taken by files. It will not report space taken by 
filesystem metadata, such as inodes, inode maps, or disk maps. inode/disk maps and other reserved areas for filesystem use will take up a negligable portion of the filesystem space, but the areas reserved for inodes can be substantial, and is ultimately based on the NBPI (Number of Bytes Per Inode) chosen when the filesystem was created. Each inode uses 128 bytes of filesystem space, so the amount of space taken for inode use will be the percentage defined below:
 
  (128 / NBPI) * 100 
 
By default, a filesystem will use a NBPI of 4096, so the general overhead for
a filesystem will be about 3%. 
 
 
To determine what the NBPI is for the filesystem in question, issue the lsfs 
command with the -q option on the mount point of the filesystem
 
Example: 
# lsfs -q /
Name      Nodename   Mount Pt    VFS   Size    Options   Auto  Accounting
/dev/hd4  --         /           jfs   81920   --        yes   no 
(lv size: 24576, fs size: 24576, frag size: 4096, nbpi: 4096, compress: no, bf : false, ag: 8)
 
 
du will only show allocated information about files it can reference. There are two cases where du may not show information about allocated storage. 
 
The file is hidden because a filesystem or file has been mounted on top of this entry. If you had a file that was stored in /bobby, and then mounted a 
filesystem on top of /bobby, then du would no longer see what was in the 
directory /bobby. It would only see the information in the filesystem that was mounted over /bobby.
 
The file is open by other applications, and the file has been removed. In this case, the storage for that file will remain allocated until all references to that file have been closed. Without a filesystem entry, du will not show allocated space for that file, though df will show this space taken from the filesystem as a whole.
 
 
 
Determining files that are using space in a filesystem
 
To address the situation presented in case 1 in the previous section, mount 
the primary mount point of the desired filesystem on a secondary mount point. 
This has the effect of negating any filesystem mounted under the primary 
mount point. 
Example: 
 
   mount / /mnt
 
 
In the above example, we mounted the / filesystem over /mnt . The effect is 
that if we go into /mnt , we see all the information about the / filesystem 
and no other filesystem mounted under / . If we run cd /mnt/tmp , then we are 
actually in the directory /tmp in the / filesystem, and not in the /tmp
filesystem. Also, if we run du -sk /mnt , it should closely match the %Used 
for / from the df command. If it does not, then this indicates that case 2 
may be occurring. For now, we will proceed with case 1. 
 
We can now investigate disk usage accurately. First, go into the root directory of this filesystem. 
 
   cd /mnt
 
Run the du command to get an accurate accounting of the space that can be seen for all accessible files in this filesystem. 
 
# du -sk /mnt
11778   /mnt
 
This will report the accounted space taken for files in kilobytes. If you add
the overhead of the filesystem described above, this figure should closely
match that given by the df command. 
 
Example: 
# df -vk /
Filesystem  1024-blocks  Used  Free  %Used  Iused   Ifree %Iused  Mounted on
/dev/hd4    12288        12220  68     99%   1823    1249   23%      /
 
The overhead in this case will be (128/4096) * 12288K = 368K 
 
The total space that can be accounted for is 11778K + 368K = 12146K versus the reported space taken from df of 12220K. This outcome is accurate and indicates that any space seen is accounted for in a file somewhere in the filesystem. If the difference between the two is large, then this indicates case 2 is more likely to be occurring. The next section , "Resolving Space Taken by Open Files That Have Been Deleted" addresses the situation in case 2. If not, continue with the following steps.
 
Run the following command on the new mount point of this filesystem to get 
a sorted disk usage of the filesystem's root directory. 
 
 
   ls -A . | while read name; do du -sk $name; done | sort -nr
 
Example: 
 
   # ls -A . | while read name; do du -sk $name; done | sort -nr
   2168    etc
   192     lpp
   168     sbin
    40     dev
    28     export
    12     smit.log
     4     var
     4     usr
     4     tmp
     4     tftpboot
     4     src
     4     smit.script
     4     mnt
     4     .sh_history
     4     .profile
     0     unix
     0     u
     0     lib
     0     bootrec
     0     bin
This command sorts disk usage for all files in the current directory by size, 
in decreasing order. If the file we suspect happens to be a directory, we can 
then change into that directory, and re-run the preceding command to determine what is taking up space within that directory. Continue these steps until you find the desired file or files, at which point you can take appropriate actions. 
 
 
Resolving space taken by open files that have been deleted
 
In case 2, there are files within the filesystems that are opened by 
applications but have been removed from the filesystem tree. This behavior 
is documented in the unlink() system call as follows.
When all links to a file are removed and no process has the file open, all 
resources associated with the file are reclaimed, and the file is no longer 
accessible. If one or more processes have the file open when the last link 
is removed, the directory entry disappears.
However, the removal of the file contents is postponed until all references
to the file are closed. 
 
You can use the fuser command with the -dV flag on the full path to the 
device on which the filesystem resides. This will display files that have
been removed but are still open. It will also report the inode number and 
size of such files. Using the process ID returned for these files, you can
instruct the source application to close these files, or you can exit the 
application. Once this has occurred, and fuser no longer shows this deleted 
file, the space will be returned to the filesystem for general use. 

No comments:

Post a Comment

Twitter Bird Gadget