The Curious Case of Missing Quota Report on Volume

A week ago one of my old colleagues ask some help with an error he has on one of his servers, I was happy to oblige.

He had several servers all working exactly the same, same hardware, same OS and same configuration (apart from IP’s).
One of his server crashed and after the reboot one of his batch scripts stopped working specifically on this server.
The script tried to determind if a specific volume (for example: /mnt/volA) exist and have enougth free space,
This he did by invoking the “df” command and the “quota” command (Or their equivalent in the specific language).

What he found was:
While running: “df -k /mnt/volA” he got the correct line and response,
But on the other hand when he tried to run “quota -v |grep “/mnt/volA” the quota command didn’t return the volume.
This was particulary wierd because this volume was NFS volume with specified quota (set on the NFS server) – moreover the volume did apeer on the other servers when you looked for it when you’ve used “quota”

I’ve looked again and again on his configuration without having the slight idea what was the issue.
Then I’ve remembered one of my bosses once saying to me – “did you RTFM? no? then why do we have this conversation – go read the fucking manual!” then I’ve said well I’ve got nothing to loose, let’s read quota’s manual.

Following the manual I’ve found:

1
2
<b>-A, --all-nfs
    </b>report quotas for all NFS filesystems even if they report to be on the same device.

So I’ve tested “quota -vA” and that did the trick! The volume was there on the output.

So I’ve checked further on why the heck this volume didn’t showed before with the “-v” flag.
Something caught my eye in the manual: “report quotas for all NFS filesystems even if they report to be on the same device.

Then it hit me – I’ve looked to the mounts:
server.xxx:/vol1/volB on /mnt/volB
server.xxx:/vol1/volA  on  /mnt/volA

So what happened? My friend use autofs to mount it’s NFS volume, this meant that the volume mount sequence didn’t always performed the same and the volume would be sorted the same way.
Adding to that is the fact that volA and volB both are just a directory inside a volume on the NFS server – thus meant they both have the same quota (quota set on volumes/groups/users not directories.
So for the past couple of years his script has luck! Because volA always mounted before volB – but after the last restart he did to the server volB was the first to mount by chance then it did show!

When I’ve checked that I found this is the case on all his servers:

1
2
3
4
5
6
7
8
9
Server A (problematic server):
# quota -v |grep volA
# quota -v |grep volB
server.xxx:/vol1/volB  on  /mnt/volB
 
Server B (Other working server):
# quota -v |grep volA
server.xxx:/vol1/volA  on /mnt/volA
# quota -v |grep volB

So what is the conclusion?
* When you create a volume on your NFS server make sure you create a volume for each mount you are going to export (or create a smart directory) – so you can set quota on every mount.
* Always use “-A” alongside “-v” when you try to see NFS mount on “quota”
* RTFM

 

Leave a Reply

Your email address will not be published. Required fields are marked *