[Networker] Problem with mmrecov after /nsr array failure

This past Saturday, my NetWorker 7.4 server's /nsr storage arrayfailed. The server runs Solaris 9 and the array is an old Sun A1000the array is one of two that was connected to the server. I placed aservice call have have the problem fixed. On Saturday morning, Ifound myself in a computer room while a hardware fixer upper guyfixed the array ... so we thought. To make a long story short, fsckran for 24 hours and by Sunday night, I was up and running again withthe fixed array, but NetWorker crashed within seconds of restarting,so I decided to hold off until Monday morning to address the problemso I could get some sleep.

So on Monday, rebooting the server didn't produce any SCSI errors atall and it came backup fine, so I did a mmrecov from Friday'sbootstrap tape. I restarted NetWorker and all was fine. Later thatevening, the same array died on me again. Sigh! On Tuesday, we didmore array repairs, but nothing we tried worked. The broken A1000disk array is one of two we had sitting on my backup server. Thesecond one is /nsr2 (which contains some CFI data for a few largeclients), but it was only 20% full and the the /nsr array onlycontained 21GB worth of data. Since the /nsr2 array had somethinglike 150GB free on it, so my boss and I decided to create a directorycalled nsr on the /nsr2 array and we disconnected the faulty /nsrarray from the SCSI chain and powered it off. So /nsr now sits onthe /nsr2 array and all the /nsr2 array's cfi data is still visibleto NetWorker as /nsr2. I hope this makes sense.

This all works and I get no SCSI errors at all when I rebooted theserver twice. Since this scheme wiped out the entire contents of /nsr, I used jbconfig to configure a tape library resource so I couldread the bootstrap tape. Then I used mmrecov to recover the samebootstrap saveset from the same tape I used on Monday. This worked,except for one problem. When I did the mmrecov, instead of recoveringto /nsr/res.R it recovered the data to /nsr/res and when I restartedNSR, the tape library that's connected to our server appeared twicein the NetWorker management console window and each instance of thetape library had two device resources for every physical device onthe library (14 physical devices), except for the five devices thatwe use for NDMP which only had one device resource each. This serveralso has a Linux storage node connected to a totally differentlibrary, and that library's resource information is fine. I spent twohours tonight trying to fix this issue, including doing anothermmrecov, which also dumped its data into /nsr/res instead of /nsr/res.R.

I tried deleting the second tape library resource, but this did nothelp. As a result, tape mount requests are not being satisfied forthe main tape library, but they are for the tape library on mystorage node. I don't know if its relevant, but the tape library is aSony PetaSite with 14 S-AIT1 drives and its fibre channel connectedto my NetWorker server. We do not do drive or tape library sharing.The inquire command also shows exactly the same thing it showedbefore we disconnected the broken A1000 array (except of course, forthe missing array).

If anyone has any idea how to correct this problem, please let meknow; otherwise, I intend to open up a support case with EMC in themorning (since I am too exhausted to do it now).


--
Stan Horwitz
stan AT temple DOT edu

CONFIDENTIALITY STATEMENT: The information contained in this e-mail,including attachments, is the confidential information of, and/or isthe property of, Temple University. The information is intended foruse solely by the individual or entity named in the e-mail. If youare not an intended recipient or you received this in error, then anyreview, printing, copying, or distribution of any such information isprohibited. Please notify the sender immediately by reply e-mail andthen delete this e-mail from your system.


To sign off this list, send email to listserv AT listserv.temple DOT edu and type 
"signoff networker" in the body of the email. Please write to networker-request 
AT listserv.temple DOT edu if you have any problems with this list. You can access the 
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER