VM: VM-A
Time: 9PM 30 May,2013
Analysis:
-VM backup fails during create snapshot.
2013-05-30 21:07:06 avvcbimage Info <14632>: snapshot create completed
2013-05-30 21:07:06 avvcbimage Info <14634>: New snapshot 'VDP-1369919135de3a2a20f17d33073b17a0fe31acc41fbd2efdfe', moref is 'snapshot-569'.
2013-05-30 21:11:13 avvcbimage Error <14690>: No response from Avamar Administrator to retain snapshot request, please check Avamar Administrator log during period in the vicinity of this timestamp for root cause
2013-05-30 21:11:22 avvcbimage Info <0000>: Starting graceful (staged) termination, MC did not respond to retain snapshot request (wrap-up stage)
2013-05-30 21:11:33 avvcbimage Error <9759>: createSnapshot: snapshot creation failed
-On checking avamar admin log, we do see an exception during that point of time.
05/30-21:13:15.00134 com.avamar.mc.datatap.ConnectionPool.closeConnection FINE: mcdbPool: Freeing 492577680 caller = com.avamar.mc.datatap.MCDataAccessObject.setClientMsgOffset#5295
05/30-21:13:15.00206 com.avamar.mc.dpn.DPNProxyService.handleEvent FINE: DPN_POLLING_RECOVERED: <event-source NodeID="sgequsvdp" ProgramName="com.avamar.mc.dpn.ClientMessageCache" ddr-id="" ddr-name="" hardware-id="" source-hardware-id=""/>
05/30-21:13:15.00206 com.avamar.mc.message.MessageEventService.receiveMessage FINE: Event 22401 processing time = 1 age: 71 05/30-21:13:15.00238 com.avamar.mc.util.MCException.logException
WARNING: java.lang.NullPointerException The stack trace of the Exception is empty, use that of the current thread, instead.
at com.avamar.mc.um.UserManagerService.getClientOnlyListing(UserManagerService.java:2786)
at com.avamar.mc.um.UserManagerService.getAllClients(UserManagerService.java:2979)
at com.avamar.mc.sdk.mo.ClientMO.findReplicateClientDescriptorByFqn(ClientMO.java:442)
at com.avamar.mc.sdk.mo.ClientMO.findReplicateClientDescriptorById(ClientMO.java:333)
at com.avamar.mc.sdk.mo.ClientMO.getClientDescriptor(ClientMO.java:315)
-Further on checking err.log, we noticed the blackout window had initiated during which the mc server won’t take any requests for backup and therefore the request was terminated.
2013/05/30-20:59:39.39573 {0.0} [perfbeat.5:193] WARN: <1060> perfbeat::outoftolerance mask=[] average=2388.18 limit=238.8182 mbpersec=115.90
2013/05/30-21:00:58.37298 {0.0} [perfbeat.0:183] WARN: <0963> server node 0.0 is swapping: check configuration
2013/05/30-21:14:39.59909 {0.0} [perfbeat.5:193] WARN: <1060> perfbeat::outoftolerance mask=[] average=2388.18 limit=238.8182 mbpersec=129.30
2013/05/30-21:15:01.41611 {0.0} [sched.gc:3372] <4200> starting scheduled garbage collection
2013/05/30-21:16:28.47383 {0.0} [perfbeat.0:183] WARN: <0963> server node 0.0 is swapping: check configuration
2013/05/30-21:20:02.10546 {0.0} [perfbeat.1:185] WARN: <1060> perfbeat::outoftolerance mask=[gc] average=33.43 limit=3.3433 mbpersec=0.49
Recommendations:
-Increase the backup window of VDP.