Timestamp: 17th Dec,2013 07:30 local time
Analysis:
- As per view connection manager logs, users were not able to connect to the desktops since they were not powered on followed by ldap issues.
2013-12-17T07:16:59.846+07:00 INFO (0858-11B0) <TP-Processor6> [AuthorizationFilter] (SESSION:ecbb_***_9e5c) User VMware\Aakash has successfully authenticated to VDM
2013-12-17T07:16:59.858+07:00 INFO (0858-11B0) <TP-Processor6> [Audit] (SESSION:ecbb_***_9e5c) BROKER_LOGON:USER:VMware\Aakash;USERSID:S-1-5-21-3129790736-3909135156-1495698044-26157;USERDN:CN=S-1-5-21-3129790736-3909135156-1495698044-26157,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;
2013-12-17T07:17:02.956+07:00 INFO (0D70-0698) <Thread-26> [g] (Request78) User Aakash connected to the Secure Gateway Server - session ID: D621_***_604E
2013-12-17T07:17:04.298+07:00 WARN (0858-11B4) <TP-Processor7> [LdapContextManager] (SESSION:ecbb_***_9e5c) Context failed test - closing it: Connection reset
2013-12-17T07:17:04.317+07:00 WARN (0858-11B4) <TP-Processor7> [DesktopsHandler] (SESSION:ecbb_***_9e5c) Pool control for desktop pool003 reports machine assigned to VMware\Aakash is unavailable
2013-12-17T07:17:04.317+07:00 ERROR (0858-11B4) <TP-Processor7> [DesktopsHandler] (SESSION:ecbb_***_9e5c) Failed to launch desktop cn=pool003,ou=applications,dc=vdi,dc=vmware,dc=int for user Aakash: The assigned desktop source for this desktop is not currently available. Please try again later.
- Further we noticed, that the View manager pools were not gettng trusts from domains followed by ldap errors as below.
2013-12-17T07:26:49.566+07:00 WARN (0838-1748) <MessageFrameWorkDispatch> [ws_winauth] getTrustedDomains: Cannot find domain controller for domain "Domain"
2013-12-17T07:26:49.761+07:00 INFO (0858-0F80) <DesktopControlJMS> [MachineInformation] The machine previously using the IP hostname Desktop1.vmware.asia.blog has changed it to PC1.vmware.asia.blog
2013-12-17T07:26:50.490+07:00 WARN (0838-1748) <MessageFrameWorkDispatch> [ws_winauth] getTrustedDomains: Cannot find domain controller for domain "Domain2"
2013-12-17T07:27:19.625+07:00 WARN (0838-1748) <MessageFrameWorkDispatch> [ws_winauth] getTrustedDomains: Cannot find domain controller for domain "VMware"
2013-12-17T07:27:30.596+07:00 ERROR (0838-17BC) <WSAdminDomainTimerThread> [ws_admin] OpenObject could not bind to LDAP://vmw.co.in/rootDSE (A local error has occurred.)
2013-12-17T07:27:31.379+07:00 ERROR (0838-17BC) <WSAdminDomainTimerThread> [ws_admin] OpenObject could not bind to LDAP://vmware3.blog.Asia/rootDSE (The server is not operational.)
2013-12-17T07:27:32.947+07:00 WARN (0858-11B0) <TP-Processor6> [DesktopsHandler] (SESSION:31cd_***_9423) Pool control for desktop pool003 reports machine assigned to VMware\Aakash is unavailable
- The SSO logs provided below are the oldest event and there is no event prior to this which means it doesnot correspond to the time of issue. But it clearly mentions the issue that SSO is not able to connect to domain controller ' Domain.vmware.asia.blog and ' Domain2.blog.Asia'.
ssoAdmin Server log
[2013-12-17 13:28:44,679 WARN opID= DomainKeepAliveThread com.vmware.vim.sso.admin.server.impl.KeepAlive] Unexpected exception in KeepAlive attempt.com.rsa.common.ConnectionException: Error connecting to the identity source
Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection Domain.vmware.asia.blog:3268 [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection Domain.vmware.asia.blog:3268]
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection Domain.vmware.asia.blog:3268
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection Domain.vmware.asia.blog:3268
Caused by: javax.naming.CommunicationException: Domain.vmware.asia.blog:3268 [Root exception is java.net.UnknownHostException: Domain.vmware.asia.blog]
Caused by: java.net.UnknownHostException: Domain.vmware.asia.blog
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jndi.ldap.Connection.createSocket(Unknown Source)
at com.sun.jndi.ldap.Connection.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapClient.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
[2013-12-17 14:01:19,922 TRACE opID=c8c2448b-af80-4ba9-850e-772e47e6590a pool-13-thread-12 com.vmware.vim.vmomi.server.impl.InvocationTask] (11) Invoking com.vmware.vim.binding.sso.admin.PrincipalDiscoveryService.findPersonUser() [MORef principalDiscoveryService] with parameters: com.vmware.vim.binding.sso.PrincipalId:
name = vdiadm,
domain = vmware.asia.blog
inherited from com.vmware.vim.binding.sso.PrincipalId@13b5b562
[2013-12-17 14:01:19,924 INFO opID=c8c2448b-af80-4ba9-850e-772e47e6590a pool-13-thread-12 com.vmware.vim.sso.admin.vlsi.PrincipalDiscoveryServiceImpl] Vmodl method 'PrincipalDiscoveryService.findPersonUser' invoked by [ User {Name: vCenterServer_2013.03.31_143609, Domain: System-Domain} with role RegularUser] [caller:/0:0:0:0:0:0:0:1] Find person user {Name: vdiadm, Domain: vmware.asia.blog}
[2013-12-17 14:01:19,940 DEBUG opID=c8c2448b-af80-4ba9-850e-772e47e6590a pool-13-thread-12 com.vmware.vim.sso.admin.server.ims.impl.DefaultCommandExecutor] Command com.rsa.admin.SearchIdentitySourcesCommand was executed successfully
[2013-12-17 14:01:20,375 ERROR opID=c8c2448b-af80-4ba9-850e-772e47e6590a pool-13-thread-12 com.vmware.vim.sso.admin.vlsi.PrincipalDiscoveryServiceImpl] Error connecting to the identity source com.rsa.common.ConnectionException: Error connecting to the identity source
Caused by: javax.naming.NamingException: getInitialContext failed. javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain2.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection GSSAPI [Root exception is javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain2.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection GSSAPI]
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create a managed connection 'ldap://Domain2.vmware.asia.blog:3268' with 'GSSAPI' Reason: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection GSSAPI
Caused by: javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection GSSAPI
Caused by: javax.naming.AuthenticationException: GSSAPI [Root exception is javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))]]
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))]
Caused by: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null))
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Unknown Source)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.access$000(Unknown Source)
at javax.security.auth.login.LoginContext$5.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokeCreatorPriv(Unknown Source)
at javax.security.auth.login.LoginContext.login(Unknown Source)
at sun.security.jgss.GSSUtil.login(Unknown Source)
at sun.security.jgss.krb5.Krb5Util.getTicket(Unknown Source)
at sun.security.jgss.krb5.Krb5InitCredential$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Unknown Source)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown Source)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
imsSystem.2013-12-17-16-25.log
2013-12-17 14:45:01,323, 1041d75916640a0a4d0e723313685678,e7fdddd510ea480a32f9f91de7de8c7f,,10.10.100.22,RIAT_REPLICATION_STARTUP,21001,WARN,REPLICATION_DISABLED,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,,,,,,,
2013-12-17 14:46:44,554, 21db4e4c16640a0a075806649077ddbf,e7fdddd510ea480a32f9f91de7de8c7f,,10.10.100.22,ACCESS_DIRECTORY,16045,FAIL,UNEXPECTED_LDAP_EXCEPTION,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,SYSTEM,,ldap://Domain.vmware.asia.blog:3268,,,,,
javax.resource.spi.ResourceAdapterInternalException: Unable to create managed connection Domain.vmware.asia.blog:3268
at com.rsa.ims.connectionpool.jca.ldap.ManagedConnectionFactoryImpl.createLdapContext(ManagedConnectionFactoryImpl.java:170)
at com.rsa.ims.connectionpool.jca.ldap.ManagedConnectionFactoryImpl.newManagedConnection(ManagedConnectionFactoryImpl.java:82)
at com.rsa.ims.connectionpool.jca.common.AbstractManagedConnectionFactory.createManagedConnection(AbstractManagedConnectionFactory.java:277)
at org.apache.geronimo.connector.outbound.MCFConnectionInterceptor.getConnection(MCFConnectionInterceptor.java:49)
at org.apache.geronimo.connector.outbound.SinglePoolMatchAllConnectionInterceptor.internalGetConnection(SinglePoolMatchAllConnectionInterceptor.java:88)
at org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:80)
at org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:43)
at org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39)
at org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:77)
at com.rsa.ims.connectionpool.jca.common.InitialContextFactoryImpl.getInitialContext(InitialContextFactoryImpl.java:99)
at com.rsa.ims.connectionpool.manager.DelegatingInitialContextFactory.getInitialContext(DelegatingInitialContextFactory.java:78)
at com.rsa.ims.connectionpool.manager.DelegatingInitialContextFactory.getInitialContext(DelegatingInitialContextFactory.java:78)
at com.rsa.ims.instrumentation.monitor.InstrumentedContextFactory.getInitialContext(InstrumentedContextFactory.java:76)
at com.rsa.ims.connectionpool.manager.impl.ConnectionManagerImpl.getLDAPConnection(ConnectionManagerImpl.java:262)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy21.getLDAPConnection(Unknown Source)
Caused by: javax.naming.CommunicationException: Domain.vmware.asia.blog:3268 [Root exception is java.net.ConnectException: Connection timed out: connect]
at com.sun.jndi.ldap.Connection.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapClient.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source)
at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(Unknown Source)
Recommendations:
- Check why the domain controllers were having connectivity issues with SSO and View Connection Manager.
Reference kb: http://kb.vmware.com/kb/1006300