Why Can't My DMGR See a Successful Federated Ex-Standalone Node?

Sometimes when we are federating a standalone IBM WebSphere Application Server v8.5.5 for z/OS node with a cell, you can observe a strange situation: the BBOWADDN job calling the addNode command is successful complete, but the DMGR doesn't connect to the new federated node.

The node (CZNODEB) is running and the address space exists in the z/OS image.

Analysis

Let's have a look at the FFDC logs. The application server writes node-level logs into the /wasv85config/CELL/NODE/AppServer/profiles/PROFILE/logs/ffdc/ directory. A tip: you can just sort files in the directory by change-date descending and open only top ones. The problem description should look like the following:

[6/30/15 7:01:44:582 GMT] FFDC Exception:com.ibm.websphere.management.exception.ConnectorException SourceId:com.ibm.ws.management.RoutingTable.Accessor.getConnector ProbeId:583 Reporter:com.ibm.ws.management.RoutingTable$Accessor@e153aa4c
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host HOST at port PORT.
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:635)
at com.ibm.websphere.management.AdminClientFactory.access$000(AdminClientFactory.java:127)
at com.ibm.websphere.management.AdminClientFactory$1.run(AdminClientFactory.java:210)
...
Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)
at org.apache.soap.rpc.Call.WASinvoke(Call.java:510)

The FFDC logs for the DMGR is the similar:

[6/26/15 6:45:11:707 GMT] FFDC Exception:com.ibm.websphere.management.exception.ConnectorException SourceId:com.ibm.ws.management.RoutingTable.Accessor.getConnector ProbeId:583 Reporter:com.ibm.ws.management.RoutingTable$Accessor@b44063cd
com.ibm.websphere.management.exception.ConnectorException: ADMC0016E: The system cannot create a SOAP connector to connect to host HOST at port PORT.
at com.ibm.websphere.management.AdminClientFactory.createAdminClientPrivileged(AdminClientFactory.java:635)
at com.ibm.websphere.management.AdminClientFactory.access$000(AdminClientFactory.java:127)
at com.ibm.websphere.management.AdminClientFactory$1.run(AdminClientFactory.java:210)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63)
at com.ibm.websphere.management.AdminClientFactory.createAdminClient(AdminClientFactory.java:206)
at com.ibm.ws.management.RoutingTable$Accessor.getConnector(RoutingTable.java:1102)
at com.ibm.ws.management.RoutingTable$Accessor.getConnector(RoutingTable.java:1218)
at com.ibm.ws.management.RoutingTable$Accessor.resetSession(RoutingTable.java:1264)
at com.ibm.ws.management.RoutingTable.addChild(RoutingTable.java:397)

Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(SOAPHTTPConnection.java:475)

So, the DMGR and the Node aren't able to set a secured connection using the HTTPS protocol between each other. Since the DMGR must give credence to the Node, his truststore has to contain the root digital certificate for the Node Agent. Similarly, Node Agent's truststore has to contain the root digital certificate for the DMGR.

Because WebSphere Application Server for z/OS is the case, digital certificates usually keep in z/OS System Access Facility (SAF) Keyring The management console page Security -> SSL certificate and key management -> Key stores and certificates is used for key rings set up.

Solution

In the picture above the cznodeb Node is configured for safkeyring:///WASKeyring.WSBCELL SSL keystore using. The keystore was created for the standalone node and actually it was empty. Because the safkeyring:///WASKeyring.CZCELL SSL keystore is used by other application servers in the cell, it is a good idea to switch the SSL keystore for the cznodeb Node to this value.

After the changes saved and synchronized between all nodes (so, please be careful, the cznodeb is not able automatically synchronize with the DMGR, because there isn't a connection between them, you should manually copy the /wasv85config/CELL/DMGR/DeploymentManager/profiles/default/config/cells/CELL/security.xml file in the Node configuration directory), the cznodeb Node Agent must be restarted. When the Node Agent became started, the connection is established and a green arrow is shown on the System administration -> Node agents, Node agents page.

The problem is solved and the Node now is being managed by the DMGR.

Would you like to give a 'Like'? Please follow me on Twitter!