Wednesday, August 23, 2017

Oracle WebLogic Cluster Causes Network Storm When a Problem Happens

Not every system administrator is aware of such WebLogic Server capability as Message Forwarding to Domain Logs. In addition to writing messages to the server log file, each server instance forwards a subset of its messages to a domain-wide log file. This domain-wide log file certainly helps to the system administrator understand the situation on a large domain, for instance, when several dozen servers belong to the domain, it is very helpful having all server logs sit in one place. But the convenience comes at a price.

If there are some problems caused by applications deployed on the server, the server log tends to get stuck by diagnostic messages and large stack traces. The problem is these messages and traces are being written not only to the server log file but also being forwarded to the administration server via the network and causes network storm. The administration server, as a result, might become inaccessible.

For a medium domain (4 - 8 WebLogic Server instances belong to) the idea just to disable the message forwarding capability can be considered. The Domain log broadcaster:, Severity level property value on the Environment -> Servers -> SERVER -> Logging -> General, Advanced page for every managed server must be set to Critical or higher.

For a large domain, the above approach is too radical; if domain log broadcaster disabled at all, the system administrator will have to investigate incidents walking by the server log of every WebLogic Server instance. The solution is to enable such capability as log filters.

In order to enable log filters, one or more filters must be created and configured. The filters are domain-wide and are configured on the DOMAIN -> Configuration -> Log Filters page.

Only the name of the created filter should be specified.

Then drill down to the filter and configure the log filter expression.

The Save button must be clicked to apply the new filter. In the Log Filters, you can see all the filters and expressions in the domain, including the once just created.

Then the filter can be assigned on the Domain log broadcaster:, Filter property value on the Environment -> Servers -> SERVER -> Logging -> General, Advanced page.

After these changes have been applied, the log storm should be collapsed significantly if not completely avoid. Then root cause analysis could be performed.

Would you like to give a 'Like'? Please follow me on Twitter!

No comments:

Post a Comment