Purpose of this post is to share how you can debug ONS - FAN events notification if you need to setup Weblogic / Data Source to be aware of the ONS FAN events
ONS is a very useful option if you are using Oracle RAC architecture but the prior actual configuration of GridLink in the Weblogic make sure that your ONS working from the local and remote side because it may lead to an issue in the Weblogic and high CPU utilization
Also, check Metalink for a bug related to the jdbc driver and ONS feature because it may require to patch Weblogic prior to implementing it.
Please check the below reference link in order to get familiar with ONS and what has been changed in 18c compared to 12c
Oracle Clusterware 18c introduces the following features & enhancements compare to 12.2 ( as per the documentation )
- Cluster Domain architectures enhancements:
- Conversions, from Standalone Cluster to Member Cluster, and from one storage access configuration to another for Database Member Clusters.
- Oracle ACFS Remote Service: enable Oracle ACFS file systems on local Member Clusters utilizing remotely hosted ASM storage services on the Domain Services Cluster.
- Single network support for Application Member Clusters.
- Shared SCAN – the optional election of SCAN VIP’s on one cluster to act as the SCAN VIP’s for other clusters, reducing the number of IP addresses required by using a single SCAN VIP setup for multiple clusters.
- Node VIP-less Clusters – optionally, deploy new clusters without Node VIP’s, thus freeing up cluster resources and reducing the number of IP addresses required.
- Cross-Cluster Dependency Proxies – support for Clusterware resource dependencies that span clusters.
12c Fast Application Notification in 12c
Please check this document and follow the instruction in order to avoid an issue with configuration
How to Monitor FAN events
Testing WLS and ONS Configuration
In case of GI upgrade from 12.1 to 12.2
Recommended upgrade of Oracle Notification Server (ONS) to 12.2 from an earlier version 12.1 Grid Infrastructure (Doc ID 2376962.1)
LOCAL on database server
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/jdk/bin/javac ~/fanWatcher.java
Note: /home/grid/fanWatcher.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/jdk/bin/javac -Xlint:deprecation -nowarn ~/fanWatcher.java
warning: [deprecation] Subscriber(String,String) in Subscriber has been deprecated s = new Subscriber(eventType, "");
// subscribe to service events only ^ /home/grid/fanWatcher.java:103:
warning: [deprecation] receive(boolean) in Subscriber has been deprecated Notification e = s.receive(true);
// blocking wait for notification receive ^ /home/grid/fanWatcher.java:108:
warning: [deprecation] print() in Notification has been deprecated e.print(); ^
Once you have compiled java you should see fanWatcher.class file which has been created
[grid@aixrac1 ~]$ ls -la | grep fanWatcher.class
-rw-r--r--. 1 grid oinstall 5812 Aug 14 22:55 fanWatcher.class
Now you need to set ONS in debug mode
/opt/app/18.0.0/grid/opmn/bin/onsctl set target=debug comp='ons[all,!workers,!servers]'
/opt/app/18.0.0/grid/opmn/bin/onsctli query target=debug
No target for set or query request
If you see this message, try to stop and start ONS
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/opmn/bin/onsctl stop onsctl stop
shutting down ons daemon ...
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/opmn/bin/onsctl start
onsctl start: ons started
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/opmn/bin/onsctl debug
HTTP/1.1 200 OK Connection:
close Content-Type: text/html Response:
== aixrac1:6200 31573 (2) 18/08/14 23:04:16
== Build: ONS_188.8.131.52.0_LINUX.X64_171214.2047 2017/12/15 6:9:53 UTC
You have now 7 connections
Now you can execute java program
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/jdk/bin/java -Doracle.ons.oraclehome=$CRS_HOME fanWatcher crs
Subscribing to events of type:
if you have the above message, you can run again debug to check if there are new connection allocated
[grid@aixrac1 ~]$ /opt/app/18.0.0/grid/opmn/bin/onsctl debug
as you can see connecting now is 8 which means that you have a new client which is our program
Now you are ready to test ONS service by triggering event as stop or start instance or service which is part of the Cluster
Also, keep in mind that ONS should be tested from Application server - Weblogic in order to make sure that connection works as expected and you don’t any FW issue in regards to the port for ONS
In case of FW issue, you should add special rules in the FW
for Weblogic testing - please check document - Testing WLS and ONS Configuration from the above list, because there are some options which you may need to configure or to be aware like wallet and it really depends on what kind of architecture you have :)
You can follow the below documents because there are a different approach for one or other situation
Supported AGL Data Source URL Formats
AGL data sources only support long format JDBC URLs. The supported long format pattern is:
Easy Connect (short) format URLs are not supported for AGL data sources. The following is an example of an Easy Connect URL pattern that is not supported for use with AGL data sources:
Recommendations for AGL Data Source URLs
The following section provides general recommendations when creating AGL data source URLs.
Use a single
DESCRIPTION. Avoid a
DESCRIPTION_LISTto avoid connection delays.
ADDRESS_LISTfor each RAC cluster or DataGuard database.
DESCRIPTIONlevel so that all
ADDRESS_LISTentries use the same value.
RETRY_DELAYspecifies the delay, in seconds, between the connection retries. This attribute is new in the Oracle 184.108.40.206 release.
RETRY_COUNTis used to specify the number of times an
ADDRESSthe list is traversed before the connection attempt is terminated. The default value is 0. When using
RETRY_COUNTto a value of 2 means that if you had 3
SCAN IPaddresses, each would be traversed three times each, resulting in a total of nine connect attempts (3 * 3).
LOAD_BALANCE=onfor each address list to balance the
The service name should be a configured application service, not a PDB or administration service.
CONNECT_TIMEOUTis used to specify the overall time used to complete the Oracle Net connect. Set
CONNECT_TIMEOUT=90or higher to prevent login storms. For JDBC driver 220.127.116.11 and earlier,
CONNECT_TIMEOUTis also used for the TCP/IP connection timeout for each address in the URL. When considering TCP/IP connections, a shorter
CONNECT_TIMEOUTis preferred though secondary to overall timeout requirements.
Do not set the
oracle.net.CONNECT_TIMEOUTdriver property on the data source because it is overridden by the URL property.
Application Continuity Statistics
Starting with Oracle JDBC thin driver 18.104.22.168, there are 11 statistics available for related to replay processing. WebLogic Server exposes this information to application users via a runtime MBeans. Sample statistics
TotalRequests = 1
TotalCompletedRequests = 1
TotalCalls = 19
TotalProtectedCalls = 19
TotalCallsAffectedByOutages = 3
SuccessfulReplayCount = 3
FailedReplayCount = 0
ReplayDisablingCount = 0
TotalReplayAttempts = 3
Figure 6: Sample Application Continuity Statistics.
The statistics will be available via a new runtime MBean that is accessible via the data source runtime MBean.
» It is available for Generic and AGL data sources. It is not available (null is returned) for MDS, PROXY, and UCP data sources.
» It is available only if running with the 22.214.171.124 or later Oracle thin driver. It is not available (null is returned) for earlier driver versions.
» It is available only if the data source is configured to use the replay driver. It is not available (null is returned) for non-replay drivers (e.g. Oracle driver, XA driver).
» The runtime MBean will initially have no statistics set (they will be initialized to -1). The refreshStatistics() operation on the MBean must be called to update the statistics before getting them.
» Refreshing the statistics is a heavy operation - it locks the entire pool and runs through all reserved and unreserved connections aggregating the statistics. Running this operation often will impact the performance of the data source. The same is true for clearing the statistics.
The information on the runtime MBean can be access via the WLST scripting language or via any program that provides access to the MBean tree (e.g., jconsole or JRMC).
import sys, socket, os
hostname = socket.gethostname()
datasource='JDBC GridLink Data Source-0'
svr='myserver' connect("weblogic","welcome1","t3://"+hostname+":7001") serverRuntime()
cd('/JDBCServiceRuntime/' + svr + '/JDBCDataSourceRuntimeMBeans/' +
datasource + '/JDBCReplayStatisticsRuntimeMBean/' +
datasource + '.ReplayStatistics') cmo.refreshStatistics()
ls() total=cmo.getTotalRequests() cmo.clearStatistics()
If you have done some of the tests from the above process and you have issue, error or just feedback add in the below as comments