¯\_(ツ)_/¯: Weblogic

Showing posts with label Weblogic. Show all posts

Sunday, October 30, 2016

Oracle WebLogic Server - Part 2 - Node Manager and its interaction with WebLogic Server instances

In this post, I am going to describe node manager process and its interaction with WebLogic server instances. This post covers mostly theoretical parts, it does not includes instructions on how to perform tasks related to node manager.

In WebLogic Server environment, the Admin Server does not connect to the managed servers directly. A separate utility named "node manager" should be running in each machine for helping this purpose.

The main task of node manager is to provide a remote controller interface for controlling of the WebLogic Server instances (Admin Server and managed server instances) which are located in the same machine as the node manager process. It is not mandatory to have a node manager in a machine if you do not want to control managed server instances through Admin Server. Also it is possible to have only one node manager for all the domains in a machine.

It is recommended to install node manager as an operating system service or daemon. Then the node manager will be automatically restarted when a system failure or reboot of the machine happens.

Node manager is required if you want following features,

If you want to start managed servers from Admin console.
For supporting High Availability.
For Whole Server Migration.
For Automatic Service Migration.

Purpose of Node Manager Process:

Allow a remote or a local client to stop/start WebLogic server instances in the machine which runs the Node Manager process (mainly, but not necessarily from the Admin Server)

start, shutdown Admin Server instance
start, suspend, shutdown managed server instances

Automatically restart failed WebLogic server instances which were abnormally exited, if following conditions met,

the failed server instance was started by the node manager.
AutoRestart value of this server instance is set to true. (AutoRestart property of nodemanager's startup.properties file in <DOMAIN_HOME>/servers/<server_name>/data/nodemanager folder )
Number of max allowed restart is not yet passed. (RestartMax property of nodemanager's startup.properties file).

Allow monitoring of the health status of WebLogic servers from a remote or the local machine. (mainly, but not necessarily from the Admin Server)
Retrieve server logs and node manager logs.

Usually, the node manager gets the start/shutdown commands for starting/shut downing managed servers from the Admin Server. Then node manager is the process that actually start or shutdown the relevant managed server.

Tip: you can access the node manager (tested for a Plain Node Manager in WL 10.3.6) through a program like netcat, and then execute plain text commands to control the servers and monitor servers.

e.g:

~# nc <MACHINE_IP> 5556
domain <your_domain_name>
+OK Current domain set to ‘<your_domain_name>’
getStates
-ERR Must login to a domain first
user <weblogicuser>
+OK NM usr set to '<weblogicuser>'
pass <weblogicuser_password>
+OK Password accepted
getStates
+OK managed1=RUNNING managed2=UNKNOWN AdminServer=UNKNOWN

There are two versions of node manager processes.

Versions of Node Manager

Java based node manager

Java-based node manager process runs in a JVM instance. A Java-based node manager provides more security than a script based node manager.

Java-based node manager supports both Windows and UNIX operating systems.
Database-less leasing (consensus leasing) is faster when using a Java-based node manager.
Using inetd in UNIX, java-based node managers can be configured to restart automatically when receiving a request on a configured port.
Determines its configuration from nodemanager.properties file.

See https://docs.oracle.com/cd/E14571_01/web.1111/e13740/java_nodemgr.htm#NODEM177 for more details.

Script based node manager

You can managed remote server instances in a network through ssh using a script-based node manager (a script named wlscontrol.sh), using the scripts provided in this node manager version.

Script-based node manager only supports UNIX operating systems.
Automatic server migration is only supported by script-based node manager.
The node manager can control servers in remote machines through a set of scripts copied to those remote machine.
It requires only a smaller foot print than java-based node manager.
It supports SSH for security and setting up security is easier than setting up SSL for java based node manager.
It supports SSH or RSH for connection.

Following command line option should be added to Admin Server startup parameters in order to use SSH type of script based node manager.

-Dweblogic.nodemanager.ShellCommand='ssh -o PasswordAuthentication=no %H wlscontrol.sh -d %D -r %R -s %S %C'

where %H = host name of ssh server, %S = WebLogic server name, %D = WebLogic domain name, %R = domain directory, %C = node manager script command. For RSH type, the ShellCommand to use is relevant RSH command.

See http://docs.oracle.com/cd/E12840_01/wls/docs103/nodemgr/script_nodemgr.html and "Running Script-based Node Manager" in https://docs.oracle.com/cd/E24329_01/web.1211/e21050/starting_nodemgr.htm#NODEM214 for more details about Script based node manager.

Node Manager Types

There are several node manager types which supports different level/type of security for node manager communication.

Type	Description	Java Based Node Manager	Script Based Node Manager
Plain	Communication is not encrypted.	✓	✘
SSL	Use SSL for encrypting communication. You need to setup a certificate for this. See http://www.veriton.co.uk/roller/fmw/entry/ssl_for_weblogic_server_11g and http://weblogic-wonders.com/weblogic/2015/04/23/configure-ssl-nodemanager-12c/	✓	✘
SSH	Use SSH session to do the communication. An OS user should be created for connecting using SSH in each machine and setup SSH settings for connection. See https://docs.oracle.com/cd/E23943_01/web.1111/e13740/script_nodemgr.htm#NODEM208	✘	✓
RSH	This type of node manager uses RSH for communication. But RSH is an older protocol which does not use encryption when sending data (such as passwords). Therefore RSH is not secure. It seems it is now rare to use this type of node manager, as not much information about it available in the internet.	✘	✓

Node Manager Operations

Please see https://docs.oracle.com/cd/E28280_01/web.1111/e13740/overview.htm#NODEM137 for more detailed explanation.

Starting Admin Server using Node Manager

A node manager client authenticate to the Node Manager using node manager user name and password (e.g. nmConnect from WLST). Node Manager authenticates the given node manager user credentials using nm_password.properties file.
Then Node Manager receives a command to start Admin Server from the node manager client (e.g. nmStart from WLST). This command provides the server name, domain name, server user credentials and any additional start-up properties.
Node Manager determines the domain directory for the Admin Server from nodemanager.domains file and authenticates the given user credentials using a local file which contains the encrypted user credentials.
Node Manager creates relevant directories (server_name/logs, server_name/security, server_name/data/nodemanager etc.) and obtains start-up properties for the Admin Server.

Then it saves encrypted user name, password etc in boot.properties file.
It saves the startup properties obtained from Admin Server configuration in startup.properties.

Node Manager creates Admin Server process and start monitoring it.
Admin Server process gets domain configuration data from its config folder.

Starting a managed server using Node Manager

This process can be done either from Admin Console, WLST console or another node manager client. Following steps mentions how to start the server from Admin console.

A user issues a start command from Admin Console to start a managed server.
Then Admin Server connects to the node manager which is located on the same machine as the targeted managed server, authenticate to the node manager.
Then Admin Server sends the start command for starting the managed server to the node manager (with server name, username/password for connecting to server and other start-up properties).
Node manager starts the managed server process.
Managed server contacts the Admin Server to check for updates to its configuration information.
If there is any update to the domain configuration, managed server updates its local cache of configuration from the update it got from the Admin Server.

Restart Admin Server using Node Manager

Node Manager determines from the exit code of Admin Server process that it needs to restart the Admin Server.
Node Manager obtains the user name and password value needed for starting the Admin server from the boot.properties file, and the start-up properties from <servername>/data/nodemanager/startup.properties file.
Node Manager uses the above retrieved information and starts the Admin Server process.
Admin Server reads its configuration data and starts up.

Restart a managed server using Node Manager

Node Manager determines from the last known status of the managed server process (+ other node manager settings) that it needs to restart that managed server.
Node Manager retrieves the user name and password value needed for starting the managed server from the boot.properties file, and the startup properties from <servername>/data/nodemanager/startup.properties file.
Node Manager uses the above retrieved information and starts the managed Server process.
Managed server contacts the Admin server to check for updates of the configuration data. If it get any updates, it update its local cache of config directory with the received updates.
If managed server fails to contact the Admin Server and if the Managed Server Independence Mode (MSI) is enabled, the managed server uses its locally cached configuration data.

Shutdown a server instance using Node Manager

An authorized user connects to the Admin console and issues the shutdown command for a managed server.
Then Admin Server issues shutdown command directly to the managed server. Then managed server perform the shutdown sequences for Graceful Shutdown procedure.
If Admin server failed to contact the managed server in above step, then it connects to the node manager and issues the shutdown command to the node manager for shutting down the managed server.
Node manager asks Operating System to kill the managed server process.
Operating system kills the managed server process.

Important files and folders related to node manager

File	Folder	Purpose
nodemanager.properties	<NODEMANAGER_HOME>/	The configuration file used by Java based node manager
nodemanager.domains	<NODEMANAGER_HOME>/	Contains domain names and corresponding directories of the domains managed by the node manager
nm_data.properties	<NODEMANAGER_HOME>/	This file stores a symmetric encryption key. It is stored in encrypted form.
nm_password.properties	<DOMAIN_HOME>/config/nodemanager	This file stores the username and password of the node manager
boot.properties	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	This files contains the user credentials needed for starting server instances.
startup.properties	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	Each managed server instance contains its own copy of startup.properties file which contains the properties that control the startup/shutdown/restart of the server instance by the node manager.
<server_name>.addr	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	This file contains the IP address added when the server started or migrated.
<server_name>.pid	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	This file contains the process id of the server instance. Node manager checks this process id during crash recovery. This is generated by the server instance.
<server_name>.state	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	This file contains the current state of the server instance. This file is generated by the server instance and monitored by the node manager. Without this file, node manager cannot determine the current state of the server instance.
<server_name>.lck	<DOMAIN_HOME>/servers/<server_name>/data/nodemanager	This file is generated by the server instance and contains a internal lock id.
nodemanager.log	<NODEMANAGER_HOME>/	Node manager log file (only for Java based node manager). This file contains logs about all of domains controlled by the node manager. This log can be viewed using nmLog command of WLST and "Machines --> Monitoring --> Node Manager Log" in Admin Console.
<server_name>.out	<DOMAIN_HOME>/servers/>server_name</logs	Node Manager Server instance log file. This file contains the messages written to stderr and stdout by the server instance. You can view this log by either nmServerLog WLST command or "Diagnostics --> Log Files".
<server_name>.log	<DOMAIN_HOME>/servers/<server_name>/logs	WebLogic server log files.

References:

https://docs.oracle.com/cd/E28280_01/web.1111/e13740/overview.htm#NODEM127
http://docs.oracle.com/cd/E13222_01/wls/docs90/server_start/nodemgr.html
Java-based Node Manager - https://docs.oracle.com/cd/E14571_01/web.1111/e13740/java_nodemgr.htm#NODEM177
Script-based Node Manager - http://docs.oracle.com/cd/E12840_01/wls/docs103/nodemgr/script_nodemgr.html
Running Script-based Node Manager - https://docs.oracle.com/cd/E24329_01/web.1211/e21050/starting_nodemgr.htm#NODEM214
Running Script-based Node Manager - http://docs.oracle.com/cd/E13222_01/wls/docs90/server_start/nodemgr.html#1081981
WLST commands for Node Manager - https://docs.oracle.com/middleware/1212/wls/WLSTC/reference.htm#WLSTC470

Saturday, October 15, 2016

Oracle WebLogic Server - Part 1 - Main components and their relationship

In this post series, I am planning to write about the important things I learned about the WebLogic server during past few years (specially about version 10.3.6). Main purpose of these posts is to keep it as a future reference.

The WebLogic Server is an "application server" developed by the Oracle Corporation, and it supports many powerful features such as high availability, scalability, high performance, reliability, etc.

The WebLogic Server has following main components.

Domain

A WebLogic domain is a logically related group of WebLogic server resources. A WebLogic domain has a special server instance known as “Admin Server”, and zero or more server instances known as “managed servers”.

All WebLogic server instances in a WebLogic domain must have same WebLogic major and minor version. The server instances in a WebLogic domain can be distributed through out several machines.

Admin Server

This is the component which helps the configuration, management, and monitoring of the servers and resources in the whole WebLogic Server domain. You can access the Admin Server through,

The "WebLogic Admin Console" web application Or
An UNIX shell like command line interface known as "WLST" Or
Through JMX API

All the configurations settings related to the Admin Server, managed servers, clusters, and machines are stored and managed in the Admin Server, mainly in a file named config.xml.

Using Admin Server, You can

Start/stop managed servers,
Deploy/update/delete applications,
Update settings of managed servers/admin server/cluster/etc.,
Create/edit/delete more managed servers/clusters/machines etc.
Monitor health of the servers.

Managed Servers

In a WebLogic domain, you can have zero or more servers known as "managed servers". These are separate JVM processes.

A "managed server" is the server process where usually you deploy your application. Although it is possible to deploy an application in Admin Server, it is not the intended usage of the Admin Server.

A “managed server” can be a part of WebLogic cluster, but it is not mandatory.

Clusters

A cluster is a set of managed servers grouped to achieve Scalability (Horizontal scalability) and High Availability. A cluster appears to the client as a single WebLogic server instance (when client access the cluster through a load balancer or use the cluster address to connect to the EJBs in the cluster). All managed server instances in a cluster must belong to the same WebLogic domain. You can create a WebLogic cluster distributed in one or more server machines.

WebLogic cluster has following capabilities in order to achieve the Scalability and High Availability.

Application Fail Over Protection
Load Balancing
Supporting automatic and manual migration of resources between servers in the cluster.

Machines

A “machine” in WebLogic is a logical representation of a physical machine or a virtual machine, which contains one or more WebLogic server instances (Admin Server or managed servers).

In each machine, there is a special process known as “Node Manager” in order for Admin Console to start/stop “managed server” instances in the respective machine, without actually login to that machine. “Machine” configuration includes the listen address and the port of the Node Manager of the respective “machine”.

Machine Types:

Unix
Virtual
Other

Node Manager

In WebLogic Server environment, the Admin Server does not connect to the managed servers directly. In each machine, a separate process named "node manager" should be running for helping this purpose.

A "node manager" process is necessary for managing server instances through Admin Server or similar remote client and to support restarting of failed server instances.

But it is not mandatory to have a node manager process if you do not want above features.

Purpose of Node Manager Process:

Allow to stop/start/restart WebLogic server instances in the machine which has the Node Manager process from a remote/local machine (mainly, but not necessarily from the Admin Server)

If a server instance started using a node manager fails and AutoRestart attribute of the server set to true, the node manager automatically restart that server instance.

Allow monitoring of the health status of WebLogic servers from a remote machine. (mainly, but not necessarily from the Admin Server)
Retrieve server logs and node manager logs.

(In the above diagram, Admin Server can be at only one location depicted above in a given domain, not in the both places at once).

Please see my next blog post "Oracle WebLogic Server - Part 2 - Node Manager and its interaction with WebLogic Server instances" for more information about the node manager and its interaction with other components.

References:

Wednesday, August 24, 2016

Check DB connection to an Oracle DBMS from a remote machine

Using SQLPLUS:

Execute following command in a command line:
sqlplus "<DB_USERNAME>/<DB_PASSWORD>@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=<HOST_NAME>)(Port=1521))(CONNECT_DATA=(SID=<SID>)))"

You will be able to login to the database in the remote machine, if connection is successful.

Using Weblogic DBPing utility

Execute following command in a command line:
java -classpath <WL_HOME>/<WL_SERVER>/server/lib/weblogic.jar utils.dbping ORACLE_THIN <DB_USERNAME> <DB_PASSWORD> <HOST_NAME>:<DB_PORT>:<SID>

You will get following message if connection is successful.
**** Success!!! ****

You can connect to the database in your app using:

java.util.Properties props = new java.util.Properties();
props.put("user", "<DB_USERNAME>");
props.put("password", "<DB_PASSWORD>");
java.sql.Driver d =
Class.forName("oracle.jdbc.OracleDriver").newInstance();
java.sql.Connection conn =
Driver.connect("jdbc:oracle:thin:@<HOST_NAME>:<DB_PORT>:<SID>", props);

References:

https://docs.oracle.com/cd/E13222_01/wls/docs/techstart/dbping.html
http://javaeesupportpatterns.blogspot.sg/2011/03/network-adapter-could-not-establish.html
http://dba.stackexchange.com/questions/13075/how-to-use-sqlplus-to-connect-to-an-oracle-database-located-on-another-host-with

Monday, March 7, 2016

Remote debugging in Weblogic server using Eclipse

In Weblogic server,
Select relevant managed server and go to "Server Start" tab/ Then in the "Arguments:" text box, append following parameters

   -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n

Then, in Eclipse,

Add a new "Remote Java Application" in the Debug Configurations (via the menu: Run -> Debug Configurations).
Make sure the correct project is selected (Browse button) and fill in server name in "Host" and 8000 in "Port".
To avoid Eclipse stopping debug at "Daemon Thread [Timer-xxx] (Suspended (exception RuntimeException)) TimerThread.run()",

Go to Preferences -> Java -> Debug
Uncheck "Suspend execution on uncaught exceptions"

Note: It is not necessary to wait until managed server finish startup for the debugging. You can start the debugging after you initiate the managed server startup. This means you can debug post start/activate actions of the managed server also.

Friday, February 12, 2016

Tips on Tools Usage - WLST (For viewing MBeans)

Locate the wlst.sh

solaris:

find <a_parent_folder_in_path_to_wls> -name wlst.sh

linux:

locate wlst.sh

Storing weblogic admin username password in a file for easy access:

Use storeUserConfig([userConfigFile], [userKeyFile], [nm]) command.
After this, you do not have to specify username and password when connecting to a wlst session.

Example session for viewing custom MBeans (MBeans in your application):

cd /<path_to_wls>/wlserver_10.3/common/bin
sh ./wlst.sh
wls:/offline> connect(url='t3://hostname:managedserverpot')
custom()
ls()

Saturday, July 20, 2013

How to avoid "java.lang.NoClassDefFoundError: Could not initialize class" when using JFreeChart in a headless server

Recently I encountered following exception when using JFreeChart to generate a graphical chart in a product deployed in a weblogic server.

]] Root cause of ServletException.
java.lang.NoClassDefFoundError: Could not initialize class sun.awt.X11GraphicsEnvironment
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68)
        at sun.awt.X11.XToolkit.(XToolkit.java:89)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at java.awt.Toolkit$2.run(Toolkit.java:834)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:826)
        at sun.swing.SwingUtilities2$AATextInfo.getAATextInfo(SwingUtilities2.java:126)
        at javax.swing.plaf.metal.MetalLookAndFeel.initComponentDefaults(MetalLookAndFeel.java:1556)
        at javax.swing.plaf.basic.BasicLookAndFeel.getDefaults(BasicLookAndFeel.java:130)
        at javax.swing.plaf.metal.MetalLookAndFeel.getDefaults(MetalLookAndFeel.java:1591)
        at javax.swing.UIManager.setLookAndFeel(UIManager.java:541)
        at javax.swing.UIManager.setLookAndFeel(UIManager.java:581)
        at javax.swing.UIManager.initializeDefaultLAF(UIManager.java:1343)
        at javax.swing.UIManager.initialize(UIManager.java:1432)
        at javax.swing.UIManager.maybeInitialize(UIManager.java:1420)
        at javax.swing.UIManager.getDefaults(UIManager.java:660)
        at javax.swing.UIManager.getColor(UIManager.java:702)
        at org.jfree.chart.JFreeChart.(JFreeChart.java:261)
        at org.jfree.chart.ChartFactory.createTimeSeriesChart(ChartFactory.java:1918)

Reason:
During the issue investigation, I found out that the above mentioned Weblogic server is located inside a headless server. (A headless server is a server which does not have a monitor, a key board and a mouse. The control and access to a headless server is done using a network card (remote connection).) It turned out the reason for the above exception is that.

The problem happens because JFreeChart class has some static variables which are initialized using local graphic environment properties retrieved from javax.swing.UIManager.
javax.swing.UIManager in turn tries to get these default properties from local graphic environment.
But because there is no graphic environment in a headless server , initializing sun.awt.X11GraphicsEnvironment class (local graphic environment class) fails and then the loading of JFreeChart class also fails.

Solution:
The solution for this problem is to add -Djava.awt.headless=true to the list of server start arguments of the managed server. You can change it by,

Login to the Weblogic admin console.
Expand Environment and select Servers.
In the Servers table, click the name of the managed server you want to configure.
Select Configuration > Server Start.

If this is another server or some other java application, you can add this property to wherever place you can specify the JVM arguments for the start of that server or application.

Read more about Java headless mode from: http://www.oracle.com/technetwork/articles/javase/headless-136834.html

Friday, June 21, 2013

How to solve "javax.naming.CommunicationException [Root exception is java.net.ConnectException" thrown when connecting to a EJB in weblogic server

Recently during a test done on an EJB, I got the following exception.

javax.naming.

CommunicationException [Root exception is java.net.ConnectException: t3://server01:7101: Bootstrap to server01/10.10.58.26:7101 failed. It is likely that the remote side declared peer gone on this JVM]
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://server01:7101: Bootstrap to server01/10.10.58.26:7101 failed. It is likely that the remote side declared peer gone on this JVM]
   at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:40)
   ......
Caused by: java.net.ConnectException: t3://server01:7101: Bootstrap to server01/10.10.58.26:7101 failed. It is likely that the remote side declared peer gone on this JVM
   at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:216)
..............
   ... 25 more
Caused by: java.rmi.ConnectException: Bootstrap to server01/10.10.58.26:7101 failed. It is likely that the remote side declared peer gone on this JVM
   at weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:334)
   at weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:254)
   at weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:197)
   at weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:238)
   at weblogic.rjvm.RJVMFinder.findOrCreateInternal(RJVMFinder.java:200)
   ... 31 more

It turned out that the reason for this exception is, the listen address (lets say "lstnr01") declared in the managed server in Weblogic Admin console is different from server name (although it is a correct address to that machine, I can use it to connect to weblogic console and do ssh) I used for initiating the connection to EJB. You can check the listen address of the server by browsing to "Environments"->Servers->YourServer->Configuration->General. In that page, there is a text box to enter "Listen Address".

Now after finding the reason for the problem, next problem I got is I cannot use this listen address "lstnr01" to connect to my machine, because that is not registered in a DNS server in the network. The mapping of "lstnr01" to that machine was only configured in the "/etc/hosts" file of the server.
Therefore to fix that issue, I added the same IP host name mapping entry to map "lstnr01" to server IP in the C:/Windows/system32/drivers/etc/hosts/hosts file.

10.10.58.26 lstnr01

Then I executed the "ipconfig /flushdns" command using the command line. It refreshed the DNS cache of the local machine and updated the cache with newly added entry in the hosts file.
Then I used "lstnr01" as the remote server address instead of "server01" when connecting to the EJB.
Then the above exception was gone and I was able to connect to the EJB and run the EJB method successfully.

Tuesday, November 13, 2012

A JPA issue which was caused by the Oracle DB's treatment of empty strings in VARCHAR fileds as NULL

Yesterday I encountered a strange issue when I was doing a performance test of my current project using JMeter. My project uses EJB 3 on Weblogic 10.3 server and an Oracle 11g database. I executed a JMeter test with 25 threads with 4000 iterations. But on each test execution, the test cases failed with a UniqueConstraintException after running the test cases for about 10 minutes.

That unique constraint violation exception happened because system was trying to insert a record which was already inserted to the database in setUpTest() method in JMeter java sampler client. But this was not supposed to happen because in actual test cases (in runTest() methods), the system runs a find query to check whether a record with matching value is already existing in the database before inserting any record of the specific type to the database. Another strange thing was this happened only to the test cases executed after the first 10 minutes of the test.

So after some investigation, I found that in the setUpTest() method, one field was set to an empty string in the specific entity object (record) I mentioned earlier. But when I checked the relevant record in the Oracle database, it contained NULL value instead of an empty string for the relevant field.

After some googling, it turned out that when JPA (or using normal SQL query) insert a empty string to Oracle database, Oracle converts it to a NULL before storing it in the database.

So the reason behind my issue is when JPA insert an entity object which has an empty string as the value for a its property, Oracle changes that field value to NULL and store NULL value in the database instead of an empty string. But in the JPA cache, the value of the relevant property of the inserted object is still an empty string, not a NULL value. When system runs the JPQL find query, it checks for the relevant object in the cache first. Because the relevant newly inserted object still has an empty string for the value of the relevant property, the find query returns that object, so system doesn't try to create a new object with the same values again. Therefore we don't get an unique constraint violation exception. But after about 10 minutes, the JPA cache expires and cache is reloaded from the database. At this time, the value for the previously mentioned field of the reloaded specific object is no longer an empty string, because what is in the database for that field is NULL. So from now on, when the system tries to find an object with a value of empty string for this field, it is no longer able to find it. Therefore it assumes this record doesn't exist in the database and tries to re-insert the record. But because Oracle changes the empty string to NULL when it inserts the relevant record to the database and because a record with NULL already exists in the database, it throws an unique constraint violation exception.

See this discussion in stackoverflow for more details on why Oracle treats VARCHAR empty strings as NULL.

http://stackoverflow.com/questions/203493/why-does-oracle-9i-treat-an-empty-string-as-null