Sunday, October 30, 2016

Oracle WebLogic Server - Part 2 - Node Manager and its interaction with WebLogic Server instances

In this post, I am going to describe node manager process and its interaction with WebLogic server instances. This post covers mostly theoretical parts, it does not includes instructions on  how to perform tasks related to node manager.

 In WebLogic Server environment, the Admin Server does not connect to the managed servers directly. A separate utility named "node manager" should be running in each machine for helping this purpose.

The main task of node manager is to provide a remote controller interface for controlling of the WebLogic Server instances (Admin Server and managed server instances) which are located in the same machine as the node manager process. It is not mandatory to have a node manager in a machine if you do not want to control  managed server instances through Admin Server. Also it is possible to have only one node manager for all the domains in a machine.

It is recommended to install node manager as an operating system service or daemon. Then the node manager will be automatically restarted when a system failure or reboot of the machine happens.

Node manager is required if you want following features,

Purpose of Node Manager Process:

  • Allow a remote or a local client to stop/start WebLogic server instances in the machine which runs the Node Manager process (mainly, but not necessarily from the Admin Server)
    • start, shutdown Admin Server instance
    • start, suspend, shutdown managed server instances
  • Automatically restart failed WebLogic server instances which were abnormally exited, if following conditions met,
    • the failed server instance was started by the node manager.
    • AutoRestart value of this server instance is set to true. (AutoRestart property of nodemanager's startup.properties file in <DOMAIN_HOME>/servers/<server_name>/data/nodemanager folder )
    • Number of max allowed restart is not yet passed. (RestartMax property of nodemanager's startup.properties file).
  • Allow monitoring of the health status of WebLogic servers from a remote or the local machine. (mainly, but not necessarily from the Admin Server)
  • Retrieve server logs and node manager logs.

Usually, the node manager gets the start/shutdown commands for starting/shut downing managed servers from the Admin Server. Then node manager is the process that actually start or shutdown the relevant managed server.

Tip: you can access the node manager (tested for a Plain Node Manager in WL 10.3.6) through a program like netcat, and then execute plain text commands to control the servers and monitor servers.
e.g:
~# nc <MACHINE_IP> 5556
domain <your_domain_name>
+OK Current domain set to ‘<your_domain_name>’
getStates
-ERR Must login to a domain first
user <weblogicuser>
+OK NM usr set to '<weblogicuser>'
pass <weblogicuser_password>
+OK Password accepted
getStates
+OK managed1=RUNNING managed2=UNKNOWN AdminServer=UNKNOWN
There are two versions of node manager processes.

Versions of Node Manager

Java based node manager

Java-based node manager process runs in a JVM instance. A Java-based node manager provides more security than a script based node manager.
  • Java-based node manager supports both Windows and UNIX operating systems.
  • Database-less leasing (consensus leasing) is faster when using a Java-based node manager.
  • Using inetd in UNIX, java-based node managers can be configured to restart automatically when receiving a request on a configured port.
  • Determines its configuration from nodemanager.properties file.
See https://docs.oracle.com/cd/E14571_01/web.1111/e13740/java_nodemgr.htm#NODEM177 for more details.

Script based node manager

You can managed remote server instances in a network through ssh using a script-based node manager (a script named wlscontrol.sh), using the scripts provided in this node manager version.
  • Script-based node manager only supports UNIX operating systems.
  • Automatic server migration is only supported by script-based node manager.
  • The node manager can control servers in remote machines through a set of scripts copied to those remote machine.
  • It requires only a smaller foot print than java-based node manager.
  • It supports SSH for security and setting up security is easier than setting up SSL for java based node manager.
  • It supports SSH or RSH for connection.
Following command line option should be added to Admin Server startup parameters in order to use SSH type of script based node manager.
-Dweblogic.nodemanager.ShellCommand='ssh -o PasswordAuthentication=no %H wlscontrol.sh -d %D -r %R -s %S %C'
where %H = host name of ssh server, %S = WebLogic server name, %D = WebLogic domain name, %R = domain directory, %C = node manager script command. For RSH type, the ShellCommand to use is relevant RSH command.

See http://docs.oracle.com/cd/E12840_01/wls/docs103/nodemgr/script_nodemgr.html  and "Running Script-based Node Manager" in https://docs.oracle.com/cd/E24329_01/web.1211/e21050/starting_nodemgr.htm#NODEM214 for more details about Script based node manager.

Node Manager Types

There are several node manager types which supports different level/type of security for node manager communication.
TypeDescriptionJava Based Node ManagerScript Based Node Manager
PlainCommunication is not encrypted.
SSLUse SSL for encrypting communication. You need to setup a certificate for this. See http://www.veriton.co.uk/roller/fmw/entry/ssl_for_weblogic_server_11g and http://weblogic-wonders.com/weblogic/2015/04/23/configure-ssl-nodemanager-12c/
SSHUse SSH session to do the communication. An OS user should be created for connecting using SSH in each machine and setup SSH settings for connection. See https://docs.oracle.com/cd/E23943_01/web.1111/e13740/script_nodemgr.htm#NODEM208
RSHThis type of node manager uses RSH for communication. But RSH is an older protocol which does not use encryption when sending data (such as passwords). Therefore RSH is not secure. It seems it is now rare to use this type of node manager, as not much information about it available in the internet.

Node Manager Operations

Starting Admin Server using Node Manager

  1. A node manager client authenticate to the Node Manager using node manager user name and password (e.g. nmConnect from WLST). Node Manager authenticates the given node manager user credentials using nm_password.properties file.
  2. Then Node Manager receives a command to start Admin Server from the node manager client (e.g. nmStart from WLST). This command provides the server name, domain name, server user credentials and any additional start-up properties.
  3. Node Manager determines the domain directory for the Admin Server from nodemanager.domains file and authenticates the given user credentials using a local file which contains the encrypted user credentials.
  4. Node Manager creates relevant directories (server_name/logs, server_name/security, server_name/data/nodemanager etc.) and obtains start-up properties for the Admin Server.
    • Then it saves encrypted user name, password etc in boot.properties file.
    • It saves the startup properties obtained from Admin Server configuration in startup.properties.
  5. Node Manager creates Admin Server process and start monitoring it.
  6. Admin Server process gets domain configuration data from its config folder.

Starting a managed server using Node Manager

This process can be done either from Admin Console, WLST console or another node manager client. Following steps mentions how to start the server from Admin console.
  1. A user issues a start command from Admin Console to start a managed server.
  2. Then Admin Server connects to the node manager which is located on the same machine as the targeted managed server, authenticate to the node manager.
  3. Then Admin Server sends the start command for starting the managed server to the node manager (with server name, username/password for connecting to server and other start-up properties).
  4. Node manager starts the managed server process.
  5. Managed server contacts the Admin Server to check for updates to its configuration information.
  6. If there is any update to the domain configuration, managed server updates its local cache of configuration from the update it got from the Admin Server.

Restart Admin Server using Node Manager

  1. Node Manager determines from the exit code of Admin Server process that it needs to restart the Admin Server.
  2. Node Manager obtains the user name and password value needed for starting the Admin server from the boot.properties file, and the start-up properties from <servername>/data/nodemanager/startup.properties file.
  3. Node Manager uses the above retrieved information and starts the Admin Server process.
  4. Admin Server reads its configuration data and starts up.

Restart a managed server using Node Manager

  1. Node Manager determines from the last known status of the managed server process (+ other node manager settings) that it needs to restart that managed server.
  2. Node Manager retrieves the user name and password value needed for starting the managed server from the boot.properties file, and the startup properties from <servername>/data/nodemanager/startup.properties file.
  3. Node Manager uses the above retrieved information and starts the managed Server process.
  4. Managed server contacts the Admin server to check for updates of the configuration data. If it get any updates, it update its local cache of config directory with the received updates.
  5. If managed server fails to contact the Admin Server and if the Managed Server Independence Mode (MSI) is enabled, the managed server uses its locally cached configuration data.

Shutdown a server instance using Node Manager

  1. An authorized user connects to the Admin console and issues the shutdown command for a managed server.
  2. Then Admin Server issues shutdown command directly to the managed server. Then managed server perform the shutdown sequences for Graceful Shutdown procedure. 
  3. If Admin server failed to contact the managed server in above step, then it connects to the node manager and issues the shutdown command to the node manager for shutting down the managed server.
  4. Node manager asks Operating System to kill the managed server process.
  5. Operating system kills the managed server process.

Important files and folders related to node manager

FileFolderPurpose
nodemanager.properties<NODEMANAGER_HOME>/The configuration file used by Java based node manager
nodemanager.domains<NODEMANAGER_HOME>/Contains domain names and corresponding directories of the domains managed by the node manager
nm_data.properties<NODEMANAGER_HOME>/This file stores a symmetric encryption key. It is stored in encrypted form.
nm_password.properties<DOMAIN_HOME>/config/nodemanagerThis file stores the username and password of the node manager
boot.properties<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis files contains the user credentials needed for starting server instances.
startup.properties<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerEach managed server instance contains its own copy of startup.properties file which contains the properties that control the startup/shutdown/restart of the server instance by the node manager.
<server_name>.addr<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the IP address added when the server started or migrated.
<server_name>.pid<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the process id of the server instance. Node manager checks this process id during crash recovery. This is generated by the server instance.
<server_name>.state<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the current state of the server instance. This file is generated by the server instance and monitored by the node manager. Without this file, node manager cannot determine the current state of the server instance.
<server_name>.lck<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file is generated by the server instance and contains a internal lock id.
nodemanager.log<NODEMANAGER_HOME>/Node manager log file (only for Java based node manager). This file contains logs about all of domains controlled by the node manager. This log can be viewed using nmLog command of WLST and "Machines --> Monitoring --> Node Manager Log" in Admin Console.
<server_name>.out<DOMAIN_HOME>/servers/>server_name</logsNode Manager Server instance log file. This file contains the messages written to stderr and stdout by the server instance. You can view this log by either nmServerLog WLST command or "Diagnostics --> Log Files".
<server_name>.log<DOMAIN_HOME>/servers/<server_name>/logsWebLogic server log files.

References:

1 comment:

Anonymous said...
This comment has been removed by a blog administrator.